JP2011182330A - Image processing method, image processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve estimation accuracy of a restoration image by suppressing increase in computational complexity during DCT coding. <P>SOLUTION: While utilizing the feature that, for a DCT coded image, observation data are given as a DCT coefficient and convolution in a space domain corresponds to a product, in a DCT domain, for each element of the DCT domain, an SR approach is applied in the DCT domain, an inputted quantized matrix is linearly interpolated, and an inverse for each element of the quantized matrix is multiplied as a weight to mainly restore a low-frequency component of a small quantization error. If a quantization error occurs, a minimum value of a matching degree evaluation value is set to 0, and fitting is performed using such a combination as to minimize a differential factor of evaluation values at two points of both ends used for fitting. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing method, an image processing apparatus, and a program, and more particularly, to an image processing method, an image processing apparatus, and a program for performing high-speed and high-precision super-resolution processing on a compressed image.

  In a digital image, if the resolution is high, the edge and texture portion of an object can be accurately expressed. Therefore, the demand for high-resolution images is increasing in various fields such as medical images and aerial photographs.

  There are two methods for improving the image resolution: hardware performance improvement and software processing. In the method by improving the hardware performance, a high-resolution image can be obtained by increasing the arrangement density of the imaging elements. However, there are problems in hardware performance limitations and development costs.

  On the other hand, methods using software processing have been proposed in various studies. In recent years, super resolution (SR) technology that restores a high-resolution image from a plurality of observed images including misalignment has attracted attention (for example, see Non-Patent Document 1).

  On the other hand, images and moving images taken with a camera are generally compressed to reduce transmission costs and storage costs. Among them, a frequently used image compression technique is DCT (Discrete Cosine Transformer) coding standardized by JPEG or the like, and is stored in a state where it has been deteriorated by quantization. However, the normal compression operation is irreversible, and even if the SR method is directly applied to this, there is a limit to improving the restoration accuracy. For this reason, it has been studied to apply the SR method to a compressed image.

  However, the SR method for application to a DCT encoded image has the following two problems.

  (1) A reconstruction algorithm such as MAP (Maximum A Posteriori) when restoring an SR image can stably estimate a high-resolution image, but generally requires a large amount of calculation (calculation cost). Here, MAP is a method for obtaining the most probable reconstructed image by statistical estimation processing.

  (2) It is difficult to improve the accuracy of the SR restored image with respect to the compressed image.

  There are three conventional methods for solving these problems, namely, the SR acceleration method, the SR method for DCT-encoded compressed images, and the alignment method, which will be described below.

1: Speeding up method:
From the viewpoint of reducing the number of pixel value estimation calculations for calculating the evaluation function, this method aligns multiple low-resolution images, sets discretization points in the high-resolution image space, and places them near the discretization points. The average pixel value of the corresponding pixel is used. At this time, by considering the number of pixels corresponding to the vicinity of the discretization point as a weight, the calculation can be performed at high speed without reducing the estimation accuracy (see, for example, Non-Patent Document 2). Moreover, high-speed calculation can be performed by performing optimization calculation in the frequency domain (for example, refer nonpatent literature 3).

FIG. 11 shows a block diagram of the error term of the frequency domain optimization method and its differential calculation. The symbol “×” in the figure represents multiplication for each element, and F and F ⁻¹ represent Fourier transform and inverse Fourier transform, respectively. The average pixel represents an image composed of the average pixel value of the observed image, and the weighted image represents the number of elements of each pixel of the average image. PSF is an abbreviation for point spread function.

  On the other hand, there is also a method of performing high-speed calculation by dividing an image into spatial regions (see, for example, Non-Patent Document 4). As shown in FIG. 12, the divided images a to d are processed independently. At this time, in order to avoid a boundary error, it is necessary to overlap each divided image.

2. SR for DCT encoded compressed images:
As the SR method, a DCT encoding process is incorporated into an observed image acquisition model, and an error due to quantization is assumed to be Gaussian noise, thereby increasing the resolution (for example, see Non-Patent Document 5). In addition, assuming that the quantization error is not Gaussian noise, the error term of the evaluation function is weighted with the product of the Gaussian function and the quantization matrix (see, for example, Non-Patent Document 6). FIG. 13 shows a quantization error model of the conventional method. FIG. 4A shows a model of Non-Patent Document 5, and FIG. 4B shows a model of Non-Patent Document 6.

3. Alignment method:
As one of conventional alignment methods, there is a method using a fitting function given from the coincidence evaluation value for each pixel as shown in FIG. In this method, the subpixel displacement can be obtained with a small amount of calculation (see, for example, Non-Patent Document 7). In addition, in the registration method for the DCT-encoded image, the estimation accuracy is improved mainly by using only the low frequency component as compared with the registration method using the DCT positive / negative code (see, for example, Non-Patent Document 6). ).

S. C. Park, M. K. Park, M. G. Kang, "Super-Resolution Image Reconstruction: A Technical Overview", IEEE, MAY, 2003. Masayuki Tanaka and Masatoshi Okutomi, "High-speed algorithm for reconfigurable super-resolution processing and its accuracy evaluation", IEICE Transactions D-II Vol. J88-D-II No.11 pp. 2200-2209, 2005 Masayuki Tanaka and Masatoshi Okutomi, "Acceleration of MAP-type super-resolution processing by frequency domain optimization", Information Processing Society of Japan Vol. 47 No. SIG10, July 2006. D. Zhang, H. Li, M. Du, "Fast MAP-based multiframe super-resolution image reconstruction", Image and Vision computing 23 pp. 671-679, 2005. Bahadir K. Gunturk, Y. Altunbasak, Russell M. Merserreau, "SR reconstruction of compressed video Using Transform-Domain Statistics", IEEE, 2004. Y. Yamaguchi and N. Hamada, "Super Resolution for DCT-based Compressed Images", Proceedings of NCSP '09, March, 2009. Masao Shimizu, Masatoshi Okutomi, "Precise Subpixel Estimation Method for Image Matching", IEICE Transactions D-II Vol. J84-D-II No.7 pp. 1409-1418, 2001.

  However, the above conventional techniques have the following problems.

1. Challenges of high-speed algorithm:
In the conventional frequency domain optimization method (Non-patent Document 3), the number of pixels corresponding to the vicinity of the discretization point is used as a weight in order not to reduce the estimation accuracy. However, for that purpose, it is necessary to return from the frequency domain to the spatial domain in the process of optimization calculation, and the Fourier transform and inverse Fourier transform must be calculated. In addition, the operation of applying a weighted image is not only a meaningless operation for a DCT-encoded compressed image, but conversely causes a reduction in SR accuracy. This is because the quantization error is not taken into consideration.

  Further, in the spatial region division processing (Non-Patent Document 4), a boundary error due to a difference in boundary conditions occurs. Therefore, in order to prevent this from occurring, at least (PSF kernel size-1) pixel overlap is performed. There is a need.

2. Quantization error compensation challenges:
In the technique of Non-Patent Document 5 described above, since the quantization error model is assumed in units of 8 × 8 blocks, the frequency resolution becomes low and the quantization error cannot be compensated with high accuracy. Further, in the technique of Non-Patent Document 6 described above, it is necessary to experimentally determine the variance value of the Gaussian function, and this value is not practical considering that it greatly affects the SR accuracy.

3. Challenges of alignment method:
In the conventional method, since the quantization error is not taken into consideration, only one pixel period has been studied. However, the alignment with respect to the DCT-encoded compressed image is 8 pixels in consideration of compression in units of 8 × 8 blocks. Periodic examination is required.

  The present invention has been made in view of the above points. An image processing method, an image processing apparatus, and an image processing method capable of reducing the calculation cost and improving the estimation accuracy of a restored image in super-resolution processing for a DCT encoded image, and The purpose is to provide a program.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is an image processing apparatus that uses a super-resolution method (SR method) when performing image compression by DCT encoding,
The DCT encoded image is obtained by using the SR method in the DCT domain by using the fact that the observation data is given by DCT coefficients and the convolution in the spatial domain corresponds to the product of each element of the DCT domain in the DCT domain. DCT region processing means 110 for applying
Linear interpolation means 120 for linearly interpolating the input quantization matrix and multiplying a reciprocal for each element of the quantization matrix as a weight to restore mainly a low frequency component with a small quantization error;
Positioning means 130 for performing fitting using a combination in which the difference rate between the evaluation values at the two ends used for fitting is minimized, with the minimum value of the matching degree evaluation value being 0 when the quantization error has occurred. , At least one of the following.

In addition, the present invention (Claim 2) is the registration unit 130 of the image processing apparatus according to Claim 1,
As the evaluation value, the square sum of the density differences is used.

According to the present invention (Claim 3), in the image processing apparatus according to Claim 1 or 2,
The DCT-transformed prior information image C and PSF (point spread function) image B are acquired from the DCT region processing unit 110, the quantization matrix Q is acquired from the linear interpolation unit 120, and the average pixel of the observation image is acquired from the alignment unit 130 Obtaining an average image G of values and a regularization parameter α,
Super resolution image F
F = {BG} / {(1-α) ² B ² + Q (αC) ² }
The super-resolution synthesis means 150 obtained by

The present invention (Claim 4) is an image processing method using a super-resolution method (SR method) when performing image compression by DCT encoding,
The image processing device
The DCT encoded image is obtained by using the SR method in the DCT domain by using the fact that the observation data is given by DCT coefficients and the convolution in the spatial domain corresponds to the product of each element of the DCT domain in the DCT domain. DCT domain processing step applying
A linear interpolation step for linearly interpolating the input quantization matrix and preferentially restoring low frequency components with a small quantization error by multiplying the inverse of each element of the quantization matrix as a weight;
An alignment step in which fitting is performed using a combination in which a difference rate between two evaluation values at both ends used for fitting is minimized when a quantization error has occurred and a minimum value of the matching evaluation value is set to 0; Perform at least one of the steps.

The present invention (Claim 5) is the image processing method according to Claim 4,
In the alignment step, the evaluation value uses a square sum of density differences.

  The present invention (Claim 6) is an image processing program for causing a computer to function as each means constituting the image processing apparatus according to any one of Claims 1 to 3.

  As described above, according to the present invention, in the image processing method (apparatus) that provides the super-resolution method (SR method) in the image compression by DCT encoding, the SR method is applied in the DCT region, and the quantization error is small. By carefully restoring low-frequency components and selecting evaluation values for fitting so that estimation errors are unlikely to occur, it is possible to suppress an increase in calculation cost and improve the estimation accuracy of restored images Become.

It is a principle block diagram of this invention. 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention. It is an acceleration algorithm in one embodiment of the present invention. It is a figure which shows the relationship with the pixel count of the high resolution image in one embodiment of this invention. It is a figure which shows the relationship between this invention and the number of PSF kernels in a prior art. It is a figure which shows the alignment result with respect to the DCT coding image in one embodiment of this invention. It is a figure which shows the case where an error generate | occur | produces largely. It is a figure which shows the case where an error does not generate | occur | produce largely. It is a figure which shows the subpixel estimation in one embodiment of this invention. It is a figure which shows the comparison with the conventional method and this invention. It is a block diagram of the error term of a frequency domain optimization method, and its differential calculation. It is a figure which shows the method of dividing | segmenting an image in a space area | region. It is a figure which shows the quantization error model of the conventional method. It is a figure which shows the positioning method of the conventional method (nonpatent literature 7).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 2 shows a configuration of an image processing apparatus according to an embodiment of the present invention.

  The image processing apparatus shown in the figure includes an alignment processing unit 110, an overlay unit 120, a DCT region conversion processing unit 130, a linear interpolation unit 140, and a super-resolution composition unit 150.

  In the figure, a plurality of DCT region conversion processing units 130 are shown, but a single DCT region conversion processing unit may be used.

  Further, although not shown, the image processing apparatus is connected to input means for inputting a regularization parameter, a quantization matrix, prior information, a PSF kernel, a low resolution image, and the like.

  The alignment processing unit 110 calculates the amount of misalignment of the input low resolution image. At this time, the input low-resolution image is fitted using a combination that minimizes the difference rate. Details will be described later.

  The superimposing unit 120 superimposes a plurality of low resolution images whose positional deviations have been corrected by the alignment processing unit 110 to synthesize a high resolution image. The superimposing unit 120 performs an average value process as necessary. For example, when the number of low-resolution images is not sufficient, a portion without a superimposed signal (a portion where the signal is missing) appears. At this time, correction (average value processing) is performed using surrounding signals. In addition, when there is no overlay signal, it is not limited to the average value processing.

  The DCT region conversion processing unit 130 c performs DCT conversion of the high-resolution image generated using the alignment results of the alignment unit 130 and the superposition unit 140.

  The DCT domain conversion processing unit 130b performs DCT conversion of the PSF kernel.

  The DCT region conversion processing unit 130a performs DCT conversion of the prior information (prior information image) of the spatial region.

  The linear interpolation unit 140 linearly interpolates the quantization matrix by using the fact that the reciprocal for each quantization matrix has a larger value for lower frequency components.

  The super-resolution composition unit 150 includes an average image (G) output from the DCT region conversion processing unit 130c, a quantization matrix (Q) linearly interpolated to the image size output from the linear interpolation unit 140, and a DCT region conversion processing unit. Using the prior information image (C) output from 130a, the PSF image (B) output from the DCT region conversion processing unit 130, and the input regularization parameter (α), an operation for obtaining a high resolution image is performed.

  FIG. 3 shows an acceleration algorithm in an embodiment of the present invention.

  In the present embodiment, the DCT region transform processing unit 130 uses the fact that observation data is given as DCT coefficients for DCT encoded images, and that convolution in the spatial domain corresponds to a product for each element in the DCT region. A high-speed calculation method for solving the SR problem in the DCT domain is realized.

  Further, the linear interpolation unit 140 is robust against quantization error by multiplying the weight in the DCT region, and can eliminate the operation of returning to the conventional spatial region. For this reason, it is possible to further increase the speed.

  In addition, by performing division processing in the DCT region in the DCT region conversion processing unit 130, it is possible to eliminate the occurrence of boundary errors without overlapping as in the spatial region.

  The number of multiplications necessary for the steepest descent method in the conventional method (spatial domain), the conventional method (frequency domain), and the present invention (DCT domain) is shown below.

According to the present invention, the relationship between the number of pixels of the high-resolution image and the number of multiplications is as shown in FIG. 4, and covers the required number of high-resolution pixels in general as compared with the conventional high resolution in the spatial domain or frequency domain. Since the number of multiplications can be reduced, the amount of calculation and cost required for higher resolution are reduced.

  Further, the relationship between the number of PSF kernels and the number of multiplications is as shown in FIG. 5, and the number of multiplications can be reduced over the required number of PSF kernels compared to the conventional high resolution in the spatial domain or frequency domain. The amount of calculation and cost required for higher resolution are reduced.

  Next, compensation for quantization error in the linear interpolation unit 140 will be described.

  Since the quantization operation performs a remainder operation on the quantization matrix, the quantization error depends on the value of the quantization matrix. In general, the value of the quantization matrix is larger as the high-frequency component is larger, and a larger error than that of the low-frequency component is generated, which causes a reduction in SR accuracy. That is, it is possible to suppress the influence of the quantization error by focusing on the low-frequency component with a small quantization error.

  Therefore, in this embodiment, optimization is performed in the DCT region of the entire image in order to increase the resolution, and the reciprocal for each quantization matrix has a larger value for lower frequency components. By linearly interpolating the quantization matrix and multiplying the inverse of each element as a weight, low frequency components with a small quantization error can be reconstructed mainly.

In the super-resolution composition unit 150, the input normalization parameter α, the linearly interpolated quantization matrix Q output from the linear interpolation unit 140, the prior information image C output from the DCT region conversion processing unit 130, and the PSF image B The super-resolution image F is obtained by performing the following calculation using the average image G _{S & A} output from the DCT region conversion processing unit 130c.

However, each of the above values is a value in the DCT region.

  On the right side of the above equation (1)

Is the error term, and

Is a normalization term.

  In the conventional method, since it is a high-dimensional and complicated problem, it has been necessary to perform repetitive calculation. However, with the evaluation function of the present invention, the problem is simplified and a solution can be obtained analytically. That is, F that minimizes E can be obtained in explicit form as follows.

Solving the above equation (2) for F,

It becomes. The calculations of equations (1) to (3) are performed in the DCT domain, all matrix multiplication is element-by-element multiplication, and matrix division is element-by-element division. The calculation (3) is performed by the super-resolution composition unit 150, and a super-resolution image F is output.

  Next, the alignment for the DCT encoded image in the alignment processing unit 110 will be described.

  First, it will be verified what the alignment estimation accuracy is when the quantization error occurs.

  First, a test image is shifted by 0.1 unit in the y direction in the range of 0 to 8 pixels and compressed (compression rate 1.25) to create a total of 81 images. Pixel estimation was performed. At this time, the shift image was generated by shifting the DCT basis matrix by 2πδ and performing inverse DCT, and the square sum of the density difference was used for evaluation of the degree of coincidence. The result is shown in FIG. From this result, it can be considered as follows that the error is large when the positional deviation amount is in the range of 0 to 2 and 6 to 8.

  FIG. 7 shows a case in which a large error occurs. As shown in FIG. 7, since an evaluation value of positional deviation 0 hardly causes a quantization error, it is considered that an error occurs in other parts. It is considered that an estimation error is likely to occur toward a larger evaluation value.

  FIG. 8 shows a case in which the error is not large. Considering the same as in FIG. 7, in the case of FIG. It is thought that it is hard to generate.

  That is, it is considered that an error is less likely to occur if the evaluation values at both ends used for fitting are close.

  Therefore, in the present invention, as shown in FIG. 9, the estimation error is reduced by selecting an evaluation value at the time of fitting so that the estimation error is unlikely to occur. Specifically, in FIG. 9, the minimum value of the coincidence evaluation value is set to 0, and among the evaluation values in the range of -3 to 3, two points at both ends used for fitting are expressed as the difference rate (the evaluation value is large). Fitting is performed using a combination that minimizes (a). The evaluation value uses the following sum of squares of the density differences in order to generate the same sign and size of the evaluation value error due to the quantization error.

Here, R _i and R _j represent evaluation values at both ends used for fitting.

  The alignment processing unit 110 calculates the amount of misalignment using the above difference rate, and outputs the calculated value to the superposition unit 120 described above.

  FIG. 10 shows a comparison result of evaluation values when there is no quantization error between the conventional method and the present invention.

  FIG. 4A shows evaluation values when estimation is performed using three points of the conventional method [−1, 0, 1], and FIG. 4B illustrates 3 of [−1, 0, 2] according to the present invention. The evaluation value when estimated by points is shown. As can be seen from the figure, according to the present invention, it is difficult for an estimation error to occur.

  In the above embodiment, the high-speed calculation method (speed-up algorithm) for solving the SR problem in the DCT domain, the compensation of the quantization error, and the alignment with respect to the DCT-coded image are described. At least one of these is described. Even if one is used, the calculation cost can be reduced.

  The operations of the components of the image processing apparatus shown in FIG. 3 described above can be constructed as a program, installed in a computer used as the image processing apparatus, executed, or distributed via a network. .

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to DCT coding of image compression technology.

110 Positioning means, Positioning processing section 120 Superposition section 130 DCT area processing means, DCT area conversion processing section 140 Linear interpolation means, Linear interpolation section 150 Super-resolution composition means, Super-resolution composition section

Claims (6)

An image processing apparatus using a super resolution (SR) method when performing image compression by DCT (Discrete cosine Transformer) encoding,
The DCT encoded image is obtained by using the SR method in the DCT domain by using the fact that the observation data is given by DCT coefficients and the convolution in the spatial domain corresponds to the product of each element of the DCT domain in the DCT domain. DCT region processing means for applying
Linear interpolation means for linearly interpolating the input quantization matrix, and multiplying the inverse of each element of the quantization matrix as a weight to restore a low-frequency component with a small quantization error, and
An alignment means for performing fitting using a combination in which the difference rate between two evaluation values at both ends used for fitting is minimized when the quantization error has occurred and the minimum value of the matching evaluation value is set to 0;
An image processing apparatus comprising at least one of the following. In the alignment means,
The image processing apparatus according to claim 1, wherein the evaluation value uses a square sum of density differences. A prior information image C and a PSF (point spread function) image B are acquired from the DCT region processing means, a quantization matrix Q is acquired from the linear interpolation means, and an average composed of average pixel values of observed images from the alignment means Obtain image G and regularization parameter α,
Super resolution image F
F = {BG} / {(1-α) ² B ² + Q (αC) ² }
The image processing apparatus according to claim 1, further comprising a super-resolution composition unit that is obtained by: An image processing method that uses a super resolution (SR) method when performing image compression by DCT (Discrete cosine Transformer) encoding,
The image processing device
The DCT encoded image is obtained by using the SR method in the DCT domain by using the fact that the observation data is given by DCT coefficients and the convolution in the spatial domain corresponds to the product of each element of the DCT domain in the DCT domain. DCT domain processing step applying
A linear interpolation step for linearly interpolating the input quantization matrix and preferentially restoring low frequency components with a small quantization error by multiplying the inverse of each element of the quantization matrix as a weight;
An alignment step in which fitting is performed using a combination in which a difference rate between two evaluation values at both ends used for fitting is minimized when a quantization error has occurred and a minimum value of the matching evaluation value is set to 0;
An image processing method comprising performing at least one of the steps. In the alignment step,
The image processing method according to claim 1, wherein the evaluation value uses a square sum of density differences.   An image processing program for causing a computer to function as each means constituting the image processing apparatus according to claim 1.