JP2011248475A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for improving the spatial resolution of an image after enlargement.SOLUTION: An initial image generating part 202 enlarges a first image to a size of a second image to store in a buffer 206. A weight calculation unit 203 generates an edge-enhanced image of the enlarged image. An updating amount calculation part 207 minimizes the total value of an evaluation value obtained by weighting a value showing a change in the pixel value in the image stored in the buffer 206 by the pixel value of the edge-enhanced image and an evaluation value showing the difference between the image obtained by reducing the stored image to a size of the first image to degrades the image quality and the first image. The updating amount calculation part 207 acquires an updating amount for the stored image, the updating amount that realizes the minimization. An adder 208 updates the stored image using the determined updating amount.

Description

  The present invention relates to a technique related to image quality improvement.

  In devices that handle image data such as computers, digital cameras, and displays, image enlargement is a frequently performed operation. In enlarging an image, the enlarging method becomes a problem, and the image quality of the image after enlarging differs depending on the method. Typical examples include the nearest neighbor method, bi-linear method, and bi-cubic method.

  An image enlarged by such a method increases the number of pixels, but the spatial resolution does not change, and thus the image becomes blurred. On the other hand, the spatial resolution can be improved by performing image estimation using information on the process at the time of image data acquisition.

  In Non-Patent Document 1, a high-resolution image is generated by a method based on MAP (Maximum A Posterior) estimation. The MAP method is a method of maximizing the posterior probability for a given low-resolution image, that is, obtaining a high-resolution image that is most probable. The posterior probability is multiplied by the conditional probability (the probability that a given low-resolution image will be obtained when the image data acquisition process is applied to the high-resolution image) and the prior probability (the probability that this high-resolution image will appear). It is proportional to what you did. A high-resolution image is estimated by solving an optimization problem that maximizes this. Non-Patent Document 1 describes a method for setting a prior probability on the assumption that an image is smooth.

  On the other hand, Non-Patent Document 2 proposes a method for estimating a high-resolution image so that the sum of pixel value fluctuations of the high-resolution image is reduced while utilizing information regarding the process at the time of image data acquisition. This method is equivalent to the case where the prior probability is set on the assumption that the image is stepped in the MAP method. Similarly, in Non-Patent Document 2, the image is decomposed into a skeleton and a texture, the above-described method is applied to the skeleton image, and resolution conversion using a filter is applied to the texture, and the image is synthesized after each processing. This method has been proposed.

Sung C. P., Min K. P., “Super-Resolution Image Reconstruction: A Technical Overview”, IEEE Signal Proc. Magazine, Vol. 26, No. 3, p. 21-36, 2003 Takahiro Saito, “Super-resolution oversampling from a single image that breaks the wall of the sampling theorem”, Journal of the Institute of Image Information and Television Engineers, Vol.62, No.2, pp.181-189 (2008)

  In the method proposed in Non-Patent Document 1, there is a problem that ringing occurs in the edge portion and the edge cannot be reproduced correctly. On the other hand, the image estimation method assuming that the image proposed in Non-Patent Document 2 is stepped can correctly reproduce the edge, but the edge is formed even to the non-edge part, and the image is There is a problem of becoming unnatural. In addition, the method using image decomposition proposed in Non-Patent Document 2 can correctly reproduce edges while avoiding unnatural images, but it cannot improve spatial resolution for textures. There's a problem.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for improving the spatial resolution of an enlarged image.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is, an image processing apparatus that generates, from a first image, a second image having a resolution higher than that of the first image and having a larger image size than the first image,
Storage means for generating an image obtained by enlarging the first image to the same size as the second image, and storing the generated image in a memory of the image processing device;
Generating means for generating an edge-enhanced image in which edges in the enlarged image are enhanced;
Updating means for updating an image stored in the memory;
Means for outputting the image updated by the updating means as the second image,
The updating means includes
A first evaluation value calculated by weighting a value representing a change in pixel value between pixels in the stored image stored in the memory using a pixel value of each pixel constituting the edge enhanced image. And a second evaluation value indicating a difference between the first image and the image obtained by reducing the stored image to the same image size as the first image using a specified filter, and Calculation means for obtaining an update amount for the stored image so as to minimize the total;
Means for updating the stored image using the update amount.

  According to the configuration of the present invention, the spatial resolution of an enlarged image can be improved.

FIG. 3 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. FIG. 3 is a block diagram showing an example functional configuration of an image estimation unit 102. The figure which shows the size of a vector and a matrix. The flowchart of the process which the image estimation part 102 performs. The flowchart of the process in step S403. FIG. 3 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. The block diagram which shows the function structural example of the image estimation part 602. FIG. The figure which shows the size of a vector and a matrix. The flowchart of the process which the movement amount calculation part 603 and the image estimation part 602 perform.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
<Functional configuration example of image processing apparatus>
First, a functional configuration example of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. Since the input image (first image) is input to the image input unit 101, the image input unit 101 outputs this input image to the subsequent image estimation unit 102.

  The image estimation unit 102 generates an image (second image) having a resolution higher than that of the input image and having a larger image size than the input image from the input image. In order to generate the second image, an image estimation process is performed. This image estimation process will be described in detail below.

  Since the image estimation unit 102 sends the image generated by the image estimation process to the subsequent image output unit 103, the image output unit 103 sends this image. The transmission destination is not particularly limited, and may be transmitted to an external device via a network, or may be output to an image display device configured by a CRT, a liquid crystal screen, or the like.

<Example of Functional Configuration of Image Estimating Unit 102>
A functional configuration example of the image estimation unit 102 will be described with reference to the block diagram of FIG. In the present embodiment, an image input to the image estimation unit 102 via the image input unit 101 is referred to as an LR image (Low Resolution image). An image generated by the image estimation unit 102 from the LR image and sent to the image output unit 103 is referred to as an HR image (High Resolution image). The image size of the LR image is W (number of horizontal pixels) × H (number of vertical pixels). Further, as described above, the HR image is an enlarged image of the LR image, and the horizontal magnification is s_x and the vertical magnification is s_y. Therefore, the image size of the HR image is s_x × W (number of horizontal pixels). ) × s_y × H (vertical pixel number).

  The LR image input to the image estimation unit 102 is stored in the buffer 204 and also input to the initial image generation unit 202 and the weight calculation unit 203. The initial image generation unit 202 uses an interpolation method such as bi-cubic or cubic interpolation to generate an image in which the LR image is enlarged to the same size as the HR image. The initial image generation unit 202 stores the generated image in the buffer 206 as an intermediate HR image (stored image).

The weight calculation unit 203 uses an interpolation method such as bi-cubic or cubic interpolation to generate an image in which the LR image is enlarged to the same size as the HR image. The weight calculation unit 203 generates an image having a pixel value as a value indicating whether each pixel constituting the generated image forms an edge or a non-edge, and applies a low-pass filter to the image. By performing processing such as processing, an edge-enhanced image is generated. Although the edge-enhanced image will be described in detail later, the following description will be made assuming that the pixel value of each pixel constituting the edge-enhanced image is normalized to 0 to 1. Then, the weight calculation unit 203 stores a weight value (α _{m, n} , β _{m, n} ) described later calculated from the generated edge enhanced image in the buffer 205.

  The degradation model memory 201 stores matrix data as a “degradation model” as a specified filter for reproducing image quality degradation that has occurred in the LR image in the process of obtaining the LR image by imaging or the like. Details of this deterioration model will be described later.

  The update amount calculation unit 207 uses the deterioration model read from the deterioration model memory 201, the weight value read from the buffer 205, the LR image read from the buffer 204, and the intermediate HR image read from the buffer 206, and uses this intermediate HR image. Calculate the amount of updates. This update amount is expressed as a matrix having the same number of vertical and horizontal elements as the number of vertical and horizontal pixels of the intermediate HR image stored in the buffer 206, and the update amount stored at the position P in the matrix is the intermediate HR image. This is the update amount applied to the middle position P.

  The adder 208 converts each update amount registered in the matrix indicated by the update amount obtained by the update amount calculation unit 207 to the pixel value of the pixel at the corresponding pixel position of the intermediate HR image stored in the buffer 206. By adding the values, a new intermediate HR image is generated.

  The selector 209 determines whether or not to complete the update process of the intermediate HR image by the update amount calculation unit 207 and the adder 208. If it is determined to complete, the new intermediate HR image generated by the adder 208 is sent to the image output unit 103 as an HR image. On the other hand, when it is determined not to be completed, the new intermediate HR image generated by the adder 208 is overwritten and stored on the intermediate HR image already stored in the buffer 206. That is, the intermediate HR image stored in the buffer 206 is updated.

<About deterioration model>
Next, the deterioration model stored in the deterioration model memory 201 will be described. Image quality degradation that occurs in the process of capturing a single image using an imaging device such as a digital camera is expressed by the point spread function (PSF) of the imaging device and downsampling, which is a degradation process due to the limitation on the number of sensor pixels. To do. According to this, assuming that the LR image is y and the HR image is x, the relationship between x and y can be expressed by the following equation (1).

  In this equation (1), B is a square matrix representing degradation due to PSF. D is a matrix reflecting the magnification of image reduction by downsampling. In the following, the product DB of the matrix D and the matrix B is treated as a deterioration model. That is, the degradation model indicates a degradation process modeled by the matrices B and D. However, in the present embodiment, the matrix B is set in consideration of the optical characteristics and sensor characteristics of the imaging characteristics of the imaging apparatus that captured the LR image.

  The sizes of the matrices B and D depend on the image size of the LR image and the image size of the HR image. The image size of the LR image is W × H as described above, the image size of the HR image is as described above (s_x × W) × (s_y × H), the image reduction ratio by downsampling is 1 / s_x in the horizontal direction, and 1 in the vertical direction. Assuming / s_y, the matrices B and D have the sizes shown in FIG.

  Since y representing the LR image is handled as a vector description, the matrix size is (H × W) × 1. Since x representing the HR image is handled as a vector description, the matrix size is {(s_x × W) × (s_y × H)} × 1. However, the matrix size of the matrix D is (H × W) × {(s_x × W) × (s_y × H)}, and the matrix size of the matrix B is {(s_x × W) × (s_y × H)} × { (S_x × W) × (s_y × H)}.

<Image Estimation Processing by Image Estimation Unit 102>
Next, image estimation processing by the image estimation unit 102 will be described. In the present embodiment, image estimation processing is performed based on the MAP estimation method disclosed in Non-Patent Document 1. More specifically, image estimation processing is performed using the following equation (2).

  Here, x is an intermediate HR image, y is an LR image, x ′ is an HR image obtained by updating the intermediate HR image a plurality of times, and all of x, y, and x ′ are column vectors. Equation (3) shows a column vector representing the intermediate HR image.

This column vector is a vector configured by arranging pixel values xm _{, n} of pixels at each pixel position (m, n) in the intermediate HR image so that (m, n) is in the following order. . That is, (0,1), (0,2), (0.3) ... (1,1), (1,2), (1.3) ... (2,1), (2,2), (2.3). Thus, in the following, it is assumed that vec (s) indicates a column vector of the image s.

  In Expression (2), the first term in argmin [] is a smaller value as the difference (difference) between the image generated by applying the deterioration model (D · B) to the intermediate HR image x and the LR image y is smaller. Is an evaluation value that corresponds to the conditioning probability. σ is a standard deviation of the noise amount of the LR image y.

In Equation (2), the second term in argmin [] is an evaluation value that takes a smaller value as the “gradient of the change in pixel value between pixels in the intermediate HR image x” is smoother. Corresponds to prior probabilities. ∇ ² x _{m, n} corresponds to the second derivative of the intermediate HR image x and can be calculated by the following equation (4).

Α _{m, n} as a weight value for the second derivative ∇ ² x _{m, n} is a coefficient for weighting the contribution of ∇ ² x _{m, n} for each pixel position.

In Expression (2), the third term in argmin [] is an evaluation value that takes a smaller value as the “change in the pixel value between pixels in the intermediate HR image x is smaller”, and corresponds to the prior probability of the edge. . | ∇x _{m, n} | corresponds to the absolute value of the gradient of the pixel value in the intermediate HR image x, and can be calculated by Equation (5).

The _{| ∇x m, n | β m} , n of the weight values for the, | a coefficient for weighting the contributions for each pixel position _| ∇x _{m, n.} α _{m, n} , β _{m, n} are weight values for realizing processing according to the characteristics of the region in the intermediate HR image x, and are calculated from the LR image by the weight calculation unit 203 using a method described later. Is done.

  In other words, the first term in argmin [] is an evaluation value (first value) indicating the difference between an LR image and an image obtained by reducing the image quality by reducing the intermediate HR image to the same image size as the LR image using a specified filter. 2).

  The second and third terms in argmin [] are weighted using the pixel value of each pixel constituting the edge-enhanced image with respect to the value representing the change in pixel value between pixels in the intermediate HR image. Thus, it can be said that the evaluation value (first evaluation value) is calculated. Expression (6) below is an expression that extracts the right side of Expression (2) and uses it as an evaluation function.

  In the present embodiment, the HR image is estimated by updating the intermediate HR image x so as to minimize the evaluation function. In the present embodiment, the update amount of the intermediate HR image x is determined using the steepest descent method so as to minimize the evaluation function, and the intermediate HR image x is updated with the determined update amount. For minimization, a method other than the steepest descent method may be used.

<Processing performed by the image estimation unit 102>
Next, processing performed by the image estimation unit 102 will be described with reference to FIG. 4 showing a flowchart of the processing. First, in step S <b> 401, the image estimation unit 102 acquires an LR image from the image input unit 101. As described above, the LR image is input to the initial image generation unit 202, the buffer 204, and the weight calculation unit 203.

  In step S402, the initial image generation unit 202 generates an image (an image having the same size as the HR image) obtained by enlarging the LR image by s_y times in the vertical direction and s_x times in the horizontal direction, and the generated image (initial HR image). Are stored in the buffer 206 as intermediate HR images. As described above, this enlargement process is performed using an interpolation method such as bi-cubic or cubic interpolation.

In step S _< b _> 403, the weight calculation unit 203 calculates the weight values α _{m, n} , β _{m, n} from the LR image, and stores the calculated weight values α _{m, n} , β _{m, n} in the buffer 205. The calculation method of each weight value will be described later in detail.

  In step S404, the update amount calculation unit 207 obtains an update amount for the intermediate HR image stored in the buffer 206. As described above, this update amount is obtained by using the degradation model read from the degradation model memory 201, the weight value read from the buffer 205, the LR image read from the buffer 204, and the intermediate HR image read from the buffer 206, and Ask for minimization. More specifically, an image calculated by the following formula (7), an image in which the pixel value of the pixel position (m, n) is defined by the formula (8), and the pixel position (m by the formula (9) , N) An image (composite image) obtained by combining an image in which the pixel value is defined is set as an update amount to be obtained.

Equation (7) is the difference between the image generated by applying the deterioration model (D · B) to the intermediate HR image x and the LR image y, the standard deviation σ of the noise amount of the LR image y, and the deterioration model (D -It can be determined using B). Equation (8) can be obtained using the weight values α _{m, n} and ∇ ² x _{m, n} corresponding to the second derivative of the intermediate HR image x. Equation (9) can be obtained using the intermediate HR image x and the weight values β _{m, n} . The pixel value at each pixel position in the composite image obtained in this way is an update amount at the corresponding pixel position in the intermediate HR image x.

  In step S405, the adder 208 updates the intermediate HR image x stored in the buffer 206 by calculating the following equation (10).

That is, the pixel value of the pixel position (m, n) in the image calculated by Expression (7), the pixel value of the pixel position (m, n) calculated by Expression (8), and the expression (9) A value obtained by multiplying the total value of the pixel values at the pixel position (m, n) by the coefficient ε is added to the pixel position (m, n) in the intermediate HR image x. The coefficient ε is the width of the update. If the value is large, the convergence approaches quickly, but if it is too large, the error becomes large or diverges. By performing this for all pixel positions, the intermediate HR image x is updated, and an updated image x ′ is obtained. In Expression (10), the intermediate HR image x before update is ^expressed as x ^(t) (intermediate HR image x updated at the t-th time), and the updated intermediate HR image x is expressed as x ^{(t + 1)} .

  In step S406, the selector 209 determines whether or not the update amount calculation unit 207 and the adder 208 complete the update process of the intermediate HR image. If it is determined to complete the process, the process proceeds to step S407. In step S407, the selector 209 sends the intermediate HR image generated by the adder 208 in step S405 to the image output unit 103 as an HR image.

  On the other hand, if it is determined not to be completed, the selector 209 stores the intermediate HR image generated by the adder 208 in step S405 by overwriting the intermediate HR image stored in the buffer 206. Then, the process returns to step S404.

  In the present embodiment, the number of intermediate HR image update processes by the update amount calculation unit 207 and the adder 208 is defined in advance, and it is determined that the update process is completed when the update process is performed for the specified number of times. However, there are various criteria such as whether or not the time required for the update process exceeds a specified time, and is not limited to any one.

<Calculation of Weight Values α _{m, n} , β _{m, n} >
Next, a process (step S403) performed by the weight calculation unit 203 for obtaining the weight values α _{m, n} , β _{m, n} will be described with reference to FIG. 5 showing a flowchart of the process.

  In step S501, the weight calculation unit 203 uses an interpolation method such as bi-cubic or cubic interpolation to generate an image (enlarged image) obtained by enlarging the LR image to the same size as the HR image. In step S502, the weight calculation unit 203 generates an edge enhanced image in which the edge of the enlarged image is enhanced. As is well known, this edge-enhanced image is an image in which a value indicating whether a pixel at the pixel position P in the enlarged image constitutes an edge or a non-edge is a pixel value at the pixel position P in the edge-enhanced image. It is.

  As the edge-enhanced image, a differential image of the enlarged image may be used, or an image generated from the enlarged image using a method such as Sobel method, Prewitt method, Robert method, zero-cross method, Canny method, or the like may be used.

  In step S503, the weight calculation unit 203 performs a low-pass filter process using an LPF (Low Pass Filter) such as a Gaussian filter on the edge enhanced image, thereby generating a filtered edge enhanced image.

  In step S504, the weight calculation unit 203 refers to the pixel value of each pixel in the filtered edge-enhanced image, crops the pixel value that is greater than or equal to the threshold 1 and the pixel value that is less than or equal to the threshold 2 (<threshold 1). Make strength adjustments to clarify the area.

  In step S505, the weight calculation unit 203 normalizes the pixel value of each pixel constituting the filtered edge enhanced image whose intensity has been adjusted in step S504 to 0-1. The normalization range is not particularly limited.

Then, the pixel value at the pixel position (m, n) in the filtered edge enhanced image after the intensity adjustment normalized in step S505 is set as the weight value β _{m, n} . Further, the weight values α _{m, n} = 1−β _{m, n} are set. The weight calculation unit 203 stores the weight values α _{m, n} , β _{m, n} calculated in this way in the buffer 205.

  As described above, according to the present embodiment, the prior probabilities are adaptively set in the edge region and the non-edge region, and image estimation is performed by the MAP method, so that an HR image with a sharp edge and natural texture (high image) Resolution image) can be generated.

  In the present embodiment, buffers 204 to 206 are provided, and each buffer is used for storing different information. However, one buffer may be provided instead of the buffers 204 to 206, and the information described as being stored in each of the buffers 204 to 206 may be stored in this one buffer. That is, the number of buffers is not limited to three.

[Second Embodiment]
In the first embodiment, one HR image is generated from one LR image. In the present embodiment, one HR image is generated from a plurality of LR images. In the following description, parts different from those of the first embodiment will be described.

<Functional configuration example of image processing apparatus>
A functional configuration example of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. In FIG. 6, the same components as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

  The configuration of FIG. 6 uses the buffer 601 for storing the input image from the image input unit 101 and a plurality of input images in the buffer 601 with respect to the configuration of FIG. A movement amount calculation unit 603 for calculating (movement amount) is added. Further, in the configuration of FIG. 6, the image estimation unit 102 is replaced with an image estimation unit 602 that generates one HR image using a plurality of input images.

  With such a configuration, it is possible to perform image estimation in consideration of the amount of positional deviation for a plurality of input images. That is, the resolution of the generated image can be improved by supplementing the high frequency components that cannot be reproduced from one input image with the misaligned pixels of the plurality of input images.

<Regarding Example Functional Configuration of Image Estimating Unit 602>
A functional configuration example of the image estimation unit 602 will be described with reference to the block diagram of FIG. In FIG. 7, the same components as those in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted.

  In this embodiment, the LR image input via the image input unit 101 is stored in the buffer 601, but the image estimation unit 602 has N (N> 1) LRs stored in the buffer 601. The image is read out and stored in the buffer 703. Of course, all of the N LR images have the same image size.

  The initial image generation unit 202 reads the LR image first stored in the buffer 703, generates an initial HR image from the read LR image in the same manner as in the first embodiment, and uses the generated initial HR image as an intermediate HR image. Is stored in the buffer 206.

The weight calculation unit 203 reads the LR image first stored in the buffer 703, and calculates the weight values α _{m, n} , β _{m, n} from the read LR image in the same manner as in the first embodiment. The weight calculation unit 203 stores the calculated weight values α _{m, n} , β _{m, n} in the buffer 205.

  Note that the LR image input to the initial image generation unit 202 and the weight calculation unit 203 may not be the LR image initially stored in the buffer 703 but may be an LR image selected based on another criterion. . Further, a composite image obtained by aligning and synthesizing a plurality of LR images may be input to the initial image generation unit 202 and the weight calculation unit 203 as an LR image.

  The degradation model generation unit 701 uses the movement amount calculated by the movement amount calculation unit 603 to generate a deterioration model as described in the first embodiment. Since the movement amount calculation unit 603 calculates the movement amount between images for each LR image, if the number of LR images is N, N movement amounts can be obtained. As a result, N deterioration models are also generated. Will be. The deterioration model generation unit 701 stores the generated N deterioration models in the buffer 702. The operations of the movement amount calculation unit 603 and the degradation model generation unit 701 will be described in detail later.

  The update amount calculation unit 704 obtains an update amount for the intermediate HR image using the deterioration model in the buffer 702, the weight value in the buffer 205, the LR image in the buffer 703, and the intermediate HR image in the buffer 206. The operation of the update amount calculation unit 704 will be described later in detail.

<About deterioration model>
Next, the deterioration model generated by the deterioration model generation unit 701 will be described. Image quality degradation that occurs during the process of capturing a single image using an imaging device such as a digital camera, the point spread function (PSF) of the imaging device, downsampling, which is a degradation process due to limitations on the number of sensor pixels, and between images It is expressed by the positional deviation. According to this, if the LR image is y and the HR image is x, the relationship between x and y can be expressed by the following equation (11).

  In this equation (11), M is a matrix representing the positional deviation between images. FIG. 8 shows the size of the column vector and matrix appearing in Equation (11). y, D, B, and x are the same as those in the first embodiment. The matrix M is a matrix of (H × s_y) × (W × s_x) rows (H × s_y) × (W × s_x) columns. In the following, DBM data, which is the product of matrix D, matrix B, and second sequence M, is treated as a deterioration model. That is, in this embodiment, the deterioration model indicates a deterioration process modeled by the matrix DBM.

  The movement amount calculation unit 603 reads N LR images stored in the buffer 601 and sets one of them as a reference LR image. Then, the movement amount calculation unit 603 obtains a relative movement amount between the k (k = 1,..., N) -th LR image and the reference LR image among the N LR images. The relative movement amount is calculated by calculating the sum of the luminance differences and the square sum of the luminance differences while changing the relative positions of the kth LR image and the reference LR image, and fitting with a quadratic function to minimize the value Is calculated by obtaining. There are various methods for obtaining the movement amount between images, and any method may be used.

N row m-th column element of the matrix M, column vector x m-th element representing the HR image (pixel values), the n th element of the column vector x _s representing an HR image by applying a positional deviation It shows how much it contributes. That is, if the relative movement amount is an integer pixel, the value is 1, and if it is a non-integer pixel, the value is interpolated by an appropriate method such as bi-linear or bi-cubic. The degradation model generation unit 701 configures the matrix M in this way. When the matrix M is obtained for k = 1 to N, the matrix M is obtained for the number of LR images, that is, N.

<Image Estimation Processing by Image Estimation Unit 602>
In the first embodiment, one LR image is used to generate one HR image, and the number of degradation models is one. In the present embodiment, the number of LR images used for generating one HR image is N, and the number of degradation models is N. In the first embodiment, the HR image is generated by performing the image estimation process according to the equation (2). However, in the present embodiment, the first term of the equation (2) is modified to correspond to the N LR images. The image estimation process is performed according to the following equation (12).

y _k is the k-th LR image of the N LR images, and DBM _k is a matrix M obtained by multiplying the k-th LR image by the matrices D and B. It is a deterioration model. However, in this embodiment, the HR image is generated using the steepest descent method so as to minimize the evaluation function shown in the following formula (13).

<About the processing performed by the movement amount calculation unit 603 and the image estimation unit 602>
Next, processing performed by the movement amount calculation unit 603 and the image estimation unit 602 will be described with reference to FIG. 9 showing a flowchart of the processing. First, in step S <b> 901, the image estimation unit 602 acquires N LR images from the image input unit 101 and stores the N LR images in the buffer 703.

In step S902, the movement amount calculation unit 603 obtains a movement amount for each of the N LR images stored in the buffer 601. The obtained N movement amounts are input to the deterioration model generation unit 701. In step S903, the deterioration model generation unit 701 obtains N matrices M using N movement amounts. Then, the degradation model generation unit 701 obtains DBM _k for k = 1 to N, and stores the obtained DBM _k in the buffer 702.

  In step S <b> 904, the initial image generation unit 202 selects one of the N LR images stored in the buffer 703. The initial image generation unit 202 generates an image (an image having the same size as the HR image) obtained by enlarging the selected LR image by s_y times in the vertical direction and s_x times in the horizontal direction, and generates the generated image (initial HR image). Are stored in the buffer 206 as intermediate HR images. As described above, this enlargement process is performed using an interpolation method such as bi-cubic or cubic interpolation.

In step S <b> 905, the weight calculation unit 203 selects one of the N LR images stored in the buffer 703. Then, the weight calculation unit 203 calculates the weight values α _{m, n} , β _{m, n} from the selected LR image in the same manner as in the first embodiment _, and calculates the calculated weight values α _{m, n} , β _{m , N} are stored in the buffer 205.

  In step S <b> 906, the update amount calculation unit 704 uses the N degradation models in the buffer 702, the weight values in the buffer 205, the N LR images in the buffer 703, and the intermediate HR images in the buffer 206. The update amount for the HR image is obtained. The method for obtaining the update amount is the same as that in the first embodiment except that the following equation (14) is used instead of the above equation (7).

  In step S907, the adder 208 updates the intermediate HR image x stored in the buffer 206 in the same manner as in the first embodiment. In step S908, the selector 209 determines whether or not the update amount calculation unit 704 and the adder 208 complete the update process of the intermediate HR image. If it is determined to complete the process, the process proceeds to step S909. In step S909, the selector 209 sends the intermediate HR image generated by the adder 208 in step S907 to the image output unit 103 as an HR image.

  On the other hand, if it is determined not to be completed, the selector 209 stores the intermediate HR image generated by the adder 208 in step S907 by overwriting the intermediate HR image stored in the buffer 206. Then, the process returns to step S906.

  As described above, according to the present embodiment, a high-resolution image with a higher resolution can be obtained by using a plurality of input images. In this embodiment, buffers 205, 206, 702, and 703 are provided, and each buffer is used for storing different information. However, one buffer is provided instead of the buffers 205, 206, 702, and 703, and the information described as being stored in each of the buffers 205, 206, 702, and 703 is stored in the one buffer. Also good. That is, the number of buffers is not limited to four.

[Third Embodiment]
In the first and second embodiments, each part in the image processing apparatus shown in FIGS. 1 and 6 has been described as being configured by hardware. However, each unit excluding the buffer may be implemented by software (computer program). In this case, this software is installed in a computer such as a PC (personal computer) or a digital camera having the buffer shown in FIGS. 1 and 6 as a memory, and a CPU included in the computer executes the software. Of course, this memory may be any type of memory as long as it is a computer-readable storage medium. Thereby, this computer functions as an image processing apparatus according to the first and second embodiments.

[Fourth Embodiment]
In the above embodiment, the pixel values of the edge enhanced image are β _{m, n} , α _{m, and n} are (1−β _{m, n} ). However, as the pixel value of the edge enhanced image is greater beta _m, the value of _n is large, if the value of the larger pixel value of the edge enhanced image alpha _{m, n} becomes smaller, alpha using other methods _{m , N} , β _{m, n} may be determined.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An image processing apparatus that generates, from a first image, a second image having a resolution higher than that of the first image and having a larger image size than the first image,
Storage means for generating an image obtained by enlarging the first image to the same size as the second image, and storing the generated image in a memory of the image processing device;
Generating means for generating an edge-enhanced image in which edges in the enlarged image are enhanced;
Updating means for updating an image stored in the memory;
Means for outputting the image updated by the updating means as the second image,
The updating means includes
A first evaluation value calculated by weighting a value representing a change in pixel value between pixels in the stored image stored in the memory using a pixel value of each pixel constituting the edge enhanced image. And a second evaluation value indicating a difference between the first image and the image obtained by reducing the stored image to the same image size as the first image using a specified filter, and Calculation means for obtaining an update amount for the stored image so as to minimize the total;
Means for updating the stored image using the update amount. The first evaluation value is
An evaluation value calculated by weighting the gradient of the change in pixel value between pixels in the stored image, with a weight value that takes a smaller value as the pixel value of the pixels constituting the edge-enhanced image increases.
A weight value that takes a larger value as the pixel value of the pixels constituting the edge-enhanced image is larger, and is a sum of an evaluation value calculated by weighting a change in pixel value between pixels in the stored image. The image processing apparatus according to claim 1.   3. The filter according to claim 1, wherein the prescribed filter is a filter for reproducing image quality degradation that has occurred in the first image in a process of obtaining the first image by imaging. 4. Image processing device.   The image processing apparatus according to claim 1, wherein the calculation unit obtains the update amount using a steepest descent method.   The image processing apparatus according to claim 1, wherein the update unit generates the second image by performing the update a predetermined number of times. Generation of a second image having a resolution higher than that of the first image and having a larger image size than the first image from N (N> 1) first images having the same image size An image processing apparatus that
Storage means for generating an image obtained by enlarging one of the N first images to the same size as the second image, and storing the enlarged image in a memory of the image processing device;
Generating means for generating an image in which one of the N first images is enlarged to the same size as the second image, and generating an edge-enhanced image in which edges in the enlarged image are enhanced;
Updating means for updating an image stored in the memory;
Means for outputting the image updated by the updating means as the second image,
The updating means includes
A first evaluation value calculated by weighting a value representing a change in pixel value between pixels in the stored image stored in the memory using a pixel value of each pixel constituting the edge enhanced image. N images obtained by reducing the stored image to the same image size as the first image by using a predetermined filter for each of the N first images and the N images. Calculation means for obtaining an update amount for the stored image so as to minimize the sum of the second evaluation value indicating a difference from the first image;
Means for updating the stored image using the update amount. An image processing method performed by an image processing apparatus that generates, from a first image, a second image having a resolution higher than that of the first image and having a larger image size than the first image. ,
A storage step in which the storage means of the image processing device generates an image obtained by enlarging the first image to the same size as the second image, and stores the generated image in a memory of the image processing device;
A generation step of generating an edge-enhanced image in which the generation unit of the image processing apparatus emphasizes an edge in the enlarged image;
An update step in which the update means of the image processing apparatus updates the image stored in the memory;
An output unit of the image processing apparatus includes a step of outputting the image updated in the update step as the second image;
The update process includes:
A first evaluation value calculated by weighting a value representing a change in pixel value between pixels in the stored image stored in the memory using a pixel value of each pixel constituting the edge enhanced image. And a second evaluation value indicating a difference between the first image and the image obtained by reducing the stored image to the same image size as the first image using a specified filter, and A calculation step for obtaining an update amount for the stored image so as to minimize the total;
And a step of updating the stored image by using the update amount. Generation of a second image having a resolution higher than that of the first image and having a larger image size than the first image from N (N> 1) first images having the same image size An image processing method performed by an image processing apparatus,
The storage unit of the image processing device generates an image obtained by enlarging one of the N first images to the same size as the second image, and stores the enlarged image in the memory of the image processing device. A storing step for storing;
An edge-enhanced image in which the generation unit of the image processing apparatus generates an image obtained by enlarging one of the N first images to the same size as the second image, and emphasizes an edge in the enlarged image A generating step for generating
An update step in which the update means of the image processing apparatus updates the image stored in the memory;
An output unit of the image processing apparatus includes a step of outputting the image updated in the update step as the second image;
The update process includes:
A first evaluation value calculated by weighting a value representing a change in pixel value between pixels in the stored image stored in the memory using a pixel value of each pixel constituting the edge enhanced image. N images obtained by reducing the stored image to the same image size as the first image by using a predetermined filter for each of the N first images and the N images. A calculation step for obtaining an update amount for the stored image so as to minimize a sum of the second evaluation value indicating a difference from the first image;
And a step of updating the stored image by using the update amount.   A computer program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 6.   A computer-readable storage medium storing the computer program according to claim 9.

Images