JP6661434B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly, to an apparatus for improving the resolution of an image.

  As a method for improving the resolution of an image, a method using interpolation is used. Some fine resolution conversion methods that are less blurry than interpolation by compensating for frequency components exceeding the Nyquist frequency of a signal have been put to practical use as super-resolution techniques.

  As one of the super-resolution methods, there is a method of assuming conversion from a high-resolution image to a low-resolution image, and introducing a regularization term to enable inverse conversion, thereby realizing high resolution ( Patent Document 1 etc.).

  As another super-resolution technique, a method of synthesizing a frequency component exceeding a Nyquist frequency from a high-frequency component (high-frequency component) of an input image has been proposed. As such a method, a frequency-resolved image generated by a plurality of frequency-resolved filters is combined to create a multiplexed image, and a filter process is performed on the multiplexed image (Patent Document 2). There is a method of performing a conversion and then performing a sharpening process using a non-linear operation (Patent Document 3). In order to synthesize a frequency component exceeding the Nyquist frequency, a digital filter train (Patent Document 2) or a nonlinear function (Patent Document 3) A relatively simple operation such as 3) is used.

  Further, there is a case-based super-resolution method in which a pair of a low-resolution patch and a high-resolution patch is stored in a database, and a high-resolution image is obtained from the low-resolution image by referring to the database (Patent Document 4, etc.). ).

Japanese Patent No. 4214409 JP 2014-119949 A Japanese Patent No. 5629902 Japanese Patent No. 5743615

  However, satisfactory images have not yet been obtained with the conventional super-resolution technology. For example, in the super-resolution method using regularization, especially when trying to obtain a high-resolution image from a single frame of low resolution, the dimension number of the unknown variable greatly exceeds the dimension number of the constraint, and the regularization term It is impossible to solve the inverse problem by introducing. As a result, a high resolution result of a flat image tone in which similar pixel values are continuous is obtained.

  In a method of synthesizing a frequency component exceeding the Nyquist frequency (also referred to as an ultra-Nyquist component) from a high-frequency component of an input signal as in the super-resolution method of Patent Document 2 or Patent Document 3, the super-Nyquist component is reliably added. Therefore, it is possible to obtain a fine sense of appearance. However, since the method of synthesizing the super-Nyquist component from the high-frequency component of the input image is simple, an unnatural super-Nyquist component may be generated. In particular, in the method of Patent Document 2, the edge of the pattern of the super Nyquist component tends to be thicker than that of the original pattern.

  The technique of referring to the database can obtain a relatively natural super-resolution result if a sufficient database is constructed. However, if the database is insufficient, high resolution is performed based on the high resolution patch corresponding to the low resolution patch that does not sufficiently match, and a high-resolution image that looks strange is generated. Would. The high-resolution patch used at this time was a pattern that actually existed as an image, and it was misused (such as applying an image of another person's eyes to the face image) to achieve high resolution, so the sense of incongruity was enormous. It is.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention, which has been made in view of the above-described problems, is to provide an image processing apparatus capable of obtaining a high-resolution image having a more natural and detailed feeling.

In order to solve the above problem, an image processing apparatus according to the present invention includes a high-frequency generation unit for generating a high-frequency component of the input low-resolution image from the low-resolution image, the low-resolution image and the high-frequency image. A high-resolution image having a higher resolution than the low-resolution image by band-synthesizing the components, and outputting the high-resolution image. Comparing the plurality of acquired difference values for each pixel position, and determining a high-frequency component value of the pixel position based on a difference value having a minimum absolute value of the difference value as a result of the comparison. .

  In the image processing apparatus, the high-frequency generation unit includes a first high-frequency generation unit that generates a high-frequency component in the horizontal direction, a second high-frequency generation unit that generates a high-frequency component in the vertical direction, A third high-frequency generation unit that generates an angular high-frequency component, wherein the band synthesis unit receives the input low-resolution image, the horizontal high-frequency component, the vertical high-frequency component, In addition, it is preferable to generate each pixel of the high-resolution image by performing an operation based on the diagonal high-frequency component.

  In the image processing apparatus, it is preferable that the operation of the band synthesis unit is band synthesis based on a two-dimensional Haar basis.

  In the image processing apparatus, the first high-frequency generation unit and the second high-frequency generation unit obtain at least two types of spatial differences having a phase difference, and the third high-frequency generation unit It is desirable to acquire at least four types of spatial differences having a phase difference.

  Further, in the image processing apparatus, the first to third high-frequency generation units may obtain two difference values having the same absolute value but different signs as the difference value having the smallest absolute value of the difference value. In addition, it is desirable to further include a process of setting the high-frequency component value of the high-frequency generation unit to 0.

  According to the image processing apparatus of the present invention, the frequency component exceeding the Nyquist frequency is generated from the input low-resolution video, and the phenomenon that the edge of the processed image becomes wider than the original pattern should be suppressed, so that more natural and fine It is possible to obtain a high-resolution image with a feeling.

FIG. 1 is a block diagram illustrating an example of an image processing device according to the present invention. FIG. 3 is a block diagram illustrating an example of a high-frequency generation unit of the image processing device according to the present invention. It is a figure explaining a spatial difference operation. FIG. 9 is a block diagram illustrating an example of a conventional high-frequency generation unit. 5 is a comparison of the results of image enlargement performed by the interpolation processing, the conventional method, and the image processing apparatus according to the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment)
FIG. 1 is a block diagram illustrating an example of an image processing apparatus according to the present invention. The image processing device 1 converts the input low-resolution image LL into a high-resolution image and outputs it as a high-resolution image S. In FIG. 1, the image processing apparatus 1 includes three high-frequency generation units 2 (2 _{1 to} 2 ₃ ) and a band synthesis unit 3.

High-frequency creating unit 2 ₁ a high-frequency component HL, the high-frequency creating unit 2 ₂ high-frequency component LH, also the high-frequency creating unit 2 ₃ respectively generate high-frequency component HH. For example, the high frequency component HL is a horizontal high frequency component, that is, a vertical edge component, the high frequency component LH is a vertical high frequency component, that is, a horizontal edge component, and the high frequency component HH is a diagonal high frequency component, that is, It is assumed that each represents an oblique edge component.

  The band synthesis unit 3 performs band synthesis on the input low-resolution image LL and the generated high-frequency components HL, LH, and HH to generate a high-resolution image S having a higher resolution than the low-resolution image LL. And output.

  Typically, the resolution of each of the generated high frequency components HL, LH and HH is the same as the resolution of the low resolution image LL, and the high resolution image has a resolution of 2 for both the horizontal and vertical resolutions of the low resolution image. The resolution is double (that is, four times the number of pixels). Hereinafter, the resolutions of the low-resolution image LL and the high-frequency components HL, LH, and HH are referred to as horizontal resolution A and vertical resolution B, respectively, and the resolution of the high-resolution image S is referred to as horizontal resolution 2A and vertical resolution 2B. Note that both A and B are natural numbers, but the conditions under which the image processing apparatus 1 according to the present invention operates effectively are A ≧ 3 and B ≧ 3.

  Hereinafter, the pixel value of the image P at the image coordinates (x, y) is described as P (x, y), and the image coordinates are expressed as an integer of 0 or more. The image coordinates (0, 0) are the pixel position at the upper left corner of the image, the first component x of the image coordinates (x, y) is horizontal right, and the second component y is vertical down.

  The band synthesis unit 3 performs band synthesis using, for example, a two-dimensional Haar basis. That is, the band synthesizing unit 3 generates and outputs the high-resolution image S according to the following equation (1).

  Alternatively, the high-resolution image S may be generated by the following equation (2). Equations (1) and (2) are equivalent. By the processing of the band synthesizing unit 3, a frequency component exceeding the Nyquist frequency is generated.

  The band synthesis method is not limited to the two-dimensional Haar base described above, and may be band synthesis based on, for example, an n-order Daubechies base. The use of the above formula (1) of the Haar basis can simplify the circuit configuration and reduce the circuit scale.

FIG. 2 is a block diagram showing an example of the high-frequency generation unit 2 (2 _{1 to} 2 ₃ ) in FIG. High-frequency generator 2 of FIG 2 includes an N-number of spatial difference calculation section 4 _{(4 1} to _{4 N),} N number of absolute value calculating section 5 _{(5 1} to _{5 N),} a comparing unit 6, selection A section 7 and an amplification section 8 are provided.

Note that N is a natural number of 2 or more. The value of N to not be the same all the high-frequency creating unit 2 ₁ to 2 _3, may be the same. Most effective is, N = 2 for the high-frequency creating unit 2 ₁ to generate the horizontal high-frequency component, for high-frequency creating unit 2 ₂ which generates a vertical high-frequency component N = 2, Further, for the high-frequency creating unit 2 ₃ for generating a diagonal direction high frequency component and N = 4.

The spatial difference calculation unit 4 (4 _{1 to} 4 _N ) calculates a spatial difference (pixel value difference) at different phases (positions).

The absolute value calculator 5 _n (n is an integer of 1 or more and N or less) takes an absolute value for each pixel position and outputs the spatial difference value output from the corresponding spatial difference calculator 4 _n .

The comparison unit 6 compares the absolute values output by the absolute value calculation units 5 (5 _{1 to} 5 _N ) for each pixel position, and specifies the absolute value calculation unit 5 _Nmin that outputs the smallest absolute value among them. Then, the identifier N _min is output for each pixel position.

Selector 7, for each pixel position, selects and outputs the output as specified by the identifier N _min from the spatial difference value output of the spatial difference computing section 4 (4 ₁ to 4 _N).

The amplification unit 8 multiplies the output of the selection unit 7 by the gain a, and outputs the result as a high-frequency component output. The gain a is a real number, and may be a positive value, a negative value, or 0. The value of the gain a is selected according to the degree of emphasizing the edge portion. Also, the gain a is may be all not be the same in the high-frequency creating unit _{2 1} to _{2 3} (high frequency component HL, LH, and respectively for HH independent gain _{a HL,} the _{a LH,} and _{a HH} May be set) or may be the same. However, gain a must not be all zero for the high-frequency creating unit 2 ₁ to 2 _3.

Hereinafter, each high-frequency generator _{2 1} to _{2 3,} spatial difference calculation section 4 _{(4 1} to ₄ N), absolute value calculation section 5 _{(5 1} to ₅ N), each of the comparison unit 6, and the selector 7 The implementation will be specifically illustrated.

(Implementation of the high-frequency creating unit _{2 1)}
High-frequency creating unit 2 _1, as comprising N = 2 pieces of spatial difference calculation section 4 (4 ₁ and 4 _2), described their implementation.

Both spatial difference calculating unit 4 ₁ and spatial difference calculating section 4 ₂ performs spatial difference calculation in the horizontal direction, but performs difference calculation shifted by one pixel in the horizontal direction. For example, spatial difference HL 1 the output of the spatial difference calculating unit 4 _1, when the output of the spatial difference calculating section 4 ₂ placed between the spatial difference HL _2, the following equation (3), to obtain respective results.

FIG. 3 is a diagram illustrating the spatial difference calculation. FIG. 3A shows the positional relationship of pixels related to the calculation of the spatial differences HL ₁ and HL ₂ when the target pixel A is set to the coordinates (1, 1), and FIG. An example of the calculation result of the difference (x direction) is shown. HL ₁ is a difference value between the target pixel A and its left adjacent pixel, HL ₂ is the difference value of the pixel of interest A and the pixel of the right side.

Absolute value calculation unit _{5 1,} to the input value _HL 1 (x, y), the absolute value _{| HL 1 (x, y)} | outputs a. The absolute value calculating unit _{5 2,} to the input value _HL 2 (x, y), the absolute value _{| HL 2 (x, y)} | outputs a.

The comparing unit 6 compares the absolute value | HL ₁ (x, y) | and the absolute value | HL ₂ (x, y) | for each pixel position (x, y), and identifies the identifier N whose value is the minimum. _{Find min} (x, y). That is, N _min (x, y) is represented by the following equation (4). In Equation (4), arg min is an abbreviation of argument of the minimum (an argument that gives a minimum value), and is a set formed by all elements of a domain in which a function takes the minimum value.

In the example of FIG. 3B, since | HL ₁ (1,1) |> | HL ₂ (1,1) |, N _min (1,1) = 2.

Selector 7, the pixel position (x, y) for each, spatial difference value outputted from the spatial difference calculating unit _{4 1 HL} 1 (x, y) and the spatial difference value _HL 2 which outputs the spatial difference calculating unit _{4 2} ( x, y), the output of the spatial difference operation unit 4 _Nmin specified by the identifier N _min (x, y) is selected, and the result is output as HL ′ (x, y). Therefore, HL ′ (x, y) is represented by the following equation (5).

  Here, the selection of a value having a small absolute value of the spatial difference value means that the edge of the pattern is thinned in the processed image.

The amplification unit 8 obtains a high-frequency component HL (x, y) by multiplying the output value HL ′ (x, y) of the selection unit 7 by the gain a _HL . The value of the gain _aHL is, for example, _aHL = 0.5. This value is desirably obtained experimentally so that a clear high-definition image can be obtained. In general, a value of about 0.3 to 0.7 is selected. The high frequency component HL (x, y) is given by Expression (6).

In the normal image processing, the calculation of the high frequency component HL is performed as described above, but the spatial difference value HL ₁ (x, y) and the spatial difference value HL ₂ (x, y) have the same absolute value. When the signs are opposite, it is desirable to add processing for setting the value of the high frequency component HL (x, y) to 0. If the absolute value of the spatial difference value is the same in the adjacent pixels and the sign is inverted, employing one spatial difference value may generate an unnatural image. As such processing, when the comparing unit 6 determines that the absolute values of the two spatial difference values are the same and the signs are opposite, HL ′ (x, y) is replaced with 0 in Expression (5). Alternatively, only under such conditions, the value of the gain a _HL may be set to 0 in Expression (6).

The operations of the absolute value calculation unit 5 (5 ₁ and 5 ₂ ), the comparison unit 6, the selection unit 7, and the amplification unit 8 can be summarized and calculated by a single means. It may be executed (implemented based on equation (7)).

  In this implementation example, the difference is obtained from the pixel adjacent to the target pixel in the horizontal direction with N = 2. However, for example, the pixel adjacent to the target pixel in the vertical and diagonal directions is also set as N = 4. Thus, obtaining a difference in the horizontal direction is one method.

(Implementation of the high-frequency creating unit _{2 2)}
High-frequency creating unit 2 _2, as comprising N = 2 pieces of spatial difference calculation section 4 (4 ₁ and 4 _2), described their implementation.

Both spatial difference calculating unit 4 ₁ and spatial difference calculating section 4 ₂ performs spatial difference calculation in the vertical direction, but performs difference calculation shifted by one pixel in the vertical direction. For example, spatial difference LH 1 the output of the spatial difference calculating unit 4 _1, when the output of the spatial difference calculating section 4 ₂ placed between the spatial difference LH _2, the following equation (8), to obtain respective results.

FIG. 3A shows the positional relationship of the pixels related to the calculation of the spatial differences LH ₁ and LH ₂ when the target pixel A is set to the coordinates (1, 1). LH ₁ is a differential value of the target pixel A and the pixel thereon, LH ₂ is the difference value between pixels of the lower target pixel A.

Absolute value calculation unit _{5 1,} to the input value _LH 1 (x, y), the absolute value _{| LH 1 (x, y)} | outputs a. The absolute value calculating unit _{5 2,} to the input value _LH 2 (x, y), the absolute value _{| LH 2 (x, y)} | outputs a.

The comparing unit 6 compares the absolute value | LH ₁ (x, y) | and the absolute value | LH ₂ (x, y) | for each pixel position (x, y), and identifies the identifier N whose value is the minimum. _{Find min} (x, y). That is, N _min (x, y) is represented by the following equation (9).

Selector 7, the pixel position (x, y) for each spatial difference computing section _{4 1} of the output spatial difference value _LH 1 (x, y) and the spatial difference value _LH 2 to output spatial difference calculating unit _{4 2} ( x, y), the output of the spatial difference calculator 4 _Nmin specified by the identifier N _min (x, y) is selected, and the result is output as LH ′ (x, y). Therefore, LH '(x, y) is represented by the following equation (10).

The amplification unit 8 obtains a high-frequency component LH (x, y) by multiplying the output value LH ′ (x, y) of the selection unit 7 by the gain a _LH . The value of the gain a _LH is, for example, a _LH = 0.5. This value is desirably obtained experimentally so that a clear high-definition image can be obtained. In general, a value of about 0.3 to 0.7 is selected. The high frequency component LH (x, y) is given by Expression (11).

In the calculation of the high-frequency component LH, when the absolute values of the spatial difference value LH ₁ (x, y) and the spatial difference value LH ₂ (x, y) are the same and the signs are opposite, the high-frequency component LH ( It is desirable to add a process for setting the value of (x, y) to 0. As such processing, when the comparing unit 6 determines that the absolute values of the two spatial difference values are the same and the signs are opposite, LH ′ (x, y) is replaced with 0 in equation (10). Alternatively, only under such conditions, the value of the gain a _LH may be set to 0 in Expression (11). This is similar to the horizontal component processing.

The absolute value calculation section 5 (5 ₁ and 5 _2), the comparing unit 6, summarizes the operation of the selector 7, and the gain portion 8, it is also possible to calculate by one means, the following equation (12) You can do it.

  In this implementation example, the difference is obtained from the pixel adjacent to the target pixel in the vertical direction with N = 2. However, for example, the pixel adjacent to the target pixel in the left, right, and diagonal directions is used with N = 4. Then, obtaining a difference in the vertical direction is one method.

(Implementation of the high-frequency creating unit _{2 3)}
High-frequency creating unit 2 _3, as comprising N = 4 pieces of spatial difference calculating unit 4 (4 ₁ to 4 _4), is described for their implementation.

While performing the spatial difference calculation of the spatial difference computing unit 4 ₁ to spatial difference calculating unit 4 ₄ also pairs Any angular and spatial difference calculating section 4 ₂ to spatial difference calculating section 4 ₄ against spatial difference calculating unit 4 ₁ The difference calculation is performed by shifting one pixel in the horizontal and vertical directions and in both the horizontal and vertical directions. For example, when the output of the spatial difference calculation unit 4 _n is set as the spatial difference HH _n ( _n {1, 2, 3, 4}), the result of each calculation unit is obtained by the following equation (13).

FIG. 3A shows the positional relationship of the pixels related to the calculation of the spatial differences HH ₁ , HH ₂ , HH ₃ , and HH _{4 when} the target pixel A is set to the coordinates (1, 1).

The absolute value calculation unit 5 _n ( _n {1, 2, 3, 4}) outputs the absolute value | HH _n (x, y) | for the input value HH _n (x, y).

The comparing unit 6 calculates an absolute value | HH ₁ (x, y) |, an absolute value | HH ₂ (x, y) |, an absolute value | HH ₃ (x, y) | for each pixel position (x, y). , And an absolute value | HH ₄ (x, y) |, and an identifier N _min (x, y) having the minimum value is obtained. That is, N _min (x, y) is represented by the following equation (14).

Selector 7, the pixel position (x, y) for each spatial difference value _HH 1 to the output of the spatial difference calculating unit _{4 1} to spatial difference calculating unit _{4 4} (x, _y), HH 2 (x, y), The output of the spatial difference calculation unit 4 _Nmin specified by the identifier N _min (x, y) is selected from HH ₃ (x, y) and HH ₄ (x, y), and the result is HH ′ ( x, y). Therefore, HH '(x, y) is represented by the following equation (15).

The amplification unit 8 obtains the high-frequency component HH (x, y) by multiplying the output value HH ′ (x, y) of the selection unit 7 by the gain a _HH . The value of the gain a _HH is, for example, a _HH = 0.1. It is desirable to experimentally determine this value so that a clear high-definition image can be obtained.

In the calculation of the high frequency component HH, there are two spatial difference values HH _n (x, y) whose absolute values are minimized, and the two spatial difference values HH _n have the same absolute value and opposite signs. It is desirable to add a process for setting the value of the high frequency component HH (x, y) to 0. As such processing, when the comparing unit 6 determines that the absolute values of the two spatial difference values are the same and the signs are opposite, HH ′ (x, y) is replaced with 0 in Expression (15). Or, only under such conditions, the value of the gain a _HH may be set to 0 in Expression (16).

Further, the operations of the absolute value calculation unit 5 (5 _{1 to} 5 ₄ ), the comparison unit 6, the selection unit 7, and the gain unit 8 can be summarized and calculated by a single means. You can do it.

  Hereinafter, a comparative example of a result of high resolution by the image processing apparatus according to the present invention and a result of high resolution by the conventional method (conventional wavelet super-resolution) will be described.

Here, as the conventional method, a case is shown in which the conventional high frequency generation unit 20 shown in FIG. 4 is used instead of the high frequency generation unit 2 (2 _{1 to} 2 ₃ ) in FIG. The conventional high-frequency generation unit 20 includes a spatial difference calculation unit 21 and a convolution unit 22. Note that the convolution unit 22 may be implemented as a simple multiplication of gain.

  In the conventional method, Gaussian is convolved in the convolution unit 22. However, in the conventional high-frequency generation, an excessively thick edge is regarded as a problem, and is a problem to be solved in the present invention. Therefore, in the following, the convolution unit 22 performs a simple gain multiplication (corresponding to an impulse multiplication) with the convolution Gaussian being the sharpest limit.

Incidentally, the conventional high frequency generator 20 of the high-frequency creating unit 2 for _1, a conventional high frequency generator 20 of the high-frequency creating unit conventional high frequency generator 20 for 2 _2, and the high-frequency generator 2 ₃ The high-frequency components HL, LH, and HH respectively output by are defined as in the following equation (18), and a comparative experiment was performed. Equation (18) corresponds to one of the wavelet super-resolution using the Haar basis.

  FIG. 5 shows the results of the comparative experiment. FIG. 5A shows a result obtained by upsampling and performing 2 × 2 image enlargement by interpolation using Lanczos-3 as an interpolation function. In the interpolation interpolation, since the information at the Nyquist frequency or higher in the image before the enlargement cannot be reproduced, the result is conspicuously blurred.

  FIG. 5B shows the result of 2 × 2 image enlargement by the conventional method (conventional wavelet super-resolution) shown in FIG. 4 and equation (18). Although there is a finer feeling than the result of FIG. 5 (a), there is a feeling of edging near the outline, a white line is seen outside the boundary, and a black line is seen inside, and the outline tends to be thick. . This is due to the thick edges of the generated high frequency components.

  FIG. 5C shows the result of 2 × 2 image enlargement performed by the image processing apparatus according to the present invention. 5 (a) and 5 (b), and the edge-like artifacts shown in FIG. 5 (b) are also reduced. Therefore, the contour portion looks sharper and a natural image is obtained. The present invention can greatly improve an image with a small number of calculations (small circuit scale).

  Note that a computer can be suitably used to function as the above-described image processing apparatus 1. Such a computer stores a program describing processing contents for realizing each function of the image processing apparatus 1 in a storage unit of the computer. And the program is read out and executed by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiment, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiment can be combined into one, or one constituent block can be divided.

First image processing apparatus _{2, 2 1,} 2 2, _{2 3} high frequency generator 3 band synthesis section _{4, 4 1,} 4 2, ..., _{4 N} spatial difference calculating unit _{5,5 1,} 5 2, ..., _{5 N} Absolute value calculation unit 6 Comparison unit 7 Selection unit 8 Amplification unit 20 Conventional high frequency generation unit 21 Spatial difference calculation unit 22 Convolution unit

Claims (5)

A high-frequency generation unit for generating a high-frequency component of the image from the input low-resolution image,
The low-resolution image and the high-frequency component by band-synthesizing, comprising a band synthesis unit that generates and outputs a high-resolution image with a higher resolution than the low-resolution image ,
The high-frequency generation unit obtains a plurality of spatial differences having a phase difference with respect to an image, compares the plurality of obtained difference values for each pixel position, and, as a result of the comparison, a difference value having the smallest absolute value of the difference value An image processing device for determining a high-frequency component value at the pixel position based on the image data.   2. The image processing device according to claim 1, wherein the high-frequency generation unit generates a high-frequency component in a horizontal direction and a second high-frequency generation unit generates a high-frequency component in a vertical direction. 3. And a third high-frequency generation unit that generates a diagonal high-frequency component, wherein the band synthesis unit receives the input low-resolution image, the horizontal high-frequency component, and the vertical high-frequency component. An image processing apparatus that performs an operation based on a band component and the diagonal high band component to generate each pixel of a high-resolution image.   3. The image processing apparatus according to claim 2, wherein the operation of the band synthesis unit is band synthesis based on a two-dimensional Haar basis.   4. The image processing device according to claim 2, wherein the first high-frequency generation unit and the second high-frequency generation unit obtain at least two types of spatial differences having a phase difference, and An image processing apparatus, wherein the high-frequency generation unit acquires at least four types of spatial differences having a phase difference.   5. The image processing device according to claim 2, wherein the first to third high-frequency generation units determine that the absolute value of the differential value is the minimum and the sign of the absolute value is the same as the differential value having the minimum absolute value. An image processing apparatus further comprising a process of setting a high-frequency component value of the high-frequency generation unit to 0 when two different difference values are obtained.