JP2015121884A - Image processing apparatus, method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide super-resolution processing of preventing noise deterioration from being visually perceived so as to improve sharpness over the entire image.SOLUTION: An image processing apparatus acquires noise information according to a plurality of types of resolution conversion processing, calculates, for the noise information and an input image, a noise map showing how noise is easily perceived in the input image, determines an addition ratio of a plurality of images subjected to resolution conversion processing, and composes the plurality of images subjected to the resolution conversion processing according to the addition ratio.

Description

  The present invention relates to an image processing apparatus for converting image resolution, a method thereof, and a program.

  Along with the high resolution of images displayed on a display device such as a television or a display, the situation in which video content input to the display device is less than the resolution of the displayed image is increasing. For this reason, the importance of super-resolution technology for generating high-resolution video from low-resolution video is increasing. There are two main types of super-resolution techniques: learning and reconstruction. Reconstruction-type super-resolution technology first sets an initial high-resolution image, then estimates a low-resolution image based on the camera model, and minimizes the error between the estimated image pixel value and the actual observed pixel value. In this way, a super-resolution image is acquired by updating a high-resolution image. On the other hand, learning-type super-resolution is a method of obtaining a high-resolution image by comparing an input image with a previously learned high-resolution image database and replacing it with a high-resolution pattern. Note that, for example, Patent Document 1 is an example of a document that describes a reconstruction-type super-resolution technique, and Patent Document 2 is an example of a document that describes a learning-type super-resolution technique.

  On the other hand, it is known that the super-resolution technique has a trade-off problem between noise and sharpness. For example, in the learning type super-resolution technique, an image with high sharpness is generally generated, while noise is easily mixed with an output image, and a broken image may be generated. On the other hand, in the reconstruction type super-resolution, generally only an image with a relatively low sharpness can be generated, but no noise is generated in the output image.

  As a technique for solving such a trade-off problem, a method of properly using a plurality of super-resolution techniques according to images has been proposed. For example, Patent Document 3 attempts to achieve both noise and sharpness by switching super-resolution processing according to the amount of edges included in a region.

JP 2008-140012 A Japanese Patent No. 3321915 JP 2006-221221 A

S. Daly, "The Visible Differences Predictor: An algorithm for the assessment of image fidelity," in A.B.Watson, editor, Digital Image and Human Vision, pages 179-206, Cambridge, MA, MIT Press, 1993.

  It is well known as a visual masking effect that the perception (viewing) of visual noise varies depending on the background image. Therefore, by changing the super-resolution processing according to the image area, the above-described problem of sharpness and noise trade-off can be solved. However, in the known technique, an effect can be obtained only in a part of an image in which vision is easy to perceive noise. For example, according to Patent Document 3, since the super-resolution processing is switched according to the edge amount, the same super-resolution processing is performed in a region other than the edge. However, the noise masking effect due to the background also occurs in a region other than the edge, such as a texture region such as a distant forest. That is, the technique according to Patent Document 3 has a problem that it cannot be said that the sharpness and noise trade-off problem has been sufficiently solved.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a super-resolution process that can improve sharpness in a whole image without noise degradation being perceived by vision.

  As a means for achieving the above object, an image processing apparatus of the present invention comprises the following arrangement. That is, a plurality of conversion units that perform different resolution conversion processes on the input image, an acquisition unit that acquires noise information according to the resolution conversion process, and the input image based on the noise information and the input image Calculating means for calculating a noise map representing the ease of perception of noise, determining means for determining an addition ratio of a plurality of images subjected to the resolution conversion processing based on the noise map, and the addition ratio And a synthesizing unit for synthesizing the plurality of images subjected to the resolution conversion processing.

  It is possible to realize super-resolution processing that improves sharpness without perceiving noise degradation in the entire image.

The block diagram which shows the structure of the image processing apparatus by some embodiment. The block diagram which shows the structure of the image process part by 1st Embodiment. The block diagram which shows the structure of the noise map calculation part by 1st Embodiment. The schematic diagram which shows a spatial visual characteristic. The schematic diagram which shows the characteristic of the circumferential direction of a band-division filter. The schematic diagram which shows the characteristic of the radial direction of a band division filter. The schematic diagram showing the frequency band division by the band division filter. The schematic diagram which shows masking intensity | strength. The block diagram which shows the structure of the image synthetic | combination part by 1st Embodiment. The block diagram which shows the structure of the image addition part by 1st Embodiment. The block diagram which shows the structure of the image process part by 2nd Embodiment. The block diagram which shows the structure of the image synthetic | combination part by 2nd Embodiment. The block diagram which shows the structure of the noise map calculation part by 3rd Embodiment. The block diagram which shows the structure of the image process part by 4th Embodiment. The block diagram which shows the structure of the noise information acquisition part by 4th Embodiment. The block diagram which shows the structure of the noise map calculation part by 4th Embodiment. The block diagram which shows the structure of the noise map calculation part by 5th Embodiment. The schematic diagram showing the frequency band division by the band division filter by 5th Embodiment.

[First Embodiment]
In this embodiment, two super-resolution processes are performed on the input low-resolution image, and the output image after the super-resolution process is synthesized so as to increase the resolution while suppressing noise. At this time, the mixing ratio of the two super-resolution output images is determined based on a noise map obtained by mapping the ease of perceiving noise according to the image structure of the input image. As a result, super-resolution processing with less noise can be assigned to areas where noise is easily perceived, and super-resolution processing with high resolution can be assigned to areas where noise is difficult to perceive. Resolution becomes possible. Note that Non-Patent Document 1 describes the noise sensitivity model in detail.

<Configuration of image processing apparatus>
FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the present embodiment. FIG. 1 shows a configuration of a digital camera as an image processing apparatus. The imaging unit 101 includes a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, and a sensor such as a CMOS or a CCD, and detects the amount of light of the subject. The A / D conversion unit 102 converts the light amount of the subject into a digital value. The signal processing unit 103 performs demosaicing processing, white balance processing, gamma processing, and the like on the digital value converted by the A / D conversion unit 102 to generate a digital image. The image processing unit 104 performs noise reduction processing on the digital image generated by the signal processing unit 103. The encoder unit 105 performs processing for converting the digital image after noise reduction into a video compression format such as JPEG. The media interface 106 is an interface for connecting to 107 media (for example, hard disk, memory card, CF card, SD card, USB memory).

  The CPU 108 is involved in all the processes of each configuration, and sequentially reads the instructions stored in the ROM 109 and the RAM 110 and executes the processes. The CPU 108 controls each component via the system bus 115. The ROM 109 and RAM 110 provide the CPU 108 with programs, data, work areas, and the like necessary for processing. The imaging control unit 111 performs control of the imaging system instructed by the CPU 108 such as focusing, opening a shutter, and adjusting an aperture. The operation unit 112 corresponds to a button, a mode dial, and the like, and receives a user instruction input via these buttons. User instructions include shooting settings such as ISO sensitivity setting, shutter speed setting, and F value setting. These shooting settings are processed by the CPU 108, reflected in the shooting conditions of the digital camera, and stored in the RAM 110. The D / A conversion unit 113 performs analog conversion on the digital image whose noise has been reduced by the image processing unit 104. In general, a liquid crystal display is widely used as the display unit 114 and displays a captured image received from the D / A conversion unit 113.

<Configuration of Image Processing Unit 104>
FIG. 2 is a diagram illustrating a configuration of the image processing unit 104 according to the present embodiment. A digital image signal generated by the signal processing unit 103 is input to the terminal 201. The parameter acquisition units 202A and 202B acquire the setting parameters of the respective processes specified from the first super-resolution processing unit 206 and the second super-resolution processing unit 207 from the RAM 110 via the system bus 115, respectively. To do. The noise characteristic storage units 203A and 203B associate the noise characteristics corresponding to the process setting parameters of the first super-resolution processing unit 206 and the second super-resolution processing unit 207 using a table or a relational expression, respectively. Remember. In addition, the noise characteristic in this embodiment consists of a noise amount and an autocorrelation function. The noise information acquisition units 204A and 204B acquire noise characteristics from the noise characteristic storage units 203A and 203B, respectively, based on the setting parameters acquired from the parameter acquisition units 202A and 202B, respectively.

  Then, the noise information acquisition units 204A and 204B transmit the acquired noise characteristics as noise information included in the image to the noise map calculation units 205A and 205B, respectively. The noise map calculation units 205A and 205B express the ease of perception of noise in the image based on the noise characteristics acquired as noise information by the noise information acquisition units 204A and 204B and the digital image input from the signal processing unit 103, respectively. A noise map is generated. The first super-resolution processing unit 206 and the second super-resolution processing unit 207 perform resolution conversion processing on the digital images obtained from the signal processing unit 103 via the system bus 115, respectively. The image composition unit 208 calculates an addition ratio based on the images obtained by the first super-resolution processing unit 206 and the second super-resolution processing unit 207, combines the images with the calculated addition ratio, A single high-resolution image is output. From the terminal 209, a digital image subjected to super-resolution processing is output. Note that 202A to 205A in FIG. 2 may be configured as the same blocks as 202B to 205B, respectively. In this case, processing corresponding to the first super-resolution processing unit 206 and the second super-resolution processing unit 207 is performed in parallel or switched.

ここでαは、ノイズ情報取得部204Aで取得されたノイズ量に関する補正項である。また、ノイズ情報取得部204Aで取得されたノイズの自己相関関数のフーリエ変換結果と、c(x,y)のパワースペクトラムが一致するように、c(x,y)は求められる。 <Configuration of Noise Map Calculation Units 205A and 205B>
FIG. 3 is a diagram illustrating a configuration of the noise map calculation unit 205A according to the present embodiment. The noise map calculation unit 205B has the same configuration as the noise map calculation unit 205A. A digital image generated by the signal processing unit 103 is input to the terminal 301. Let this digital image be i _in (x, y). Noise information is input to the terminal 302 from the noise information acquisition unit 204A. The noise image generation unit 303 generates a white noise image n _w (x, y) at the same position as the digital image so as to have a predetermined power. The filter unit 304 performs the following calculation on the generated white noise image n _w (x, y) to estimate the camera noise image n (x, y).

Here, α is a correction term related to the amount of noise acquired by the noise information acquisition unit 204A. Also, c (x, y) is obtained so that the Fourier transform result of the autocorrelation function of noise acquired by the noise information acquisition unit 204A matches the power spectrum of c (x, y).

ここで、p(mm)は画素ピッチ、R(mm)は視距離、Nxは入力画像の横画素数、Nyは入力画像の縦画素数である。これらの値は、一般的なユーザー観察環境に基づいて定められる。なお、VTFsx(u)並びにVTFsy(v)は、空間視覚特性

を視距離と画面ピッチと画素数に応じて周波数空間でサンプリングしたものであり、基本形は同じである。 The Fourier transform units 305A and 305B perform Fourier transform on the digital image and the noise image to generate spatial frequency information. Hereinafter, the Fourier transform result of the digital image i _in (x, y) is I _in (u, v), and the Fourier transform result of the noise image n (x, y) is N (u, v). Spatial visual characteristic multiplication units 306A and 306B, for the free transformation result I _in (u, v) of the digital image and the Fourier transformation result N (u, v) of the noise image, VTF (u, Multiply v) and obtain IV (u, v) and NV (u, v) as the multiplication results. VTF (u, v) is defined as VTF (u, v) = VTFsx (u) × VTFsy (v), where VTFsx (u) is the spatial visual characteristic in the horizontal direction and VTFsy (v) is in the vertical direction. Spatial visual characteristics. These are respectively defined by the following equations.

Here, p (mm) is the pixel pitch, R (mm) is the viewing distance, Nx is the number of horizontal pixels of the input image, and Ny is the number of vertical pixels of the input image. These values are determined based on a general user observation environment. VTFsx (u) and VTFsy (v) are spatial visual characteristics.

Is sampled in the frequency space according to the viewing distance, the screen pitch, and the number of pixels, and the basic shape is the same.

  FIG. 4 is a schematic diagram of the spatial visual characteristic VTF (u, v). In FIG. 4, the visual sensitivity to the spatial frequency is shown as the spatial visual characteristic. According to FIG. 4, it can be seen that the visual sensitivity increases as the spatial frequency of the image increases, that is, the image becomes easier to see. It can also be seen that the higher the spatial frequency of the image, the lower the visual sensitivity, that is, the image becomes difficult to see.

The band division units 307A and 307B respectively apply the band division filter cortex _{k, l} (u, v) to the spatial frequency information IV (u, v) and NV (u, v) which are multiplication results by the spatial visual characteristic multiplication unit 306. ) To divide the bandwidth. k is an index of the band in the radial direction, and l is an index of the band in the circumferential direction. The spatial frequency information after the band division is expressed by the following equations, respectively. Is _{k, l} (u, v) is band division information for a digital image, and Ns _{k, l} (u, v) is band division information for a white noise image.

ここで、cortex_pol_k,l(ρ,θ) はcortex_k,l(u,v)を極座標で表現したものである。Kは半径方向、Lは円周方向の帯域分割数であり、一般にはK=6,L=6を用いる。 The band division filter cortex _{k, l} (u, v) combines the radial direction and the circumferential direction and is defined as follows.

Here, cortex_pol _{k, l} (ρ, θ) represents cortex _{k, l} (u, v) in polar coordinates. K is the radial direction, L is the number of band divisions in the circumferential direction, and generally K = 6 and L = 6 are used.

ここで

である。図５は、L=6の場合のfanl(θ)の模式図である。図５において、横軸は正規化周波数、縦軸は応答である。 In the above equation, fanl (θ) is defined as follows.

here

It is. FIG. 5 is a schematic diagram of fanl (θ) when L = 6. In FIG. 5, the horizontal axis represents the normalized frequency, and the vertical axis represents the response.

ここで

である。図６は、K=6の場合のdom_k(ρ)の模式図である。図６において、横軸は正規化周波数、縦軸は応答である。 The definition of domk (ρ) is as follows.

here

It is. FIG. 6 is a schematic diagram of dom _k (ρ) when K = 6. In FIG. 6, the horizontal axis represents the normalized frequency, and the vertical axis represents the response.

ここで、

である。図７は、帯域分割フィルタcortex_k,l(u,v)による帯域分割を、横軸、縦軸とも正規化周波数空間で示した模式図である。図７において、各フィルタの半値周波数が太い実線で示されている。 The definition of base (ρ) is as follows.

here,

It is. FIG. 7 is a schematic diagram showing band division by the band division filter cortex _{k, l} (u, v) in the normalized frequency space on both the horizontal axis and the vertical axis. In FIG. 7, the half-value frequency of each filter is indicated by a thick solid line.

Inverse Fourier transform units 308A and 308B perform inverse Fourier transform on the spatial frequency information after the band division filter by the band division unit 307 to generate a band division image. In the following, the inverse Fourier transform result of the band division frequency information Is _{k, l} (u, v) for the digital image is is _{k, l} (x, y), and the band division frequency information Ns _{k, l} (u, v) for the white noise image Let _{n k, l} (x, y) be the inverse Fourier transform result of v).

ここで、

であり、本実施形態ではW=6,Q=0.7,b=4,s=0.8とする。 The noise mask unit 309 is based on the band-divided image is _{k, l} (x, y) that is output from the inverse Fourier transform unit 308A, and the band-divided noise image ns _{k, l} that is output from the inverse Fourier transform unit 308B. Mask (x, y). Then, the noise mask unit 309 calculates a perceived noise image P _{k, l} (x, y), which is a noise image after masking, for each band. First, the masking strength Te _{k, l} (x, y) is calculated by the following equation.

here,

In this embodiment, W = 6, Q = 0.7, b = 4, and s = 0.8.

と算出される。ここで、mはi_in(x,y)の平均値である。 FIG. 8 is a graph showing the relationship between the masking strengths Te _{k, l} (x, y) of is _{k, l} (x, y) in the present embodiment. In FIG. 8, the horizontal axis represents the intensity of is _{k, l} (x, y) on a logarithmic scale, and the vertical axis represents the intensity of Te _{k, l} (x, y) on a logarithmic scale. Using this Te _{k, l} (x, y), the noise image P _{k, l} (x, y)

Is calculated. Here, m is an average value of i _in (x, y).

なお、上記処理と同様にして、ノイズマップ算出部205BはノイズマップNM2(x,y)を算出する。 The noise map synthesis unit 310 synthesizes the perceptual noise image P _{k, l} (x, y) for each band based on the following equation for all bands, and calculates the noise map NM1 (x, y).

Note that the noise map calculation unit 205B calculates the noise map NM2 (x, y) in the same manner as the above processing.

<Configuration of Image Composition Unit 208>
FIG. 9 is a diagram illustrating the configuration of the image composition unit 208 according to the present embodiment. The terminal 901 receives a digital image i _in1 (x, y) after the first super-resolution processing from the first super-resolution processing unit 206. A digital image i _in2 (x, y) after the second super-resolution processing is input from the second super-resolution processing unit 207 to the terminal 902. The terminal 903 receives the noise map NM1 (x, y) for the first super-resolution processing calculated by the noise map calculation unit 205A. A noise map NM2 (x, y) for the second super-resolution processing calculated by the noise map calculation unit 205B is input to the terminal 904.

画像加算部906は、重みマップW1(x,y)及びW2(x,y)に従って、後述のようにデジタル画像i_in1(x,y)とデジタル画像i_in2(x,y)の合成処理を施す。 The weight determination unit 905 includes a weight map W1 (x, y) indicating a mixture weight of two pixel values i _in1 (x, y) and i _in2 (x, y) according to the image position (x, y) and W2 (x, y) is generated based on the noise map information NM1 (x, y) and NM2 (x, y). Here, the weight maps W1 (x, y) and W2 (x, y) can be calculated by the following equations, for example. However, the present embodiment is not limited to the following calculation method.

The image adding unit 906 performs a synthesis process of the digital image i _in1 (x, y) and the digital image i _in2 (x, y) as described later according to the weight maps W1 (x, y) and W2 (x, y). Apply.

<Configuration of Image Adder 906>
FIG. 10 is a diagram illustrating the configuration of the image adding unit 906 according to the present embodiment. The image addition unit 906 uses the weight maps W1 (x, y) and W2 (x, y) calculated by the weight calculation unit 905 to use the digital image i _in1 (x, y) and the digital image i _in2 (x, Perform the weighted addition process of y). A digital image i _in1 (x, y) is input to the terminal 1001, and a digital image i _in2 (x, y) is input to the terminal 1002. Also, a weight map W1 (x, y) for the digital image i _in1 (x, y) is input to the terminal 1006, and a weight map W2 (x, y) for the digital image i _in2 (x, y) is input to the terminal 1007. Is done. Coefficient setting units 1008A and 1008B set coefficients corresponding to the coordinates to multiplication units 1003A and 1003B with reference to weight maps W1 (x, y) and W2 (x, y), respectively. Multipliers 1003A and 1003B read pixel values from the input digital image and multiply by preset coefficients. The adder 1104 adds each pixel value multiplied by the weight.

  As described above, according to the present embodiment, two noise maps expressing the ease of perception of noise are calculated, and two super-resolution results are weighted and added based on these maps. As a result, it is possible to realize super-resolution processing that can improve the sharpness while suppressing the adverse effects of noise degradation. In addition, although this embodiment described regarding the case where two super-resolution processes are synthesize | combined, it is not limited to this. Even when there are three or more super-resolution processes, a weight map is calculated and combined for each process, thereby realizing a super-resolution process that can improve the sharpness while suppressing the effects of noise degradation. I can do it.

[Second Embodiment]
In the first embodiment, a noise map is calculated for each of the two super-resolution processes, and a weight is determined from the two noise maps. This method has a problem that the amount of calculation and the circuit scale are increased while the area where noise is perceived can be determined with high accuracy. Therefore, in the present embodiment, the processing is simplified so as to determine the weight from one noise map. Below, the difference from 1st embodiment is described.

<Configuration of image processing unit>
FIG. 11 is a diagram illustrating a configuration of the image processing unit 104 according to the present embodiment. The configuration of the parameter acquisition unit 202, noise characteristic storage unit 203, noise information acquisition unit 204, and noise map calculation unit 205 is the same as that of 202B to 205B in FIG. That is, the noise map calculation unit 205 calculates the noise map NM2 (x, y) based on the setting parameter for the second super-resolution processing, while the noise map NM1 based on the setting parameter for the first super-resolution processing. (x, y) is not calculated. The image composition unit 210 calculates an addition ratio based on the image obtained by the second super-resolution processing unit 207, combines the images based on the calculated addition ratio, and outputs a single high-resolution image.

画像加算部1206は、重みマップに従って、デジタル画像i_in1(x,y)とデジタル画像i_in2(x,y)の合成処理を施す。 <210 composition of image composition part>
FIG. 12 is a diagram illustrating the configuration of the image composition unit 210 according to the present embodiment. A digital image i _in1 (x, y) after the first super-resolution processing is input from the first super-resolution processing unit 206 to the terminal 1201. The terminal 1202 receives the digital image i _in2 (x, y) after the second super-resolution processing from the second super-resolution processing unit 207. A noise map NM2 (x, y) for the second super-resolution processing calculated by the noise map calculation unit 205 is input to the terminal 1203. The weight determination unit 1205 is a weight map W1 (x, y) indicating a mixture weight of two pixel values i _in1 (x, y) and i _in2 (x, y) according to the image position (x, y). W2 (x, y) is generated based on the noise map NM2 (x, y) and a predetermined threshold value th. Here, the weight maps W1 (x, y) and W2 (x, y) can be calculated by the following equations, for example. However, the present invention is not limited to the following calculation method.

The image adding unit 1206 performs a synthesis process of the digital image i _in1 (x, y) and the digital image i _in2 (x, y) according to the weight map.

  As described above, according to the present embodiment, one noise map is calculated for two super-resolution processes, and two super-resolution results are weighted and added based on this map. As a result, the same effects as those of the first embodiment can be obtained while reducing the calculation amount and the circuit scale as compared with the first embodiment. In the present embodiment, an example in which the weight map is calculated based on the noise map NM2 (x, y) has been described, but the noise map NM1 (x, y) based on the setting parameter of the first super-resolution processing is described. The weight map may be calculated based on the above.

[Third Embodiment]
In the first embodiment, the noise map calculation units 205A and 205B generate noise band-divided images nsk _{, l} (x, y) from the actually generated noise images n (x, y). . However, in this method, it is necessary to calculate the noise image n (x, y), and there is a problem that the calculation amount is large. Therefore, the present embodiment simplifies the process by directly generating the noise band-divided image nsk _{, l} (x, y) based on the information on the noise characteristics. Hereinafter, the noise map calculation unit 211A changed from the noise map calculation unit 205A in the first embodiment will be described. Note that the noise map calculation unit 211B changed from the noise map calculation unit 205B has the same configuration as the noise map calculation unit 211A.

<Configuration of Noise Map Calculation Unit 211A>
FIG. 13 is a diagram illustrating a configuration of the noise map calculation unit 211A according to the present embodiment. A digital image i _in (x, y) is input from the signal processing unit 103 to the terminal 1301. Noise information is input to the terminal 1302 from the noise information acquisition unit 204. The processes of the Fourier transform unit 305A to the inverse Fourier transform unit 308A are the same as those in FIG.

ここで、αはノイズ情報取得部204Aで取得したノイズ量に関する補正項であり、cortex_k,l(u,v)は第１実施形態において述べた帯域分割フィルタである。さらにNPS_k,l(u,v)の総和をとって、帯域毎のパワーNP_k,lを算出する。ここでUは横方向の周波数分割数、Vは縦方向の周波数分割数である。

帯域毎のパワーから、帯域分割画像ns_k,l(x,y)を一様なベタ画像として次の様に生成する。

The band dividing unit 1303 generates a band divided image ns _{k, l} (x, y) based on the information regarding the noise characteristics acquired by the noise information acquiring unit 204A. First, the noise auto-correlation function acquired by the noise information acquisition unit 204A is Fourier-transformed to calculate the noise power spectrum C (u, v). Based on C (u, v), the noise power spectrum NPS _{k, l} (u, v) for each band is calculated as follows. Here, k is a radial index, and l is a circumferential index.

Here, α is a correction term related to the amount of noise acquired by the noise information acquisition unit 204A, and cortex _{k, l} (u, v) is the band division filter described in the first embodiment. Further _, the power NP _{k, l} for each band is calculated by taking the sum of NPS _{k, l} (u, v). Here, U is the frequency division number in the horizontal direction, and V is the frequency division number in the vertical direction.

From the power for each band, the band-divided image ns _{k, l} (x, y) is generated as a uniform solid image as follows.

The noise mask unit 309 performs masking on the band-divided noise image ns _{k, l} (x, y) based on the band-divided image is _{k, l} (x, y) output from the inverse Fourier transform unit 308A. The perceived noise image P _{k, l} (x, y) is calculated for each band. The noise map synthesis unit 310 synthesizes perceptual noise images P _{k, l} (x, y) for each band and calculates a noise map NM1 (x, y). The processing after the noise map synthesis unit 310 is the same as the processing described in the first embodiment, and a description thereof will be omitted.

As described above, according to the present embodiment, the noise image n (x, y) is not calculated, and the noise band-divided image ns _{k, l} (x, y) is directly generated. As a result, the same effects as those of the first embodiment can be obtained while reducing the calculation amount and the circuit scale as compared with the first embodiment.

[Fourth Embodiment]
In the first embodiment, the noise amount and the autocorrelation function are used as the noise information acquired by the noise information acquisition units 204A and 204B. However, this information is only statistical information and is different from the noise that is actually mixed. Therefore, in this embodiment, a noise image calculated from an actual digital image is used as noise information. Hereinafter, the changed part from 1st Embodiment is demonstrated.

<Configuration of Image Processing Unit 104>
FIG. 14 is a diagram illustrating a configuration of the image processing unit 104 according to the present embodiment. A digital image processed by the signal processing unit 103 is input to the terminal 1401. The noise information acquisition units 1402A and 1402B compare the digital image output from the signal processing unit 103 with the digital images output from the first super-resolution processing unit 206 and the second super-resolution processing unit 207, respectively. Calculate a noise image. The noise map calculation units 1403A and 1403B express the ease of perception of noise in the image based on the noise image acquired as noise information by the noise information acquisition units 1402A and 1402B and the digital image from the signal processing unit 103. Generate a noise map. The image synthesis unit 208 synthesizes the images obtained by the first super-resolution processing unit 206 and the second super-resolution processing unit 207, and outputs a single high-resolution image. A terminal 1404 outputs a digital image subjected to super-resolution processing.

<Configuration of noise information acquisition unit 1402A>
FIG. 15 is a diagram illustrating a configuration of the noise information acquisition unit 1402A according to the present embodiment. Note that the noise information acquisition unit 1402B has the same configuration as the noise information acquisition unit 1402A. A digital image from the signal processing unit 103 is input to the terminal 1501. The noise reduction unit 1502 performs high-frequency removal processing on the digital image, and generates a noise-removed image from which noise has been removed. The image subtracting unit 1503 calculates a noise image by subtracting the noise-removed image from the noise reducing unit 1502 from the digital image from the signal processing unit 103. A terminal 1504 outputs a noise image.

<Configuration of noise map calculation unit 1403A>
FIG. 16 is a diagram illustrating a configuration of the noise map calculation unit 1403A according to the present embodiment. Note that the noise map calculation unit 1403B has the same configuration as the noise map calculation unit 1403A. A digital image i _in (x, y) is input from the signal processing unit 103 to the terminal 1601. The noise image n (x, y) is input as noise information from the noise information acquisition unit 1402 to the terminal 1602. The difference from the first embodiment is that the noise image n (x, y) is directly input from the terminal 1602 to the Fourier transform unit 305B. The processing after Fourier transform units 305A and 305B is the same as the processing described in the first embodiment, and a description thereof will be omitted.

  As described above, according to the present embodiment, a noise image calculated from an actual digital image is used as noise information. As a result, the same effects as those of the first embodiment can be obtained while reducing the calculation amount and the circuit scale as compared with the first embodiment in which noise information is calculated.

[Fifth Embodiment]
In the first embodiment, a filter known as a cortex filter is used for the processing of the band dividing units 307A and 307B in the noise map calculation units 205A and 205B. However, according to this method, since the frequency characteristic is a very special filter, the calculation processing becomes heavy. Therefore, in this embodiment, the arithmetic processing is lightened by replacing the cortex filter in the first embodiment with a simple band-splitting filter in which normal vertical and horizontal filters are combined. Hereinafter, the noise map calculation unit 212A changed from the noise map calculation unit 205A in the first embodiment will be described. Note that the noise map calculation unit 212B changed from the noise map calculation unit 205B has the same configuration as the noise map calculation unit 212A.

<Configuration of Noise Map Calculation Unit 212A>
FIG. 17 is a diagram illustrating a configuration of the noise map calculation unit 212 according to the present embodiment. A digital image i _in (x, y) is input to the terminal 1701 from the signal processing unit 103. Noise information is input to the terminal 1702 from the noise information acquisition unit 204A. Then, similarly to the processing described in the first embodiment, the noise image n (x, y) of the camera is estimated by the noise image generation unit 303A and the filter unit 304A.

帯域分割フィルタfilterbank_k,l(u,v)は、次のように定義される。

ここで、Kは横方向、Lは縦方向の帯域分割数であり、本実施形態ではK=6,L=6を用いる。 The band division unit 1703 performs band division using a band division filter to generate band division information Is _{k, l} (u, v). Here, k is a horizontal index, and l is a vertical index. The spatial frequency information after the band division is expressed by the following equations, respectively. Is _{k, l} (u, v) is band division information for a digital image, and Ns _{k, l} (u, v) is band division information for a white noise image.

The band division filter filterbank _{k, l} (u, v) is defined as follows.

Here, K is the horizontal direction and L is the number of band divisions in the vertical direction. In this embodiment, K = 6 and L = 6 are used.

FIG. 18 is a schematic diagram showing band division by the band division filter filterbank _{k, l} (u, v) in the normalized frequency space on both the horizontal axis and the vertical axis. The half-value frequency of each filter is indicated by a thick solid line. The noise mask unit 309 performs masking on the noise image ns _{k, l} (x, y) that has been band-divided based on the band-divided image is _{k, l} (x, y), and perceived noise image P _{k, l} (x, y) is calculated for each band. The noise map synthesis unit 310 synthesizes perceptual noise images P _{k, l} (x, y) for each band and calculates a noise map NM1 (x, y). The processing after the noise map synthesis unit 310 is the same as the processing described in the first embodiment, and a description thereof will be omitted.

  As described above, according to the present embodiment, the band division process is performed using the simple band division filter in which the vertical and horizontal filters are combined. As a result, the same effects as those of the first embodiment can be obtained while reducing the calculation amount and the circuit scale as compared with the first embodiment using the cortex filter. In the present embodiment, band division is performed by multiplication in the frequency domain, but it can also be realized by convolution calculation in the spatial domain. Furthermore, a known band division filter such as a wavelet can be applied to the present embodiment.

[Other Embodiments]
In the embodiments described so far, a plurality of super-resolution images are synthesized based on the weight map calculated from the noise map in the processing of the image synthesis unit. However, the present invention is not limited to this, and the processing of the image composition unit may be, for example, the following processing.

ここで、Lはローパスフィルタを表し、*はたたみこみ演算を表す。 In the first embodiment, the weight map is directly calculated from the noise map. However, in this method, there is a problem that the variation in weight between adjacent pixels is large, and noise is generated due to image synthesis. Therefore, after the low-pass filter is applied to the noise map, the weight map may be calculated to suppress the variation in weight between adjacent pixels. Specifically, the weight maps W1 and W2 of the first embodiment may be defined as follows.

Here, L represents a low-pass filter, and * represents a convolution operation.

と表わされる。 In the embodiments described so far, the weight map is calculated for each pixel based on the noise map. However, these methods have a problem in that a large amount of calculation is required for calculating weights and combining processing. Therefore, the image may be divided into regions, and the synthesis weight may be determined for each region. For example, a super-resolution image having a small noise map average value may be selected and integrated for each region such as 16 × 16 or 32 × 32. Specifically, the first and second super-resolution processed images divided into regions are i _{in1_sub} (x, y) and i _{in2_sub} (x, y), and the noise maps are NM _{1_sub} (x, y) and NM _{2_sub.} (x, y) and the output image is i _out (x, y)

It is expressed as

  As described above, according to the embodiments described so far, it is possible to perform super-resolution processing that can improve sharpness in a whole image without noise degradation being perceived visually. It is also possible to combine the embodiments described so far.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

An image processing apparatus,
A plurality of conversion means for applying different resolution conversion processes to the input image,
Acquisition means for acquiring noise information according to the resolution conversion process;
Calculation means for calculating a noise map representing ease of perception of noise in the input image based on the noise information and the input image;
Determining means for determining an addition ratio of a plurality of images subjected to the resolution conversion processing based on the noise map;
An image processing apparatus comprising: a combining unit configured to combine a plurality of images subjected to the resolution conversion processing according to the addition ratio.   The image processing apparatus according to claim 1, wherein the calculation unit calculates a plurality of the noise maps corresponding to each of the plurality of conversion units.   The image processing apparatus according to claim 1, wherein the calculation unit calculates one of the noise maps corresponding to one of the plurality of conversion units. A storage means for storing the noise information;
The image processing apparatus according to claim 1, wherein the acquisition unit acquires the noise information from the storage unit according to a setting parameter for the resolution conversion process.   The image processing apparatus according to claim 4, wherein the noise information is a noise amount and an autocorrelation function.   4. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the noise information by comparing the input image and an image subjected to the resolution conversion process. 5. . The calculating means includes
Band division means for generating a noise image that has been divided into bands based on the noise information, and generating an input image that has been divided into bands based on the input image;
Mask means for masking the band-divided noise image based on the band-divided input image;
7. The apparatus according to claim 1, further comprising: a generating unit configured to combine the band-divided noise image output from the mask unit and generate the noise map in all bands. An image processing apparatus according to 1.   The image processing apparatus according to claim 7, wherein the band-divided noise image is generated based on a white noise image having a predetermined power and the noise information.   The image processing apparatus according to claim 7, wherein the band-divided noise image is directly generated from the noise information.   The band dividing means generates the band-divided noise image and the band-divided input image based on a filter whose frequency characteristic is defined by a combination of a circumferential characteristic and a radial characteristic. The image processing apparatus according to claim 7, wherein the image processing apparatus is characterized.   The band dividing unit generates the band-divided noise image and the band-divided input image based on a filter whose frequency characteristic is defined by a combination of a vertical characteristic and a horizontal characteristic. The image processing apparatus according to any one of claims 7 to 9. A control method for an image processing apparatus, comprising:
Applying different resolution conversion processing to each input image;
Obtaining noise information according to the resolution conversion process;
Calculating a noise map representing the ease of perception of noise in the input image based on the noise information and the input image;
Determining an addition ratio of a plurality of images subjected to the resolution conversion processing based on the noise map;
And a step of synthesizing a plurality of images subjected to the resolution conversion processing according to the addition ratio.   A program that causes a computer to function as the image processing apparatus according to any one of claims 1 to 11.

Images