JP2016082496A - Image processing apparatus, imaging device, image processing system, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which allows for simultaneous reduction of a plurality of adverse effects incident to high resolution of the image, and to provide an imaging device, image processing system, image processing method and program.SOLUTION: An image processing apparatus has first acquisition means 103a for acquiring a first image, i.e., a high resolution input image, and a second image having a smaller resolution than that of the first image, selection means 103b for selecting a target pixel from the first image, second acquisition means 103c for acquiring first data including the pixel corresponding to the target pixel, and a plurality of second data from the second image, determination means 103d for determining the weight for each of the plurality of second data, based on the correlation of the first and second data, third acquisition means 103e for acquiring the signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image, and generation means 103f for generating output pixels corresponding to the target pixels, based on the signal value calculated from the signal values and weights of the plurality of reference pixels.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing device, an imaging device, an image processing system, an image processing method, and an image processing program for the purpose of reducing adverse effects caused by high-resolution images.

  In recent years, there has been a demand for higher image quality as display devices have higher definition. In order to improve the image quality, it is important to increase the resolution of the image and reduce noise. High resolution includes simple sharpness processing, processing using an inverse filter such as a Wiener filter, or super-resolution processing that performs estimation based on the least square method. However, these processes improve the sense of resolution of the image, but cause adverse effects such as noise amplification and ringing, leading to a reduction in image quality.

  Therefore, in Patent Document 1, when performing high resolution using an inverse filter, the high resolution intensity is changed at the edge portion of the image and other areas to generate a high resolution image while suppressing noise amplification. A technique is disclosed.

  Patent Document 2 proposes a noise reduction method that uses self-similarity of a subject space called an NLM (Non-local Means) filter. The NLM filter is a technique for reducing noise by replacing a signal value of a target pixel with a weighted average signal value of a plurality of pixels located around the target pixel. At this time, the weight used in the weighted average is determined according to the distance between the vector having each signal value in the partial region near the target pixel as a component and the vector similarly generated from the pixels located around the target pixel. The Thereby, it is possible to remove noise from the image while maintaining the resolution of the edge.

JP 2011-71930 A U.S. Pat. No. 8,427,559

  However, in the method disclosed in Patent Document 1, noise amplification and ringing occur in the edge region as in the conventional case only by reducing the strength of high resolution in the non-edge region. In addition, when the technique disclosed in Patent Document 2 is used for a high-resolution image, amplified noise is reduced, but strong ringing is determined to be an edge and is removed. I can't.

  In view of such problems, the present invention provides an image processing apparatus, an imaging apparatus, an image processing system, an image processing method, and an image processing program capable of simultaneously reducing a plurality of adverse effects that occur in association with an increase in image resolution. The purpose is to provide.

  An image processing apparatus according to one aspect of the present invention is configured to acquire a first image obtained by increasing a resolution of an input image, and a second image having a resolution smaller than that of the first image. Means for selecting a pixel of interest from the first image; first data including a pixel corresponding to the pixel of interest; and a plurality of second data from the second image. Two acquisition means, a determination means for determining a weight for each of the plurality of second data based on a correlation between the first and second data, and the plurality of second data from the first image And a third acquisition means for acquiring signal values of a plurality of reference pixels corresponding to, and generating an output pixel corresponding to the target pixel based on the signal values calculated from the signal values of the plurality of reference pixels and the weights And generating means.

  According to another aspect of the present invention, there is provided an imaging apparatus that includes an imaging unit that captures an image of a subject formed by an imaging optical system and outputs the input image, and a first resolution that is a high resolution of the input image. First acquisition means for acquiring an image and a second image having a resolution smaller than that of the first image, selection means for selecting a target pixel from the first image, and the second image From the first data including the pixel corresponding to the target pixel, the second acquisition means for acquiring a plurality of second data, and the correlation between the first and second data, Determining means for determining a weight for each second data; third acquiring means for acquiring signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image; Based on the signal value of the reference pixel and the signal value calculated from the weight, the attention And having a generating means for generating an output pixel corresponding to element, the.

  According to another aspect of the present invention, there is provided an image processing system that captures a subject image formed by an imaging optical system and outputs the input image; An image generating means for generating a second image having a resolution smaller than that of the first image, a first acquiring means for acquiring the first and second images, and the first image Selecting means for selecting a target pixel from the second image, first data including a pixel corresponding to the target pixel from the second image, second acquisition means for acquiring a plurality of second data, and the second Determining means for determining a weight for each of the plurality of second data based on the correlation between the first and second data; and a plurality of reference pixels corresponding to the plurality of second data from the first image. Third acquisition means for acquiring a signal value; and a plurality of reference pixels Based on the signal value calculated issue value from the weight, and having a generating means for generating an output pixel corresponding to the pixel of interest.

  According to another aspect of the present invention, there is provided an image processing method of obtaining a first image obtained by increasing the resolution of an input image and a second image having a resolution smaller than that of the first image. Selecting a target pixel from the first image; obtaining first data including a pixel corresponding to the target pixel from the second image; and a plurality of second data from the second image. 2, obtaining a weight for each of the plurality of second data based on the correlation between the first and second data, and the plurality of second data from the first image. Obtaining signal values of a plurality of reference pixels corresponding to the data of the plurality of pixels, and generating an output pixel corresponding to the pixel of interest based on the signal values calculated from the signal values of the plurality of reference pixels and the weights And having And features.

  An image processing program according to another aspect of the present invention obtains a first image obtained by increasing the resolution of an input image, and a second image having a resolution smaller than that of the first image. Selecting a target pixel from the first image; obtaining first data including a pixel corresponding to the target pixel from the second image; and a plurality of second data from the second image. 2, obtaining a weight for each of the plurality of second data based on the correlation between the first and second data, and the plurality of second data from the first image. Obtaining signal values of a plurality of reference pixels corresponding to the data of the plurality of pixels, and generating an output pixel corresponding to the pixel of interest based on the signal values calculated from the signal values of the plurality of reference pixels and the weights And the Characterized in that to execute the Yuta.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing system, an image processing method, and an image processing program that can simultaneously reduce a plurality of adverse effects that occur with high resolution of an image.

1 is an external view of an image pickup apparatus according to Embodiment 1. FIG. 1 is a block diagram of an imaging apparatus according to Embodiment 1. FIG. 3 is a flowchart of an image processing method according to the first exemplary embodiment. FIG. 3 is a configuration diagram of an image processing unit according to the first embodiment. It is explanatory drawing regarding the resolution degree of an image. FIG. 3 is a relationship diagram between a first image and a second image according to the first exemplary embodiment. It is explanatory drawing of an evil reduction process. 6 is a flowchart of an image processing method according to the second embodiment. FIG. 6 is a relationship diagram between a first image and a second image of Example 2. FIG. 10 is an external view of an image processing system according to a third embodiment. FIG. 10 is a block diagram of an image processing system according to a third embodiment. 6 is an external view of an imaging system according to Embodiment 4. FIG. FIG. 10 is a block diagram of an imaging system according to a fourth embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  Before proceeding to the description of a specific embodiment, an outline of a method for reducing the harmful effects of high resolution according to the present invention will be described. For simplicity, the description will be made with a monochrome image as an example. In the case of a color image, the same processing may be executed for each component.

  First, a first image with a high resolution is generated using an inverse filter or super-resolution processing. At this time, adverse effects (noise amplification, ringing, etc.) associated with high resolution occur in the first image.

  Next, a target pixel that is a target for reducing the harmful effect is selected from the first image. Then, a partial region including the pixel corresponding to the target pixel (target corresponding pixel) is acquired from the second image as target corresponding data (first data). Here, the second image is an image obtained by capturing the same subject space as the first image and having a resolution smaller than that of the first image. In other words, the second image has less adverse effect on the resolution than the first image, or the harmful effect does not exist in the first place. Examples of the second image include the input image itself and an image obtained by applying a higher resolution to the input image, which is weaker than the first image. The resolution refers to the contrast and frequency distribution of the image, and details thereof will be described later.

  Next, a plurality of reference correspondence data (second data), which are partial areas, are obtained from the second image, and a correlation value with each of the attention correspondence data is calculated. The weight of each reference correspondence data is determined according to the correlation value. The weight is determined so as to increase as the correlation increases, that is, as the reference correspondence data is similar to the attention correspondence data.

  Next, the signal value of the reference pixel corresponding to the reference correspondence data is acquired from the first image, and the weighted average value of the signal value is calculated using the weight calculated by the reference correspondence data. Then, by replacing the signal value of the target pixel with the weighted average value, the reduction of the harmful effect is completed.

  As described above, by calculating the correlation value from the second image with little (or no) harmful effects, it is possible to obtain highly accurate weights in which the adverse effects are suppressed. By averaging using this weight, in addition to the noise reduction effect of the conventional NLM filter, an effect of suppressing other harmful effects such as ringing can be obtained.

  In this embodiment, a case where the image processing method of the present invention is applied to the imaging apparatus 100 will be described. FIG. 1 is an external view of the imaging apparatus 100, and FIG. 2 is a block diagram of the imaging apparatus 100.

  The image acquisition unit 101 includes an imaging optical system and an image sensor. Examples of the image pickup device include a charge coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS). At the time of shooting, the light incident on the image acquisition unit 101 is collected by the imaging optical system and converted into an analog electrical signal by the imaging device. The analog electrical signal is converted into a digital signal by the A / D converter 102 and input to the image processing unit 103. The image processing unit (image processing apparatus) 103 performs a high-resolution process and a process for reducing adverse effects associated therewith in addition to a predetermined process. These processes will be described later. In these processes, information about the optical characteristics of the image acquisition unit 101 stored in the storage unit 104 and the shooting conditions of the image acquisition unit 101 detected by the state detection unit 109 are used. Here, the optical characteristics refer to information (for example, optical transfer function and point image intensity distribution) regarding aberration, diffraction, or defocus in the imaging optical system in the image acquisition unit 101. The shooting condition refers to the state of the image acquisition unit 101 at the time of shooting, for example, the state of the aperture, the focus position, the blurring locus of the imaging device 100 during exposure, the focal length of the zoom lens, or the like. The state detection unit 109 may obtain information regarding the shooting conditions from the system controller 107 or the control unit 108. The processed image is stored in the image recording medium 106 in a predetermined format. At this time, information regarding the photographing conditions may be stored at the same time. Alternatively, an image already stored in the image recording medium 106 may be read out, and the image processing unit 103 may perform high resolution and adverse effect reduction processing. When viewing an image stored in the image recording medium 106, the image is output to the display unit 105 such as a liquid crystal display. The series of controls described above is performed by the system controller 107, and the mechanical drive of the image acquisition unit 101 is performed by the control unit 108 according to an instruction from the system controller 107.

  Next, a reduction processing method for the adverse effect of high resolution performed in the image processing unit 103 will be described in detail with reference to the flowchart of FIG. The method shown in FIG. 3 can be embodied as an image processing program for causing a computer to execute the function of each step. FIG. 4 is a configuration diagram of the image processing unit 103.

  In step S101, the first acquisition unit 103a acquires the input image obtained by the image acquisition unit 101. The input image includes information on the subject space due to various factors at the time of shooting, for example, aberration and diffraction of the imaging optical system in the image acquisition unit 101, sampling at the imaging device, or shaking of the imaging device 100 during exposure, and the like. Is lost. Furthermore, noise generated by an image sensor or the like exists in the input image.

  In step S102, the first acquisition unit 103a generates two high-resolution images having different resolutions from the input image. Of these, the higher resolution is called the first image, and the lower resolution is called the second image. However, the second image may be the input image itself or an image obtained by subjecting the input image to low resolution processing such as a smoothing filter. However, it is desirable that the resolution of the second image is greater than or equal to the resolution of the input image, as will be described later.

  Here, the degree of resolution represents how much information an object has in the subject space. For example, the index includes the contrast of the image, the maximum resolution frequency, or the frequency distribution. Here, the maximum resolution frequency refers to the absolute value of the maximum spatial frequency at which the spectrum intensity is equal to or greater than a threshold value. The threshold value may be determined from the image quality required for the image. The larger the value of the contrast and the maximum resolution frequency, the greater the resolution. Further, as an example of determining the resolution degree with high accuracy, an integrated value of the frequency distribution of the image can be given. The outline is shown in FIG. In FIG. 5, for the sake of simplicity, only the spectral intensity of the spatial frequency in a certain one-dimensional direction of the image is shown. The area of the horizontal line portion represents an integral value with respect to the spectral intensity 211 of the first image 200, and the area of the vertical line portion represents an integral value with respect to the spectral intensity 311 of the second image 300. Actually, since the spatial frequency is two-dimensional, the integral value corresponds to the volume. The larger these integral values, the greater the resolution. As another method for expressing the resolution, there is a method using a parameter for high resolution. However, in this case, if the first image 200 and the second image 300 are not high-resolution by the same method, it is impossible to determine the degree of resolution based on the parameters.

  Some examples of high resolution applied to an input image are given. Examples of the high-resolution technique include super-resolution processing such as RL (Richardson-Lucy) method and posterior probability maximization method in addition to processing using an inverse filter such as a Wiener filter. Degradation information generated in the input image necessary for these processes is acquired from design values and measurement values of the imaging optical system when the degradation is aberration or diffraction, and is mounted on the imaging apparatus 100 in the case of blur during exposure. Can be obtained from a gyro sensor or the like. Alternatively, a technique called Blind Devolution may be used in which degradation information is estimated from an input image to achieve high resolution.

  However, even if any of the above-described methods is used, noise amplification or ringing occurs as a detrimental effect on a high-resolution image. These adverse effects increase as the resolution is increased. Therefore, adverse effects are likely to appear in the first image 200 with high resolution. Therefore, an object of the present invention is to suppress adverse effects while maintaining the resolution of the first image 200.

  Next, regarding each high resolution technique, a parameter indicating the resolution is described. For example, the processing by the Wiener filter is expressed by the following expression (1) on the frequency space.

G is the frequency distribution of the input image, H is the optical transfer function representing the degradation that has occurred in the input image, H ^* is the complex conjugate of H, and F is the frequency distribution of the high-resolution image. If F is subjected to Fourier transform (or inverse Fourier transform), a high-resolution image can be obtained. Here, Γ is a parameter representing the strength of high resolution. The closer this value is to 0, the higher the resolution is.

  Next, consider the super-resolution processing in real space as expressed by the following equation (2).

  X represents an input image, K represents a point image intensity distribution representing degradation that has occurred in the input image, Y represents a high-resolution image, and * represents a convolution operation. A (Y) is a term called a regularization term, and plays a role of suppressing the harmful effects associated with high resolution. However, this does not completely suppress the harmful effects. Examples of regularization terms include a primary average norm and a TV (Total Variation) norm. Here, λ is a parameter indicating how much the effect of the regularization term is strengthened. Generally, the closer to 0, the higher the resolution.

  Furthermore, in a technique such as the above-described RL method in which high resolution is repeated using an image once subjected to high resolution as a new initial image, the number of repetitions can also be a parameter indicating the resolution. That is, the high-resolution image in the middle of the iterative calculation to generate the first image 200 can be used as the second image 300. In this case, the calculation load for generating the second image 300 separately is reduced.

  In step S103, the selection unit 103b selects the target pixel 201 from which the adverse effect is to be removed from the first image 200 as shown in FIG. However, the position of the target pixel 201 is not limited to this, and a plurality of pixels (for example, a 2 × 2 pixel group) may be selected at a time.

  In step S104, the second acquisition unit 103c is a partial region including pixels corresponding to the target pixel 201 of the first image 200 to the target image 201 (referred to as the target corresponding pixel) as shown in FIG. Attention corresponding data (first data) 303 is acquired. That is, the target pixel 201 and the target corresponding pixel 301 are present at the same position with respect to each image, and when the target pixel 201 in the first image 200 is selected, the target correspondence in the second image 300 is selected. Pixel 301 and attention correspondence data 303 are determined. The size and shape of the attention correspondence data 303 are not limited to those shown in FIG. 5, but the attention correspondence data 303 needs to have information regarding the distribution of signals, and thus must be formed of a plurality of pixels. If the attention corresponding pixel 301 is formed of a plurality of pixels, the attention corresponding pixel 301 and the attention corresponding data 303 may match.

  In step S105, first, as shown in FIG. 6, a reference corresponding data acquisition region 304 representing a range in which reference corresponding data (second data) 305a to 305c is acquired is set around the target corresponding pixel 301. However, the size and shape of the reference correspondence data acquisition area 304 are not limited to this, and the entire second image 300 may be used as the reference correspondence data acquisition area 304. However, since there is a high possibility that the similar structure of the attention correspondence data 303 exists around the attention correspondence data 303, it is desirable to limit the reference correspondence data acquisition area 304 to the periphery of the attention correspondence data 303 in order to reduce the calculation load. Next, the second acquisition unit 103c acquires reference corresponding pixels 302a to 302c and a plurality of reference corresponding data 305a to 305c including the reference corresponding pixels 302a to 302c from the reference corresponding data acquisition region 304. The sizes and shapes of the reference correspondence pixels 302a to 302c and the reference correspondence data 305a to 305c do not necessarily match the attention correspondence pixel 301 and the attention correspondence data 303, respectively. This is because the size and shape of each other can be matched using reduction conversion or the like when calculating the correlation value in step S106. However, when this conversion is performed, it is necessary to perform the same conversion on the reference pixels used in the weighted average calculation in step S109. Further, when the input image is a color image, the reference correspondence data may be acquired from a color component different from the attention correspondence data. For example, even when attention corresponding data is selected from R components in an RGB (Red, Green, Blue) image, reference corresponding data may be acquired from the G component and the B component. This is because the detrimental strength associated with high resolution generally does not change so much between RGB components. In the present embodiment, three reference pixels and three pieces of reference correspondence data are acquired, but more may be acquired.

In step S106, a correlation value is calculated for the attention correspondence data 303 with each of the plurality of reference correspondence data 305a to 305c. For the calculation of the correlation value, a feature-based method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Features) or a region-based method described later may be used. Since the feature-based method focuses on the feature amount, the correlation value can be calculated even if the attention-corresponding data and the reference-corresponding data have different numbers of pixels. On the other hand, the region-based method focuses on the difference between the signal values of each other. However, it is desirable to use a region-based method because similarity calculation can be performed with higher accuracy than region-based correlation calculation. The following two examples are given as examples of the region-based correlation calculation formula, but the calculation method is not limited to this.
<Example 1 of correlation calculation formula>
In example 1 of the correlation calculation formula, the root mean square of the signal difference between the attention correspondence data and the reference correspondence data is used. Attention corresponding data and reference corresponding data as partial area of the image, that is when dealing with matrix, the correlation calculation formula g ₁ is expressed by the following formula (3).

T is a matrix in which the signal value of each pixel in the attention correspondence data is a component T _ij , N is the number of rows of T, M is the number of columns of T, and R _k is the signal value of the kth reference correspondence data. It is a matrix. Ρ satisfies the following expression (4), and Ρ _ij represents the component.

N _Rk is the number of rows of R _k , and M _Rk is the number of columns of R _k . Also, σ (R _k , N / N _Rk , M / M _Rk ) is a conversion (enlarged in the case of an image) that makes the number of rows of matrix R _k N / N _Rk times and the number of columns M / M _Rk times (Or reduction). Bilinear interpolation, bicubic interpolation, or the like may be used for σ conversion.

  When each of the target data and the reference data is handled as a vector of components, Expression (3) is rewritten as Expression (5) below.

t is a vector in which the signal values of the target corresponding data with component t _i, r _k is a vector that as a component of each signal value of the k-th reference corresponding data, [rho is rearranged each component of the matrix Ρ a one-dimensional In the vector, the component of ρ is ρ _i .

  Since the correlation calculation formulas represented by the equations (3) and (5) look at the difference between the attention correspondence data and the reference correspondence data, the closer the value is to 0, the higher the similarity between the two.

  Here, a direct current component (which indicates an average value and corresponds to the brightness of the image) may be subtracted from the signals of the attention correspondence data and the reference correspondence data. What is desired to be determined by the correlation calculation is how similar the structures of the attention correspondence data and the reference correspondence data are, and therefore the brightness (DC component) is irrelevant. Further, the contrast of the reference correspondence data may be adjusted so that the correlation between the two becomes the highest. This is equivalent to multiplying the AC component of the reference correspondence data by a scalar. At this time, the expression (3) is rewritten as the following expression (6).

T _ave and Ρ _ave are average values of the signal values in the matrices T and そ れ ぞ れ, respectively. The average value may be calculated with a uniform weight or a weighted average. c is a coefficient for adjusting the contrast, and is expressed by the following equation (7) from the least square method.

When the correlation value is calculated by Expression (6), it is necessary to similarly adjust the brightness and contrast when calculating the weighted average in step S109.
<Example 2 of correlation calculation formula>
SSIM (Structure Similarity) may be used as an example of the correlation calculation formula. The correlation calculation formula using SSIM is expressed by the following formula (8).

  L, C, and S are evaluation functions related to brightness, contrast, and other structures, and take values from 0 to 1. The closer each value is to 1, the closer the two signals being compared. α, β, and γ are parameters for adjusting the weight of each evaluation item. Here, if α = 0, correlation calculation is performed by subtracting the DC component (brightness), and if β = 0, it is not necessary to consider scalar multiplication (contrast adjustment) of the AC component when calculating correlation. For this reason, the same evaluation as in the expression (6) can be performed.

  A correlation value may be calculated by combining a plurality of correlation calculation formulas. In addition, when using region-based correlation calculation, for example, Example 1 or Example 2 of the correlation calculation formula, isometric conversion may be performed on the reference correspondence data so that the correlation value with the attention correspondence data becomes the highest. The isometric conversion is, for example, identity conversion, rotation conversion, or inversion conversion. At this time, the conversion having the highest correlation value is also performed at the time of calculating the weighted average in step S109. By finding reference correspondence data with higher similarity, the effect of reducing harmful effects can be improved. However, since the amount of calculation increases, it is preferable to determine whether to perform the isometric conversion by comparing the effect of reducing the harmful effect with the amount of calculation.

  In step S107, the determination unit 103d determines a weight for each of the plurality of reference correspondence data 305a to 305c from the correlation value calculated in step S106. Since the reference correspondence data is similar to the attention correspondence data as the correlation is higher, the weight is set to be larger. For example, the weight is determined using the equation (5) as in the following equation (9).

w _k represents the weight corresponding to the k-th reference correspondence data, and h represents the strength of the filter. Z is a normalization factor of the weight w _k and satisfies the following formula (10).

  However, the method for determining the weight is not limited to this. For example, a weight table corresponding to the correlation value may be held, and the weight may be determined with reference to the table.

  In step S108, the third acquisition unit 103e acquires the signal values of the plurality of reference pixels 202a to 202c corresponding to the plurality of reference corresponding pixels 302a to 302c in the first image 200 to the second image 300, respectively. The reference corresponding pixels 302a to 302c and the reference pixels 202a to 202c are located at the same position with respect to each image. Note that this step may be executed at any time between step S105 and step S109. In this embodiment, three reference pixels are acquired. However, more reference pixels may be acquired.

  In step S109, a weighted average of the signal values of the reference pixels is calculated using the weight determined in step S107. Then, the generation unit 103f replaces the signal value of the target pixel with the calculated weighted average, and generates an output pixel in which the harmful effect is removed from the target pixel 201. Since the signal value of the structure similar to the structure in the vicinity of the target pixel is weighted and averaged, it was amplified with high resolution while maintaining the structure of the target pixel (in other words, the edge resolution). Noise is reduced. Furthermore, since the weight is calculated from the second image with little ringing, the ringing can be distinguished from the original structure of the subject space, and the ringing is reduced by averaging as with the noise.

The flow of processing so far is shown in FIG. An average signal value 203 that is a weighted average of the reference pixels 202a to 202c is calculated using the weights w _k (k = 1, 2, 3,...) Calculated from the second image 300. Then, the signal value of the target pixel 201 is replaced with the average signal value 203. Here, the weighted average signal value s _ave was calculated as in the following equation (11).

s _k is a signal value in the kth reference pixel. If the reference pixel and the pixel of interest has a plurality of pixels, s _k and s _ave is a vector quantity. However, the method of calculating the weighted average is not limited to this, and for example, non-linear combination may be used.

  Further, in the calculation of the correlation value in step S106, when the subtraction of the DC component and the adjustment of the contrast are performed, the weighted average must be calculated after adjusting the brightness and the contrast corresponding to the reference pixel. The same applies to the size conversion and equal-length conversion in equation (4).

  In this embodiment, replacement processing is used to reduce the harmful effects. However, a learning-type adverse effect reduction process may be used with reference to a weighted average of signal values.

  In step S110, it is determined whether the predetermined area of the input image has been processed. If all the pixels for which adverse effects are to be reduced have been processed, the process ends. If there are any remaining pixels, the process returns to step S103 to select a new pixel of interest.

  By the processing as described above, it is possible to simultaneously reduce a plurality of adverse effects that occur as a result of higher image resolution.

  Next, desirable conditions for enhancing the effect of the present invention will be described.

  First, it is desirable that the resolution of the second image is equal to or higher than the resolution of the input image. If the resolution of the second image is too small compared to the resolution of the first image, the output image generated by the weighted average has a lower resolution than the first image. This is because the weight is calculated from the second image having a smaller resolution than the first image, so that the weight is increased even in a structure with low similarity. Therefore, it is preferable to avoid reducing the resolution of the second image rather than the input image.

  In addition, it is desirable to determine the resolution of the second image according to the resolution of the first image. This is because if the resolutions of the first and second images are close to each other, the harmful effects appearing in both images will be of the same level, and it will be difficult to obtain the harmful effect. Conversely, if the resolution of the first and second images is too far apart, the resolution of the output image will deteriorate as described above.

  In addition, it is desirable to determine the resolution of the second image from the shooting conditions when the input image is shot. More preferably, the photographing conditions include information on ISO sensitivity and luminance level. This is because the noise generated in the input image changes depending on the ISO sensitivity at the time of shooting and the luminance level of the input image. In other words, the greater the noise generated in the input image, the greater the noise that is amplified at the time of high resolution. Therefore, it is better to use an image with a smaller resolution, in other words, an image with less noise amplification as the second image. The noise reduction effect can be improved.

  Also, it is desirable to change the weight determined in step S107 according to the resolution of the second image. More desirably, the larger the difference in resolution between the first and second images, the larger is the value obtained by dividing the weight of the reference correspondence data having a strong correlation with the attention correspondence data by the weight of the weak correlation. . By adjusting the parameters so that the weight of the reference correspondence data that has a strong correlation with the attention correspondence data becomes larger, only the structure with higher similarity is emphasized, and the deterioration of the resolution can be prevented. it can. For example, in equation (9), this corresponds to bringing h close to 0.

  With the configuration as described above, it is possible to provide an imaging apparatus that can simultaneously reduce a plurality of adverse effects that occur with high image resolution.

  The case where the image processing method of the present embodiment in which the process can be switched according to the magnitude of the harmful effect on the target pixel is applied to the imaging apparatus 100 will be described with reference to FIGS. FIG. 8 is a flowchart of the image processing method of the present embodiment, and FIG. 9 is a relationship diagram between the first image 500 and the second image 600 of the present embodiment. In the following description, the same parts as the processing flow of the first embodiment are omitted. Further, the method shown in FIG. 8 can be embodied as an image processing program for causing a computer to execute the function of each step.

  Steps S201 to S204 are the same as steps S101 to S104 in the first embodiment, and a description thereof will be omitted.

  In step S <b> 205, the second acquisition unit 103 c acquires attention data (third data) 503 including the attention pixel 501 in the first image 500. Then, the second acquisition unit 103 c calculates a correlation value between the attention data 503 and the attention correspondence data 603 including the attention correspondence pixel 601 in the second image 600. However, the size, shape, and position of the target pixel 501 and the target data 503 are not limited to this. For the calculation of the correlation value, the method described in the first embodiment may be used. Further, the shape and size of the attention data 503 do not necessarily match the attention correspondence data 603.

  In step S206, the second acquisition unit 103c determines whether the correlation value calculated in step S205 satisfies a predetermined condition. If the correlation value between the attention data 503 and the attention correspondence data 603 satisfies the prescribed condition (high correlation), the process proceeds to step S207. If the condition is not satisfied (correlation is low), the process proceeds to step S210. In this embodiment, the second acquisition unit 103c calculates the correlation value between the attention data 503 and the attention correspondence data 603, and determines whether the calculated correlation value satisfies a predetermined condition. However, other configurations may be used.

  Here, when it is determined that the prescribed condition is satisfied, it means that the adverse effect of the high resolution in the attention data 503 is weak, and thus the same processing as that of the conventional NLM filter is executed. On the other hand, if it is determined that the predetermined condition is not satisfied, the adverse effect is strong. Therefore, in steps S210 to S213, the harmful effect reduction process is executed in the same manner as steps S105 to S108 in the first embodiment.

  In step S207, reference data 505a to 505c, which are partial regions including the plurality of reference pixels 502a to 502c and the reference pixels 502a to 502c, are acquired from the first image 500. The size and shape of the reference data 505a to 505c do not necessarily match the attention data 503. Further, a reference data acquisition region 504 may be set around the pixel of interest 501 and a plurality of reference data may be acquired from the reference data acquisition region 504. In this embodiment, three reference pixels and three reference data are acquired, but more may be acquired.

  In step S208, the correlation value between the attention data 503 and the reference data 505a to 505c is calculated by the same method as that described in step S106 of the first embodiment.

  In step S209, the weight of each reference data is determined from the correlation value calculated in step S208 by the same method as that described in step S107 of the first embodiment.

  In step S214, a weighted average of the signal values of the reference pixels 502a to 502c is calculated using the determined weight. Then, using the calculated weighted average, an output pixel from which the harmful effect is removed from the target pixel 501 is generated in the same manner as in step S109 of the first embodiment.

  Step S215 returns to Step S203 when there is a pixel that is desired to be removed from the harmful effects similarly to Step 110 of the first embodiment.

  With the configuration as described above, it is possible to provide an imaging apparatus that can simultaneously reduce a plurality of adverse effects that occur with high image resolution.

  In the present embodiment, an example in which the image processing method described in the first and second embodiments is applied to an image processing system will be described. In the image processing system of the present embodiment, there are an image processing apparatus and an image processing apparatus that perform an image processing method, respectively, and by connecting them, the adverse effect of a high-resolution image is reduced. FIG. 10 is an external view of the image processing system of this embodiment, and FIG. 11 is a block diagram of the image processing system.

  The imaging device 401 acquires an input image by capturing a subject image formed by the imaging optical system. An input image acquired by the imaging device 401 is input to the image processing device 402 via the communication unit 403. At this time, information on the optical characteristics (optical transfer function and point image intensity distribution) of the imaging apparatus 401 and the shooting conditions (focal length, aperture value, ISO sensitivity, etc.) at the time of shooting are stored in the storage unit 404 as necessary. . The high resolution unit 405 generates a first image and a second image having different resolutions from the input image. However, the imaging device 401 may perform high resolution processing and input the first image and the second image that have already been generated to the image processing device 402. The second image may be the input image itself. The first and second images are input to the adverse effect reduction unit 406, and image processing (defect reduction processing) is executed. The processed output image is output to one or more of the display device 407, the recording medium 408, and the output device 409 via the communication unit 403. The display device 407 is, for example, a liquid crystal display or a projector. The user can perform work while confirming an image being processed via the display device 407. The recording medium 408 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 409 is a printer or the like. The image processing apparatus 402 may have a function of performing development processing and other image processing as necessary.

  With the configuration as described above, it is possible to provide an image processing system that can simultaneously reduce a plurality of adverse effects that occur with high image resolution.

  In this embodiment, an example in which the image processing method described in Embodiments 1 and 2 is applied to an imaging system will be described. FIG. 12 is an external view of the imaging system of the present embodiment, and FIG. 13 is a block diagram of the imaging system. The image pickup system of the present embodiment transfers an image to a server to which the image pickup apparatus is connected wirelessly, and performs high resolution in the server and an effect reduction process associated therewith.

  The server 703 has a communication unit 704 and is connected to the imaging device 701 via the network 702. When shooting is performed by the imaging device 701, an input image is input to the server 703 automatically or manually. At that time, information on the optical characteristics and imaging conditions of the imaging device 701 is also input as necessary. Note that the image input to the server 703 may be the first image and the second image instead of the input image. The input image input to the server 703 is stored in the storage unit 705. Thereafter, the input image is subjected to high resolution processing and adverse effect reduction processing by the image processing unit 706, and an output image is generated. The output image is output to the imaging device 701 or stored in the storage unit 705.

  With the configuration as described above, it is possible to provide an imaging system that can simultaneously reduce a plurality of adverse effects that occur with higher image resolution.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

103 Image processing unit (image processing apparatus)
103a first acquisition unit 103b selection unit 103c second acquisition unit 103d determination unit 103e third acquisition unit 103f generation unit 200 first image 201 target pixel 202a to 202c reference pixel 203 average signal value 300 second image 301 Attention corresponding pixel 303 Attention corresponding data (first data)
304 Reference correspondence data acquisition areas 305a to 305c Reference correspondence data (second data)

Claims (15)

First acquisition means for acquiring a first image obtained by increasing the resolution of an input image, and a second image having a smaller resolution than the first image;
Selecting means for selecting a pixel of interest from the first image;
From the second image, first data including a pixel corresponding to the target pixel, and second acquisition means for acquiring a plurality of second data;
Determining means for determining a weight for each of the plurality of second data based on a correlation between the first and second data;
Third acquisition means for acquiring signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image;
An image processing apparatus comprising: a generation unit configured to generate an output pixel corresponding to the target pixel based on a signal value calculated from the signal values of the plurality of reference pixels and the weight.   The image processing apparatus according to claim 1, wherein the first acquisition unit generates the first and second images based on an input image.   3. The image processing according to claim 2, wherein the first acquisition unit generates the second image so that a resolution of the second image is equal to or higher than a resolution of the input image. apparatus.   The image processing apparatus according to claim 2, wherein the first acquisition unit determines a resolution degree of the second image according to a resolution degree of the first image.   The said 1st acquisition means determines the resolution degree of a said 2nd image based on the imaging conditions at the time of imaging | photography of the said input image, The any one of Claim 2 to 4 characterized by the above-mentioned. Image processing device.   The image processing apparatus according to claim 5, wherein the photographing conditions are an ISO sensitivity and a luminance level. The first acquisition unit generates the first and second images by performing an iterative operation on the input image,
The number of times of the iterative operation applied to the first image is greater than the number of times of the iterative operation applied to the second image. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the determination unit determines the weight according to a resolution of the second image.   The determination means correlates the weight of the second data, which has a strong correlation with the first data, as the difference in resolution between the first image and the second image increases. The image processing apparatus according to claim 8, wherein the weight is determined so that a value obtained by dividing the second data by the weight of the weak second data is increased.   Whether the second acquisition means acquires third data including the target pixel from the first image, and acquires the second data based on a correlation between the first and third data The image processing apparatus according to claim 1, wherein the determination is performed. An imaging means for imaging a subject image formed by the imaging optical system and outputting an input image;
First acquisition means for acquiring a first image obtained by increasing the resolution of the input image and a second image having a resolution smaller than that of the first image;
Selecting means for selecting a pixel of interest from the first image;
From the second image, first data including a pixel corresponding to the target pixel, and second acquisition means for acquiring a plurality of second data;
Determining means for determining a weight for each of the plurality of second data based on a correlation between the first and second data;
Third acquisition means for acquiring signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image;
An image pickup apparatus comprising: a generation unit configured to generate an output pixel corresponding to the target pixel based on a signal value calculated from the signal values of the plurality of reference pixels and the weight. An imaging means for imaging a subject image formed by the imaging optical system and outputting an input image;
Image generating means for generating a first image by resolving the input image and generating a second image having a resolution smaller than that of the first image;
First acquisition means for acquiring the first and second images;
Selecting means for selecting a pixel of interest from the first image;
From the second image, first data including a pixel corresponding to the target pixel, and second acquisition means for acquiring a plurality of second data;
Determining means for determining a weight for each of the plurality of second data based on a correlation between the first and second data;
Third acquisition means for acquiring signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image;
An image processing system comprising: generating means for generating an output pixel corresponding to the target pixel based on a signal value calculated from the signal values of the plurality of reference pixels and the weight. Obtaining a first image obtained by resolving an input image and a second image having a resolution smaller than that of the first image;
Selecting a pixel of interest from the first image;
Obtaining first data including a pixel corresponding to the target pixel from the second image;
Obtaining a plurality of second data from the second image;
Determining a weight for each of the plurality of second data based on the correlation of the first and second data;
Obtaining signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image;
Generating an output pixel corresponding to the pixel of interest based on signal values calculated from the signal values of the plurality of reference pixels and the weights. Obtaining third data including the pixel of interest from the first image;
Obtaining a correlation between the first data and the third data;
The image processing method according to claim 11, further comprising a step of determining whether to perform the step of acquiring the plurality of second data based on the correlation. Obtaining a first image obtained by resolving an input image and a second image having a resolution smaller than that of the first image;
Selecting a pixel of interest from the first image;
Obtaining first data including a pixel corresponding to the target pixel from the second image;
Obtaining a plurality of second data from the second image;
Determining a weight for each of the plurality of second data based on the correlation of the first and second data;
Obtaining signal values of a plurality of reference pixels corresponding to the plurality of second data from the first image;
An image processing program causing a computer to execute an output pixel corresponding to the target pixel based on a signal value calculated from the signal values of the plurality of reference pixels and the weight.