JP2015119427A - Image processing method, image processing system, imaging apparatus, image processing program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method for highly precisely performing high resolution processing by using fractal encoding.SOLUTION: An image processing method includes: a step for extracting a first image being a part of an object image depending on a photographed image; a step for calculating each correlation value of the first image and a plurality of second images; a step for selecting a third image corresponding to the first image from the plurality of second images in response to the correlation value; a step for acquiring information showing a correlation relation between the first image and the third image; and a step for making the object image into high resolution by using the third image and information. The step for making the object image into the high resolution includes a step for switching processing in accordance with precision of the third image, and performs repetitive operation for repeating the step for making the object image into the high resolution for the plurality of times. When n is set to be a natural number of 2 or more and m is the natural number smaller than n, precision of the third image is determined based on the object image used in m-th repetitive operation in n-th repetitive operation.

Description

  The present invention relates to an image processing method for a captured image.

  In recent years, with higher definition of display devices, higher resolution of images is desired. High resolution is a process of enlarging an image and removing blur from the image. Here, the blur is deterioration caused by aberration of the optical system, diffraction, defocus, or blurring during exposure, which occurs when an image is acquired by the imaging apparatus.

  The MTF (Modulation Transfer Function) of the subject in the image is zero or very small at a certain spatial frequency due to deterioration such as rough sampling or blur, and it is necessary to restore these MTFs for high resolution. However, since this problem is an inverse problem of so-called failure setting, it is difficult to obtain a solution with high accuracy.

  Patent Document 1 discloses a method for obtaining an enlarged image using fractal self-similarity of an image. In other words, the image is fractal-encoded, the resolution is increased by decoding, and the decoding process is repeated to converge the result.

JP 2009-26032 A

  However, Patent Document 1 does not describe a constraint condition when the decoding (high resolution) processing is repeated. Since high resolution is an inverse problem of defect setting, the solution is not uniquely determined, and an error occurs in the obtained result. Further, the difference between the images generated before and after the high resolution cannot be distinguished as to whether the difference is an error or restored high frequency information. For this reason, if there is no constraint that suppresses errors when repeating high resolution, the next high resolution image is estimated from the image with the error, and errors may gradually accumulate. There is. That is, the destination converged by the iterative calculation is not always the solution with the error suppressed. For this reason, the high-resolution image finally obtained cannot guarantee sufficient restoration accuracy.

  Therefore, the present invention provides an image processing method, an image processing device, an imaging device, an image processing program, and a storage medium that can perform high resolution using fractal coding with high accuracy.

  An image processing method according to one aspect of the present invention includes a step of extracting a first image that is a part of a target image based on a captured image, and a correlation between the first image and each of a plurality of second images. Calculating a value, selecting a third image corresponding to the first image from the plurality of second images based on the correlation value, the first image, and the third image A step of acquiring information indicating a correlation with an image, and a step of increasing the resolution of the target image using the third image and the information, and the step of increasing the resolution of the target image Includes a step of switching processing according to the accuracy of the third image, performing an iterative operation that repeats the step of increasing the resolution of the target image a plurality of times, wherein n is a natural number of 2 or more and m is smaller than n If it is a natural number, the nth iteration In calculation, it determines the accuracy of the third image on the basis of the target image used by m th iteration operation.

  An image processing apparatus according to another aspect of the present invention includes an image extraction unit that extracts a first image that is a part of a target image based on a captured image, and each of the first image and the plurality of second images. Correlation value calculating means for calculating a correlation value with the image, means for selecting a third image corresponding to the first image from the plurality of second images based on the correlation value, and the first Information acquisition means for acquiring information indicating a correlation between one image and the third image, and high resolution means for increasing the resolution of the target image using the third image and the information; And the high-resolution means performs high-resolution processing according to the accuracy of high-resolution of the target image, and performs repetitive calculation that repeats high-resolution of the target image a plurality of times. If n is a natural number greater than or equal to 2 and m is a natural number smaller than n, then n In the eyes of the iterative calculation to determine the accuracy of the third image on the basis of the target image used by m th iteration operation.

  An imaging apparatus according to another aspect of the present invention includes an imaging element that photoelectrically converts an optical image and outputs a captured image, and an image extraction unit that extracts a first image that is a part of a target image based on the captured image. Correlation value calculating means for calculating a correlation value between each of the first image and each of the plurality of second images, and from the plurality of second images to the first image based on the correlation value. Using an image selection unit that selects a corresponding third image, an information acquisition unit that acquires information indicating a correlation between the first image and the third image, the third image, and the information High resolution means for increasing the resolution of the target image, and the high resolution means performs high resolution processing according to the accuracy of the resolution of the target image, It is configured to perform an iterative operation that repeats high resolution of a target image a plurality of times, and n is 2 Natural numbers above, if the m and n a natural number less than, the n th iteration calculation, determines the accuracy of the third image on the basis of the target image used by m th iteration operation.

  An image processing program according to another aspect of the present invention includes: a step of extracting a first image that is a part of a target image based on a captured image; and the first image and each of the plurality of second images. Calculating a correlation value; selecting a third image corresponding to the first image from the plurality of second images based on the correlation value; and the first image and the third image Image processing configured to cause a computer to execute information acquisition of information indicating a correlation with an image of the image, and high resolution of the target image using the third image and the information. In the program, the step of increasing the resolution of the target image includes a step of switching processing according to the accuracy of the third image, and repeating the step of increasing the resolution of the target image a plurality of times The When n is a natural number of 2 or more and m is a natural number smaller than n, the accuracy of the third image is set based on the target image used in the m-th iteration in the n-th iteration. judge.

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to provide an image processing method, an image processing apparatus, an imaging apparatus, an image processing program, and a storage medium capable of performing high resolution using fractal encoding with high accuracy. .

1 is a block diagram of an imaging apparatus in Embodiment 1. FIG. 1 is an external view of an image pickup apparatus in Embodiment 1. FIG. 10 is a flowchart illustrating conversion information acquisition processing (first processing) in the first to third embodiments. 12 is a flowchart illustrating a high resolution process (second process) by decoding in the first to third embodiments. In Example 1 thru | or 3, it is explanatory drawing regarding the correlation calculation of a range block and a domain block. In Example 1 thru | or 3, it is explanatory drawing regarding the precision evaluation of high-resolution. In Example 1 thru | or 3, it is explanatory drawing regarding the search of a new corresponding | compatible domain block. 6 is a block diagram of an image processing system in Embodiment 2. FIG. 6 is an external view of an image processing system in Embodiment 2. FIG. FIG. 6 is a block diagram of an imaging system in Embodiment 3. FIG. 6 is an external view of an imaging system in Embodiment 3. FIG. 6 is a configuration diagram of an imaging apparatus (imaging optical system) in Embodiment 3.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, an overview of high resolution processing (image processing method) in the present embodiment will be described. Processing is broadly divided into two types: acquisition of conversion information using fractal coding and high resolution (corresponding to decoding for fractal coding). For the sake of brevity, an outline will be given focusing on image enlargement in the resolution enhancement.

First, fractal encoding is performed to obtain conversion information. Then, a partial region R ₀ called a range block is extracted from the acquired input image, and a domain block D ₀ having a structure (shape) similar to the partial region R ₀ and having a larger number of pixels is searched. If the high domain block D ₀ of similarity found, which was the corresponding domain block A _0, conversion correlations corresponding domain block A ₀ between the range block R ₀ (the positional relationship and the number of pixels of the ratio of the image) Obtain as information. These are performed for all range blocks R ₀ to be high-resolution, and the conversion information acquisition step is completed.

Next, the process proceeds to high resolution processing. In this processing step, the target image to be increased in resolution is increased in resolution using the conversion information described above. At the first iteration, the input image is the target image. Here, the range block of the target image in the i-th iteration is defined as R _i , the domain block is defined as D _i , and the corresponding domain block is defined as A _i . First, consider the first iteration. Corresponding domain block A ₁ can be considered because it is similar to the structure of the range block R _1, increased the number of pixels (i.e., enlarged) to be the range block R _1. For this reason, the image enlargement process can be performed by replacing the range block R ₁ with the corresponding domain block A ₁ . When replacement is performed for all the range blocks R1, the _first high resolution is completed.

Next, the high resolution is repeated with this high resolution image as a new target image, and the result is converged. At this time, the corresponding domain block A _i used for the replacement has been improved in resolution by the i−1th iteration. Therefore, it is possible to high resolution the range block R _i in each iteration of iterative operations. Even in the second and subsequent iterations, the relationship between the range block R _i and the corresponding domain block A _i does not change from the initial relationship unless an error is considered. For this reason, the initial conversion information can be used as it is.

However, in practice, since error accumulation is performed as described above, in this embodiment, accuracy determination for high resolution is performed at the time of n (arbitrary natural number) iterations. This accuracy determination, corresponding domain block A _n in the n th iteration calculation, see how much different from the range block R _m of the target image in the m (n a natural number smaller than) th iteration operation. If it is determined in the accuracy determination that the accuracy is low, processing for increasing the accuracy is performed.

  Through the above processing, accumulation of errors can be avoided and a high-resolution high-resolution image can be obtained.

  Next, the image pickup apparatus according to the first embodiment of the present invention will be described. FIG. 1 is a block diagram of an imaging apparatus 100 in the present embodiment. FIG. 2 is an external view of the imaging apparatus 100.

  In FIG. 1, an image acquisition unit 101 includes an imaging optical system 101a and an image sensor 101b. The image pickup device 101b includes a charge coupled device (CCD), a complementary metal-oxide semiconductor (CMOS), and the like. The image sensor 101b photoelectrically converts an optical image (subject image) obtained via the imaging optical system 101a and outputs a captured image (image signal).

  At the time of shooting, the light incident on the image acquisition unit 101 is collected by the imaging optical system 101a and converted into an analog electrical signal by the image sensor 101b. An output signal (analog electric signal) from the image sensor 101b is converted into a digital signal by the A / D converter 102 and input to the image processing unit 103 (image processing apparatus). In addition to predetermined image processing, the image processing unit 103 performs restoration processing (high resolution processing) of frequency components lost due to shooting.

  The image processing unit 103 includes an image extraction unit 103a, an image selection unit 103b, an information acquisition unit 103c, and a high resolution unit 103d in order to perform a high resolution process. The image extraction unit 103a extracts a range block (first image) that is a part of a target image (captured image itself or an image obtained by performing high resolution processing on the captured image) based on the captured image. To do. The image selection unit 103b selects a corresponding domain block (third image) corresponding to the range block (first image) from the domain block (a plurality of second images). The information acquisition unit 103c acquires information (conversion information) indicating the correlation between the range block (first image) and the corresponding domain block (third image). The high-resolution means 103d uses the corresponding domain block (third image) and information (conversion information) to increase the resolution of the target image. Further, the high-resolution means 103d performs high-resolution processing according to the accuracy of high-resolution of the target image. The details of the high resolution processing will be described later.

  Here, the high resolution processing is processing for enlarging an image or removing blur. Image enlargement is a process of reducing the sampling pitch, and is used for up-conversion and digital zoom. The blur is deterioration caused by aberration, diffraction, defocus, or blurring at the time of exposure that occurs in the image acquisition unit 101. The image after the high resolution processing is stored in the image recording medium 104 in a predetermined format. Alternatively, an image already stored in the image recording medium 104 may be read and the high resolution processing may be performed by the image processing unit 103 as described above. When an image stored in the image recording medium 104 is viewed, the image is output to the display unit 105 such as a liquid crystal display. The series of processes described above is controlled by the system controller 106. Further, the mechanical drive of the image acquisition unit 101 is performed by the control unit 107 based on a command from the system controller 106.

  Next, the high resolution processing performed by the image processing unit 103 will be described in detail with reference to FIGS. The high resolution processing is divided into two stages: a first process for acquiring information necessary for high resolution (conversion information acquisition process) and a second process for actually increasing the resolution of the image. Yes. First, the first process (conversion information acquisition process) will be described in detail with reference to FIGS. 3 and 5.

  FIG. 3 is a flowchart showing the first processing (conversion information acquisition processing) in the present embodiment. FIG. 5 is an explanatory diagram regarding the calculation of the correlation between the range block and the domain block. Each step in FIG. 3 is mainly executed by the image processing unit 103 (image processing apparatus) based on a command from the system controller 106 of the imaging apparatus 100.

First, in step S <b> 101, the image processing unit 103 acquires an input image (captured image) obtained by the image acquisition unit 101. The input image has less information than the subject space due to degradation (rough sampling or blur) due to shooting. Subsequently, in step S102, the image processing unit 103 (image extraction unit 103a) extracts a range block R ₀ (first image) that is a part of the input image from the input image (target image based on the captured image). . For example, as illustrated in FIG. 5, the image extraction unit 103 a extracts the range block 201 (first image) from the input image 200 (captured image). However, in the present embodiment, the size, shape, and extraction position (the position of the range block 201) of the range block 201 are not limited to this. Further, when a plurality of range blocks _R0 are extracted from the input image, the plurality of range blocks _R0 may overlap each other.

Subsequently, in step S103, the image processing unit 103 acquires a domain block D ₀ (second image). Domain block D ₀ can be obtained from the input image. That is, the domain block (each of the plurality of second images) is a part of the target image based on the captured image. However, the present embodiment is not limited to this, and may be acquired from an image different from the input image. In addition, when the range block R ₀ is extracted from a certain color component (for example, Green) of the input image, the domain block D ₀ may be acquired from another color component (Red or Blue) of the input image. However, from the viewpoint of utilizing fractal object space, image to obtain the domain block D ₀ is preferably input image and the same or similar object is an image that exists. For example, as illustrated in FIG. 5, the image processing unit 103 acquires the domain block 202 as the domain block D ₀ . In FIG. 5, the domain block 202 is acquired from the input image 200 (captured image). However, the size of the domain block D _0, shape, and extraction position (position of the domain block D ₀₎ is not limited thereto.

Here, when high resolution is achieved by image enlargement, the number of pixels of the domain block D ₀ (each of the plurality of second images) is greater than the number of pixels of the range block R ₀ (first image). Set to increase. However, when the purpose is to remove blur, the presence or absence of blur existing in the block is more problematic than the number of pixels, so the number of pixels may be smaller than the range block _R0 .

Subsequently, in step S104, the image processing unit 103 (correlation value calculation unit 103e) performs correlation (correlation value) between the range block R ₀ (first image) and the domain block D ₀ (each of the plurality of second images). f (R ₀ , D ₀ )) is calculated. The correlation function f is used for calculating the correlation value. First, in order to calculate the correlation value f (R ₀ , D ₀ ), the image processing unit 103 converts the domain block D ₀ acquired in step S103 to generate a correlation calculation domain block. The transformation here is, for example, affine transformation, complex transformation, projective transformation, or the like. FIG. 5 shows an example thereof, and the image processing unit 103 performs affine transformation of resizing and rotation to generate a domain block 203 for correlation calculation.

Resizing is performed to easily calculate the correlation of the domain block D ₀ and range block R _0. Isometric transformations such as rotation and inversion are performed to make it easier to find similar shapes of the range block _R0 . However, the conversion to be performed is not limited to this. For example, when attempting to remove the blur, the image processing unit 103 generates a correlation calculation domain block by applying a blur to be removed as a domain block D _0. Further, the image processing unit 103 may convert the range block R ₀ instead of converting the domain block D ₀ . In this case, the image processing unit 103 calculates the correlation by aligning the range block R ₀ with the domain block D ₀ . However, since the range block R ₀ has a smaller amount of information than the domain block D ₀ , interpolation and estimation are required for the conversion. Therefore, from the viewpoint of accuracy, it is preferable to apply a transform to the domain block D _0.

Here, two correlation functions f are given as examples. However, the present embodiment is not limited to this. For example, when the correlation function f capable of directly calculating the correlation value of the domain block D ₀ is prepared for the range block R ₀ having a different number of pixels, the above-described generation of the correlation calculation domain block is not necessary. Examples thereof include SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Up Robust Features). However, it is preferable to use the correlation function f described later because the method of calculating the correlation by combining the number of pixels as described later has higher similarity calculation accuracy than these methods. Further, in the following description, for simplicity, the color component (RGB) is not considered, and an equation is described for the case of a single signal. However, the present invention can be similarly applied to a case having a plurality of color components.

<Example 1 of correlation function>
The total number of pixels of the range block R ₀ is k _R0 , and the signal value vector whose component is the signal value of each pixel is r ₀ . Similarly, the total number of pixels of the domain block D ₀ is k _D0 and the signal value vector is d ₀ . Since the search target is the domain block D ₀ including a structure (shape) similar to the range block R ₀ , the DC component of the signal (indicating an average value and corresponding to the brightness of the image) is irrelevant. For this reason, the DC component is subtracted from the signal vector. At this time, the signal vector ρ ₀ of the range block R ₀ used for correlation calculation is expressed as the following equation (1).

In Equation (1), r _{0, ave} is an average signal value of the range block R ₀ , and e is a k _R0 dimensional vector with each component being 1. r _{0, ave} may be calculated with a uniform weight or a weighted average.

Similarly, the signal vector δ ₀ of the correlation calculation domain block is expressed by the following equation (2).

In Equation (2), δ _{0, ave} is an average signal value of the domain block D ₀ , σ (d ₀ , k _R0 / k _D0 ) is a transformation for resizing the signal vector d ₀ from the k _D0 dimension to the k _R0 dimension, ε Is an isometric transformation. For resizing, bilinear interpolation or bicubic interpolation may be used. The isometric conversion includes, for example, identity conversion, rotation conversion, or inversion conversion. Here, the isometric conversion is performed after resizing to suppress the calculation load, but the order may be reversed. Other affine transformations, complex transformations, or projective transformations may be used.

Since the dimensions of the signal vector ρ ₀ and the signal vector δ ₀ coincide with each other by the equations (1) and (2), the sum of absolute values of the signal (each component of the vector) difference in each pixel is obtained. The correlation calculation formula g in this case is expressed by the following formula (3).

In Equation (3), ρ _{0, j} is the j-th component of the signal vector ρ ₀ , δ _{0, j} is the j-th component of the signal vector δ ₀ , and w _j is the weight of the j-th component.

At this time, the contrast may be adjusted so that the correlation between the signal vector δ ₀ and the signal vector ρ ₀ is the highest. That is, the coefficient c that minimizes | ρ ₀ −cδ ₀ | is determined. Such a coefficient c is expressed by the following equation (4) from the least square method.

When considering the contrast adjustment in Expression (4), the absolute value in the correlation calculation expression g in Expression (3) may be set to ρ _{0, j} −cδ _{0, j} .

  The smaller the value of the correlation calculation formula g in equation (3), the stronger the correlation between the two signal vectors. The correlation function f in this example is expressed by the equations (1) to (4).

<Example 2 of correlation function>
As an example of the correlation function f, SSIM (Structure Similarity) may be used. SSIM is represented by the following formula (5).

  In Expression (5), L, C, and S are evaluation functions related to brightness, contrast, and other structures, and take values of 0 to 1, respectively. The closer each value is to 1, the closer the two signals being compared. α, β, and γ are parameters for adjusting the weight of each evaluation item. Here, if α = 0, the correlation calculation from which the DC component (brightness) is subtracted is performed, so that it is not necessary to perform the calculation represented by the equation (1). Further, when β = 0, it is not necessary to consider the scalar multiplication (contrast adjustment) of the AC component when calculating the correlation, and it is not necessary to perform the calculation represented by the equation (4). The correlation function f may be used by combining a plurality of evaluation functions. The flow of steps S102 to S104 in FIG. 3 is as shown in FIG.

Subsequently, in step S105, the image processing unit 103 determines whether or not the correlation value f (R ₀ , D ₀ ) calculated in step S104 satisfies a predetermined condition (predetermined condition). When the image processing unit 103 determines that the correlation value f (R ₀ , D ₀ ) does not satisfy the predetermined condition and the correlation is low, the image processing unit 103 returns to step S103 and acquires a new domain block D ₀ . Alternatively, the image processing unit 103 may re-calculate the correlation by performing a new conversion to the domain block D _0.

On the other hand, when the image processing unit 103 determines that the correlation value f (R ₀ , D ₀ ) satisfies the condition and the correlation between the range block R ₀ and the domain block D ₀ is high, the image processing unit 103 changes the domain block D ₀ to the range block R _0. As a corresponding domain block A ₀ (third image). As described above, the image processing unit 103 (image selection unit 103b) selects the corresponding domain block (third image) corresponding to the range block (first image) from the target image (a plurality of second images). Based on the correlation value calculated by the correlation value calculation unit 103e in step S104, the image selection unit 103b corresponds to the corresponding domain block corresponding to the range block (first image) from the domain block (a plurality of second images). Select (third image). Then, the process proceeds to step S106. Corresponding domain block A ₀ corresponds to high-resolution range block R ₀ . For example, as illustrated in FIG. 5, the image processing unit 103 selects the corresponding domain block 204 as the corresponding domain block A ₀ .

In step S106, the image processing unit 103 (information acquisition unit 103c) performs information (correlation conversion) indicating the correlation between the range block R ₀ (first image) and the corresponding domain block A ₀ (third image). Information). The conversion information, the range block R ₀ and the position and extent of the corresponding domain block A ₀ in the image conversion was performed at step S104, and includes various parameters. Subsequently, in step S107, the image processing unit 103 (or the system controller 106) determines whether or not the conversion information acquisition process is completed within a predetermined area (in a predetermined area) of the input image. The predetermined area may be either the entire area of the input image or a partial area. When the conversion information acquisition process ends, this flow ends. On the other hand, if the conversion information acquisition process has not ended, the process returns to step S102, and the image processing unit 103 extracts a new range block _R0 .

  Next, the high resolution processing will be described in detail with reference to FIGS. 4, 6, and 7. In the high-resolution processing, the processing is repeated a plurality of times to converge the result. FIG. 4 is a flowchart showing the high resolution processing (second processing) by decoding in the present embodiment. FIG. 6 is an explanatory diagram regarding accuracy evaluation for high resolution. FIG. 7 is an explanatory diagram regarding a search for a new corresponding domain block. Each step in FIG. 4 is mainly executed by the image processing unit 103 (high resolution unit 103d or the like) based on a command from the system controller 106.

  First, in step S201 in FIG. 4, the image processing unit 103 acquires a high-resolution target image (a target image based on a captured image). At the first iteration, the image processing unit 103 acquires an input image (captured image) as a target image. However, the entire resolution or a part of the input image may be increased. A case where only a part of the input image is improved in resolution, particularly a case where only a region where deterioration is strong due to the influence of aberration or the like is increased is considered. At this time, processing such as edge enhancement may be applied to the input image.

Subsequently, in step S202, the image processing unit 103 acquires a range block _Ri and conversion information corresponding thereto from the target image. Here, R _i is a range block in the i-th iteration. When i = 1, the target image is the input image, so R ₁ = R ₀ . Subsequently, in step S203, the image processing unit 103 determines whether or not the number of iterations is a predetermined number (predetermined number) for performing high resolution accuracy determination. When the accuracy determination is performed because the number of iterations is a predetermined number, the process proceeds to step S204. On the other hand, when the accuracy determination is not performed, the process proceeds to step S210. Here, it is assumed that accuracy determination is performed at the time of the nth iteration as a predetermined number of times. Note that n is a natural number of 2 or more.

In step S _< b _> 204, the image processing unit 103 calculates a correlation value f (R _m , A _n ) in order to determine the accuracy of high resolution. Here, m is a natural number less than n. As described above, high resolution Zoka is performed by replacing the range block R _n with the corresponding domain block A _n. However, errors may accumulate due to repeated iterative calculations, and the high-resolution image may deviate in the wrong direction from the initial input image. Therefore, as illustrated in FIG. 6, the image processing unit 103 calculates a correlation between the corresponding domain block A _n (207) in the n-th iteration operation and the range block R _m (205) of the target image in the m-th iteration. Thus, the accuracy of high resolution is determined. In FIG. 6, the target image is shown on the left side, and the image on which the corresponding domain block A _i used for high resolution is extracted on the right side. Here, the corresponding domain block A _i is extracted from the target image itself. I in FIG. 6 indicates the number of iterations, and indicates that the iterations are progressing downward. In FIG. 6, for simplicity, m = 1 is described, but m is not limited to this. If the correlation value f (R _m , A _n ) is high, the influence of the error is small, and if the correlation is low, the error is accumulated and the accuracy of high resolution is deteriorated.

  As described above, in the present embodiment, the image processing unit 103 (high resolution unit 103d) performs an iterative calculation that repeats the step of high resolution of the target image a plurality of times. Further, the high-resolution means 103d, when n is a natural number of 2 or more and m is a natural number smaller than n, in the nth iteration, the third resolution based on the target image used in the mth iteration The accuracy of the image is determined.

Preferably, the accuracy determination (step S204) includes the first image R _m that is a part of the target image in the m-th iteration and the third image corresponding to the first image in the n-th iteration. the correlation value f (R- _{m, a n)} and a _n is performed based on. More preferably, the high resolution unit 103d determines that the accuracy is high when the correlation value f (R− _m , A _n ) is equal to or greater than a predetermined value. On the other hand, if the correlation value f (R- _{m, A n)} is smaller than a predetermined value, it is determined that less accurate. In addition, preferably, when it is determined that the accuracy is low, the high resolution unit 103d changes the third image and information (conversion information) to increase the resolution of the target image.

Subsequently, in step S205, the image processing unit 103 determines whether or not the correlation value f (R _m , A _n ) violates a predetermined condition (predetermined condition). Here, the predetermined condition is a condition in which high resolution accuracy is maintained. If it is determined that the condition is not violated, that is, the accuracy is maintained, the process proceeds to step S210. On the other hand, if it is determined that the condition is not met, the process proceeds to step S206.

In step S206, the image processing unit 103 obtains the new domain blocks _{D n.} Since it is determined in step S205 that the accuracy of the high resolution is poor, the corresponding domain block _An is updated to restore the accuracy. For example, as shown in FIG. 7, a new domain block D _n (208) is acquired by the _nth iteration. Further, without performing the search for new correspondence domain block A _n, with respect to the region determined as a poor precision may only more error so as not to perform iterative operations so as not to increase. However, in order to improve the accuracy of high-resolution, better to search for the corresponding domain block A _n are preferred. Further, the domain block D _n may be extracted from an image different from the target image.

Subsequently, in step S207, the image processing unit 103 calculates a correlation value f (R _n , D _n ) and a correlation value f (R _m , D _n ). At this time, the image processing unit 103 generates a domain block for correlation calculation as necessary, similarly to step S104. For example, as shown in FIG. 7, the correlation value is calculated for the new domain block D _n (208), the range block R _n (206), and the range block R ₁ (205) (m = 1 in FIG. 7). Find the correlation with each. By calculating two correlation values, it has a structure (similar shape) approximated to each of the range blocks R _n and R _m at the present time (nth iteration) and before error accumulation (mth iteration). Domain block D _n can be found. Thereby, the deviation from the initial image (input image) caused by the error can be corrected. In calculating the correlation value, a correlation function different from the correlation function used for acquiring the conversion information may be used.

In step S208, the image processing unit 103 determines whether the correlation value f (R _n , D _n ) and the correlation value f (R _m , D _n ) satisfy a predetermined condition. Here, the predetermined condition is a condition in which high resolution accuracy is maintained. In the evaluation of the correlation value, for example, a method of providing a threshold value for each of the correlation value f (R _n , D _n ) and the correlation value f (R _m , D _n ), a weighted average of both, or a linear sum is taken. There is a method of comparing with a threshold value. If it is determined that the correlation is high, the image processing unit 103 sets the domain block D _n as a new corresponding domain block _An , and proceeds to step S209. If the correlation is determined to be low, the flow returns to step S206, and acquires a new domain block D _n. It is also possible to re-calculate the correlation by performing a new conversion as a domain block D _n.

Subsequently, in step S209, the image processing unit 103, a conversion information of the range block R _n, it is updated based before correlation with the new corresponding domain block A _n acquired in step. Subsequently, in step S210, the image processing unit 103 uses the conversion information corresponding to the range block R _n to obtain the corresponding domain block A _n, perform high-resolution by substitution. For example, if you are using the Example 1 correlation function, as represented by the following formula (6), using the conversion parameters in the conversion information, corresponding domain block A _n signals at the converted signal vector alpha _n it may be replaced by a vector _{r n.}

In the formula (6), _{r n} and _{a n} are each signal vector corresponding domain block _{A n} the range block _{R n} in the n th iteration operation. r _{n, ave} and _{a n, ave} are respectively the average values of each component of the signal vector _{r n} and _{a n.} Alternatively, instead of the replacement process, and the corresponding domain block A _n send using techniques such as learning type super-resolution, it may be carried out with high resolution of the range block R _n.

Subsequently, in step S211, the image processing unit 103 determines whether or not the high resolution process has been completed within a predetermined area (within a predetermined area) of the target image. The predetermined area may be the entire area of the target image or a partial area. If the high resolution processing has been completed within the predetermined area, the process proceeds to step S212. On the other hand, if the high resolution processing is not finished, the process returns to step S202, the image processing unit 103 extracts the new range block R _n.

  Subsequently, in step S212, the image processing unit 103 determines whether or not an iterative calculation end condition is satisfied. As an example of the end condition, whether or not the number of iterations has reached a predetermined number (predetermined number), or whether or not the difference between the nth and n−1th computations is smaller than a predetermined value. And so on. When the end condition is satisfied, the image processing unit 103 ends the processing and outputs a high resolution image (output image). On the other hand, if the termination condition is not satisfied, the process proceeds to step S213. In step S213, the image processing unit 103 replaces the high resolution image with a new target image. After that, the process returns to step S201, and high resolution is performed again.

  As described above, in this embodiment, the image processing unit 103 (high resolution unit 103d) uses the corresponding domain block (third image) and information (conversion information) to increase the resolution of the target image (FIG. 4 steps S201 to S213). Further, the step of increasing the resolution of the target image includes a step of switching processing according to the accuracy of the third image (step S205). Through the processing as described above, high resolution using fractal coding can be performed with high accuracy.

Next, preferable conditions for enhancing the effect of the present embodiment will be described. The determination in step S208 in FIG. 4, the correlation value _{f (R n, D} n) contribution _{f (R} m, _{D n)} that is preferably greater than. When the corresponding domain block _An is searched so as to be closer to the range block R _m in the m-th iterative calculation, although the influence of the error is reduced, there is a possibility that the effect of high resolution may be reduced. It is. In order to achieve high resolution and reduction of calculation error at the same time, it is preferable that the priority of the correlation value f (R _n , D _n ) for determination is equal to or higher than f (R _m , D _n ). For example, when the correlation value is determined by a weighted average of two values, the weight of the correlation value f (R _n , D _n ) is set to be greater than or equal to the weight of the correlation value f (R _m , D _n ). Further, when a threshold value is set for each of the two correlation values, the threshold value of the correlation value f (R _n , D _n ) becomes more severe than the threshold value of the correlation value f (R _m , D _n ). In other words, it may be set so that the determination is not satisfied unless the structure (shape) is more similar.

Preferably, the natural number m of R _m used for the accuracy evaluation of high resolution (step S205) is set to be equal to or less than half of the number of repetitive operations (natural number n) at the time of evaluation. This is because if m is too close to n, errors may have already accumulated at the m-th iteration. By setting m at an early stage of the iterative calculation, evaluation with less error can be performed from an earlier input image.

  More preferably, by setting the natural number m to 1, evaluation with less error can be made from the initial input image. In addition, it is preferable that the accuracy evaluation for high resolution is performed in all the second and subsequent iterations. As a result, errors are sequentially evaluated at the time of high resolution, so that high resolution and suppression of calculation errors can be performed in a balanced manner.

In this embodiment, after obtaining conversion information, high resolution is repeated. However, conversion information may be obtained every time high resolution is repeated. Also in this case, the search for the corresponding domain block _{A n} corresponding to the range block _{R n,} the correlation value _{f (R} m, _{D n)} is preferably used. However, since it takes a calculation time to re-acquire all conversion information every time iterative calculation is performed, the processing configuration as in this embodiment is preferable.

  With the configuration as described above, according to the present embodiment, it is possible to provide an imaging apparatus and an image processing apparatus that can perform high resolution using fractal coding with high accuracy.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a block diagram of the image processing system 300 in the present embodiment. FIG. 9 is an external view of the image processing system 300. In this embodiment, the imaging device and the image processing device are provided separately, and it is possible to increase the resolution of the image by connecting them.

  An input image (captured image) acquired by the imaging device 301 is input to the image processing device 302 via the communication unit 303. The storage unit 304 stores an input image (captured image) and shooting condition information at the time of acquisition of the input image (focal length, aperture value, focus position, etc. at the time of shooting). The image high resolution unit 305 performs high resolution processing. The image (high resolution image) whose resolution has been increased by the image resolution increasing unit 305 is output to one or more of the display device 306, the recording medium 307, and the output device 308 via the communication unit 303. . The display device 306 is, for example, a liquid crystal display or a projector. The user can perform work while confirming an image being processed via the display device 306. The recording medium 307 is, for example, a semiconductor memory, a hard disk, a network server, or the like. The output device 308 is a printer or the like. The image processing apparatus 302 may have a function of performing development processing and other image processing as necessary.

  Further, a storage medium 309 such as a CD-ROM storing an image processing program (image processing software) that causes the image processing apparatus 302 (computer) to execute the image processing method of this embodiment may be used. In this case, by installing the image processing program in the image processing apparatus 302 (computer), the image processing apparatus 302 can read the image processing program and execute the image processing method.

  The processing flow (image processing method) of the image resolution increasing unit 305 in the present embodiment is as shown in FIGS. In the present embodiment, each step of FIGS. 3 and 4 is executed by the image high resolution unit 305 (having the same configuration as the image processing unit 103 of FIG. 1). In the following description, description of the same parts as those in the first embodiment will be omitted.

In step S <b> 101 of FIG. 3, the image processing apparatus 302 acquires an input image (captured image) captured by the imaging apparatus 301. When the high resolution is blur correction, in step S104 and step S207, the image processing apparatus 302 acquires blur information generated in the imaging apparatus 301 and applies it to the domain blocks D ₀ and D- _n for correlation calculation. Generate a domain block. Here, the blur information is PSF (Point Spread Function) such as aberration or diffraction, or OTF (Optical Transfer Function).

  With the above configuration, according to the present embodiment, it is possible to provide an image processing system, an image processing program, and a storage medium that can perform high resolution using fractal coding with high accuracy. it can.

  Next, with reference to FIG. 10 and FIG. 11, the imaging system in Example 3 of this invention is demonstrated. FIG. 10 is a block diagram of the imaging system 400 in the present embodiment. FIG. 11 is an external view of the imaging system 400. In this embodiment, the imaging device and the image processing device are connected wirelessly, and an image from the imaging device is transferred to the image processing device, and the image processing device performs high resolution.

  The server 403 includes a communication unit 404 and is connected to the imaging device 401 via the network 402. When shooting is performed by the imaging device 401, an input image (captured image) is input to the server 403 automatically or manually. The storage unit 405 stores an input image and shooting condition information. Thereafter, the image processing unit 406 performs high resolution processing (image processing method) on the input image (captured image) to generate an output image (high resolution image). The output image is output to the imaging device 401 or stored in the storage unit 405.

  FIG. 12 is a configuration diagram of the imaging apparatus 401 (imaging optical system) in the present embodiment. As shown in FIG. 12, the imaging device 401 has a multi-eye configuration, and four types of imaging optical systems are arranged four by four. The imaging optical systems have different focal lengths for each type. The imaging optical systems 410a to 410d are wide-angle lenses, the imaging optical systems 420a to 420d are standard lenses, the imaging optical systems 430a to 430d are medium telephoto lenses, and the imaging optical systems 440a to 440d are telephoto lenses. However, the type, number, or arrangement of the imaging optical system is not limited to this. In addition, the imaging elements corresponding to the respective imaging optical systems may have different numbers of pixels.

  The processing flow (image processing method) of the image processing unit 406 in the present embodiment is as shown in FIGS. In this embodiment, each step in FIGS. 3 and 4 is executed by the image processing unit 406 (having the same configuration as the image processing unit 103 in FIG. 1). In the following description, description of the same parts as those in the first embodiment will be omitted.

  In step S101 in FIG. 3, the image processing unit 406 (image processing apparatus) acquires an image corresponding to one imaging optical system as an input image (captured image) among images captured by the imaging apparatus 401. However, the processing of FIGS. 3 and 4 may be repeated while changing the input image, and high resolution may be performed on all of the captured images of the imaging apparatus 401.

In step S103, the image processing unit 406 acquires the domain block D _0- . At this time, the image processing unit 406 extracts the domain block D ₀ not only from the input image but also from an image obtained by another imaging optical system of the imaging device 401. The imaging optical systems of the imaging device 401 each have a different viewpoint and angle of view, but all image a common subject. For this reason, by extracting the domain block D ₀ − from these images, the effect of high resolution can be further enhanced.

In particular, when the input image is obtained through the imaging optical systems 410a to 410d that are wide-angle lenses, by extracting the domain block D ₀ − from the captured image of the imaging optical system on the telephoto side, a higher height can be obtained. A resolution effect is obtained. This is because the telephoto side has a large photographing magnification and an image including a finer structure is acquired. The above description is the same for step S206.
With the configuration as described above, according to the present embodiment, it is possible to provide an imaging system capable of performing high resolution using fractal coding with high accuracy.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

103 image processing unit 103a image extraction unit 103b image selection unit 103c information acquisition unit 103d high resolution unit 103e correlation value calculation unit

Claims (16)

Extracting a first image that is part of the target image based on the captured image;
Calculating a correlation value between the first image and each of the plurality of second images;
Selecting a third image corresponding to the first image from the plurality of second images based on the correlation value;
Obtaining information indicating a correlation between the first image and the third image;
Resolving the target image using the third image and the information, and
The step of increasing the resolution of the target image includes a step of switching processing according to the accuracy of the third image,
Performing an iterative operation to repeat the step of increasing the resolution of the target image a plurality of times,
When n is a natural number of 2 or more and m is a natural number smaller than n, the accuracy of the third image is determined based on the target image used in the m-th iteration in the n-th iteration. An image processing method characterized by that. Determination of the accuracy, said first image R _m which is a part of the target image in the iterative calculation of the m-th, the third image A _n corresponding to the first image in the n-th iteration calculation The image processing method according to claim 1, wherein the image processing method is performed based on a correlation value f (R _m , A _n ).   The image processing method according to claim 2, wherein when the accuracy is determined to be low, the third image and the information are changed to increase the resolution of the target image. When the range block of the target image in the i-th iterative calculation is R _i , when it is determined that the accuracy is low in the accuracy determination, the conversion information corresponding to the range block R _n using the range block R _m The image processing method according to claim 1, wherein the image processing method is updated. When the domain block in the i-th iterative operation is D _i , the conversion information is updated by calculating the correlation value f (R _n , D _n ) and the correlation value f (R _m , D _n ). The image processing method according to claim 4, wherein the image processing method is updated. The image processing according to claim 5, wherein the contribution to the update of the conversion information is that the correlation value f (R _n , D _n ) is greater than the correlation value f (R _m , D _n ). Method.   The image processing method according to claim 1, wherein the m is equal to or less than half of the n.   The image processing method according to claim 7, wherein the m is 1.   9. The image processing method according to claim 1, wherein the number of pixels of each of the plurality of second images is greater than the number of pixels of the first image.   The image processing method according to claim 1, wherein each of the plurality of second images is a part of the target image based on the captured image.   The image processing method according to claim 1, wherein the third image is an image including a structure similar to the first image.   The image processing method according to claim 1, wherein the determination of the accuracy is performed for all of the second and subsequent iterations. Image extracting means for extracting a first image that is a part of the target image based on the captured image;
Correlation value calculating means for calculating a correlation value between each of the first image and the plurality of second images;
Image selecting means for selecting a third image corresponding to the first image from the plurality of second images based on the correlation value;
Information acquisition means for acquiring information indicating a correlation between the first image and the third image;
High-resolution means for increasing the resolution of the target image using the third image and the information,
The high resolution means includes:
Perform high resolution processing according to the accuracy of high resolution of the target image,
It is configured to perform an iterative calculation that repeats the resolution of the target image a plurality of times,
When n is a natural number of 2 or more and m is a natural number smaller than n, the accuracy of the third image is determined based on the target image used in the m-th iteration in the n-th iteration. An image processing apparatus characterized by that. An image sensor that photoelectrically converts an optical image and outputs a captured image;
Image extracting means for extracting a first image that is a part of a target image based on the captured image;
Correlation value calculating means for calculating a correlation value between each of the first image and the plurality of second images;
Image selecting means for selecting a third image corresponding to the first image from the plurality of second images based on the correlation value;
Information acquisition means for acquiring information indicating a correlation between the first image and the third image;
High-resolution means for increasing the resolution of the target image using the third image and the information,
The high resolution means includes:
Perform high resolution processing according to the accuracy of high resolution of the target image,
It is configured to perform an iterative calculation that repeats the resolution of the target image a plurality of times,
When n is a natural number of 2 or more and m is a natural number smaller than n, the accuracy of the third image is determined based on the target image used in the m-th iteration in the n-th iteration. An imaging apparatus characterized by that. Extracting a first image that is part of the target image based on the captured image;
Calculating a correlation value between the first image and each of the plurality of second images;
Selecting a third image corresponding to the first image from the plurality of second images based on the correlation value;
Obtaining information indicating a correlation between the first image and the third image;
An image processing program configured to cause a computer to execute the step of increasing the resolution of the target image using the third image and the information,
The step of increasing the resolution of the target image includes a step of switching processing according to the accuracy of the third image,
Performing an iterative operation to repeat the step of increasing the resolution of the target image a plurality of times,
When n is a natural number of 2 or more and m is a natural number smaller than n, the accuracy of the third image is determined based on the target image used in the m-th iteration in the n-th iteration. An image processing program characterized by that.   A storage medium storing the image processing program according to claim 15.