JP4763419B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  Conventionally, a photographed image photographed by a camera using an imaging device such as a CCD (Charge Coupled Devices) in the imaging unit is a camera shake during photographing, various aberrations of the photographing optical system, or a lens constituting the photographing optical system. It is known that when there is a distortion or the like, this is a factor and the captured image deteriorates.

  As means for preventing such deterioration of the photographed image, two methods of moving the lens and a method of circuit processing the photographed image are known regarding camera shake during photographing among the causes of deterioration of the photographed image. . For example, as a method of moving the lens, camera shake is detected, and a predetermined lens in the photographing optical system is moved in accordance with the movement of the camera due to the detected camera shake, thereby moving the imaging position on the image sensor. There is known a method for suppressing the above (see Patent Document 1).

  As a circuit processing method, a change in the optical axis of the photographing optical system of the camera is detected by an angular acceleration sensor or the like, and a transfer function representing a blurring state at the time of photographing is obtained from the detected angular velocity, etc. A method is known in which an acquired transfer function is inversely transformed to restore an image without deterioration (see Patent Document 2).

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens such as a motor, and thus becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  In addition, in the case of camera shake correction described in Patent Document 2, the above-described problems are eliminated, but the following problems occur.

  First, the transfer function to be obtained is obtained based on the angular velocity detected by an angular acceleration sensor or the like, and is very vulnerable to noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. . For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice.

  Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In reality, there is a high risk that it will be impossible to solve.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus having a realistic circuit processing method and to shorten the processing time, while preventing an increase in size of the apparatus when restoring an image. It is.

  In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus having a processing unit for processing an image, and the processing unit is restored by adding corrected image data to initial image data in a frequency space. Processing to obtain image data was performed.

  According to the present invention, it is possible to provide an image processing apparatus having a realistic circuit processing method while reducing the size of the apparatus when restoring an image, and to shorten the processing time. .

ただし、「ＩＭＧ′」は撮影画像データを表わす。「Ｈ」はフィードバック関数を表わす。「Ｇ」は変化要因を表わす。「ｎ」は復元画像データが原画像データに近似したものと推定できる任意の数を表わす。 In addition to the above-described invention, another invention performs processing for obtaining restored image data by setting the initial image data as “I ₀ ” and the corrected image data as the image data represented by Expression (A). .

However, “IMG ′” represents photographed image data. “H” represents a feedback function. “G” represents a change factor. “N” represents an arbitrary number that can be estimated that the restored image data approximates the original image data.

When this configuration is adopted, the restored image data is obtained by a calculation of adding the corrected image data represented by the expression (A) to the initial image data “I ₀ ”, so that the processing time can be further shortened. it can.

ただし、「ＩＭＧ′」は撮影画像データを表わす。「Ｈ」はフィードバック関数を表わす。「Ｇ」は変化要因を表わす。「ｎ」は復元画像データが原画像データに近似したものと推定できる任意の数を表わす。 In addition to the above-described invention, another invention performs the process of obtaining restored image data by setting the initial image data to 0 and the corrected image data as image data represented by equation (B).

However, “IMG ′” represents photographed image data. “H” represents a feedback function. “G” represents a change factor. “N” represents an arbitrary number that can be estimated that the restored image data approximates the original image data.

  When this configuration is adopted, since the initial image data is 0, the restored image data is obtained by calculation using the equation (B) having a smaller number of terms than the equation (A), so that the processing time can be further shortened. it can.

  According to the present invention, it is possible to provide an image processing apparatus having a realistic circuit processing method while reducing the size of the apparatus when restoring an image, and to shorten the processing time. .

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a so-called consumer digital camera that uses a CCD as an imaging unit. However, the image processing apparatus 1 is a surveillance camera, a television camera, and an endoscope camera that use an imaging element such as a CCD as the imaging unit. The present invention can also be applied to devices other than cameras, such as cameras for other uses, image diagnostic apparatuses for microscopes, binoculars, and NMR imaging.

  The image processing apparatus 1 includes an imaging unit 2 that captures a subject such as a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4, an angular velocity sensor, and the like, and changes factor information that causes changes such as image degradation. It has a detection unit 6 for detecting, and a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

  The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD or C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is composed of an image processing processor, and is composed of hardware such as ASIC (Application Specific Integrated Circuit). The processing unit 4 includes a recording unit (not shown), and the recording unit has an arbitrary image as a base when generating image data having a high frequency component of complex space in processing image data, which will be described later, or comparison image data. Image data is saved. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 includes a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD (Digital Versatile Disk), or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the photographing optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the photographing optical system and lens distortion. However, when correcting image degradation due to camera shake, which will be described later, such information is stored. Is not used.

  Next, prior to describing the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above, an outline of the concept of the present invention will be described with reference to FIGS. 3 and 4.

In FIG. 3, “i ₀ ” is arbitrary initial image data. “I ₀ ′” indicates degraded image data of the initial image data “i ₀ ”, and is comparison image data for comparison. “G” is a change factor function (= deterioration factor information (point spread function)) detected by the detection unit 6. “Img ′” indicates data of a captured image.

“Δ” is difference data between the two image data obtained by comparing the captured image data “img ′” with the comparison image data “i ₀ ′”. “H” is a feedback function based on the change factor function “g”. “I _{0 + n} ” is restored image data (restored image data) newly generated by distributing the difference data “δ” to the initial image data “i ₀ ” based on the change factor function “g”. “Img” is data of the original image that is the basis of the captured image data “img ′”, which is a captured degraded image. That is, the original image data “img” is an image before the photographed image data “img ′” is changed, or an original image that should be obtained if the photographed image data is correctly photographed. More specifically, for example, image data when it is assumed that the image is captured in a state in which the image is not deteriorated due to camera shake or the like during the shooting operation. Here, it is assumed that the relationship between “img” and “img ′” is expressed by the following equation (E).
img ′ = img * g (E)
“*” Is an operator representing a superposition integral.
Note that the difference data “δ” may be a simple difference between corresponding pixels, but generally differs depending on the change factor function “g” and is expressed by the following equation (F).
δ = f (img ′, img, g) (F)

The processing routine of FIG. 3 starts by preparing arbitrary initial image data “i ₀ ” (step S101). As the initial image data “i ₀ ”, photographed image data “img ′” may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. In step S102, initial image data “i ₀ ” is inserted instead of “img” in the expression (1), and comparison image data “i ₀ ” that is a deteriorated image changed (deteriorated) by the change factor function “g”. ''. Next, the captured image data “img ′” and the comparison image data “i ₀ ′” are compared to calculate difference data “δ” (step S103).

Next, in step S104, it is determined whether or not the difference data “δ” is equal to or greater than a predetermined value. If the difference data “δ” is equal to or greater than the predetermined value, a process of generating restored image data (restored image data) in step S105. I do. That is, the difference data “δ” is distributed to the initial image data “i ₀ ” according to the feedback function “h” based on the change factor function “g” to generate restored image data “i _{0 + n} ”. Thereafter, in step S102, initial image data “i ₀ ” is generated as comparison image data “i _{0 + n} ′” as restored image data “i _{0 + n} ” generated in step S105. In step S103, captured image data “img” is generated. ′ ”And the comparison image data“ i _{0 + n} ′ ”are compared. Then, the size of the difference data “δ” as a result of the comparison is determined in S104.

In this way, in step S104, the processing of steps S102, S103, S104, and S105 is repeated until the difference data “δ” becomes less than the predetermined value, and the restored image data “i _{0 + n} ” is sequentially updated. The feedback function “h” is a function that makes the difference data “δ” in step S103 smaller than in the previous processing routine.

If the difference data “δ” is smaller than the predetermined value in step S104, the process ends (step S106). Then, the restored image data “i _{0 + n} ” at the time when the processing is completed is estimated as the original image data “img”.

  The concept of the above processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

Solving as an optimization problem assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same.

In other words, as shown in FIGS. 4A and 4B, the image data for comparison “i ₀ ′ (i _{0 + n} ′) that is approximate to the captured image data“ img ′ ”that is the captured image. Can be generated, the initial image data “i ₀ ” or the restored image data “i _{0 + n} ” that is the original data of the generation approximates the original image data “img” that is the source of the captured image data “img ′”. It will be a thing.

  In step S104, the value serving as the determination criterion for the difference data “δ” is, for example, “6” when each data is represented by 8 bits (0 to 255), and is smaller than “6”, that is, “5”. Processing is terminated in the following cases.

  Further, details of the concept shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are designated as n-1, n, n + 1, n + 2, n + 3,... In order from the left, and attention is paid to a certain pixel “n”. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the original image data “img” when there is no deterioration. Here, each data is represented by 8-bit (0 to 255) data.

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred for the nth pixel, 30% of the time is for the (n + 1) th pixel, and 20% of the time is for the (n + 2) th pixel. . The way of energy dispersion is as shown in the table of FIG. The change factor function “g” has a content corresponding to this energy dispersion method.

  Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table of FIG. In FIG. 8, the data indicated as “imaging result” is the original image data “img”, and the data indicated as “blurred image” is the captured image data “img ′”. Specifically, for example, “120” of the pixel “n−3” is based on the change factor function “g” that is the blur information, and the distribution ratio of “0.5”, “0.3”, and “0.2”. Thus, “60” is distributed to the “n-3” pixel, “36” to the “n-2” pixel, and “24” to the “n−1” pixel. Here, the distribution corresponding to the distribution ratio of “0.5”, “0.3”, and “0.2” is processing corresponding to the feedback function “h”. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”.

Any data can be adopted as the initial image data “i ₀ ” shown in step S 101, but for the explanation here, for example, photographed image data “img ′” is used. That is, the process is started with i ₀ = img ′. In the table of FIG. 9, “input” corresponds to initial image data “i ₀ ”. The change factor function “g” is applied to the initial image data “i ₀ ”, that is, “img ′” in step S102. That is, for example, “60” of the “n-3” pixel of the initial image data “i ₀ ” is “30” for the “n-3” pixel and “18” for the “n-2” pixel. , “12” is assigned to each pixel of “n−1”. Allocated similarly for the other pixels, the output "i _{0 '"} image data "i ₀ for comparison shown _as'" is generated. Therefore, the difference data “δ” in step S103 is as shown in the bottom column of FIG.

Thereafter, the size of the difference data “δ” is determined in step S104. Specifically, the process ends when all the difference data “δ” is 5 or less in absolute value, but the difference data “δ” shown in FIG. Proceed to That is, the difference data “δ” is distributed to the initial image data “i ₀ ” using the feedback function “h”, and the restored image data “i _{0 + n} ” shown as “next input” in FIG. Generate. In this case, since this is the first time, i _{0 + 1} is represented in FIG.

For the distribution of the difference data “δ”, for example, the data “30” of the pixel “n−3” is multiplied by “0.5” which is the distribution ratio of one's place (= “n-3” pixel). “15” is distributed to “n-3” pixels, and “0 ..” is a distribution ratio that should have come to the “n-2” pixel in the data “15” of the “n-2” pixel. “4.5” multiplied by “3”, and the distribution ratio that should have come to the pixel “n−1” in the data “9.2” of the pixel “n−1”. Allocate “1.84” multiplied by “0.2”. The total amount allocated to the pixels of “n−3” is “21.34”, and this value is added to the initial image data “i ₀ ” (here, the captured image data “img ′” is used) and restored. Image data “i _{0 + 1} ” is generated.

As shown in FIG. 11, this restored image data “i _{0 + 1} ” becomes the input image data (= initial image data “i ₀ ”) in step S102, step S102 is executed, the process proceeds to step S103, and a new The difference data “δ” is obtained. The size of the new difference data “δ” is determined in step S104. If it is larger than the predetermined value, the new difference data “δ” is distributed to the previous restored image data “i _{0 + 1} ” in step S105, and a new restoration is performed. Image data “i _{0 + 2} ” is generated (see FIG. 12). Thereafter, new comparison image data “i _{0 + 2} ′” is generated from the restored image data “i _{0 + 2} ” by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, and the process proceeds to step S105 or shifts to step S106 depending on the determination there. Such a process is repeated.

In the processing routine shown in FIG. 3 described here, instead of determining the size of the difference data “δ” in step S104, the number of processing routines in steps S102 to S105 is set. The restored image data “i _{0 + n} ” obtained when this processing routine is executed can be approximated to the original image data “img”. In other words, by setting the number of processing times to be greater than or _equal to the number of times that the restored image data “i _{0 + n} ” can be estimated to approximate the original image data “img”, the restoration is performed without determining the size of the difference data “δ”. Image data “i _{0 + n} ” can be obtained.

Then, the concept of processing in which the restored image data “i _{0 + n} ” obtained when the processing routine of the set number of times is executed is approximated to the original image data “img” is expressed by the following formula. FIG. 3 explains the concept of the processing method of the processing unit 4 simply and easily. Therefore, in FIG. 3, the description is given focusing on one pixel, but in order to obtain restored image data in all the imaging regions of the imaging unit 2, the processing routine of FIG. 3 is performed on all the pixels in the imaging region. Need to run.

First, when the initial image data in S101 is “i ₀ ”, in S102, the comparison image data “i ₀ ′” is (1) as a superposition integral of the initial image data “i ₀ ” and the change factor function “g”. It is expressed as an expression.

The difference data “δ” in S103 is expressed as in equation (2).

The restored image data “i _{0 + 1} ” in S105 is expressed as the following equation (3) assuming that the difference data “δ” is distributed to the initial image data “i ₀ ” based on the feedback function “h”.

In the next processing routine, the restored image data “i _{0 + 1} ” in equation (3) enters instead of “i ₀ ” in S102, and the comparison image data “i _{0 + 1} ′” is expressed in equation (4). Generated in the form.

Then, in S103, the difference data “δ” is expressed by equation (5).

Then, the restored image data “i _{0 + 2} ” in S105 is expressed as in equation (6).

Similarly, when “i _{0 + 3} ”,... Is calculated, the restored image data “i _{0 + n} ” is expressed as in equation (7).

That is, the restored image data “i _{0 + n} ” can be seen as the initial image data “i ₀ ” obtained by adding the expression (8), which is a term after “i ₀ ” in the expression (7), as corrected image data. it can.

In the above description of step S104 in FIG. 3, instead of determining the size of the difference data “δ”, the restored image data “i _{0 + n} ” when the processing routine is executed a predetermined number of times or more is used as the original image. It has been explained that image data approximate to the data “img” can be obtained. However, if the above equation (7) is used, the same restored image data as when the routine has been performed a predetermined number of times can be obtained by calculation without executing the processing routine of FIG. For example, in equation (7), the calculated value when k = 20 is equal to the restored image data when the processing routine of FIG. 3 is repeated 20 times.

As the number of processing routines in FIG. 3 is increased, the restored image data “i _{0 + n} ” approaches the original image data “img”, but the processing time becomes longer. On the other hand, in the above equation (7), increasing “k” corresponds to increasing the number of processing routines in FIG. However, even if “k” is increased, the calculation time is shorter than when the processing routine corresponding to “k” is executed.

By the way, in the processing routine of FIG. 3, starting the initial image data “i ₀ ” without a solid black image, that is, without inputting the image data, indicates that “i ₀ ” is “0” in the equation (7). Is equivalent to

That is, in Expression (7), when “i ₀ ” = “0”, Expression (7) is expressed as Expression (9).

That is, it can be considered that the restored image data “i _{0 + n} ” is obtained by adding the equation (9) as the corrected image data to the initial image data “i ₀ = 0”.

The restored image data “i _{0 + n} ” is calculated by adding the equation (9) as the corrected image data to the initial image data “i ₀ = 0”, and the equation (9) is added n times. Is equivalent to restored image data “i _{0 + n} ”. That is, the restored image data “i _{0 + n} ” can be calculated directly from the captured image data “img ′”, the change factor function “g”, and the feedback function “h” without considering the initial image data.

  By the way, regarding the feedback function “h”, for example, when the camera movement locus due to camera shake is represented by a line L in FIG. 13 (the change factor function is “g (x, y)”), the feedback function “ If “h” is a function such that h (−x, −y) = g (x, y), the difference data “δ” can be efficiently reduced.

For example, when the restored image data “i _{0 + n} ” is generated, the center of gravity of a change factor such as deterioration is calculated, and the difference of only the center of gravity or the scaling of the difference is calculated as the previous restored image data “i _{0 + n−1”.} To the content of the feedback function “h”. This concept will be described below based on FIG. 14 and FIG.

  As shown in FIG. 14, when the original image data “img” is composed of pixels 11 to 15, 21 to 25, 31 to 35, 41 to 45, and 51 to 55, as shown in FIG. Pay attention to the pixel 33. When the pixel 33 moves to the positions of the pixels 33, 43, 53, and 52 due to camera shake or the like, in the captured image data “img ′” that is a deteriorated image, as shown in FIG. , 43, 52, 53 are affected by the first pixel 33.

In the case of such deterioration, if the pixel 33 moves and is located at the position of the pixel 43 for the longest time, the center of deterioration, that is, the cause of the change is the pixel 33 in the original image data “img”. In the photographed image data “img ′”, the pixel 43 is located. Thereby, the difference data “δ” is calculated as the difference between the respective pixels 43 of the photographed image data “img ′” and the comparison image data “i ₀ ′” as shown in FIG. The difference data “δ” is added to the pixels 33 of the initial image data “i ₀ ” and the restored image data “i _{0 + n} ”. This is the content of the feedback function “h”.

  In the above example, the three centroids of “0.5”, “0.3”, and “0.2” are the positions of “0.5” having the largest value, and are their own positions. Therefore, allocation of “0.5” or “0.5” of the difference data “δ” is allocated to its own position without considering the allocation of “0.3” or “0.2”. Will do. Such a process is suitable when the blur energy is concentrated. That is, in this case, the content of the feedback function “h” is to allocate only “0.5” or “0.5” magnification of the difference data “δ” to its own position.

ただし、「Ｉ_０＋ｎ」は、復元画像データ「ｉ_０＋ｎ」をフーリエ変換した周波数空間における復元画像データを表わす。「Ｉ_０」は、初期画像データ「ｉ_０」をフーリエ変換した周波数空間における初期画像データを表わす。「ＩＭＧ′」は、撮影画像データ「ｉｍｇ′」をフーリエ変換した周波数空間における撮影画像データを表わす。「Ｈ」は、フィードバック関数「ｈ」をフーリエ変換した周波数空間におけるフィードバック関数を表わす。「Ｇ」は、変化要因「ｇ」をフーリエ変換した周波数空間における変化要因を表わす。 By the way, when Expression (7) and Expression (9) are Fourier transformed, Expression (7) is expressed as Expression (10), and Expression (9) is expressed as Expression (11).

However, “I _{0 + n} ” represents restored image data in a frequency space obtained by Fourier transforming the restored image data “i _{0 + n} ”. “I ₀ ” represents initial image data in a frequency space obtained by Fourier transforming the initial image data “i ₀ ”. “IMG ′” represents photographed image data in a frequency space obtained by Fourier transform of the photographed image data “img ′”. “H” represents a feedback function in a frequency space obtained by Fourier transforming the feedback function “h”. “G” represents a change factor in the frequency space obtained by Fourier transform of the change factor “g”.

Thus, by calculating in the frequency space using the equations (10) and (11) obtained by performing Fourier transform on the equations (7) and (9), the equations (7) and (9) Without performing such superposition integration, the restored image data “I _{0 + n} ” can be calculated by integration (multiplication) and addition / subtraction (addition and subtraction). For this reason, it is possible to increase the speed compared to the case of performing the superposition integration with the calculation speed first.

For restored image data “I _{0 + n} ” calculated by equation (10) or equation (11), image data in the spatial domain is obtained by inverse Fourier transform.

この（１２）式の表すところは、原画像データ「ｉｍｇ」の周波数空間における原画像データを「ＩＭＧ」としたときの、ＩＭＧ・Ｇ＝ＩＭＧ′の逆変換に他ならない。 In Expressions (10) and (11), if k → ∞, both Expressions (10) and (11) are expressed as Expression (12).

Expression (12) represents nothing but the inverse transformation of IMG · G = IMG ′ when the original image data in the frequency space of the original image data “img” is “IMG”.

  However, in the case of the expression (12), it can be seen that the restored image data (I∞) becomes unstable when “G” is 0 or a value close to 0. However, as shown in the equations (10) and (11), when the restored image data is obtained by a geometric series that repeats addition n times, even when “G” is 0 or a value close to 0, Stable restored image data can be obtained.

  The processing using the restored image data using the above equations (7), (9), (10), and (11) may be applied as follows.

  As shown in FIG. 16, when the captured image “data img ′” is composed of pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61 to 66, 1 Every other pixel is thinned out, and reduced-magnified image data “Simg ′” having a size of a quarter composed of the pixels 11, 13, 15, 31, 33, 35, 51, 53, and 55 is generated.

In this way, the photographic image data “img ′” is thinned out to generate the reduced photographic image data “Simg ′” which is the decimated data, and the reduced photographic image data “Simg ′” is expressed by Equation (7) or ( 9) The restored image data “Si _{0 + n} ” for the reduced photographed image data “Simg ′” is obtained using the equation (9). Then, a transfer function is obtained for the restored image data “Si _{0 + n} ” and the reduced photographed image data “Simg ′”, the transfer function is enlarged, the interpolated portion is interpolated, and the corrected transfer function is reduced. This is a transfer function for the captured image data “img ′” of the entire imaging region that is not. Then, using the modified transfer function, deconvolution calculation (calculation for removing blur by calculation from a group of images including blur) may be performed in the frequency space to obtain restored image data “i _{0 + n} ”. The restored image data “i _{0 + n} ” may be used as the initial image data “i ₀ ” in the equation (7).

Alternatively, the reduced photographic image data “SIMG ′” in the frequency space for the reduced photographic image data “Simg ′” is obtained by using the expression (10) or (11) for the reduced photographic image data “Simg ′”. And the restored image data “SI _{0 + n} ” is obtained. Then, a transfer function for the reduced photographed image data “SIMG ′” and the restored image data “SI _{0 + n} ” is obtained, the transfer function is enlarged, the interpolated portion is interpolated, and the corrected transfer function is reduced. This is a transfer function for the captured image data “IMG ′” of the entire imaging region that is not. Then, the restored image data “I _{0 + n} ” is obtained using this transfer function. The restored image data “I _{0 + n} ” may be used as the initial image data “I ₀ ” in the equation (10).

  Further, as shown in FIG. 17, the reduced image data “Simg ′” includes the pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61. When it is configured with ˜66, a region made up of the pixels 32, 33, 34, 42, 43, 44, which is the central region, may be a partial region. However, when the reduced photographed image data “Simg ′” is extracted in this way, it is necessary that the extracted area is sufficiently larger than the fluctuation area. In the previous example shown in FIG. 5 and the like, since it fluctuates over 3 pixels, it is necessary to extract an area of 3 pixels or more.

  The image data to be added as the corrected image data is not limited to those calculated by the equations (7), (9), (10), and (11). Alternatively, the corrected image data in the equations (7), (9), (10), and (11) can be used by further adding / subtracting (plus / minus) image data for correction. .

  Moreover, each processing method mentioned above may be programmed. The program may be stored in a storage medium, for example, a CD (Compact Disc), a DVD, or a USB (Universal Serial Bus) memory, and read by a computer. In this case, the image processing apparatus 1 has reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading a program in the storage medium.

  In the above description, the information stored in the factor information storage unit 7 is not used, but known deterioration factors (change factors) stored here, for example, data such as optical aberrations and lens distortions. May be used. In this case, for example, it is preferable to perform the calculation by considering the blur information and the optical aberration information as one change factor function (deterioration factor function), but the calculation with the change factor function based on the blur information is finished. You may make it perform the calculation by the change factor function by the information of optical aberration later. Further, the factor information storage unit 7 may not be installed, and the image may be restored by a dynamic factor at the time of shooting, for example, a change factor function relating only to blurring.

  Note that the change factor function “g” includes not only a function of a deterioration factor but also a function that simply changes an image or a function that improves an image contrary to deterioration. Further, the angular velocity detection sensor detects the angular velocity every 5 μsec, for example. In addition, the raw shake data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. It is a flowchart for demonstrating the outline | summary of the view of the processing method performed in the process part of the image processing apparatus shown in FIG. It is a figure for demonstrating the concept of the view of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing the concentration of energy when there is no hand shake. FIG. 4 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking hand shake as an example, and showing image data when there is no hand shake. FIG. 4 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, and is a diagram showing energy dispersion when camera shake occurs. FIG. 4 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, and is a diagram for explaining a situation in which comparison image data is generated from an arbitrary image. FIG. 3 is a diagram for specifically explaining an outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, comparing comparison image data with a blurred original image to be processed, and calculating difference data It is a figure for demonstrating the condition which produces | generates. FIG. 3 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, and explains a situation in which restored image data is generated by distributing difference data and adding it to an arbitrary image. FIG. FIG. 3 is a diagram for specifically explaining an outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, generating new comparison image data from the generated restored image data, It is a figure for demonstrating the condition which compares with the blurred original image and produces | generates the data of a difference. FIG. 3 is a diagram for specifically explaining the outline of the concept of the processing method shown in FIG. 3 by taking an example of camera shake, and explains a situation in which newly generated difference data is allocated and new restored image data is generated. It is a figure for doing. It is a figure which shows an example of change factor function "g". It is a figure for demonstrating the case where a feedback function is calculated | required using the gravity center of a change factor function, (A) is a figure which shows the state which pays attention to one pixel in the data of an original image, (B) is a picked-up image. In the figure which shows data, it is a figure which shows the state which the data of the focused pixel expand. It is a figure for demonstrating the case where a feedback function is calculated | required using the gravity center of the change factor function which is the processing method shown in FIG. It is a figure for demonstrating the modification in embodiment, (A) shows the picked-up image data used as a process target, (B) is a figure which shows the data which thinned out the data of (A). FIG. 6 is a diagram for explaining another modification example of the embodiment, in which (A) shows captured image data to be processed, and (B) shows data obtained by extracting a part of the data of (A). is there.

Explanation of symbols

First image processing apparatus 2 imaging unit 3 control system unit 4 processing unit 5 recording unit 6 detector 7 factor information storage unit i _{0, I 0} of the initial image data (data of any image)
i ₀ ′, I ₀ ′ Comparative image data g, G Change factor (deterioration factor)
img ′, IMG ′ Captured image data (captured image)
h, H feedback function i _{0 + n} , I _{0 + n} restored image data

Claims (2)

In an image processing apparatus having a processing unit for processing an image,
The processing unit
In the frequency space, a process for obtaining restored image data by adding the corrected image data to the initial image data,
The initial image data is "I _0", as image data represented the corrected image data by formula (A), an image processing apparatus and performs a process for obtaining the restored image data.

However, “IMG ′” represents photographed image data. “H” represents a feedback function. “G” represents a change factor. “N” represents an arbitrary number that can be estimated that the restored image data approximates the original image data. The image processing apparatus according to claim 1, wherein the initial image data is set to 0, and the corrected image data is processed as image data represented by equation (B) to perform processing for obtaining restored image data.

However, “IMG ′” represents photographed image data. “H” represents a feedback function. “G” represents a change factor. “N” represents an arbitrary number that can be estimated that the restored image data approximates the original image data.

Images