JP4606976B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  Conventionally, a photographed image photographed by a camera using an imaging device such as a CCD (Charge Coupled Devices) in the imaging unit is a camera shake during photographing, various aberrations of the photographing optical system, or a lens constituting the photographing optical system. It is known that when there is a distortion or the like, this is a factor and the captured image deteriorates.

  As means for preventing such deterioration of the photographed image, a method of moving the lens and a method of circuit processing the photographed image are known as camera shake during photographing among the causes of deterioration of the photographed image. For example, as a method of moving the lens, camera shake is detected, and a predetermined lens in the photographing optical system is moved in accordance with the movement of the camera due to the detected camera shake, thereby moving the imaging position on the image sensor. There is known a method for suppressing the above (see Patent Document 1).

  As a circuit processing method, a change in the optical axis of the photographing optical system of the camera is detected by an angular acceleration sensor or the like, and a transfer function representing a blurring state at the time of photographing is obtained from the detected angular velocity, etc. A method is known in which an acquired transfer function is inversely transformed to restore an image without deterioration (see Patent Document 2).

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens such as a motor, and thus becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  In addition, in the case of camera shake correction described in Patent Document 2, the above-described problems are eliminated, but the following problems occur.

  First, the transfer function to be obtained is obtained based on the angular velocity detected by an angular acceleration sensor or the like, and is very vulnerable to noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. . For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice.

  Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In practice, there is a high risk of being unable to solve.

  Accordingly, an object of the present invention is to prevent an increase in the size of the apparatus and to obtain a transfer function that is not easily affected by noise, blur information error, etc. when restoring an image, and to have a realistic circuit processing method. Is to provide a device.

In order to solve the above problems, an image processing apparatus according to the present invention includes an image processing apparatus having a processing unit that processes an image. The processing unit converts input image data that is image data of an image to be processed into an imaginary part. From the processing image data generation processing that is generated as the processing image data to be image data having a high frequency component and the arbitrary image data having the image data of the real part and the imaginary part, the change factor information that causes the image change Comparison image data generation processing for generating comparison image data having image data of a real part and an imaginary part using data, and a difference obtained by comparing the comparison image data with the processing image data of data, thereby generating a restoration image data the percentage of variance based on the data of the change factor information by allocating the arbitrary image data as a distribution ratio, optionally the restored image data By using the image data instead of the image data and repeating the process of generating the restored image data, the original image restored image data in which the real part of the restored image data approximates the image data of the real part of the original image is used. The original image restoration image data generation process to be generated and the transfer function generation process for generating a transfer function from the original image restoration image data and the processing image data are performed.

  According to the present invention, since the transfer function is obtained from the original image restoration image data and the processing image data, a transfer function that is not easily affected by noise, blur information error, etc. is obtained, and a realistic circuit processing method is obtained. Can be provided.

According to another aspect of the present invention, in the image processing apparatus having a processing unit that processes an image, the processing unit converts input image data that is image data of an image to be processed into image data having a high-frequency component in an imaginary part. A processing image data generation process that is generated as processing image data, and from any image data having image data of a real part and an imaginary part, using data of change factor information that causes an image change, and a real part The comparison image data generation process for generating the comparison image data having the image data of the imaginary part, and the difference data obtained by comparing the comparison image data with the processing image data are used as the data of the change factor information. the percentage of variance based on the generated restored image data by allocating the arbitrary image data as a distribution ratio, using the restored image data in place of any image data, An original image that generates original image restored image data in which the real part and imaginary part of the restored image data approximate the image data of the real part and the imaginary part of the original image by repeating the process of generating the original image data. The restored image data generation process and the transfer function generation process for generating a transfer function from the original image restored image data and the processing image data are performed.

  According to the present invention, since the transfer function is obtained from the original image restoration image data and the processing image data, a transfer function that is not easily affected by noise, blur information error, etc. is obtained, and a realistic circuit processing method is obtained. Can be provided.

  In addition to the above inventions, another invention is a reduction process in which processing image data is generated based on input image data for a part of the entire imaging area in which an image to be processed is imaged. The transfer function generation process is performed on the input image data for some areas. When this configuration is adopted, the transfer function generation process is performed on the input image data in a part of the area, so that the processing speed for obtaining the transfer function can be increased.

  In another invention, the original image restored image data is generated using the transfer function generated by the transfer function generation process of the above invention. When this configuration is adopted, the processing speed for generating the original image restored image data can be increased.

  According to the present invention, a transfer function that is hardly affected by noise, blur information error, and the like is obtained, and an image processing apparatus having a realistic circuit processing method can be provided.

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a so-called consumer digital camera that uses a CCD as an imaging unit. However, the image processing apparatus 1 is a surveillance camera, a television camera, and an endoscope camera that use an imaging element such as a CCD as the imaging unit. The present invention can also be applied to devices other than cameras, such as cameras for other uses, image diagnostic apparatuses for microscopes, binoculars, and NMR imaging.

  The image processing apparatus 1 includes an imaging unit 2 that captures a subject such as a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4, an angular velocity sensor, and the like, and changes factor information that causes changes such as image degradation. It has a detection unit 6 for detecting, and a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

  The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD or C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as ASIC (Application Specific Integrated Circuit). The processing unit 4 includes a recording unit (not shown), and the recording unit has an imaginary part high-frequency component in image data for processing to be described later, and an arbitrary image that is a base when generating comparison image data. Image data is saved. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 includes a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD (Digital Versatile Disk), or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the photographing optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the photographing optical system and lens distortion. However, when correcting image degradation due to camera shake, which will be described later, such information is stored. Is not used.

  Next, a processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described.

  First, the basic method of this processing method will be described. That is, in this processing method, the basic method is to first determine the restored image by treating the restoration of the image as an optimization problem, and then obtain the transfer function from the restored image and the image to be restored. It is said.

  Treating image restoration as an optimization problem means that “(1) the output for the input is uniquely determined”, “(2) the input is the same if the output is the same”, “(3 The solution is converged by iteratively processing while updating the input so that the output is the same.

  That is, as shown in FIG. 3A, the image data “Img” is changed by the change factor information data “H” that causes the image to change, and changed to the image data “Img ′” (recovery target). In this case, the image data “Io” is changed to “Io + n” so that the image data “Io ′” obtained by changing the arbitrary image data “Io” by the change factor information data “H” approximates the image data “Img ′”. ”(N is an integer greater than or equal to 1) and updated iteratively. That is, the image data “Io + n” in which the image data “Io + n” is changed by the change factor information data “H” may generate (obtain) the image data “Io + n” that approximates the image data “Img ′”. If possible, as shown in FIG. 3B, the image data “Io (= Io + n)” that is the original data for generating the image data “Io ′ (= Io + n ′)” is changed to the image data “Img” before the change. It can be said that this is an approximate restored image.

  Then, in the processing unit 4, the transfer function “G” from the image data “Img ′” to be restored to the image data “Io (= Io + n)” of the restored image is obtained as G = Io + n / Img ′. This is a basic processing method.

  The basic method will be described in more detail with reference to FIGS.

  In FIG. 4, “Io” is arbitrary initial image data stored in advance in the recording unit of the processing unit 4. “H” is data of change factor information (degradation factor information (point spread function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “Io ′” is comparison image data in which the initial image data “Io” is changed by the change factor information data “H”. “Img ′” is image data of a changed image to be restored. Here, the captured image data captured by the imaging unit 2 is used.

  “Δ” is difference data between the captured image data “Img ′” and the comparison image data “Io ′”. “K” is a distribution ratio based on the change factor information data “H”. “Io + n” (n is an integer equal to or greater than 1) is restored image data newly generated by distributing the difference data δ according to the distribution ratio “k” to the initial image data “Io”. “Img” is image data of the original image that is the basis of the photographed image data “Img ′”. That is, the image before the photographic image data “Img ′” is changed, or the original image that should have been obtained if the photographic image data was correctly photographed. This is data of the subject image when it is assumed that the image is taken.

Here, the relationship between “Img” and “Img ′” is expressed by the following equation (1).
Img ′ = Img × H (1)
Note that the difference data “δ” may be a simple difference between corresponding pixels, but generally differs depending on the change factor information data “H” and is expressed by the following equation (2).
δ = f (Img ′, Img, H) (2)

  The processing routine of the processing unit 4 starts by preparing initial image data “Io” (step S101). As the initial image data “Io”, captured image data “Img ′” may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. In step S102, data “Io” of an arbitrary image that is an initial image is inserted instead of “Img” in the equation (1) to obtain comparison image data “Io ′” that is a deteriorated image. Next, the photographed image data “Img ′” and the comparison image data “Io ′” are compared, and difference data “δ” is calculated (step S103).

  Next, in step S104, it is determined whether or not the difference data “δ” is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, new restored image data (= restored image data) is generated in step S105. Perform the process. That is, the difference data “δ” is distributed to arbitrary initial image data “Io” based on the change factor information data “H” to generate new restored image data “Io + n”. Thereafter, steps S102, S103, S104, and S105 are repeated.

  If the difference data “δ” is smaller than the predetermined value in step S104, the process ends (step S106). Then, the restored image data “Io + n” at the time when the processing is completed is estimated as the original image data “Img”.

  Next, details of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create an image without blur from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are set to n-1, n, n + 1, n + 2, n + 3,... In order from the left, and attention is paid to a certain pixel n. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the correct image data “Img” when there is no deterioration. Each data is represented by 8 bits (0 to 255).

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the n + 1th pixel, and 20% of time is shifted to the n + 2th pixel. . The way of energy dispersion is as shown in the table of FIG. This is the change factor information data “H”.

  Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table of FIG. In FIG. 8, the data shown as “imaging result” is the original image data “Img”, and the data shown as “blurred image” is taken image data “Img ′”. Specifically, for example, “120” of the pixel “n−3” follows the distribution ratio of “0.5”, “0.3”, and “0.2” of the change factor information data “H” that is the blur information. , "60" is distributed to "n-3" pixels, "36" is distributed to "n-2" pixels, and "24" is distributed to "n-1" pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this photographic image data “Img ′” and the change factor information data “H” shown in FIG.

  Any arbitrary initial image data “Io” shown in step S101 can be used, but in this description, the photographed image data “Img ′” is used. That is, the process starts with Io = Img ′. In the table of FIG. 9, “input” corresponds to the initial image data “Io”. The change factor information data “H” is applied to the initial image data “Io”, that is, “Img ′” in step S102. That is, for example, “60” of the “n-3” pixel of the initial image data “Io” is “30” for the “n-3” pixel, “18” for the “n-2” pixel, “12” is assigned to each pixel of “n−1”. The other pixels are similarly distributed and the comparison image data “Io ′” shown as the output “Io ′” is generated. Therefore, the difference data “δ” in step S103 is as shown in the bottom column of FIG.

  Thereafter, the size of the difference data “δ” is determined in step S104. Specifically, the process ends when all the difference data “δ” is 5 or less in absolute value, but the difference data “δ” shown in FIG. Proceed to That is, the difference data “δ” is distributed to arbitrary initial image data “Io” using the change factor information data “H”, and restored image data “Io + n” shown as “next input” in FIG. Is generated. In this case, since this is the first time, it is represented as “Io + 1” in FIG.

  The distribution of the difference data “δ” is, for example, “15” obtained by multiplying the data “30” of the pixel “n−3” by 0.5, which is the distribution ratio of the place (= “n−3” pixel). ”Is distributed to the pixel“ n-3 ”, and“ 0.3 ”, which is the distribution ratio that should have come to the pixel“ n-2 ”in the data“ 15 ”of the pixel“ n-2 ”. “4.5” multiplied by the “n−1” pixel data “9.2”, and the distribution ratio “0” that should have come to the “n−1” pixel is “0”. Allocate “1.84” multiplied by “.2”. The total amount allocated to the pixels of “n-3” is “21.34”, and this value is added to the initial image data “Io” (here, the captured image data “Img ′” is used) to restore the restored image. Data “Io + 1” is generated.

  As shown in FIG. 11, the restored image data “Io + 1” becomes the input image data (= initial image data “Io”) in step S102, and step S102 is executed. Data “δ” is obtained. The size of the new difference data “δ” is determined in step S104. If it is larger than the predetermined value, the new difference data “δ” is distributed to the previous restored image data “Io + 1” in step S105, and a new restored image is obtained. Data “Io + 2” is generated (see FIG. 12). Thereafter, new comparison image data “Io + 2 ′” is generated from the restored image data “Io + 2” by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, and the process proceeds to step S105 or shifts to step S106 depending on the determination there. Such a process is repeated.

  From the restored image data “Io + n” obtained as described above and the captured image data “Img ′” to be restored, a transfer function G from the captured image data “Img ′” to the restored image data “Io + n” is obtained. G = Io + n / Img ′.

By the way, when the photographed image data is deteriorated due to camera shake or the like and the outline portion is blurred, the high frequency component of the image data is lost, and the transfer function becomes unstable. End up.
Therefore, on the premise of the basic method described above, a stable transfer function can be obtained by using the method described below. Note that FIGS. 1 and 2 are the same as the basic method of the processing method described above, and a description thereof will be omitted.

  In FIG. 13, “A” is image data of the original image. This original image is an image before the change or an original image that should have been obtained if the image was correctly captured. In this embodiment, the image is deteriorated due to camera shake during the image capturing operation. This is the image data of the subject image when it is assumed that the image was taken in the absence.

  “A ′” is processing image data, and has real part image data “AR ′” and imaginary part image data “AI ′ · i” described below. “R” is an identifier representing the real part, and “I” is an identifier representing the imaginary part. The image data “AR ′” is captured image data captured by a digital camera, that is, image data of a real part of input image data input from the imaging unit 2 to the processing unit 4. That is, the image data “AR ′” is image data of a real part of an image in which the original image has deteriorated due to camera shake or the like. The image data “AI ′ · i” is image data having an arbitrary high-frequency component, and is stored in advance in the recording unit of the processing unit 4.

  “Bo” is arbitrary initial image data stored in advance in the recording unit of the processing unit 4. The image data of the real part of the initial image data “Bo” is “BRo”, and the image data of the imaginary part is “BIo · i”.

  “H” is data of change factor information (degradation factor information (point spread function)) detected by the detection unit 6. This change factor information data “H” is data indicating the state of camera shake at the time of photographing detected by the detection unit 6, and is stored in the recording unit of the processing unit 4 in association with the image data “AR ′”. .

  “Bo ′” is comparative image data in which the image data “Bo” is changed by the change factor information data “H”, and the real part image data “BRo ′” and the imaginary part image data “BIo ′ · i”. Have The image data “BRo ′” indicates image data in which the real part image data “BRo” of the image data “Bo” is changed by the change factor information data “H”, and is compared for comparison with the image data “AR ′”. Image data. “BIo ′ · i” indicates image data in which the imaginary part image data “BIo · i” of the image data “Bo” is changed by the change factor information data “H”, and the image data “AI ′ · i”. Is comparison image data for comparison.

  “ΔR” is difference data between the image data “AR ′” and the image data “BRo ′”. “ΔI · i” is difference data between the image data “AI ′ · i” and the image data “BIo ′ · i”.

  “K” is a distribution ratio based on the data of the change factor information data “H”.

  “Bo + n” (n is an integer equal to or greater than 1) is restored image data in which difference data “δR” and “δI · i” are allocated to image data “Bo” according to an allocation ratio “k”. “BRo + n” is image data of the real part of the restored image data “Bo + n” when the process of allocating the difference data “δR” to the image data “BRo” according to the distribution ratio “k” is repeated n times. For “BIo + n · i” (n is an integer of 1 or more), the process of allocating the difference data “δI · i” to the image data “BIo · i” according to the distribution ratio “k” is repeated n times. This is the image data of the imaginary part of the restored image data “Bo + n”.

  A processing routine of the processing unit 4 will be described. First, the image data for processing “A ′” is generated with the image data in the real part as image data “AR ′” and the image data in the imaginary part as image data “AI ′ · i” (S201). The image data “AR ′” is image data of the real part of the input image data output from the imaging unit 2 and input to the processing unit 4 by the shooting operation of the digital camera. When there is camera shake in the shooting operation, the image data “AR ′” is image data of an image deteriorated due to camera shake with respect to the image data “A”. The image data “AI ′ · i” is image data having an arbitrary high-frequency component as described above, and is stored in the processing unit 4 in advance to generate the processing image data “A ′”. Call from. The reason for generating such processing image data “A ′” will be described later.

  Next, arbitrary initial image data “Bo” stored in advance in the recording unit of the processing unit 4 is called (S202). The initial image data “Bo” includes real part image data “BRo” and imaginary part image data “BIo · i”, but the data value is arbitrary, and any data can be adopted. it can. For example, using the processing image data “A ′”, the processing may be started with “Bo” = “A ′”. In this case, the real part image data “BRo” becomes the image data “AR ′” in the real part of the captured image. The image data “BIo” of the imaginary part is “AI ′ · i”. Since the image data “Bo” is arbitrary, any image data such as black solid, white solid, gray solid, and checkered pattern may be used.

  In step S203, comparison image data “Bo ′” for the real part and the imaginary part is generated from the image data “Bo” using the change factor information data “H”. That is, the comparison part image data “BRo ′” in the real part is obtained as “BRo × H”, and the comparison part image data “BIo ′ · i” in the imaginary part is obtained as “BIo · i × H”.

  Next, the image data “AR ′” and the comparison image data “BRo ′” are compared to calculate the difference data δR (step S204), and the image data “AI ′ · i” and the comparison image are also calculated. The data “BIo ′ · i” is compared, and difference data “δI · i” is calculated (step S204).

  Subsequently, in step S205, it is determined whether or not the difference data “δR” is equal to or greater than a predetermined value. If the difference data “δR” is equal to or greater than the predetermined value, restored image data “Bo + n” is generated (step S206). That is, the difference data “δR” is distributed to the image data “BRo” based on the distribution ratio “k”, and the restored image data “BRo + n” in the real part is generated (step S206). Further, the difference data “δI · i” is also distributed to the image data “BIo · i” based on the distribution ratio “k” to generate the restored image data “BIo + n · i” in the real part (step S206). .

  Then, step S203 is executed with the restored image data “Bo + n” as arbitrary image data (= “Bo”) in step S203. Thereafter, S203, S204, S205, and S206 are repeated to sequentially generate new restored image data “Bo + n”.

  Incidentally, the distribution ratio “k” is based on the change factor information data “H” and is determined as follows, for example. When attention is paid to a certain pixel, in the state where there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration level at that time is set to “1.0”. However, when the image is blurred, the energy is distributed to the pixels in the blurred direction at a ratio of “0.5”, “0.3”, and “0.2”, for example. Then, with the distribution ratio as the distribution ratio, “δR” and “δI · i” are distributed to the pixels in which energy is distributed according to the distribution ratio.

  As described above, the difference data “δR” between the image data “BRo ′ (BRo + n ′)” obtained by repeating S203, S204, S205, and S206 and the image data “AR ′” is a predetermined value. When the image data becomes smaller, the restored image data “Bo + n” that is the source of the image data “BRo ′” is used as the original image restored image data that approximates the image data “A” of the original image. The predetermined value in step S205 is, for example, the process is ended when the difference data “δR” between the processing image data and the restored image data is 5 or less in absolute value in all pixels. If this condition is not met, the process proceeds to step S206, and the above steps S203, S204, S205, and S206 are repeated.

  In the processing method described above, the processing solution is not solved as an inverse problem, but as an optimization problem for obtaining a reasonable solution.

  Solving as an optimization problem means that, as described in the basic method described above, “(1) The output for the input is uniquely determined”, “(2) If the output is the same, the input is the same. And “(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same”.

  In other words, in FIG. 14, if comparison image data “BRo ′” that approximates the image data “AR ′” of the photographed image can be generated, the restored image data “ BRo + n ”can be said to be approximate to the image data“ AR ”of the real part of the image data“ A ”of the original image, which is the original image data“ AR ′ ”without deterioration.

  Therefore, when the restored image data “BRo + n” approximates the original image data “A”, the transfer function “G” from the processing image data “A ′” to the original image restored image data “Bo + n”. Can be obtained as Bo + n / A ′ (step S207).

  By the way, in the present invention, image data “A ′” is obtained by giving image data “AI ′ · i” having an arbitrary high-frequency component to the imaginary part image data of the input image data. The reason for obtaining the transfer function “G” to the original image approximate restoration image data is as follows.

  The input image data is an image deteriorated due to camera shake, and the outline portion of the image is blurred. Therefore, the high frequency component of the image data is lost and “AR ′” ≈0. Therefore, if only “AR ′” is used to determine the transfer function “G”, since “AR ′” ≈0 in G = B / AR ′, “G” becomes unstable. End up. Therefore, processing image data “A ′” is generated as image data “AI ′ · i” having image data in the imaginary part and arbitrary high-frequency components, and restored image data generation processing is also performed in the imaginary part. This is because the restored image data in the section is obtained, “A ′” ≠ 0, and a stable transfer function “G” can be obtained.

となり、ＡＲ′≒０による不安定性は減少する。 That means

Thus, the instability due to AR′≈0 is reduced.

  The transfer function “G” obtained as described above is recorded in the recording unit 5 together with the image data “AR ′”. Then, when an image whose blurring has been corrected is reproduced at a later date, the original image is directly generated from the image data “AR ′” using the transfer function “G”, that is, “AR ′ × G”. It is possible to obtain restored image data and reproduce an image based on this data.

  In this image processing apparatus 1, in processing, in step S <b> 205, one or both of the number of times of processing and the determination reference value of the difference data “δR” can be set in advance. For example, an arbitrary number such as 20 times or 50 times can be set as the number of times of processing. Further, the value of the difference data “δR” for stopping the processing is set to “5” in 8 bits (0 to 255), and when it becomes “5” or less, the processing is terminated, or “0.5” is set. The processing can be terminated when the value is set to “0.5” or less. This set value can be set arbitrarily. When both the number of processing times and the judgment reference value are input, the processing is stopped when either one is satisfied. When both settings are possible, the determination reference value may be given priority, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

  In the description of this embodiment, the information stored in the factor information storage unit 7 is not used. However, known deterioration factors stored here, such as optical aberrations and lens distortions, are not included. Data may be used. In this case, for example, in the processing method of the previous example (FIG. 13), it is preferable to perform processing by combining blur information and optical aberration information as one deterioration factor. You may make it correct | amend with the information of an optical aberration after it complete | finishes. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors during shooting, for example, only blurring.

  The image processing apparatus 1 according to the embodiment of the present invention has been described above, but various modifications can be made without departing from the gist of the present invention. For example, the processing performed by the processing unit 4 is configured by software, but may be configured by hardware composed of parts that perform part of the processing.

  Further, as the image to be processed, in addition to the photographed image, the photographed image may be subjected to processing such as color correction or Fourier transform. Further, as comparison image data, in addition to the data generated using the change factor information data “H”, color correction is applied to the data generated using the change factor information data “H”, or Fourier transform is performed. It is good also as the data. Further, the data of the change factor information data “H” includes not only the data of the deterioration factor information but also information that simply changes the image and information that improves the image contrary to the deterioration.

  In the above-described embodiment, in step S205, it is determined whether only the difference data “δR” for the real part image data is greater than or equal to a predetermined value, but step 205 in FIG. As indicated by ′, it may be determined whether or not the difference data “δI · i” for the image data of the imaginary part is equal to or greater than a predetermined value. That is, if the difference “δR” and “δI · i” between the image data of the real part and the imaginary part are both equal to or less than a predetermined value (step S207), the original image restoration is performed. Since the image data “Bo + n” approximates the original image “A” not only for the real part but also for the imaginary part, it is more accurate than when only the difference data “δR” of the real part is equal to or less than a predetermined value. A transfer function “G” having a high value can be obtained.

  By the way, in the process for obtaining the transfer function “G” described above, it is possible to speed up the process by extracting an image of a part of the entire imaging area of the imaging unit 2 and performing this part of the image. This is preferable in terms of points.

  For example, as shown in FIG. 15, when the pixels in the imaging region are composed of pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61 to 66, every other pixel. Pixels are thinned out, and reduced image data, which is a quarter of the total number of pixels 11, 13, 15, 31, 33, 35, 51, 53, and 55, is processed as image data “AR ′”. can do.

  Further, for example, as shown in FIG. 16, when the pixels in the imaging region are composed of pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61 to 66, The reduced image data for the pixels in the area consisting of the pixels 32, 33, 34, 42, 43, 44, which is the area of, can be processed as image data “AR ′”.

  That is, in step S201, the image data “AR ′” for a part of the entire imaging region is used as the image data in the real part of the input image data, and the image data “AI ′” The reduced image data for processing “A ′” is generated by using image data having an arbitrary high-frequency component for the image data “AR ′” in the region. Then, the processing of steps S202, S203, S204, S205, and S206 is performed on the image data for reduction processing “A ′ (= AR ′ + AI ′ · i)” to obtain a transfer function “G” (S207).

  As described above, when the transfer function “G” is calculated for a part of the imaging region, that is, the image data “A ′” for reduction processing, the processing time from step S201 to step S206 is targeted for the entire imaging region. Can be shortened. Note that the area of the image to be extracted needs to be sufficiently larger than the fluctuation area. For example, if the image blur is over 3 pixels, it is necessary to extract an area of 3 pixels or more.

  The transfer function “G” obtained for a part of the imaging areas as described above is enlarged and interpolated, and the enlarged and interpolated transfer function “G” is used to restore the original image restored image data “Bo + n” for the entire imaging area. Get. That is, the transfer function “G” can be obtained at high speed, and the original image restored image data “Bo + n” can be restored directly from the image data “AR ′” using this transfer function “G”. Therefore, according to this processing method, it is possible to improve the processing speed when processing a large image.

  By the way, the original image restored image data “Bo + n” obtained for a part of the area is different from the original image restored image data “Bo + n” obtained by performing the process on the entire area. Therefore, the transfer function “G” obtained for a part of the region is different from the transfer function “G” obtained for the entire region. Therefore, the transfer function “G” obtained for a part of the area is enlarged, the enlarged portion is interpolated, and the corrected transfer function is used to input the input image data “AR ′” for the entire imaging area. ”Is subjected to deconvolution calculation in frequency space (calculation to remove blur by calculation from a group of images including blur) to obtain original image restored image data“ Bo + n ”in the entire imaging region, which is not deteriorated. The image data “A” of the original image may be estimated.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. It is a figure for demonstrating the concept of the basic method of the processing method performed with the process part of the image processing apparatus shown in FIG. FIG. 4 is a processing flowchart for explaining a basic method (processing routine) described in FIG. 3. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 with an example of camera shake, and is a diagram showing image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 4 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 5 is a diagram for specifically explaining the processing method illustrated in FIG. 4 using camera shake as an example, and is a diagram for illustrating a situation in which comparison data is generated from an arbitrary image. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking an example of camera shake, in which the comparison data and the blurred original image to be processed are compared to generate the difference data. It is a figure for demonstrating. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking hand shake as an example. In this case, new comparison data is generated from the generated restored data, the data and the blurred original image to be processed are It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 with an example of camera shake, and a diagram for explaining a situation in which newly generated difference data is allocated and new restored data is generated. is there. FIG. 2 is a processing flowchart for explaining a processing method (processing routine) performed by a processing unit of the image processing apparatus shown in FIG. 1. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 14 is a processing flowchart for explaining a modification of the processing method (processing routine) shown in FIG. 13. FIGS. 13A and 13B are diagrams for explaining another processing method using the processing method shown in FIGS. 13 and 15, where FIG. 13A shows original image data to be processed, and FIG. 15B thins out the data in FIG. FIG. FIGS. 13A and 13B are diagrams for explaining another processing method using the processing method shown in FIGS. 13 and 15, in which FIG. 13A shows original image data to be processed, and FIG. It is a figure which shows the data which took out a part of.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image pick-up part 3 Control system part 4 Processing part 5 Recording part 6 Detection part 7 Factor information storage part A Original image data A 'Image data for processing Bo Arbitrary image data Bo' Comparison image data δR Data δI · i Difference data k Distribution ratio Bo + n Restored image data (original image restored image data)
G Transfer function H Change factor information data

Claims (4)

In an image processing apparatus having a processing unit for processing an image,
The processing unit
Processing image data generation processing for generating input image data, which is image data of an image to be processed, as processing image data that is image data having a high frequency component in an imaginary part;
Comparison that generates image data for comparison having image data of real part and imaginary part from arbitrary image data having image data of real part and imaginary part using data of change factor information that causes image change Image data generation processing,
The difference data obtained by comparing the comparison image data and the processing image data is restored by distributing the difference ratio based on the data of the change factor information to the arbitrary image data as a distribution ratio. By generating the image data and using the restored image data instead of the arbitrary image data and repeating the process of generating the restored image data, the real part of the restored image data becomes the original data. Original image restoration image data generation processing for generating original image restoration image data that approximates the image data of the real part of the image;
A transfer function generating process for generating a transfer function from the original image restored image data and the processing image data,
An image processing apparatus characterized by In an image processing apparatus having a processing unit for processing an image,
The processing unit
Processing image data generation processing for generating input image data, which is image data of an image to be processed, as processing image data that is image data having a high frequency component in an imaginary part;
Comparison that generates image data for comparison having image data of real part and imaginary part from arbitrary image data having image data of real part and imaginary part using data of change factor information that causes image change Image data generation processing,
The difference data obtained by comparing the comparison image data and the processing image data is restored by distributing the difference ratio based on the data of the change factor information to the arbitrary image data as a distribution ratio. By generating the image data and using the restored image data instead of the arbitrary image data and repeating the process of generating the restored image data, the image data of the real part and the imaginary part of the restored image data is reproduced. Is an original image restoration image data generation process for generating original image restoration image data that approximates the image data of the real part and imaginary part of the original image
A transfer function generating process for generating a transfer function from the original image restored image data and the processing image data,
An image processing apparatus characterized by   The processing image data is reduced image data generated based on the input image data for a part of the entire imaging area where the image to be processed is captured, and the input for the part of the area is input. The image processing apparatus according to claim 1, wherein the transfer function generation processing is performed on image data.   4. An image processing apparatus for generating original image restored image data using a transfer function generated by the transfer function generating process according to claim 3.

Images