JP2007124198A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of restoring an image even if the contrast of the photographed image is low. <P>SOLUTION: In a plurality of processing image data different from each other, any one of pixel image data of pixels in the same pixel position of each of the processing image data are made to be the same as the pixel image data of pixels of the same pixel position as that of restoration target image data; and of the pixels of each of the processing image data, pixel image data of pixels having no pixel image data of the restoration target image data are made to be contrast generating pixel image data. Then, processing image data in which the pixels of pixel image data having the restoration target image data and the contrast generating pixel image data are alternately arranged are generated. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus.

Conventionally, a photographed image taken by a camera that uses an image sensor such as a CCD (Charge C ₀ up Devices) in the imaging unit constitutes camera shake during shooting, various aberrations of the photographing optical system, or a photographing optical system. It is known that when there is distortion or the like of the lens to be taken, this is a factor and the captured image is deteriorated.

  As means for preventing such deterioration of the photographed image, a method of moving the lens and a method of circuit processing the photographed image are known as camera shake during photographing among the causes of deterioration of the photographed image. For example, as a method of moving the lens, camera shake is detected, and a predetermined lens in the photographing optical system is moved in accordance with the movement of the camera due to the detected camera shake, thereby moving the imaging position on the image sensor. There is known a method for suppressing the above (see Patent Document 1).

  As a circuit processing method, a change in the optical axis of the photographing optical system of the camera is detected by an angular acceleration sensor or the like, and a transfer function representing a blurring state at the time of photographing is obtained from the detected angular velocity, etc. A method is known in which an acquired transfer function is inversely transformed to restore an image without deterioration (see Patent Document 2).

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens such as a motor, and thus becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  In addition, in the case of camera shake correction described in Patent Document 2, the above-described problem is eliminated, but when the contrast of the captured image is low, the transfer function becomes unstable, and as a result, the image cannot be restored. There is.

  Therefore, an object of the present invention is to provide an image processing apparatus that can restore an image even when the contrast of a captured image is low.

  In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus having a processing unit for processing an image, wherein the processing unit is a plurality of processing image data different from each other, and each of these processing images The pixel image data of any one of the pixels at the same pixel position of the data is the same as the pixel image data of the pixel at the same pixel position of the restoration target image data, and the restoration target image is out of the pixels of each processing image data The pixel image data of the pixels that do not have the pixel image data of the data is set as the pixel image data for contrast generation, and the pixels having the pixel image data of the restoration target image data and the pixels having the pixel image data for contrast generation are alternately arranged. Generate processing image data and generate processing original image data for each processing image data It performs management, from processing the original image data, and performing the original image data restoration processing for generating a restoration original image data that approximates the original image data.

  According to this invention, since the processing image data is image data with high contrast, the image can be restored even if the contrast of the restoration target image is low.

  In another invention, in addition to the above-described invention, the processing original image data generation process generates comparison data from data of an arbitrary image by using data of change factor information that causes an image change. Then, the processing image data and the comparison data are compared, and the obtained difference data is distributed to arbitrary image data using the data of the change factor information, thereby generating the restored image data for processing. Thereafter, the processing restored image data is used in place of arbitrary image data, and the same processing is repeated to generate processing original image data.

  According to the present invention, since the restoration original image data for processing is generated by generating predetermined data using the image change factor information, a transfer function that is weak against noise, blur information error, and the like is used. The processing original image data can be obtained.

  In addition to the above-described invention, in another invention, the plurality of processing image data is two processing image data, and each of the two processing image data includes pixels having pixel image data of restoration target image data. The pixels having the contrast generating pixel image data are alternately arranged for each pixel.

  With this configuration, contrast generation pixel image data is arranged between the pixel data, and the processing image data can be image data with higher contrast.

  In order to solve the above-described problems, an image processing apparatus according to the present invention includes an image processing apparatus having a processing unit that processes an image. Image data for partial area processing, and any one pixel image data of the pixels at the same pixel position of each partial area processing image data is stored in a partial area of the restoration target image data. It is the same as the pixel image data of the pixel at the same pixel position, and among the pixels of each partial region processing image data, the pixel image data of the pixel that does not have the pixel image data of the restoration target image data is used as the contrast generation pixel image data Image data for partial area processing in which pixels having pixel image data of restoration target image data and pixels having pixel image data for contrast generation are alternately arranged. And performing partial area processing original image data generation processing for generating partial area processing original image data for each of the partial area processing image data. From the original image data, a partial area transfer function between the partial area processing image data and the partial area processing original image data is obtained, and using this partial area transfer function, a restoration source of the restoration target image data is obtained. An original image data restoration process for generating image data is performed.

  According to the present invention, since the processing original image data for a part of the area is obtained, the processing speed can be increased.

  In addition to the above-mentioned invention, the other image data generation process for partial region processing uses comparison factor information data that causes an image change, and compares the data for an arbitrary image with the comparison data. After that, the partial area processing image data and the comparison data are compared, and the obtained difference data is distributed to arbitrary image data using the change factor information data. Generate restored image data for region processing, then use the restored image data for partial region processing instead of arbitrary image data, and repeat the same process to generate original image data for partial region processing It is assumed that the process is performed.

  According to the present invention, the restoration original image data for partial area processing is generated by generating predetermined data using the factor information of the image change, so that the transfer function weak against noise, blur information error, etc. The partial region processing original image data can be obtained without using the.

  Further, in addition to the above-described invention, in another invention, the plurality of partial area processing image data are two partial area processing image data, and each of the two partial area processing image data is to be restored. The pixels having the pixel image data of the image data and the pixels having the pixel image data for contrast generation are alternately arranged for each pixel.

  With this configuration, contrast generation pixel image data is arranged between the pixel data, and the processing image data can be image data with higher contrast.

  In addition to the above-described invention, in another invention, the contrast generation pixel image data is image data having a luminance of 0 or high luminance.

  With this configuration, the contrast generation pixel image data is image data with zero or high luminance, so that a plurality of image data can be image data with high contrast.

  According to the present invention, it is possible to restore an image even if the restoration target image has a low contrast.

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a so-called consumer digital camera that uses a CCD as an imaging unit. However, the image processing apparatus 1 is a surveillance camera, a television camera, and an endoscope camera that use an imaging element such as a CCD as the imaging unit. The present invention can also be applied to devices other than cameras, such as cameras for other uses, image diagnostic apparatuses for microscopes, binoculars, and NMR imaging.

  The image processing apparatus 1 includes an imaging unit 2 that captures a subject such as a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4, an angular velocity sensor, and the like, and changes factor information that causes changes such as image degradation. It has a detection unit 6 for detecting, and a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

  The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD or C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 includes a recording unit (not shown), and the recording unit has an imaginary part high-frequency component in image data for processing to be described later, and an arbitrary image that is a base when generating comparison image data. Image data is saved. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 is composed of a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD (Digital Versatile Disk), or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the photographing optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the photographing optical system and lens distortion. However, when correcting image degradation due to camera shake, which will be described later, such information is stored. Is not used.

  Next, a processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described.

  The processing unit 4 has a processing method for obtaining a restored image by treating image restoration as an optimization problem. First, this processing method will be described.

  Treating image restoration as an optimization problem means that “(1) the output for the input is uniquely determined”, “(2) If the output is the same, the input is the same”, “(3 The solution is converged by iteratively processing while updating the input so that the output becomes the same.

That is, as shown in FIG. 3A, the original image data “img” is changed by the change factor information data “g” that causes the image to change, and the captured image data “img ′” as the restoration target image data is changed. "Given the changes (image data of the restoration of the target), the image data is changed by any of the image data" i ₀ "the change factor information data" g "," i _{0 '} "is, captured image data"img' The image data “i ₀ ” is iteratively updated as “i _{0 + n} ” (n is an integer equal to or greater than 1). That is, the image data which the image data "i _{0 + n"} is changed by the change factor information data "g", "i _{0 + n '"} is, captured image data "img' generates the image data" i _{0 + n} "so as to approximate the" ( if we ask) that, as shown in FIG. 3 (B), the image data _{"i 0 '(= i 0 +} n') " image data _{"i 0 (= i 0 + n} ) " as the original data for generating of change It can be said that this is a restored image approximated to the previous original image data “img”.

Further, the transfer function “G” from the image data “img ′” to be restored to the image data “i ₀ (= i _{0 + n} )” of the restored image can be obtained as G = i _{0 + n} / img ′.

  The original image data “img” is image data of the original image that is the basis of the photographed image data “img ′”. That is, the image before the photographed image data “img ′” is changed, or the original image that should have been obtained if the photographed image was correctly photographed. This is data of a captured image when it is assumed that the image is captured.

  The basic method will be described in more detail with reference to FIGS.

In FIG. 4, “i ₀ ” is arbitrary initial image data stored in advance in the recording unit of the processing unit 4. “G” is data of change factor information (degradation factor information (point spread function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “I ₀ ′” is comparison image data in which the initial image data “i ₀ ” is changed by the change factor information data “g”. “Img ′” is image data of a changed image to be restored. Here, the captured image data captured by the imaging unit 2 is used.

“Δ” is difference data between the captured image data “img ′” and the comparison image data “i ₀ ′”. “H” is a feedback function representing a distribution ratio based on the change factor information data “g”. “I _{0 + n} ” (n is an integer greater than or equal to 1) is restored image data newly generated by distributing the difference data δ to the initial image data “i ₀ ” according to the distribution ratio “h”.

Here, it is assumed that the relationship between “img” and “img ′” is expressed by the following equation (1).
img ′ = img * g (1)
“*” Is an operator representing a superposition integral.
Note that the difference data “δ” may be a simple difference between corresponding pixels, but generally differs depending on the change factor information data “g” and is expressed by the following equation (2).
δ = f (img ′, img, g) (2)

Processing routine of the processing unit 4 first begins providing initial image data "i _0" (step S101). As the initial image data “i ₀ ”, captured image data “img ′” may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. In step S102, data “i ₀ ” of an arbitrary image that is an initial image is input instead of “img” in the equation (1), and comparison image data “i ₀ ′” that is a degraded image is obtained. Next, the photographed image data “img ′” and the comparison image data “i ₀ ′” are compared, and difference data “δ” is calculated (step S103).

Next, in step S104, it is determined whether or not the difference data “δ” is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, new restored image data (= restored image data) is generated in step S105. Perform the process. That is, the difference data “δ” is distributed to arbitrary initial image data “i ₀ ” based on the change factor information data “g” to generate new restored image data “i _{0 + n} ”. Thereafter, steps S102, S103, S104, and S105 are repeated.

If the difference data “δ” is smaller than the predetermined value in step S104, the process ends (step S106). Then, the restored image data “i _{0 + n} ” at the time when the processing is completed is estimated as the original image data “img”.

  Next, details of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are set to n-1, n, n + 1, n + 2, n + 3,... In order from the left, and attention is paid to a certain pixel n. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the correct image data “img” when there is no deterioration. Each data is represented by 8 bits (0 to 255).

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the n + 1th pixel, and 20% of time is shifted to the n + 2th pixel. . The way of energy dispersion is as shown in the table of FIG. This is the change factor information data “g”.

  Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table of FIG. In FIG. 8, the data indicated as “imaging result” is the original image data “img”, and the data indicated as “blurred image” is the captured image data “img ′”. Specifically, for example, “120” of the pixel “n−3” follows the distribution ratio of “0.5”, “0.3”, and “0.2” of the change factor information data “g” that is the blur information. , "60" is distributed to "n-3" pixels, "36" is distributed to "n-2" pixels, and "24" is distributed to "n-1" pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this photographic image data “img ′” and the change factor information data “g” shown in FIG.

Any arbitrary initial image data “i ₀ ” shown in step S 101 can be adopted, but in this description, the photographed image data “img ′” is used. That is, the process starts with i ₀ = img ′. Those with "input" in the table of FIG. 9 corresponds to the initial image data "i _0". In step S102, the change factor information data “g” is applied to the initial image data “i ₀ ”, that is, “img ′”. That is, for example, “60” of the “n-3” pixel of the initial image data “i ₀ ” is “30” for the “n-3” pixel and “18” for the “n-2” pixel. , “12” is assigned to each pixel of “n−1”. Allocated similarly for the other pixels, the output "i _{0 '"} image data "i ₀ for comparison shown _as'" is generated. Therefore, the difference data “δ” in step S103 is as shown in the bottom column of FIG.

Thereafter, the size of the difference data “δ” is determined in step S104. Specifically, the process ends when all the difference data “δ” is 5 or less in absolute value, but the difference data “δ” shown in FIG. Proceed to That is, the difference data “δ” is distributed to arbitrary initial image data “i ₀ ” using the change factor information data “g”, and the restored image data “ i _{0 + n} ”. In this case, since it is the first time, it is represented as “i _{0 + 1} ” in FIG.

The distribution of the difference data “δ” is, for example, “15” obtained by multiplying the data “30” of the pixel “n−3” by 0.5, which is the distribution ratio of the place (= “n−3” pixel). ”Is distributed to the pixel“ n-3 ”, and“ 0.3 ”, which is the distribution ratio that should have come to the pixel“ n-2 ”in the data“ 15 ”of the pixel“ n-2 ”. “4.5” multiplied by the “n−1” pixel data “9.2”, and the distribution ratio “0” that should have come to the “n−1” pixel is “0”. Allocate “1.84” multiplied by “.2”. The total amount allocated to the “n−3” pixels is “21.34”, and this value is added to the initial image data “i ₀ ” (here, the captured image data “img ′” is used) to be restored. Image data “i _{0 + 1} ” is generated.

As shown in FIG. 11, this restored image data “i _{0 + 1} ” becomes the input image data (= initial image data “i ₀ ”) in step S102, step S102 is executed, the process proceeds to step S103, and a new The difference data “δ” is obtained. The size of the new difference data “δ” is determined in step S104. If it is larger than the predetermined value, the new difference data “δ” is distributed to the previous restored image data “i _{0 + 1} ” in step S105, and a new restoration is performed. Image data “i ₀ +2” is generated (see FIG. 12). Thereafter, new comparison image data “i _{0 + 2} ′” is generated from the restored image data “i _{0 + 2} ” by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, and the process proceeds to step S105 or shifts to step S106 depending on the determination there. Such a process is repeated.

'Since the photographed image data "img above manner obtained restored image data" i _{0 + n} "and the photographed image data is restored in the target" img "' transfer function G from" to restore the image data "i _{0 + n"} Can be determined as G = i _{0 + n} / img ′.

  By the way, when the photographed image data is deteriorated due to camera shake or the like and the outline portion is blurred, the high-frequency component of the image data is lost. Because of this influence, it is not possible to obtain so good restored original image data even if the above restoration processing is performed. Further, it is considered that the transfer function becomes unstable due to the loss of the high frequency component. Therefore, by taking the processing method described below on the premise of the processing method described above, it is possible to obtain better restored original image data and to obtain a stable transfer function.

  As shown in FIG. 13, two pieces of processing image data “img ′ (1)” and “img ′ (2)” are generated from the captured image data “img ′”. The processing image data “img ′ (1)” and “img ′ (2)” are generated based on the following concept.

  The photographed image data “img ′” has pixel image data from “aa” to “dh”, respectively, in the pixels at the pixel positions 1 to 32, for example, as shown in FIG. Stuff.

  For this photographed image data “img ′”, the processing image data “img ′ (1)” is assigned to pixel numbers “1, 3, 5,..., 32” as shown in FIG. , Have pixel image data of “aa”, “ac”, “ae”,..., “Dh”, respectively, and pixel numbers “2, 4, 6,..., 31” corresponding to pixels between these pixels. The pixel image data for contrast generation is generated so as to have pixel image data “z” having a luminance of 0. The pixel image data with 0 brightness is, for example, image data having no brightness such as black solid when the brightness of the pixel image data is expressed in 255 steps in order from the darker with 8 bits.

  Further, the processing image data “img ′ (2)” is assigned to pixel numbers “2, 4, 6,..., 31”, respectively, as shown in (B2) of FIG. , “Af”,..., “Dg”, and pixel number “1, 3, 5,..., 32” corresponding to pixels between these pixels has pixel image data “z”. To generate.

  That is, any one pixel image data of the pixels at the same pixel position in the processing image data “img ′ (1)” and “img ′ (2)” is the same pixel position in the captured image data “img ′”. This is the same as the pixel image data of the other pixel. In addition, pixel image data of pixels that do not have the pixel image data of the captured image data “img ′” of the respective processing image data “img ′ (1)” and “img ′ (2)” is pixel image data of 0 brightness. “Z”. The processing image data “img ′ (1)” and “img ′ (2)” generated in this way are generated as high-contrast image data.

  Here, the pixel image data “z” is the data having the luminance 0 (no luminance) as the pixel image data for contrast generation. However, the pixel image data “z” is not necessarily 0, and for example, the luminance 255 may be the high luminance data. Good. It is appropriate that the contrast generation pixel image data is data that produces as high a contrast as possible for as many pixels as possible with respect to the pixels adjacent to the pixels into which the contrast generation pixel image data is to be input.

The processing described with reference to FIG. 4 is performed on the processing image data “img ′ (1)” instead of the captured image data “img ′”. That is, instead of the captured image data “img ′” in step S103 of FIG. 4, the processing image data “img ′ (1)” is processed as a processing target, and processing image data “img ′ (1)” is obtained. The comparison image data “i ₀ ′” is compared, and difference data “δ” is calculated.

Next, in step S104, it is determined whether or not the difference data “δ” is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, processing restored image data is generated as new restored image data in step S105. Process. That is, based on the difference between the data "δ" in the change factor information data "g", allocated to an arbitrary initial image data "i _0", to generate a processing restored data "i _{0 + n".} Thereafter, steps S102, S103, S104, and S105 are repeated.

If the difference data “δ” is smaller than the predetermined value in step S104, the process ends (step S106). Then, the processing restored image data “i _{0 + n} ” at the time when the processing is completed is set as the processing original image data “i _{0 + n} (1)”.

That is, the processing image data “img ′ (1)” is subjected to the processing described in FIG. 4 in place of the captured image data “img ′”, whereby the processing original image data “i _{0 + n} (1)”. Is generated.

The concept of the processing original image data “i _{0 + n} (1)” is shown in FIG. 13 (C1). That is, each pixel image data “aa”, “z”, “ac”, “z”, “ae”, “z”,..., “Z”, “dh” of the processing image data “img ′ (1)”. , “Aa ′”, “z ′”, “ac ′”, “z ′”, “ae ′”, “z ′”,..., “Z ′”, “dh ′” by the processing of FIG. To be restored. Each “z ′” is not necessarily equal to each other. This processing original image data “i _{0 + n} (1)” becomes processing image data “img ′ (1)” when the change factor information data “g” acts.

Similarly to the processing image data “img ′ (1)”, the processing described in FIG. 4 is performed on the processing image data “img ′ (2)”, and the processing original image data “i _{0 + n} ( 2) ". The concept of the processing original image data “i _{0 + n} (2)” is shown in (C2) of FIG. That is, each pixel image data “z”, “ab”, “z”, “ad”, “z”, “af”, “z”,..., “Z” of the processing image data “img ′ (2)”. , “Dg”, “z” are converted into “z ′”, “ab ′”, “z ′”, “ad ′”, “z ′”, “af ′”, “z ′” by the processing of FIG. ,..., “Z ′”, “dg ′”, “z ′”. Each “z ′” is not necessarily equal to each other as described above. The processing original image data “i _{0 + n} (2)” for the processing image data “img ′ (2)” is also changed to the processing image data “img ′ (2)” when the change factor information data “g” acts. It will be.

Then, each pixel image data related to the pixel image of the captured image data of the processing original image data “i _{0 + n} (1)”, “i _{0 + n} (2)”, that is, “aa ′”, “ac ′”, “ “ae ′”,..., “dh ′” and “ab ′”, “ad ′”, “af ′”,..., “dg ′” are pixels at corresponding pixel positions of the original image data “img”. It can be said that it approximates the image data. Accordingly, each pixel image data other than “z ′” in the processing original image data “i _{0 + n} (1)” and “i _{0 + n} (2)” (each pixel image data related to the pixel image of the photographed image data “img ′”). ) To generate restored original image data “i _{0 + n} ” shown in FIG. 13D. In other words, the processing original image data “i _{0 + n} (1)” and the processing original image data “i _{0 + n} (2)” are added so that each pixel overlaps, and restoration is performed by reducing “z ′” for each pixel. Original image data “i _{0 + n} ” can be generated.

As described above, if the processing image data “img ′ (1)” and “img ′ (2)” are high-contrast image data, the captured image data “img ′” described as the basic processing method in FIG. The original image data “i _{0 + n} ” that is closer to the original image data “img” can be generated than is directly processed. That is, a small restored original image can be obtained from the original image.

Further, the transfer function “G1” from the processing image data “img ′ (1)” to the processing original image data “i _{0 + n} (1)” is obtained as G1 = i _{0 + n} (1) / img ′ (1). In this case, since the processing image data “img ′ (1)” is image data with high contrast and sufficiently includes a high-frequency component, a stable transfer function G1 can be obtained. Similarly, a stable transfer function G2 can be obtained for the transfer function “G2” from the processing image data “img ′ (2)” to the processing original image data “i _{0 + n} (2)”.

The obtained transfer functions “G1” and “G2” are recorded in the recording unit 5 together with the photographed image data “img ′”. Then, when the captured image data “img ′” is reproduced at a later date, processing image data “img ′ (1)” and “img ′ (2)” are generated from the captured image data “img ′”. By applying transfer functions G1 and G2 to the processing image data “img ′ (1)” and “img ′ (2)”, the processing original image data “i _{0 + n} (1)”, “i _{0 + n} (2) ) ”And the restored original image data“ i _{0 + n} ”of the captured image data“ img ′ ”can be obtained. That is, it is possible to obtain the restored original image data “i _{0 + n} ” of the captured image data “img ′” in a short time without going through the processing method of FIG.

By the way, in the processing method shown in the processing routine of FIG. 4, instead of determining the size of the difference data “δ” in step S104, the number of processing routines in steps S102 to S105 is set. The restored image data “i _{0 + n} ” when the processing routine is executed can be approximated to the original image data “img”. In other words, by setting the number of processing times to be greater than or _equal to the number of times that the restored image data “i _{0 + n} ” can be estimated to approximate the original image data “img”, the restoration is performed without determining the size of the difference data “δ”. Image data “i _{0 + n} ” can be obtained.

Then, the concept of processing in which the restored image data “i _{0 + n} ” obtained when the processing routine of the set number of times is executed is approximated to the original image data “img” is expressed by the following formula. FIG. 4 explains the concept of the processing method of the processing unit 4 simply and easily. Therefore, in FIG. 4, the description is given focusing on one pixel, but in order to obtain restored image data in all the imaging regions of the imaging unit 2, the processing routine of FIG. 4 is performed for all the pixels in the imaging region. Need to run.

First, when the initial image data in S101 is “i ₀ ”, in S102, the comparison image data “i ₀ ′” is (1) as a superposition integral of the initial image data “i ₀ ” and the change factor function “g”. It is expressed as an expression.

The difference data “δ” in S103 is expressed as in equation (2).

The restored image data “i _{0 + 1} ” in S105 is expressed as the following equation (3) assuming that the difference data “δ” is distributed to the initial image data “i ₀ ” based on the feedback function “h”.

In the next processing routine, the restored image data “i _{0 + 1} ” in equation (3) enters instead of “i ₀ ” in S102, and the comparison image data “i _{0 + 1} ′” is expressed in equation (4). Generated in the form.

Then, in S103, the difference data “δ” is expressed by equation (5).

Then, the restored image data “i _{0 + 2} ” in S105 is expressed as in equation (6).

Similarly, "i ₀ +3", ... when the going calculated, the restored image data "i _{0 + n"} is expressed as equation (7).

  That is, when the above equation (7) is used, the same restored image data as when the predetermined number of times of the routine is performed can be obtained by calculation without executing the processing routine of FIG. For example, in equation (7), the calculated value when k = 20 is equal to the restored image data when the processing routine of FIG. 4 is repeated 20 times.

As the number of processing routines in FIG. 4 is increased, the restored image data “i _{0 + n} ” approaches the original image data “img”, but the processing time becomes longer. On the other hand, in the above equation (7), increasing “k” corresponds to increasing the number of processing routines in FIG. However, even if “k” is increased, the calculation time is shorter than when the processing routine corresponding to “k” is executed.

Incidentally, in the processing routine of FIG. 4, the initial image data "i _0" black solid image, that is, to start with no image data is input, in the equation (7), the "i _0", "0" Is equivalent to

That is, in Expression (7), when “i ₀ ” = “0”, Expression (7) is expressed as Expression (9).

  That is, even if the above equation (9) is used, the same restored image data as that obtained when the predetermined number of times of the routine is performed can be obtained by calculation without executing the processing routine of FIG.

  As for the feedback function “h”, for example, when the camera movement trajectory due to camera shake is represented by the line L in FIG. 13 (the change factor function is “g (x, y)”), the feedback function “ If “h” is a function such that h (−x, −y) = g (x, y), the difference data “δ” can be efficiently reduced.

ただし、「Ｉ_０＋ｎ」は、復元画像データ「ｉ_０＋ｎ」をフーリエ変換した周波数空間における復元画像データを表わす。「Ｉ_０」は、初期画像データ「ｉ_０」をフーリエ変換した周波数空間における初期画像データを表わす。「ＩＭＧ′」は、撮影画像データ「ｉｍｇ′」をフーリエ変換した周波数空間における撮影画像データを表わす。「Ｈ」は、フィードバック関数「ｈ」をフーリエ変換した周波数空間におけるフィードバック関数を表わす。「Ｇ」は、変化要因「ｇ」をフーリエ変換した周波数空間における変化要因を表わす。 By the way, when Expression (7) and Expression (9) are Fourier transformed, Expression (7) is expressed as Expression (10), and Expression (9) is expressed as Expression (11).

However, “I _{0 + n} ” represents restored image data in a frequency space obtained by Fourier transforming the restored image data “i _{0 + n} ”. "I _0" represents the initial image data an initial image data "i _0" in the frequency domain of Fourier transform. “IMG ′” represents photographed image data in a frequency space obtained by Fourier transform of the photographed image data “img ′”. “H” represents a feedback function in a frequency space obtained by Fourier transforming the feedback function “h”. “G” represents a change factor in the frequency space obtained by Fourier transform of the change factor “g”.

Thus, by calculating in the frequency space using the equations (10) and (11) obtained by performing Fourier transform on the equations (7) and (9), the equations (7) and (9) Without performing such superposition integration, the restored image data “I _{0 + n} ” can be calculated by integration (multiplication) and addition / subtraction (addition and subtraction). For this reason, it is possible to increase the calculation speed as compared with the case of performing the superposition integration described above.

For restored image data “I _{0 + n} ” calculated by equation (10) or equation (11), image data in the spatial domain is obtained by inverse Fourier transform.

The processing image data “img ′ (1)”, “img ′ (2)” is used for processing by using the formulas (7), (9), (10), or (11) described above. The original image data “i _{0 + n} (1)”, “i _{0 + n} (2)” may be generated.

  Note that the number of processing image data is not limited to two, and may be three or more. For example, as shown in FIG. 14, from the captured image data “img ′” of FIG. 14A, three processing image data “img ′” shown in (B1), (B2), and (B3) of FIG. (1) "," img '(2) "," img' (3) "may be generated.

  Further, as shown in FIG. 15, the processing image data “img ′ (1)”, “B” shown in (B1) and (B2) of FIG. As in “img ′ (2)”, the pixel image data of the photographed image data “img ′” may be continuously input to two or more pixels. For example, pixel images “aa” and “ab” are continuously stored in the pixels 1 and 2 in FIG. However, if the pixel image data of the photographed image data “img ′” and the pixel image data for contrast generation “z” are alternately arranged, the image data for processing with high contrast can be obtained.

  When the direction in which the contrast decreases is one direction, the pixel image data of the captured image data and the pixel image data for contrast generation are arranged alternately in only one direction, and the contrast is decreased. For the other direction that is not present, the arrangement of the pixel image data of the captured image data may be left as it is.

  By the way, the processing for obtaining the transfer functions “G1” and “G2” described above is performed on a part of the entire imaging area of the imaging unit 2, that is, a part of the captured image data “img ′”. As a target, it is preferable to perform this partial area in terms of speeding up the processing.

For example, pixel numbers 1, 2, 3, 4, 5, 9, 10, 11, 12, and 13 corresponding to a quarter of the number of pixels from 1 to 32 shown in FIG. With respect to a partial area composed of pixels, partial area processing image data “img ′ (1s)” and “img ′ (2s)” shown in (B1) and (B2) of FIG. 16 are generated. Then, the image data for partial region processing “img ′ (1s)”, “img ′ (2s)” by the processing method of FIG. 4 or the equations (7), (9), (10), and (11). ”Are generated as partly processed original image data“ i _{0 + n} (1s) ”and“ i _{0 + n} (2s) ”shown in (C1) and (C2) of FIG. Then, the partial area processing image data “img ′ (1s)”, “img ′ (2s)” and the partial area processing original image data “i _{0 + n} (1s)”, “i _{0 + n} (2s)”. Are determined as partial area transfer functions “G1s” and “G2s”.

Then, the partial area transfer functions “G1s” and “G2s” obtained for a part of the imaging areas are enlarged and interpolated to process image data “img ′ (1)” and “img ′ (2)”. It is obtained as transfer functions “G1” and “G2” between the original image data “i _{0 + n} (1)” and “i _{0 + n} (2)”. In this way, since the process shown in FIG. 4 only needs to be performed for a part of the area, the processing speed when processing a large image can be improved.

The obtained transfer functions “G1” and “G2” are recorded in the recording unit 5 together with the photographed image data “img ′”. Then, when the captured image data “img ′” is reproduced at a later date, processing image data “img ′ (1)” and “img ′ (2)” are generated from the captured image data “img ′”. By applying transfer functions G1 and G2 to the processing image data “img ′ (1)” and “img ′ (2)”, the processing original image data “i _{0 + n} (1)”, “i _{0 + n} (2) ) ”And the restored original image data“ i _{0 + n} ”of the captured image data“ img ′ ”is obtained.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. It is a figure for demonstrating the concept about the processing method calculated | required by treating a restored image as an optimization problem. FIG. 4 is a processing flowchart for explaining a specific technique (processing routine) of the concept described in FIG. 3. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 with an example of camera shake, and is a diagram showing image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 4 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 5 is a diagram for specifically explaining the processing method illustrated in FIG. 4 using camera shake as an example, and is a diagram for illustrating a situation in which comparison data is generated from an arbitrary image. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking an example of camera shake, in which the comparison data and the blurred original image to be processed are compared to generate the difference data. It is a figure for demonstrating. FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 4 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 by taking hand shake as an example. In this case, new comparison data is generated from the generated restored data, the data and the blurred original image to be processed are It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 4 with an example of camera shake, and a diagram for explaining a situation in which newly generated difference data is allocated and new restored data is generated. is there. It is a figure for demonstrating the production | generation of the image data for a process in the image processing apparatus of FIG. It is a figure for demonstrating the other example about the production | generation of the image data for a process in the image processing apparatus of FIG. It is a figure for demonstrating the other example about the production | generation of the image data for a process in the image processing apparatus of FIG. It is a figure for demonstrating the other example about the production | generation of the image data for a process in the image processing apparatus of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image pick-up part 3 Control system part 4 Processing part 5 Recording part 6 Detection part 7 Factor information storage part img Original image data img 'Photographed image data img' (1), img '(2), img' ( 3) processing image data i _{0 + n} restored original image data _{i 0 + n (1),} i 0 + n (2) processing the original image data img '(1s), img' (2s) for a partial region processed image data i ₀ + n (1s ), I _{0 + n} (2s) Partial image processing original image data z Contrast generation pixel image data δR Difference data δi · i Difference data h Feedback function (distribution ratio)
G1, G2 Transfer function g Change factor information data

Claims (7)

In an image processing apparatus having a processing unit for processing an image,
The processing unit
A plurality of processing image data different from each other, and any one pixel image data of the pixels at the same pixel position of each processing image data is converted into pixel image data of a pixel at the same pixel position of the restoration target image data. The pixel image data of the pixels that do not have the pixel image data of the restoration target image data among the pixels of the processing image data is the pixel image data for the contrast generation, and the pixel image of the restoration target image data Generating processing image data in which pixels having data and pixels having contrast generation pixel image data are alternately arranged;
Perform processing original image data generation processing for generating processing original image data for each of the processing image data,
An image processing apparatus for performing original image data restoration processing for generating restored original image data that approximates the original image data from the processing original image data. The processing original image data generation process includes:
Using the data of the change factor information that causes the image change, the comparison data is generated from the data of any image, and then the processing image data and the comparison data are compared, and the obtained difference Is distributed to the data of the arbitrary image using the data of the change factor information, and then the restored image data for processing is generated, and then the restored image data for processing is used instead of the arbitrary image data. The image processing apparatus according to claim 1, wherein processing for generating original image data for processing is performed by repeating similar processing. The plurality of processing image data are two processing image data,
In each of the two processing image data, the pixel having the pixel image data of the restoration target image data and the pixel having the contrast generating pixel image data are alternately arranged for each pixel. The image processing apparatus according to claim 1. In an image processing apparatus having a processing unit for processing an image,
The processing unit
A plurality of different partial area processing image data generated for a partial area of the restoration target image data, and any one of the pixels at the same pixel position in each of the partial area processing image data One pixel image data is the same as the pixel image data of a pixel at the same pixel position in a partial area of the restoration target image data, and the restoration target image data among the pixels of the partial area processing image data The pixel image data of the pixels having no pixel image data is used as contrast generation pixel image data, and pixels having the pixel image data of the restoration target image data and pixels having the contrast generation pixel image data are alternately arranged. Generate image data for partial area processing,
Perform partial area processing original image data generation processing for generating partial area processing original image data for each of the partial area processing image data,
A partial area transfer function between the partial area processing image data and the partial area processing original image data is obtained from the partial area processing image data and the partial area processing original image data,
An image processing apparatus for performing original image data restoration processing for generating restored original image data of the restoration target image data using the partial area transfer function. The partial area processing original image data generation process includes:
Using the data of the change factor information that causes the image change, the comparison data is generated from the data of an arbitrary image, and then the partial region processing image data is compared with the comparison data. The partial difference processing restored image data is generated by distributing the obtained difference data to the arbitrary image data using the change factor information data, and then the partial area processing restored image data. 5. The image processing apparatus according to claim 4, wherein a process for generating original image data for partial region processing is performed by repeating the same process using the image data instead of the arbitrary image data. The plurality of partial area processing image data are two partial area processing image data,
In each of the two partial area processing image data, the pixel having the pixel image data of the restoration target image data and the pixel having the contrast generating pixel image data are alternately arranged for each pixel. 6. The image processing apparatus according to claim 4, wherein the image processing apparatus is characterized in that:   The image processing apparatus according to claim 1, wherein the contrast generation pixel image data is image data having a luminance of 0 or high luminance.

Images