WO2007083621A1

WO2007083621A1 - Image processing device

Info

Publication number: WO2007083621A1
Application number: PCT/JP2007/050486
Authority: WO
Inventors: Fuminori Takahashi
Original assignee: Nittoh Kogaku K.K
Priority date: 2006-01-18
Filing date: 2007-01-16
Publication date: 2007-07-26
Also published as: JP4885150B2; CN101375591A; JPWO2007083621A1

Abstract

Provided is a practical circuit processing method which prevents increase of the device size for restoring an image. An image processing device compares image data I0’on a comparison image generated from image data I0 on an arbitrary image to image data Img on an original image to be processed. Difference data D obtained by the comparison is appropriately distributed image data on an arbitrary image so as to generate image data I0+n on the restored image.

Description

Specification

Image processing device

Technical field

[0001] The present invention relates to an image processing apparatus.

Background art

[0002] Conventionally, it is known that image degradation occurs when an image is taken with a camera or the like. Causes of image degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, etc.

In order to correct camera shake during shooting, a method of moving a lens and a method of circuit processing are known. For example, as a method for moving a lens, a method is known in which camera shake is detected and correction is performed by moving a predetermined lens in accordance with the detected camera shake (see Patent Document 1).

[0004] Further, as a circuit processing method, a change in the optical axis of the camera is detected by an angular acceleration sensor, and a transfer function representing a blurring state at the time of photographing is detected and obtained for a photographed image. A method is known in which the transfer function is inversely transformed to restore the image (see Patent Document 2).

[0005] Patent Document 1: Japanese Patent Laid-Open No. 6-317824 (see abstract)

Patent Document 2: JP-A-11 24122 (see abstract)

Disclosure of the invention

Problems to be solved by the invention

[0006] A camera adopting the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens such as a motor, and becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are required, which increases costs.

[0007] In addition, the camera shake correction described in Patent Document 2 has the following problems although the above-described problems are eliminated. In other words, it is theoretically possible to perform image restoration by inverse transformation of the acquired transfer function, but as a practical problem, image restoration is difficult for the following two reasons. [0008] First, the value of the transfer function to be obtained fluctuates greatly due to these slight fluctuations that are very vulnerable to noise information errors. For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice. Secondly, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. Therefore, there is a high risk that it will not be solved in practice.

[0009] As described above, an object of the present invention is to provide an image processing apparatus having a realistic circuit processing method while preventing an increase in size of the apparatus when restoring an image.

Means for solving the problem

In order to solve the above problems, an image processing apparatus according to the present invention includes an image processing apparatus having a processing unit that processes an image, and the processing unit uses data of change factor information that causes an image change. Thus, the image data power of an arbitrary image is also generated as a result of this comparison by generating image data of a comparison image and then comparing the image data of the original image to be processed with the image data of the comparison image. The difference data is distributed to the image data of an arbitrary image using the data of the change factor information to generate the image data of the restored image, and whether the image data of the restored image is within the limit value. If the image data limit of the restored image is exceeded and the range of the restored image is exceeded,

V, based on the data that exceeds the limit value of the original restored image data! / The image data of the corrected and restored image that has been corrected is generated, then the image data of the corrected and restored image is used in place of the image data of any image, and the same processing is repeated, so that the original A process for generating image data of a restored image that approximates the original image is performed.

[0011] According to the present invention, when restoring an image, it is possible to prevent an increase in size of the apparatus and to provide an image processing apparatus having a practical circuit processing method.

[0012] In addition to the above-described invention, in another invention, the limit value is a limit value of the amount of energy that can be stored in a pixel. When this configuration is adopted, it is possible to prevent the restoration image data exceeding the amount of energy that can be stored in the pixel from being generated. Furthermore, according to another invention, the correction amount of the correction of the original restored image performed based on the excess data is obtained using the data of the change factor information. When this configuration is adopted, the amount of distribution to the original restored image data is determined using the data of the change factor information, so that the restored image data can be efficiently accommodated within the limit value.

[0014] Further, in another invention, the distribution of the difference data to the image data of an arbitrary image performed to generate the image data of the restored image is obtained by dividing the difference data by the change factor information. The distribution amount is calculated using the value. When this configuration is adopted, the difference data can be efficiently made smaller than the predetermined value.

[0015] According to another invention, the processing unit further performs a process of removing an influence on the unprocessed pixels of the distribution amount of the pixels subjected to the correction / restoration image processing with respect to the unprocessed pixels. When this configuration is adopted, the amount of distribution to the original restored image data is determined using the data of the change factor information, so that the restored image data can be efficiently accommodated within the limit value.

The invention's effect

According to the present invention, when restoring a deteriorated image, it is possible to prevent an increase in size of the apparatus and to provide an image processing apparatus having a realistic circuit processing method.

Brief Description of Drawings

FIG. 1 is a block diagram showing a main configuration of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is an external perspective view showing an outline of the image processing apparatus shown in FIG. 1, and is a view for explaining an arrangement position of angular velocity sensors.

FIG. 3 is a processing flow diagram (processing routine) performed by the processing unit of the image processing apparatus shown in FIG. 1 for explaining the basic concept.

4 is a diagram for explaining the concept of the processing method shown in FIG.

FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 3 using hand shake as an example, and a table showing energy concentration when there is no hand shake.

FIG. 6 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram showing image data when there is no camera shake. FIG. 7 is a diagram for specifically explaining the processing method shown in FIG. 3 with an example of camera shake, and is a diagram showing energy dispersion when camera shake occurs.

FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, and is a diagram for explaining a situation in which image data for comparison is generated with an arbitrary image force.

[FIG. 9] A diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. Comparison image data is compared with a blurred original image to be processed, and difference data is obtained. It is a figure for demonstrating the condition to produce | generate.

FIG. 10 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, and explains a situation in which restored image data is generated by allocating difference data and adding it to an arbitrary image. FIG.

[FIG. 11] A diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, and generating new comparison image data from the generated restored image data, and the data and processing target It is a figure for demonstrating the condition which produces | generates difference data by comparing with the blurred original image.

[FIG. 12] A diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, and explaining a situation in which newly generated differential data is allocated and new restored image data is generated. It is a figure for doing.

FIG. 13 is a processing flow for explaining the distribution of difference data.

FIG. 14 is a diagram illustrating the contents of each pixel image data in the image data handled by the image processing apparatus shown in FIG. 1.

FIG. 15 is a diagram showing a relationship between corrected pixel image data handled by the image processing apparatus shown in FIG. 1 and pixel image data of comparison image data.

FIG. 16 is a diagram for explaining the distribution amount of difference data for pixels after pixel n handled by the image processing apparatus shown in FIG. 1.

[FIG. 17] A diagram for explaining another processing method using the processing method shown in FIG. 3 or FIG. 13, in which (A) shows data of an original image to be processed, and (B) shows (A) It is a figure which shows the data of the arch IV, and the data.

[FIG. 18] To explain another processing method using the processing method shown in FIG. 3 or FIG. (A) shows data of an original image to be processed, and (B) shows data obtained by extracting a part of the data of (A).

FIG. 19 is a diagram for explaining a modification of the processing method shown in FIG. 18, and is a diagram showing that the data of the original image is divided into four, and a part of the area for iterative processing is extracted from each divided area. It is.

Explanation of symbols

[0018] 1 Image processing apparatus

2 Imaging unit

3 Control system

4 Processing section

5 Recording section

6 Detector

7 Factor information storage

I

0 Initial image data (image data of any image)

b Initial image data (image data of any image)

I '

0 Comparison image data

b Image data for comparison

G Change factor information data

Img 'original image data (image data of the captured image)

b 'Restored image data (Restored image data)

D Differential data

k Allocation ratio

I

0 + n Restored image data (Restored image data)

Img No degradation! Original correct image data (base image data)

a No deterioration! Original correctness! /, Image data (base image data)

e Excess data

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the image processing apparatus 1 according to the first embodiment of the present invention will be described with reference to the drawings. Will be explained. The image processing apparatus 1 is a so-called digital camera for consumer use that uses a CCD for the image pickup unit. However, the image processing device 1 is for a surveillance camera, a TV camera, and an endoscope that use an image pickup device such as a CCD for the image pickup unit. It can also be applied to devices other than cameras, such as cameras for other uses, imaging diagnostic devices for microscopes, binoculars, and NMR imaging.

[0020] The image processing apparatus 1 includes an imaging unit 2 that captures an image of a person or the like, a control system unit 3 that drives the imaging unit 2, a processing unit 4 that processes an image captured by the imaging unit 2, have. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records the image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

The imaging unit 2 is a part that includes an imaging optical system having a lens and an imaging element such as a CCD or C-MOS that converts light that has passed through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

[0022] The processing unit 4 is composed of an image processing processor, and is configured by an ASIC (Application Specific

It is composed of hardware such as (Integrated Circuit). The processing unit 4 may store an image serving as a base when generating image data for comparison images (hereinafter referred to as comparison image data), which will be described later. The processing unit 4 may be configured to process with software rather than configured as hardware such as an ASIC. The recording unit 5 may employ magnetic recording means such as a hard disk drive made of semiconductor memory, or optical recording means using a DVD or the like.

As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X and Y axes that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. Is provided. By the way, camera shake when shooting with the camera is the force that also causes movement in the X, Y, and Z directions and rotation around the Z axis. Rotation and rotation around the X axis. These two variations are only a slight variation, and the captured image is greatly blurred. Therefore, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. Z for strength and completeness An additional angular velocity sensor around the axis or a sensor that detects movement in the X or Y direction may be added. In addition, the sensor used may be an angular acceleration sensor that is not an angular velocity sensor.

[0024] The factor information storage unit 7 is a recording unit that stores change factor information such as known change factor information, such as aberrations of the optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the optical system and lens distortion. The information is used when restoring blurring of camera shake described later. Wow! /

Next, an outline of the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described with reference to FIG.

In FIG. 3, “I” is image data of an arbitrary image (hereinafter, arbitrary image data)

0

This is image data stored in advance in the recording section of section 4. ΓΙ ¹ ”is an arbitrary image

0

Shows the data of the change image of data I.

0

Comparison image data). “G” is data of change factor information (= change factor information (point spread function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “rim g ′” refers to the captured image, that is, the data of the changed image that has changed due to the change factor, and is the image data of the original image that will be processed in this processing (hereinafter referred to as the original image data).

“D” is difference data between the original image data Img ′ and the comparison image data. "

0

kj is the distribution ratio based on the data G of the change factor information. "I" is optional image data I

This is image data of a restored image newly generated by distributing the difference data D to O + n 0 based on the data of the change factor information (hereinafter, restored image data). “Img” is the original image data before the change (hereinafter referred to as “basic image data”) that is the basis of the original image data Img ′. Here, the relationship between Img and Img 'is expressed by the following equation (1).

Img, = Img water G (1)

“*” Is an operator representing a superposition integral.

The difference data D may be a simple difference between corresponding pixels, but in general, it differs depending on the data G of the change factor information and is expressed by the following equation (2).

D = f (Img ', I, G) ~ (2) [0028] The processing routine of the processing unit 4 first uses the arbitrary image data I as the initial image data I.

0 0 Starts with the intention (step S 101). This initial image data I, that is, arbitrary image data

0

Data I may be the original image data Img 'that is the image data of the captured image.

0

In addition, any image data such as black solid, white solid, gray solid, and pine pattern may be used. In step S102, the initial image data I is inserted instead of Img in equation (1), and the change image is displayed.

0

Image data for comparison I ′, which is an image, is obtained. Next, the original image data Img 'and the comparison image

0

Compared with the image data I ′, difference data D is calculated (step S103).

0

Next, in step S 104, it is determined whether or not the difference data D is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, new initial image data (= restored image data) is determined in step S 105. ) Process to generate. That is, the difference data D is allocated to the initial image data I based on the data G of the change factor information V and the restored image data I is generated. Then restore the image

0 O + n

Steps S102, S103, and S104 are repeated using image data I as initial image data I.

O + n 0

[0030] If the difference data D is smaller than the predetermined value in step S104, the process is terminated (step S106). Then, the restored image data I at the end of processing is corrected and the image

O + n

That is, the base image data Img without deterioration is estimated, and the data is recorded in the recording unit 5. The recording unit 5 records initial image data I and change factor information data G.

0

If necessary, it may be passed to the processing unit 4.

[0031] The concept of the above processing method is summarized as follows. In other words, in this processing method, the processing solution is not solved as an inverse problem, but as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

[0032] Solving as an optimization problem is based on the following conditions.

That is,

(1) The output corresponding to the input is uniquely determined.

(2) If the output is the same, the input is the same.

(3) The solution is converged by iteratively updating the input so that the output is the same.

In other words, as shown in FIGS. 4A and 4B, it is approximate to the original image data Img ′. Comparison image data (I

If 0 O + n ') can be generated, the initial image data I or the restored image data I that is the original data for that generation is

0 O + n

It approximates the data Img.

In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 seconds. In addition, the value used as a determination criterion for the difference data D is “6” in this embodiment when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. In addition, the shake data detected by the angular velocity detection sensor does not correspond to actual shake when the sensor itself is not calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, a correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

Next, the details of the processing method shown in FIGS. 3 and 4 will be described with reference to FIGS. 5, 6, 7, 8, 8, 9, 10, 11 and 12. FIG.

[0036] (Image stabilization algorithm)

When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed in the blurred pixels during the exposure time. In addition, if the blur during the exposure time is known, the way the light energy is dispersed during the exposure time can be understood, so that it is possible to produce a blur-free image.

[0037] Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. In order from the left,..., N—1, n, n + 1, η + 2, η + 3,. When there is no blur, the light energy during the exposure time is concentrated on that pixel, so the light energy concentration is “1.0”. Figure 5 shows this state. The table of Fig. 6 shows the shooting results at this time. The basic image data Img when the power shown in Fig. 6 is not deteriorated. Each data is represented by 8-bit data (0 to 255).

[0038] There is a blur during the exposure time, 50% of the exposure time is at the nth pixel, 30% is at the n + 1st pixel, 20% is at the n + 2th pixel, Suppose that each was blurred. The way in which light energy is dispersed is as shown in the table in Figure 7. This becomes the data G of the change factor information. [0039] Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table in FIG. The data shown as “ideal image” in FIG. 8 is the basic image data Img, and the data shown as “blurred image” is the image data of the photographed image, that is, the original image data Img ′. Specifically, for example, “12 0” of the pixel “n−3” is the distribution ratio of “0.5”, “0.3”, “0.2” of the data G of the change factor information that is blur information. Accordingly, “60” is distributed to the “n−3” pixel, “36” is distributed to the “n−2” pixel, and “24” is distributed to the “n−1” pixel. Similarly, “60” which is the data of the pixel “n−2” is distributed as “30” in “n−2”, “18” in “n−l”, and “12” in “n”. This original image data Img 'and the change factor information data G shown in Fig. 7 will be calculated as an ideal image (base image data Img) with no blurring.

[0040] Any initial image data I shown in step S101 can be used.

0

In this explanation, the original image data Img ′ is used. That is, I = Img '

0

Start processing. In the table of FIG. 9, “input” corresponds to the initial image data I.

0

This initial image data I, that is, the original image data Img '

0

Use data G of information. That is, for example, the “n-3” image of the initial image data I

0

For the prime “60”, “30” is assigned to the n−3 pixel, “18” is assigned to the “n-2” pixel, and “12” is assigned to the “n-1” pixel. The other pixels are allocated in the same way, and are set as “Output”.

0 for comparison I

0 'is generated. As a result, the difference data D in step S103 is as shown in the bottom column of FIG.

[0041] Thereafter, the size of the difference data D is determined in step S104. Specifically, the process is terminated when all the difference data D is 5 or less in absolute value, but the difference data D shown in FIG. 9 does not meet this condition, so the process proceeds to step S105. . That is, the difference data D is allocated to the initial image data I using the data G of the change factor information, and “

0

The restored image data I shown as “next input” is generated. In this case, it is the first time

0 + n

For this reason, in FIG.

0+ 1

[0042] The distribution of the difference data D is, for example, the data “30” of the pixel “n−3” (= “n

“15”, which is the distribution ratio of “3” pixel), is distributed to “n—3” pixel “15”, and “n—2” is assigned to “n 2” pixel data “15”. It is the distribution ratio that should have come to the pixels of 0.3 The distribution ratio that should have come to the “n−l” pixel is assigned to the data “9.2” of the “n−l” pixel. Allocate “1. 84” multiplied by 2. The total amount allocated to the “n—3” pixel is “21. 34”, and this value is the initial image data I (here,

0

Using the shadow image data Img '), the restored image data I is generated. This

0 +1

The restored image data I corresponds to “next input” in the table of FIG.

0 +1

[0043] As shown in FIG. 11, the restored image data I is the input image data in step S102.

0+ 1

(= Initial image data I), and step S102 is executed. In the table in Fig. 11, “Input I

0 0 +

"Corresponds to the restored image data I. Then move on to step S103

1 0+ 1

Minute data D is obtained. The size of the new difference data D is determined in step S104. If it is larger than the predetermined value, the new difference data D is converted to the previous restored image data I in step S105.

Allocate to 0 + and generate new restored image data I. This restored image data I is shown in Fig. 1.

1 0 + 2 0 + 2

This corresponds to “next entry” in the table of 2. After that, by performing step S102, the restored image data I

New image data for comparison with 0 + 2 force I

0 + 2 'is generated. As described above, after steps S102 and S103 are executed, the process goes to step S104, and depending on the determination, the process goes to step S105, or the process proceeds to step S106. Such a process is repeated.

In this image processing apparatus 1, before processing, in step 104, either or both of the number of processes and the determination reference value of the difference data D can be set in advance. For example, the number of processings can be set to any number such as 20 or 50 times. Also, set the value of the difference data D that stops processing to “5” in 8 bits (0 to 255). When it becomes 5 or less, the processing is terminated, or “0.5” is set to “0”. The process can be terminated when 5 or less. This set value can be set arbitrarily. If both the number of processing times and the criterion value are entered, the processing is stopped when either one is satisfied. When both settings are possible, the judgment reference value may be prioritized, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

[0045] In the description of this embodiment, the force that did not use the information stored in the factor information storage unit 7 is a known change factor stored here, such as optical aberration and lens distortion. Data may be used. In this case, for example, in the processing method in the previous example (Fig. 3), the blur information and the optical aberration information are combined and regarded as one change factor. However, it may be preferable to correct the optical aberration information after the blur information processing is completed. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors during shooting, for example, only blurring.

This image processing apparatus 1 adopts the following processing method based on the above-described concept of the processing method. This method will be described with reference to FIGS. This method is intended to optimize the amount of distribution of difference data D to initial image data I.

0

Image data (I

0 O + n ′) can be quickly approximated to the original image data Img ′, and as a result, the number of processing routines shown in FIG. 3 can be reduced.

A method for distributing the difference data D will be described along the processing flow of FIG. Processing flow S2 01 force S204 is a processing flow corresponding to processing flow S101 to S104 in FIG. That is, the input initial image data I (S201) force also generates comparison image data I '(S2

0 0

02), each image of the original image data Img 'which is the change image data and the comparison image data I'

0

Difference data D for the prime is calculated (S203). Then, the size of the difference data D is determined for each pixel (S204). If the size is appropriate for all the pixels, it is determined that the image has been restored, and the process is terminated (S205).

The processing contents from S201 to S205 will be described with reference to FIG. This figure is similar to FIG. 5 and FIG. 12, for the sake of simplicity, the processing of the horizontal one-dimensional pixels n− 3, n−2, n−1, n, n + 1, n + 2,. Indicates the contents.

First, the initial image data I for pixels n−3, n−2, n−l, n, n + 1, n + 2,.

0 b, b, b, b, b, b,..., And processing routines up to S201 and S202 (n-3 n-2 n-1 n n + 1 n + 2 in FIG. 3)

Equivalent to the processing routine from S101 to S102), the comparison image data I

B ^r as ^{0 ', b r, b,} b, b r, b r, ... obtained.

n-3 n-2 n-1 n n + 1 n + 2

[0050] This comparison image data I ′ (= b ^r , b ^r , b, b, b ^r , b ...

0 n-3 n-2 n-1 n n + 1 n + 2

) Is the initial image data I (= b, b, b, b, b, b,.

0 n-3 n-2 n-1 n n + 1 n + 2

According to data G, it is generated by being distributed to itself and other pixels. Here, the change factor information data G is described as α, β, γ (α>β> γ, α + β + γ = 1). In other words, the light energy of each pixel is a ratio of α, β, γ to the pixel in the blur direction. Will be distributed.

[0051] For example, when the pixel image data b of the pixel n in the initial image data I is considered,

0 n

In the image data b, a-b is dispersed in the pixel n, β-b is dispersed in the pixel n + 1, and y-b is dispersed in the pixel n + 2. Similarly, the energy is distributed to the pixels indicated by arrows in the figure according to the data G of the change factor information. Therefore, b, b, b, b, b which are initial image data of each pixel (n−3, n−2, n−2, n−l, n, n + 1, n + 2,...) , b, n-3 n-2 n-1 n n + 1 n + 2 are distributed according to the data G of the change factor information, and b ′, b ′, b are respectively used as comparison image data for each pixel. ', b', b ', b', ...

n-3 n-2 n-1 n n + 1 n + 2

[0052] Comparative image data I ′ (b ′, b ′, b ′, h ′, b ′, b ′,...)

0 n-3 n-2 n-1 n n + 1 n + 2

For example, regarding the pixel image data f of the pixel n, the dispersion amount of γ of the pixel image data b of the pixel n−2, the dispersion amount of β of the pixel image data b of the pixel η−1, and the own pixel. The content is influenced by the amount of dispersion of α for pixel image data b of η. In other words, h '= a -b + | 8 -b + γ -b. Similarly, the pixel image data of the other pixels of the comparison image data I ′ are also changed according to the change factor information data G, according to the change factor information data G (in FIG. This is the content affected by the amount of dispersion of the pixel on the left and the pixel on the left.

[0053] Initial image data I to comparison image data I '

Figure 9 shows the process that 0 generates.

This is the same as described in 0. Initial image data I

0 is the input I in Figure 9

This corresponds to 0, and the image data for comparison corresponds to the output in Fig. 9.

0 0

[0054] Pixel image data power a, a, a, a, a, a n-3 n-2 n-1 n n + 1 n + 2 of each pixel of the base image data Img

, ..., the pixel image data of each pixel of the original image data Img 'is a', a 'n-3 n-2

, a ', a', a ', a

n + 2 ', ... The original image data Img 'is, for example, pixel n n-1 n n + 1

For the pixel image data a ′ of the base image data Img, the dispersion amount for the proportion of γ in the pixel image data a of the pixel n−2 of the base image data I and the dispersion amount of the proportion of β for the pixel image data a of the pixel η—1 Furthermore, the content is influenced by the amount of dispersion corresponding to the proportion of oc in the pixel image data a of its own pixel η. In other words, it can be seen as a '= a -a + β -a + y -a. Similarly for the other pixels of the original image data Img ', the self-pixel, the previous one and the other two previous pixels (the pixel on the left in FIG. In addition, the content is influenced by the amount of dispersion of the base image data Img corresponding to the pixel on the left.

[0055] The image data of each pixel of the original image data Img 'is a', a ', a', a ', a n-3 n-2 n-1 n n + 1

When ', a', ..., the image data of each pixel of the original image data Img 'and the comparison image n + 2

The difference data D, which is the difference between the data and the image data of each pixel, becomes δ, δ, δ, δ, δ, δ,... (S203). In S204, the difference data n n + 1 n + 2 of each pixel

Whether each of δ, δ, δ, δ, δ, δ,... is an appropriate value is determined as n-3 n-2 n-1 n n + 1 n + 2

If it is determined that the size of the difference data D is not appropriate, the process proceeds to S206 and subsequent steps described below. If it is appropriate, it is determined that the original image has been properly restored, and the process ends (S205). ). In the processing after S206, the difference data D to the initial image data I

0

Set an appropriate distribution amount.

[0056] First, using the difference data D, an arbitrary distribution amount that is considered to be an appropriate distribution amount is calculated (S206). This distribution amount is calculated as follows. For example, regarding pixel n, pixel image data b ′ is the sum of and | 8′b and a′b, so the difference data δ is the initial value of pixels n−2, n-1, and n. Pixel image data in image data I b, bn 0 n-2 n-1

, And the amount of dispersion from. In addition, for the pixel n in the original image data Img ′, since the previous pixel image data a ′ is the sum of γ · _& and jS′a and a′a, the difference data δ 〖 It also includes the amount of dispersion of the pixel image data a_, a, a force in the base image data Img of -2, n-1, n.

That is, the difference data _δη includes a′a as the amount of dispersion of the pixel image data a of the base image data Img at the pixel η and the amount of dispersion of the initial image data I from the pixel image data b. a -b is included. Then, it is considered that the difference data δ includes the dispersion amount of the pixel image data a and the pixel image data b force at a ratio of K (0≤K≤1). In other words, it can be considered as the following equation (3).

α · Ά — a ^# b = Κ · o ··· \ 3)

From this equation (3)

a = b + Κδ / α

It becomes. In other words, the distribution amount h allocated from the difference data δ to the initial image data b is expressed as Κ · δ Let it be Z a (S206). Here, the value obtained by dividing (dividing) the difference data _δη by the data α of the change factor information is used as the distribution amount h. Then, this distribution amount h = K ′ δ Ζ α is distributed to the pixel image data b of the pixel n of the initial image data I for n 0 n, and b + Κ · δ Ζ α is the pixel image data of the restored image for the pixel η (S20 7).

In the processing flow of FIG. 3 described above, the pixel image data b + Κ · δ Ζ α becomes the pixel image data for the pixel n of the new restored image data I, and the pixel image data

O + n

The data b + Κ · δ / α is processed as new initial image data (S105). Thereafter, the process returns to S102, and whether the restored image data I approximates the base image data Img.

O + n

The processing flow (S103, S104) for determining whether or not is executed. However, in the processing flow in Fig. 13, S207 force and S212 are instructed to determine whether or not the appropriate amount of distribution is として · δ / α force as the self amount h. If it is not appropriate, correct it so that the appropriate distribution amount is obtained.

[0059] The determination as to whether the distribution amount Κ · δ δ α is an appropriate distribution amount is based on whether the pixel image data “b + Κ · δ Ζ α” for the pixel η obtained in S207 is a pixel of the restored image data This is performed based on the judgment (S208) whether the image data is appropriate. This decision is made as follows.

[0060] The pixel image data “b + Κ · δ Ζ α” corresponds to an image that approximates the pixel image data a. Therefore, the range of limit values that can be assumed for the pixel image data a (min≤a≤ max). In this embodiment, the range of the limit value is a limit value of the amount of light energy that can be accumulated in the pixel. For example, if the pixel image data is represented by 8 bits (0 to 255) as the intensity of light energy, the pixel image data a should be within the range of 0≤a≤255. .

Accordingly, it is determined that the pixel image data “b +」 · δΖα ”that exceeds the limit value range (min ≦ a ≦ max) of the pixel image data a is inappropriate as the pixel image data. That is, when the pixel image data “b + Κ δ Ζ α” is below the lower limit of the limit value range of the pixel image data a or exceeds the upper limit value, the pixel image data “b + b · “Δ / H” is determined to be inappropriate as pixel image data. That is, the pixel image data “b If “+ Κ · δ Ζα” is inappropriate, it can be determined that the distribution amount Κ · δ Ζα is inappropriate. In other words, when it falls outside the range of b + Κ δ Ζ α force min≤b + Κ δ Za ≤ max in Eq. (4), Κ δ 〖is the source of the generation of pixel image data f. Shadow of the distribution amount from the pixel image data b and b for the pixels n− 1 and n− 2 of the original restored image data n−1 n−2

It can be considered that the sound is greatly included.

Therefore, in order to reduce the distribution amount of the pixel image data b 1, b 2, and force, the pixel image data n−1 n−2

N-1 n-2 n-1 n-2 The data b, b is reduced and the amount of distribution to the pixel image data b, b force pixel n is reduced.

Like that. For this purpose, first, a correction amount for subtracting the pixel image data b and b is set. N-1 n-2

(S209). This correction amount is calculated as follows.

[0063] pixel image data "b + Κ · δ Ζα" range data (min≤b + Κ · δ Za≤max) amounts exceeding (hereinafter, beyond content data) is calculated e _n. The excess data e is considered to be included in Κ · δ _η Ζα, and the excess data e is considered to be the amount of dispersion of the pixel image data b and b force. That is, such a state is considered to be because the pixel image data b and b from which the pixel image data “b + Κ · δ / α” is restored are inappropriate.

Accordingly, the pixel image data b 1 and b 2 are corrected so that the pixel image data b + Κ · δ Ζα does not exceed the range (min≤b + Κ · δ / a≤max)! /. This correction amount is determined based on the following concept. As shown in FIG. 15, assuming that the correction amount data of the pixel image data b and b are correction data e and e, respectively, the pixel image data b ′ of the comparison image data at the pixel n is expressed by the equation (5). It is expressed as follows.

b '= γ-(b _ + e _) + j8-(b + e) + a · (b + e)

= yh _ ₂ + β -b _ a -b + γ -e _ _≥ + β -ea -e, (5) In other words,

γ -e _ + β -e + a -e = 0, (6)

Of time, the pixel image data b in the differential data [delta] _eta pixels n, the influence of the dispersion amount from b and summer small. Therefore, the restored image data at the pixel n can be set to an appropriate value by setting the correction data e_, e so that the expression (6) is satisfied. That is, it is possible to set で · δ Z a that is the distribution amount based on the difference data δ to an appropriate value. it can. The specific setting of the correction data e _{n 2} , based on equation (6), can be considered as follows, for example.

[0065] (ii) When e = e.

From Equation (6), e_ (γ + j8) = − e · α. Therefore,

e _ ₂ (= e _) =-a -e / (γ + β) =-a -ea)

It can be done.

[0066] (Mouth) Exceeding data It is assumed that the amount of influence on e depends on the change factor, and e: e = γ: β.

e: e = γ β

It becomes. Therefore, from equation (6)

y ^# e / β + β -e = — α · Θ

_C

[0067] Since Eq. (8) and Eq. (9) are α + | 8 + γ = 1, they can be approximated as j8 ² + γ ² = (1— α) ² and e _ = (-α · β / (ΐ- a) ^Z ) -e --- (10)

e _ = (-α · γ / (ΐ- α) ² ) · Θ ~ (11)

As good as.

[0068] (c) is small effect on the over amount data e _n, pixel Nitsu Te, for example, when set to 0 the effects of the e _n, as e = 0,

n-2

From equation (6)

It can be done.

In addition to the above, various methods for setting the correction data e 1 and e 3 can be considered. Then, correction is performed by adding the correction data e and e set as described above to the pixel image data of the pixels n−2 and n−1, and pixel image data corrected for the pixels n−2 and n−1 is generated. (S210).

In the processing in S210, it is further determined whether or not the pixel image data of the pixels n−2, n−1 is appropriate as the pixel image data of the restored image data, as in S208. That is, for pixel n-2, within the pixel image data "b + e" force range (min≤ n-2 n-2

b + e ≤max). For pixel n-l, pixel image n-2 n-2

It is determined whether the image data “b _ + e _” is within the range (min ≤ b + e _ ≤ max).

If it is determined that the corrected pixel image data “b + e” and “b _ + e” are not within the range, the pixel images of the pixels n−2 and n−l are further displayed. The pixel image data of the original restored image data on the left side that is the source of the data, for example, returning to b and the above S20 n-3

6 forces are also processed in S210 (S211).

[0072] The processing described above is performed for all pixels in the pixel order for each pixel, so that the restored image data falls within a predetermined range from the difference data D to the initial image data I.

0

Set the allocation amount. As a result, the comparison image data I ′ (I

0 O + n ') can be approximated to the original image data Img' quickly.

[0073] Note that when calculating the distribution amount for the unprocessed pixels that have not been subjected to the processing from S206 to S210 after pixel n, before calculating the pixel distribution amount, the distribution amount "b. The influence based on “δ / a + e” is removed (S212). As a result, a more appropriate distribution amount can be calculated for pixels after pixel n.

That is, as shown in FIG. 16, when the distribution amount at pixel n is (Κ · δ / a) + e, the influence of this distribution amount on pixel n + 1 is Κ · β · δ α + β -e, and the influence on the pixel n + 2 can be considered as Κ · γ · δ / a + ye.

[0075] Therefore, 差分 ·· δ Ζα + jS'e is removed from the difference data δ of pixel n + 1, and n + l n n

Also, remove Κ · γ-δ / a + ye from the difference data δ of pixel n + 2, and replace this with each n + 2 n n

Let the difference data δ and δ for pixels n + l and n + 2. In this way, the pixels η + 1, η n + 1 n + 2

+ More appropriate distribution amount can be calculated for 2 pixels. By performing the same process for pixels n + 3, n + 4, ..., the distribution of difference data D to initial image data I at pixels n + 2, n + 3, n + 4, ... It can be carried out. This makes it efficient

0 The comparison image data (I ') is more quickly approximated to the original image data Img'

0 O + n

Can do.

As described above, the difference data D is distributed to the initial image data I of each pixel.

0

Thereafter, the process returns to S202, and the above-described processes from S206 to S211 are repeated until the comparison (S203) with the original image data Img ′ as the photographed image data becomes small (S204).

Although the image processing apparatus 1 according to the embodiment of the present invention has been described above, various modifications can be made without departing from the gist of the present invention. For example, the processing performed by the processing unit 4 may be configured by hardware composed of components in which a part of processing is shared by each of the forces configured by software.

In addition to the captured image, the original image to be processed may be processed such as color-corrected or Fourier-transformed. In addition to the data generated using the change factor information data G, the comparison image data includes color correction for data generated using the change factor information data G, or Fourier transform. It is also good as data that has been processed. In addition, the change factor information data includes information that simply changes the image, and information that improves the image, as opposed to deterioration.

[0079] When the number of processing iterations is automatically or fixedly set on the image processing apparatus 1, the set number of times may be changed by the data G of the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.

[0080] Furthermore, during the iterative process, if the difference data D diverges, that is, if the difference data D becomes larger, the process may be stopped. For example, a method of judging whether or not the force is diverging can be determined by looking at the average value of the difference data D and determining that the diverging force is greater than the previous value. In addition, during an iterative process, if an attempt is made to change the input to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed exceeds 255, processing is stopped. Also, during an iterative process, when trying to change an input that is new data to an abnormal value, that value may be used instead of the normal value. For example, let's assume that a value exceeding 255 within the 8-bit range of 0 to 255 is input data. When this happens, it is processed as 255, which is the maximum value.

[0081] Further, when the restored image data to be the output image is generated, the change factor information data G may generate data that goes out of the area of the image to be restored. . In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable that the data also bring the opposite side force.

[0082] When the restored image data I is generated, the distribution ratio k is not used, and the corresponding pixel

O + n

The difference data D can be directly used as the previous restored image data I, or the corresponding pixel

O + n— 1

It is added after scaling the difference data D, or the data kD (value indicated as “update amount” in FIGS. 10 and 12) after the difference data D is allocated is scaled to restore the previous restored image. It may be added to data I. If you use these processing methods well, the processing speed will be

O + n— 1

Get faster.

[0083] When the restored image data I is generated, the center of gravity of the change factor is calculated, and

O + n

Difference of the image or the magnification of the difference is the previous restored image data I

O + n— You may add a little more. In the previous example, the three centroids of “0.5”, “0.3”, and “0.2” are the highest value of “0.5”, which is their own position. Therefore, without assigning “0.3” or “0.2”, only “0.5” or 0.5 magnification of difference data D is assigned to its own position. Such a process is suitable when the blur energy is concentrated.

[0084] Furthermore, each processing method described above can be automatically selected according to the content of the data G of the change factor information. For example, as shown in FIGS. 5 to 12, as processing methods, (1) a method of allocating difference data D using an allocation ratio k (allocation ratio allocation method), (2) a corresponding pixel difference, or Processing unit program that can execute the three methods of scaling the difference data D (corresponding pixel method) and (3) detecting the center of gravity of the change factor and using the data of the center of gravity (centroid method) 4) Save the data in the analysis, analyze the status of the change factors, and select one of the three methods based on the analysis results. Alternatively, you can select any of the three methods and use them alternately in each routine, or process them in one method for the first few times, and then process them in another method. !

[0085] Further, there is a method combined with the inverse problem in order to increase the speed of the restoration process. That is, an iterative process is performed on the reduced data, and a transfer function from the reduced original image to the reduced restored image data is calculated. Then, the calculated transfer function is enlarged and interpolated, and the enlarged and interpolated transfer function is used to obtain restored image data of the original image. This processing method is advantageous for processing large images.

[0086] As such a processing method, two methods are conceivable. The first method is to reduce the data by thinning out the data. When thinning, for example, as shown in FIG. 17, when the original image data ImgZ force consists of pixels 11-16, 21-26, 31-36, 41-46, 51-56, 61-66, There is a method of thinning out every other pixel to generate a reduced size Img 'of the pixels 11, 13, 15, 31, 33, 35, 51, 53, 55, which is a quarter size.

[0087] In this way, the original image data Img 'is thinned out, and the reduced Img', which is the thinned data, is generated, and the iterative Img 'is used to perform the iterative process shown in FIG. Retrieved image data I

Get O + n. Reduced image data I

O + n is satisfactory, but it is only an approximation. Therefore, the restored image data I and the original image data Img

O + n

The transfer function of 'is not the transfer function used in the iterative processing of the reduced data. Therefore, the reduced and restored image data I and the reduced Img 'force that is the reduced original image data also calculate the transfer function.

O + n

The calculated transfer function is enlarged, and the enlarged transfer function is interpolated. The corrected transfer function is used as the transfer function for the original image data Img ′ as the original data. Then, using the modified transfer function, deconvolution calculation is performed in the frequency space (calculation that removes blur by calculating the image group force including blur), and the complete restored image data I

O + n is obtained, and it is degraded and estimated to be the correct image data Img.

[0088] In this process, the obtained correct image and the restored image data I estimated are shown in FIG.

O + n

It is used as initial image data I for the processing shown in Fig.

0

The original image data Img 'may be used for further processing.

[0089] The second method of using reduced data is a method of obtaining reduced data by extracting data of a partial area of original image data Img '. For example, as shown in FIG. 18, the original image data ImgZ force consists of pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61 to 66! There is a method of generating a reduced Img ′ by taking out the area of the pixels 32, 33, 34, 42, 43, and 44, which is the central area, from the area. [0090] In this way, the entire image area is not restored by iterative processing, but a part of the area is iteratively processed to obtain a good restored image, which is used to obtain a transfer function for that part, and the transfer function itself Alternatively, the entire image is restored using a modified version (enlarged). However, the area to be extracted must be sufficiently larger than the fluctuation area. In the previous example shown in Fig. 5 etc., it fluctuates over 3 pixels, so it is necessary to extract an area of 3 pixels or more.

[0091] In the method of extracting the reduced area, the original image data Img 'is divided into four parts as shown in Fig. 19, for example. It is possible to make the original whole image by iteratively processing each of the four reduced Ims, restoring each of the four divided areas, and combining the restored four divided images into one. In addition, when dividing into a plurality of regions, it is preferable to always have an overlapping region (overlap region) over a plurality of regions. In addition, it is preferable to perform processing such as using an average value or smoothly connecting the overlap areas of the restored images in the overlap areas.

[0092] Further, when the processing methods of Figs. 3 and 13 are actually adopted, it has been found that convergence of an image with a sharp change in contrast to a good approximate restored image is slow. Thus, depending on the nature of the subject that is the original image, it may be necessary to increase the number of iterations at which the convergence speed of the iteration process is slow. In the case of such a subject, it is estimated that this problem can be solved by adopting the following processing method.

[0093] The method is as follows. In other words, a subject with a sudden change in contrast uses an iterative restoration process using the processing methods shown in Fig. 3 and Fig. 13, and if it tries to obtain something similar to the original image, the number of iterations is extremely high. And the restored image data that approximates the original subject after processing many times.

Cannot generate O + n. Therefore, the data of the original image (blurred image) Img '〖こ, the data B of the change factor information at the time of shooting is generated from the data B of the known image, and the data of the blurred image is generated. Negotiate and make "Img '+ Β'". After that, the superimposed image is restored by the process shown in FIG. 3, and a known added field image is obtained from the result data C as the restored image data I.

O + n

Data B is removed, and base image data Img of the restored image to be obtained is taken out. [0094] In this method, the base image data Img includes a rapid contrast change, but by adding the data B of a known image, this rapid contrast change can be reduced, and the restoration process is performed. The number of iterations can be reduced.

In addition, other processing methods can be adopted as a processing method for a subject that is difficult to restore and a high-speed processing method. For example, if the number of iterations of restoration processing is increased, it takes time to perform force processing that can be brought closer to a good restored image. Therefore, by using the image obtained with a certain number of iterations, the error component included in the image is calculated, and the calculated error is removed from the restored image including the error. Get

O + n comes out.

[0096] This method will be specifically described below. Assume that the obtained image is A, the captured original image is A, the image restored from the original image is A + D, and the blurred comparison image data generated from the restored image data is + D ′. If the original image "A" is added to this "A ^ + D '" and restored, it becomes "A + D + A + D + D", which is "2A + 3D" Also, “2 (Α + α) + α”. Since “A + D” has been obtained in the previous restoration process, “2 (A + D) + D− 2 (A + D)” can be calculated, and “D” is obtained. Therefore, by removing “D” from “A + DJ”, the image A can be obtained! /, Correct! /.

[0097] Each processing method described above, that is, (1) a method of distributing the difference data D using the distribution ratio k (basic method), (2) a corresponding pixel difference, or scaling of the difference data D (3) Method of detecting the centroid of the change factor and using the data of the centroid (centroid method), (4) Difference data explained with reference to FIGS. 13 to 16 Method of optimizing the distribution of D (Method of optimizing the distribution of difference data), (5) Method of thinning out data and combining with inverse problem (inverse problem thinning out method), (6) Extracting the reduced area and inverse problem (7) Method of superimposing a predetermined image and performing an iterative process, and then removing the predetermined image (bad image countermeasure overlay method), (8) Method to remove the calculated error from the restored image including (error extraction method) The processing method program may be stored in the processing unit 4 so that the processing method can be automatically selected according to the user's selection or the type of image.

[0098] Further, any one of these (1) to (8) is stored in the processing unit 4 and selected by the user. Alternatively, the processing method may be automatically selected according to the type of image. Also

Select any one of these 8 methods and use them alternately or in sequence for each routine, or process them in one method for the first few times and then process them in another method. Also good. Note that the image processing apparatus 1 may have a different processing method in addition to any one or a plurality of (1) to (8) described above.

Moreover, each processing method mentioned above may be programmed. A program may be stored in a storage medium such as a CD, a DVD, or a USB memory so that it can be read by a computer. In this case, the image processing apparatus 1 has a reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading the program in the storage medium.

Claims

The scope of the claims

[1] In an image processing device with a processing unit that processes images! In a hurry

The processing unit

Using the data of the change factor information that causes the image change, the image data of the arbitrary image

After that, the image data of the original image to be processed is compared with the image data of the comparison image,

The difference data obtained as a result of this comparison is distributed to the image data of the arbitrary image using the data of the change factor information, thereby generating image data of the restored image.

Determine whether the image data of this restored image is within the limit value,

If the image data of the restored image exceeds the range of the limit value, the image data of the original restored image that is the source of the generation of the image data of the restored image exceeds the range of the limit value. Image data of the corrected restored image corrected based on the minute data,

After that, the image data of the corrected restored image is used in place of the image data of the arbitrary image, and the same processing is repeated to generate the image data of the restored image that approximates the original image before the change. An image processing apparatus characterized in that the processing is performed.

2. The image processing apparatus according to claim 1, wherein the limit value is a limit value of an amount of light energy that can be accumulated in a pixel.

[3] The correction amount of the correction of the original restored image performed based on the excess data is obtained using the data of the change factor information. Image processing apparatus.

[4] The distribution of the difference data to the image data of the arbitrary image performed to generate the image data of the restored image is a distribution calculated using a value obtained by dividing the difference data by change factor information. The image processing apparatus according to claim 1, wherein the force is an amount.

[5] The processing unit further performs processing for the corrected and restored image processing on unprocessed pixels. 5. The image processing device according to any one of claims 1 to 4, wherein a process for removing an influence of an unallocated amount on the unprocessed pixels is performed.