JP5007234B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  Conventionally, it is known that image degradation occurs when an image is taken with a camera or the like. Factors of image degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, and the like.

  In order to correct camera shake during shooting, a method of moving a lens and a method of circuit processing are known. For example, as a method for moving a lens, a method is known in which camera shake is detected and a predetermined lens is corrected by moving it in accordance with the detected camera shake (see Patent Document 1).

  In addition, as a circuit processing method, a change in the optical axis of the camera is detected by an angular acceleration sensor, a transfer function indicating a blurring state at the time of shooting is acquired from the detected angular velocity, etc., and the acquired transfer function is obtained for a shot image. A method is known in which an image is restored by performing an inverse transformation of (see Patent Document 2).

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens, such as a motor, and becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  In addition, in the case of camera shake correction described in Patent Document 2, the above-described problems are eliminated, but the following problems occur. That is, it is theoretically possible to perform image restoration by inverse transformation of the acquired transfer function, but as a practical problem, image restoration is difficult for the following two reasons.

  First, the transfer function to be acquired is very weak against noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice. Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In practice, there is a high risk of being unable to solve.

  As described above, an object of the present invention is to provide an image processing apparatus having a realistic circuit processing method as well as preventing an increase in size of the apparatus when restoring an image.

In order to solve the above-described problems, an image processing apparatus according to the present invention includes an image processing apparatus having a processing unit that processes an image. The comparison data is generated from the image data, the original image data to be processed is compared with the comparison data, and the image from the surrounding pixels for a certain pixel is obtained from the obtained difference data. The restoration data is generated by removing the influence of the change according to the data of the change factor information, and dividing the value by the data of the change factor information and dividing the value by less than 1 to add the comparison data. using the data in place of any image data, by repeating the same processing, the recovered data as the image data to be approximate to the original image or before the change of the original image and form raw.

  According to the present invention, since the restoration data that approximates the original image is generated only by generating predetermined data using the factor information of the image change, there is almost no increase in hardware, and the device Does not increase in size. In addition, the comparison data is created from the restored data, and the comparison data is compared with the original image data to be processed, so that the restored data that is close to the original image before the original image is gradually obtained. It will be a realistic restoration work. In addition, since the difference data excluding the influence of the surroundings is divided by a value less than 1 and the divided value is returned, high-quality restored data can be obtained with a small number of iterations. For this reason, when restoring an image, an image processing apparatus having a realistic circuit processing method can be obtained.

According to another aspect of the present invention, there is provided an image processing apparatus having a processing unit that processes an image, wherein the processing unit uses data of change factor information that causes an image change, from data of an arbitrary image. The comparison data is generated, the original image data to be processed is compared with the comparison data, and the obtained difference data is directly or multiplied by a value less than 1 as the change factor information data. In addition, a return amount to be added to the comparison data by dividing by a value less than 1 is determined as the value of the restoration data of the specific pixel, and the value of the restoration data is already calculated for the pixel for which the value of the restoration data has not been calculated. in accordance with the data affected by the image change of the variation factor information from a particular pixel is, the process of removing from the data of the difference by repeating for all pixels, thereby generating a restoration data, the restoration data It was used in place of the arbitrary image data, by repeating the same processing, and generates the restored data as the image data to approximate pre-change image or to that of the original image.

  According to the present invention, since the restoration data that approximates the original image is generated only by generating predetermined data using the factor information of the image change, there is almost no increase in hardware, and the device Does not increase in size. In addition, the comparison data is created from the restored data, and the comparison data is compared with the original image data to be processed, so that the restored data that is close to the original image before the original image is gradually obtained. It will be a realistic restoration work. In addition, since the difference data excluding the influence of the surroundings is divided by a value less than 1 and the divided value is returned, high-quality restored data can be obtained with a small number of iterations. For this reason, when restoring an image, an image processing apparatus having a realistic circuit processing method can be obtained.

  Furthermore, in addition to the above-described invention, the processing unit may stop if the difference data when the number of repetitions reaches a predetermined number or less than a predetermined value or less than a predetermined value during the repetition process, If it exceeds the predetermined value or exceeds the predetermined value, the process is further repeated a predetermined number of times. In the present invention, since the number of times of processing and the difference value are combined, the image quality and processing are compared with the case where the number of times of processing is simply limited or the difference value is limited. The processing can balance the shortness of time.

According to another aspect of the present invention, there is provided an image processing apparatus having a processing unit for processing an image, wherein the processing unit uses data of change factor information that causes an image change to obtain data from a predetermined image. Generate comparison data, compare the comparison data with the original image data whose image to be processed has changed. If the obtained difference data is less than or equal to the specified value, stop the process and compare If the difference is greater than or equal to the specified value, the specified image that is the source of the data is treated as an image before or near the original image . the removed according to the data the effect of image change change factor information from surrounding pixels, the return amount to be added to the comparison data by dividing the value in the data a and 1 less than the value of the change factor information And in, thereby generating a restoration data, and performs a process of repeating the same process by replacing the restored data to a predetermined image.

  According to the present invention, by using change factor information such as image degradation, comparison data is generated, compared with the original image, and restored data that approximates the original image is generated only when the difference is large. Therefore, there is almost no increase in hardware, and the apparatus does not increase in size. In addition, by making comparison data from the restored data and comparing the comparison data with the original image data to be processed, the restored data that is close to the original image before the original image is gradually obtained. Restoration work. In addition, since the difference data excluding the influence of the surroundings is divided by a value less than 1 and the divided value is returned, high-quality restored data can be obtained with a small number of iterations. Therefore, when restoring a deteriorated image, an image processing apparatus having a realistic circuit processing method can be obtained.

According to another aspect of the present invention, there is provided an image processing apparatus having a processing unit for processing an image, wherein the processing unit uses data of change factor information that causes an image change to obtain data from a predetermined image. Generate comparison data, compare the comparison data with the original image data whose image to be processed has changed. If the obtained difference data is less than or equal to the specified value, stop the process and compare If the difference image is treated as a pre-change image or an image similar to the original image, and the difference is greater than or equal to the predetermined value, the difference data is left as it is or a value less than 1 was the data at a by and returning amount divided by the addition to the comparison data with a value less than 1 for a value change factor information multiplied, determined as the value of the recovered data for a particular pixel, the value of the recovered data is calculated By repeating the process of removing the influence of the image change from the specific pixel whose restoration data value has already been calculated for all the pixels according to the change factor information data for all the pixels, The restoration data is generated, and the restoration data is replaced with a predetermined image, and the same processing is repeated.

  According to the present invention, by using change factor information such as image degradation, comparison data is generated, compared with the original image, and restored data that approximates the original image is generated only when the difference is large. Therefore, there is almost no increase in hardware, and the apparatus does not increase in size. In addition, by making comparison data from the restored data and comparing the comparison data with the original image data to be processed, the restored data that is close to the original image before the original image is gradually obtained. Restoration work. In addition, since the difference data excluding the influence of the surroundings is divided by a value less than 1 and the divided value is returned, high-quality restored data can be obtained with a small number of iterations. Therefore, when restoring a deteriorated image, an image processing apparatus having a realistic circuit processing method can be obtained.

  In another invention, in addition to the above-described invention, the processing unit performs a process of stopping when the number of repetitions reaches a predetermined number during the repetition process. When this configuration is adopted, the processing is stopped regardless of whether the difference becomes “0”, so that it is possible to prevent the processing from taking a long time. Further, since the processing is continued up to a predetermined number of times, the restored data becomes closer to the original image before the change that is the original image. Furthermore, when there is noise or the like, a situation where the difference does not become “0” tends to occur in reality, but in such a case, the process is repeated infinitely, but if this configuration is adopted , No such problem occurs.

  In addition to the above-described invention, still another invention includes a detection unit that detects change factor information and a factor information storage unit that stores known change factor information. By adopting this configuration, it is possible to obtain corrected restoration data that takes into account both external and internal factors of image change.

  In addition, it is preferable that the value used when dividing by the data of the change factor information is a value having the largest weight among the data of the change factor information. When this configuration is adopted, the processing speed is further improved.

  Furthermore, it is preferable that the order of calculating the return amount depends on the data characteristics of the change factor information. When this configuration is adopted, an optimum processing method can be selected depending on the data characteristics of the change factor information.

  According to the present invention, it is possible to prevent an increase in the size of an apparatus when restoring an image that has changed due to deterioration or the like, and to provide an image processing apparatus having a realistic circuit processing method.

1 is a block diagram showing a main configuration of an image processing apparatus according to a first embodiment of the present invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. FIG. 2 is a processing flowchart for explaining a basic concept of a processing method (processing routine) performed by a processing unit of the image processing apparatus shown in FIG. 1. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which comparison data is generated from an arbitrary image. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. The situation in which the comparison data is compared with the blurred original image to be processed to generate difference data. It is a figure for demonstrating. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, in which new comparison data is generated from the generated restored data, the data and the blurred original image to be processed, It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and a diagram for explaining a situation in which newly generated difference data is allocated and new restoration data is generated. is there. It is a figure for demonstrating the problem of the processing method shown in FIGS. FIG. 4 is a diagram for explaining an algorithm performed by the image processing apparatus shown in FIG. 1, and a diagram for explaining the contents of a partially changed algorithm while adopting the concept of the processing method shown in FIG. 3. It is a figure for demonstrating the algorithm used for the image processing apparatus which concerns on the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,1A Image processing apparatus 2 Imaging | photography part 3 Control system part 4 Processing part 5 Recording part 6 Detection part 7 Factor information storage part Io Initial image data (data of arbitrary images)
Io 'comparison data G Change factor information data (degradation factor information data)
Img 'Original image data (captured image)
σ Difference data k Distribution ratio Io + n Restored data (Restored image data)
Img Original correct image data without deterioration (original image)

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a consumer camera. However, the image processing apparatus 1 may be a camera for other uses such as a monitoring camera, a television camera, an endoscopic camera, or an image such as a microscope, binoculars, or NMR imaging. It can also be applied to devices other than cameras, such as diagnostic devices.

  The image processing apparatus 1 includes a photographing unit 2 that shoots a video of a person, the like, a control system unit 3 that drives the photographing unit 2, and a processing unit 4 that processes an image photographed by the photographing unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a factor information storage unit 7 for storing known change factor information that causes image degradation and the like. Note that a display unit including a monitor or the like may be provided in the image processing apparatus 1.

  The imaging unit 2 includes an imaging optical system having a lens and an imaging element such as a CCD (Charge Coupled Devices) and a C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 may store an image serving as a base when generating comparison data described later. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 includes a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD (Digital Versatile Disk), or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the optical system and lens distortion. However, when restoring blurring of camera shake described later, the information is used. Not done.

  Next, the basic concept of the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described with reference to FIG.

  In FIG. 3, “Io” is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4. “Io ′” indicates data of a degraded image of Io of the initial image data, and is comparison data for comparison. “G” is data of change factor information (= deterioration factor information (point image function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “Img ′” refers to captured image data, that is, data of a degraded image, and is data of an original image to be processed in this processing.

“Σ” is difference data between the original image data Img ′ and the comparison data Io ′. “K” is a distribution ratio based on the data G of the change factor information. “Io + n” is restored image data (restored data) newly generated by distributing the difference data σ to the initial image data Io based on the data G of the change factor information. “Img” is an original image that is based on original image data Img ′, which is a deteriorated image that has been taken, and that is the original correct image data without deterioration. Here, it is assumed that the relationship between Img and Img ′ is expressed by the following equation (1).
Img ′ = Img × G (1)
Note that the difference data σ may be a simple difference between corresponding pixels, but generally, the difference data σ differs depending on the data G of the change factor information, and is expressed by the following equation (2).
σ = f (Img ′, Img, G) (2)

  The processing routine of the processing unit 4 starts by preparing arbitrary image data Io (step S101). As the initial image data Io, the photographed degraded image data Img ′ may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. In step S102, data Io of an arbitrary image that is an initial image is inserted instead of Img in the equation (1), and comparison data Io ′ that is a degraded image is obtained. Next, the difference image σ is calculated by comparing the original image data Img ′, which is a deteriorated image, with the comparison data Io ′ (step S103).

  Next, in step S104, it is determined whether or not the difference data δ is equal to or larger than a predetermined value. If the difference data δ is equal to or larger than the predetermined value, a process for generating new restored image data (= restored data) in step S105. I do. That is, the difference data δ is distributed to arbitrary image data Io based on the change factor information data G to generate new restored data Io + n. Thereafter, steps S102, S103, and S104 are repeated.

  If the difference data δ is smaller than the predetermined value in step S104, the process is terminated (step S106). Then, the restored data Io + n at the time when the processing is completed is estimated as data Img of a correct image, that is, an image without deterioration, and the data is recorded in the recording unit 5. The initial image data Io and change factor information data G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.

  The concept of the above processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

Solving as an optimization problem assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same.

  In other words, as shown in FIGS. 4A and 4B, if comparison data Io ′ (Io + n ′) that is approximate to data Img ′ of the original image that is the captured image can be generated, The initial image data Io or the restored data Io + n, which is the original data for generation, is the correct image data Img that is the original of the original image data Img ′ or an approximation thereof.

  In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 μsec. In addition, in this embodiment, the value that is the determination criterion for the difference data σ is “6” when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. In addition, the raw shake data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

  Next, specific examples of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are set to n-1, n, n + 1, n + 2, n + 3,... In order from the left, and attention is paid to a certain pixel n. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the correct image data Img when no deterioration occurs. Each data is represented by 8 bits (0 to 255).

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the n + 1th pixel, and 20% of time is shifted to the n + 2th pixel. . The way of energy dispersion is as shown in the table of FIG. This becomes the data G of the change factor information.

  Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table of FIG. The data shown as “imaging result” in FIG. 8 is the original correct image data Img, and the data shown as “blurred image” is the taken degraded image data Img ′. Specifically, for example, “120” of the pixel “n−3” is in accordance with the distribution ratio of “0.5”, “0.3”, and “0.2” of the data G of the change factor information that is the blur information. It is distributed such that “60” is distributed to “n-3” pixels, “36” is distributed to “n-2” pixels, and “24” is distributed to “n-1” pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this deteriorated image data Img ′ and the change factor information data G shown in FIG.

  Any data Io of the arbitrary image shown in step S101 can be adopted, but in this description, the photographed original image data Img ′ is used. That is, the process starts with Io = Img ′. In the table of FIG. 9, “input” corresponds to the initial image data Io. This data Io, ie, Img ′ is multiplied by the change factor information data G in step S102. That is, for example, “60” of the “n-3” pixel of the initial image data Io is “30” for the n-3 pixel, “18” for the “n-2” pixel, “12” is allocated to each pixel of “1”. The other pixels are similarly distributed, and comparison data Io ′ shown as “output Io ′” is generated. Therefore, the difference data δ in step S103 is as shown in the bottom column of FIG.

  Thereafter, the size of the difference data δ is determined in step S104. Specifically, the processing ends when all the difference data σ is 5 or less in absolute value, but the difference data σ shown in FIG. 9 does not meet this condition, and thus the process proceeds to step S105. That is, the difference data σ is distributed to arbitrary image data Io using the change factor information data G to generate restoration data Io + n shown as “next input” in FIG. In this case, since this is the first time, it is represented as Io + 1 in FIG.

  The distribution of the difference data σ is, for example, “15” obtained by multiplying the data “30” of the pixel “n−3” by 0.5 which is the distribution ratio of the place (= “n−3” pixel). “N-3” pixels are distributed, and “n-2” pixel data “15” is multiplied by 0.3 which is the distribution ratio that should have come to the “n-2” pixel. 4.5 ”is allocated, and the data“ 9.2 ”of the pixel“ n−1 ”is multiplied by 0.2 which is the distribution ratio that should have come to the pixel“ n−1 ”. Allocate “1.84”. The total amount allocated to the pixels of “n-3” is “21.34”, and this value is added to the initial image data Io (here, the captured original image data Img ′ is used) for restoration. Data Io + 1 is generated.

  As shown in FIG. 11, this restored data Io + 1 becomes the input image data (= initial image data Io) in step S102, step S102 is executed, and the process proceeds to step S103 to obtain new difference data σ. . The size of the new difference data σ is determined in step S104. If it is larger than the predetermined value, the new difference data σ is distributed to the previous restored data Io + 1 in step S105 to generate new restored data Io + 2 (FIG. 12). reference). Thereafter, new comparison data Io + 2 ′ is generated from the restored data Io + 2 by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, and the process proceeds to step S105 or shifts to step S106 depending on the determination there. Such a process is repeated.

  In such an algorithm, the present inventor has confirmed that the restoration data Io + n becomes a very good approximation to the original original image when a very large number of loops of tens of thousands of iterations are performed. ing. However, when considering commercialization, tens of thousands of iterations are not realistic. The reason why iterative processing must be repeated tens of thousands of times is because the convergence speed is very slow.

  The reason why the convergence speed is slow is considered as follows. That is, since the restoration data Io + n is evaluated in the degraded image space, the difference in the original original image space is dispersed around and the value is also reduced. Further, the feedback update amount calculated from the difference is also calculated using the data G of the change factor information. However, the feedback update amount becomes a smaller value, and the difference that should not be fed back is also included in the update amount. Therefore, the feedback update amount is small, and in addition, there is an error (an amount that should not be fed back), so the convergence speed is considered to be very slow.

  This will be described with reference to FIG. FIG. 13A shows an example in which the data G of the change factor information is deterioration factor information, and the pixel value “1” is distributed at 10 locations by “0.1”. FIG. 13B is a diagram showing how the pixels A are dispersed based on the data G of the change factor information in FIG. FIG. 13C shows “kσ” in “Io + kσ” when “0” is entered as an arbitrary image in step S101 in FIG.

  When attention is paid to the pixel A in FIG. 13B, the feedback target is set to 10, but the feedback is only “1” as shown in FIG. 13C. Further, the difference that is originally the influence of the pixel A is fed back to the pixels B, C,... Yes. As described above, in the algorithms shown in FIGS. 5 to 12, the feedback amount is calculated even in a region where the feedback amount is very small with respect to the target and should not be fed back originally.

  In this way, when the distribution method shown in FIGS. 5 to 12 is adopted as the distribution ratio k in the processing routine shown in FIG. 3, the amount of feedback is so small that the error once generated does not disappear easily, The effect spreads to the surroundings like a ripple, although the effect is weakening. This is considered to be the cause of the slow convergence speed and the cause of a phenomenon called ringing that occurs near the edge of the image.

  The image processing apparatus 1 according to the present embodiment uses a repetitive process (loop process) that is the basis of the process routine shown in FIG. 3 and dramatically increases the convergence speed. The improved algorithm devised “kσ”, which is the distribution amount.

  First, an outline of the improved algorithm will be described. The deterioration of the image is that the surrounding data is distributed by the data G of the change factor information. Therefore, if the data G of the change factor information is known, it is possible to roughly determine how much data is included in the data of the pixel having the deteriorated image. Therefore, since the ratio of itself in the difference data σ can be roughly estimated, the difference in the original image space (the space between Img and Img ′) can be predicted. Therefore, the feedback amount becomes a value close to the difference in the original image space. In addition, since the predicted difference data σ is used for feedback to its own pixel, feedback that adversely affects the surroundings is not performed.

  The above concept will be specifically described with reference to FIG.

  As shown in FIG. 14A, the data G of the change factor information is composed of α, β, γ, and there is α amount at its own position, β is allocated to the next adjacent pixel, and Assume that γ is distributed to pixels. Here, “α + β + γ = 1”.

  The data of the pixel ao of the original correct image data Img is distributed to the pixels ao ′, a1 ′, a2 ′ of the deteriorated image data Img ′. The pixel ao ′ is ao × α, ao × β is allocated to the pixel a1 ′, and ao × γ is allocated to a2 ′. Similarly, the comparison data Io + n ′ obtained by adding the change factor information data G to the restored data Io + n is a pixel row of bo ′, b1 ′, b2 ′. The pixel bo ′ is bo × α, bo × β is allocated to the pixel b1 ′, and bo × γ is allocated to the pixel b2 ′.

  The value of the pixel a1 is distributed to each pixel a1 ′, a2 ′, a3 ′. A1 × α is allocated to the pixel a1 ′, a1 × β is allocated to the pixel a2 ′, and a1 × γ is allocated to a3 ′. Similarly, the value of the pixel b1 is also distributed to the pixels b1 ′, b2 ′, b3 ′ in the ratio of α, β, γ. The values of the pixels a2 ′, a3 ′, a4 ′... Are similarly distributed, and the values of the pixels b2 ′, b3 ′, b4 ′.

  The difference data amount do is “ao′−bo ′ = ao × α−bo × α = (ao−bo) × α”. As a result, ao−bo = do ÷ α, and ao = bo + do ÷ α. That is, the amount returned to the pixel bo ′ is “do ÷ α”. Similarly, the amount to be returned to the pixel b1 is calculated as follows. That is, “d1 = a1′−b1 ′ = (ao × β + a1 × α) − (bo × β + b1 × α) = (ao−bo) × β + (a1−b1) × α”. Then, “d1− (ao−bo) β = (a1−b1) × α”, and “a1 = b1 + (d1− (ao−bo) × β) ÷ α)”. Here, “ao−bo” is regarded as the difference data amount do and becomes “a1 = b1 + (d1−do × β) ÷ α)”, and the amount returned to the pixel b1 is “(d1−do × β)”. ÷ α) ”.

  Similarly, the amount returned to the pixel b2 is “(d2−d1 × β−do × γ) ÷ α”. When such a return amount is generalized, the return amount to be returned to the pixel bn is (dn−dn−1 × β−dn−2 × γ) ÷ α ”. In this way, in step S105 in the processing flow shown in FIG. 3, as an amount to be restored to the previous restoration data Io + n, the influence of the surroundings is removed from the difference amount of the difference data σ according to the data G of the change factor information, and the value is calculated. By adopting the value divided by the data G of the change factor information, the number of repetitions of the processing routine of FIG. 3 can be greatly reduced. According to the experiment of the present inventor, the processing algorithm of FIGS. 5 to 12 can hardly approximate the original image with several tens of iterations, but the algorithm shown in FIG. It is almost converged by processing.

  In the above case, the feedback amount (return amount) to bn is obtained by dividing the value obtained from the difference data σ by “α”, which is one of the change factor information data G. When this method is adopted, it is possible to efficiently converge when α has the largest weight (the ratio is high) among α, β and γ. For example, when β has the largest weight (the ratio is high), the restoration using the difference data amounts d1, d2, and d3 is “a1 = b1 + (d2−do × γ−d2 × α) / The return amount to the pixel b1 is preferably “(d2−do × γ−d2 × α) ÷ β” using the expression “β”. When this is generalized, the return amount to be returned to the pixel bn is “(dn + 1−dn−1 × γ−dn + 1 × α) ÷ β”. Similarly, when γ has the largest weight (the ratio is high), the return amount to be returned to the pixel bn is “dn + 2−dn + 2 × α−dn + 1 × β) ÷ γ”. When efficient processing is not considered so much, it is not necessary to divide by the one with the largest weight (high ratio).

  In the above example, the change factor information data G is three, α, β and γ, but may be five, seven or ten as long as there are two or more. When each value is generalized so that α is expressed as Psfo and β is expressed as Psf1, it becomes Psfn (n is an integer value of 0 or more). Even when generalized in this way, the above-described concept can be applied, and the return amount is calculated.

  As described above, in the image processing apparatus 1 according to the first embodiment, the difference data σ is divided by the change factor information data G equal to or less than “1”, and thus the return amount is a very large value. End up. For this reason, even if there is a small noise, it expands. In order to cope with this problem, in this embodiment, the surrounding influence included in the data amount of the specific difference is removed, and the data after the removal is used as the data G of change factor information (actually one of them). I'm giving them feedback.

  In addition to such measures, in relation to the reliability of data, the return amount obtained by the above-mentioned concept is multiplied by a predetermined ratio such as 0.3, 0.5, or 0.7. The amount of return may be reduced. When the value less than 1 is multiplied in this way, the convergence speed is slowed down, but it can be restored more reliably. On the other hand, when the obtained return amount is multiplied by a value exceeding 1, the convergence speed can be further improved.

  Next, an image processing apparatus according to a second embodiment will be described with reference to FIG. This image processing apparatus is basically the same in component configuration and processing routine as the image processing apparatus 1 according to the first embodiment, and is different from the above-described algorithm (how to perform the return amount). Only. Therefore, in the following description, the same members as those in the first embodiment are denoted by the same reference numerals. In addition, although the symbol “1A” is assigned to the apparatus according to the second embodiment, this symbol “1A” does not appear on the drawing.

  The image processing apparatus 1A according to the second embodiment uses an iterative process (loop process) that is the basis of the process routine shown in FIG. However, the return amount (kσ in FIG. 3) in the process is as follows. That is, in the data G of the change factor information, pay attention to the heaviest place (the place where the ratio is the highest in the previous example), trust the difference or trust a certain percentage of the difference, and set the value Divide by the data G of the change factor information. Then, the divided value is set as the return amount to the restored data Io + n. Thereafter, the influence of the restored value on the surroundings is removed from the difference data amount (return amount) according to the data G of the change factor information, and the process proceeds to the next pixel. By repeating this, one-time restoration data Io + n for all pixels is obtained.

  This will be specifically described below. Assuming data G of change factor information as shown in FIG. 15A, the data relationship shown in FIG. 15B is obtained. Under such circumstances, if the initial value that is the basis of the comparison data Io + n ′ is “0”, Io ′ is also “0”. Then, σ = Img ′. Examining the four pixels shown in FIG. 15B, “do = ao × α”, “d1 = ao × β + a1 × α”, and “d2 = ao × γ + a1 × β + a2 × α”. d3 is “a1 × γ + a2 × β + a3 × α”.

  As a result, the relationship from the degraded image data Img ′ to the original correct image data Img is “ao = do ÷ α”, “a1 = (d1−ao × β) ÷ α”, “a2 = (d2−a1). × β−ao × γ) ÷ α ”,“ a3 = (d3−a2 × β−a1 × γ) ÷ α ”. When calculating a specific return amount, first, “ao” is calculated from “do” and “α” (both known), so “do ÷ α” is set as the return amount. Next, “a1 = (d1−ao × β) ÷ α” is considered. At this time, “d1”, “α”, and “β” are known, and “ao” is obtained one before. Therefore, “a1” is obtained by substituting the obtained value. Therefore, the value of “a1” is used as the return amount. “D1−ao × β” in this value is obtained by removing “ao × β”, which is the influence of “ao” that is the previous restored data, from d1 that is the difference data amount. is there.

Next, the adjacent pixel a2 (that is, b2) is obtained. As described above, “a2 = (d2−a1 × β−ao × γ) ÷ α”, among which “d2”, “α”, “β”, and “γ” are known. In addition, since “ao” and “a1” are obtained by the previous two processes, “a2” can be obtained. Here, “d2−a1 × β−ao × γ” is “ao × γ” that is an influence of “ao” that is restoration data obtained two times before from d2 that is a difference data amount, This is obtained by removing “ a1 × β” which is an influence by “a1” which is the restored data obtained one time before.

  Similarly, “a3” is obtained from “a3 = (d3−a2 × β−a1 × γ) ÷ α”. In this way, each pixel value of the restoration data Io + n is obtained from the next to the next. In the next loop, the comparison data Io + n ′ and the degraded image (original image) data Img ′ are compared to obtain new difference data σ. If α, β, γ are correct and do, d1,..., Ao, a1,... Are also correct values, Io + 1 = Img is obtained by one restoration process. This is not the case because it contains errors. Therefore, the same repeating process as in FIG. 3 is performed.

  As described above, do, which is the difference data amount between the comparison data Io + 1 ′ (generally Io + n ′) and the degraded image (original image) data Img ′, is “(ao−bo) × α”. Therefore, “ao = bo + do ÷ α”, and the amount returned to the known pixel bo is “do ÷ α”. Similarly, the amount returned to the pixel b1 is “(d1− (ao−bo) × β) ÷ α” as described above. Here, “d1”, “α”, “β”, and “bo” are known, and “ao” is also known by the previous process, and the amount to be returned to the pixel b1 can be obtained. The total value of the returned amount and the amount of the pixel b1 is the value of the pixel a1.

  Similarly, the amount returned to the pixel b2 is “(d2− (a1−b1) × β− (ao−bo) × γ) ÷ α” as described in the description of the first embodiment. Become. Here, “d2”, “α”, “β”, “γ”, “ao”, “bo”, and “b1” are known, and “a1” is also known by the previous processing, and the pixel b2 The amount to return is determined. “D2− (a1−b1) × β− (ao−bo) × γ” in the value of the amount to be returned is the influence of the two previous pixels and the previous one from d2 which is the difference data amount. This value is obtained by removing the influence of pixels. In this way, the more accurate feedback amount is determined by using the value obtained in the previous processing and removing the influence based on the data G of the change factor information from other pixels. It is an algorithm of the image processing apparatus 1A according to the second embodiment. As in the first embodiment, the feedback amount is divided by the data G of the change factor information, the return amount increases, the number of loops decreases, and the convergence speed increases.

  In the above description, for example, ao = do ÷ α, and all the calculated values are fed back. However, the return amount is “(do ÷ α) × 0.5 in relation to the reliability of the value of“ do ”. ”May be halved, 60%, or 30% to increase the reliability of restoration. In the above description, the sizes of α, β, and γ were not described. However, when the weight of the position of itself is large (the ratio is high), that is, when α is larger than β and γ, When α> β> γ, the above-described algorithm (divided by α) is most suitable. However, as in the first embodiment, when β or γ is larger than α, the algorithm is divided by α. Can be used.

  In the image processing apparatuses 1 and 1A, in processing, in step S104, one or both of the number of processes and the determination reference value of the difference data σ can be set in advance. For example, an arbitrary number of times such as three times or ten times can be set as the number of processing times. Further, the value of the difference data σ for stopping the processing is set to “5” in 8 bits (0 to 255), and when it becomes 5 or less, the processing is ended, or “0.5” is set to “0”. .5 "or less, the process can be terminated. This set value can be set arbitrarily. When both the number of processing times and the judgment reference value are input, the processing is stopped when either one is satisfied. When both settings are possible, the determination reference value may be given priority, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

  In the description of each of the embodiments, the information stored in the factor information storage unit 7 is not used. However, known deterioration factors stored here, such as optical aberrations and lens distortions, are stored. May be used. In this case, for example, in the processing method of FIG. 3, it is preferable to perform processing by combining blur information and optical aberration information as one deterioration factor. It is also possible to perform correction with this information. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors at the time of shooting, for example, only blurring.

  The image processing apparatuses 1 and 1A according to the embodiments of the present invention have been described above, but various modifications can be made without departing from the gist of the present invention. For example, the processing performed by the processing unit 4 is configured by software, but may be configured by hardware composed of parts that perform part of the processing.

  In each of the above-described embodiments, the restoration process is performed in order from the left end to the right end. For example, when the change factor information data G flows in the direction from the right end to the left end, from the right end. It is preferable to carry out in order at the left end. Similarly, even if the direction of the data G of the change factor information is from right to left, if α <β <γ, it is preferable to use γ to set the processing order from left to right. Thus, it is preferable to determine the processing order in accordance with the nature of the data G of the change factor information.

  In addition to the photographed image, the original image to be processed may be a photographed image subjected to processing such as color correction or Fourier transform. In addition to the data generated using the change factor information data G, the comparison data includes data generated using the change factor information data G, color correction, or Fourier transform. It is also good. Further, the data of the change factor information includes not only the data of the deterioration factor information but also information that simply changes the image and information that improves the image contrary to the deterioration.

  Further, when the number of processing iterations is set automatically or fixedly on the image processing apparatus 1, 1 </ b> A side, the set number of times may be changed by the data G of the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.

  Furthermore, during the iterative process, if the difference data σ diverges, that is, if it becomes larger, the process may be stopped. For example, it is possible to adopt a method of determining whether or not the light is diverging by observing the average value of the difference data σ and determining that the light is diverging if the average value is larger than the previous value. In addition, if the divergence occurs once, the process may be stopped immediately. However, if the divergence occurs twice, the method may be stopped, or the process may be stopped if the divergence continues for a predetermined number of times. . In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed exceeds 255, the process is stopped. Further, during an iterative process, when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 within the 8-bit range of 0 to 255 is used as input data, it is processed as a maximum value of 255. That is, if the restored data includes an abnormal value (a value exceeding 255 in the above example) other than an allowable value (0 to 255 in the above example), the process is stopped or the restored data When an abnormal numerical value other than the allowable numerical value is included, the abnormal numerical value can be changed to an allowable numerical value and the processing can be continued.

  In addition, when generating restoration data to be an output image, depending on the data G of the change factor information, there may be data that goes out of the area of the image to be restored. In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable to bring that data from the opposite side. For example, when data allocated to a lower pixel is generated from the data of the pixel XN1 (N rows and 1 column) located at the bottom in the area, the position is outside the area. Therefore, the data is processed to be allocated to the pixel X11 (one row and one column) located directly above the pixel XN1. Similarly, the pixel XN2 (N rows and 2 columns) adjacent to the pixel XN1 is assigned to the pixel X12 in the uppermost column directly above (= 1 row and 2 columns adjacent to the pixel X11). In this way, when data that is outside the restoration target area is generated when the restoration data is generated, the restoration target at a position opposite to one of the vertical, horizontal, and diagonal directions of the data generation position is generated. If it is arranged within the area, it is possible to reliably restore the target area to be restored.

  In addition, any one of the various algorithms described above, for example, processing in the opposite direction, processing of multiplying a certain ratio, and the like are stored in the processing unit 4 to select the user or the type of image or the change factor The processing method may be selected automatically or manually according to the property of the information data G. Also, any one of these methods can be selected and used alternately or in sequence every routine, or the first few times can be processed in a certain method, and then the other methods can be processed. good. The image processing apparatuses 1 and 1A may have a processing method different from any one or more of the various algorithms described above.

  Moreover, each processing method mentioned above may be programmed. The program may be stored in a storage medium such as a CD (Compact Disc), a DVD, or a USB (Universal Serial Bus) memory so that it can be read by a computer. In this case, the image processing apparatus 1 has reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading a program in the storage medium.

Claims (9)

  In an image processing apparatus having a processing unit that processes an image, the processing unit generates comparison data from arbitrary image data using data of change factor information that causes image change, The original image data and the comparison data are compared, and the influence of the image change from the surrounding pixels on a certain pixel is removed from the obtained difference data according to the data of the change factor information. The restoration data is generated by dividing the value by the data of the change factor information and the value less than 1 and added to the comparison data, and the restoration data is used instead of the arbitrary image data. Then, by repeating the same processing, the image processing apparatus performs processing for generating restoration data that becomes data of an image before the original image change or an image similar thereto In an image processing apparatus having a processing unit that processes an image, the processing unit generates comparison data from arbitrary image data using data of change factor information that causes image change, The original image data and the comparison data are compared, and the obtained difference data is directly or multiplied by a value less than 1 and divided by the value of the change factor information and less than 1. An image from a specific pixel whose restoration data value has already been calculated for a pixel whose restoration data value has not been calculated. according to the data of the change factor information affected by the change, the process of removing from the data of the difference by repeating for all pixels, thereby generating a restoration data, image the restored data of the arbitrary It was used instead of chromatography data, by repeating the same processing, image processing apparatus and performs a process of generating restored data as the data of an image to be approximate to the image or before the change of the original image.   The processing unit stops if the difference data when the number of repetitions reaches a predetermined number is less than or equal to a predetermined value or less than a predetermined value, and exceeds or exceeds a predetermined value. 3. The image processing apparatus according to claim 1, wherein the processing is further repeated a predetermined number of times.   In an image processing apparatus having a processing unit for processing an image, the processing unit generates data for comparison from data of a predetermined image using data of change factor information that is a factor of image change. The comparison image is compared with the original image data in which the image is changed, and if the obtained difference data is less than or equal to a predetermined value or smaller than the predetermined value, the process is stopped, A predetermined image is treated as an image before the change of the original image or an image similar thereto, and when the difference is larger than a predetermined value or larger than a predetermined value, the surrounding data for a certain pixel is calculated from the difference data. A return that removes the influence of the image change from the pixel according to the data of the change factor information, divides the value by the data of the change factor information and less than 1 and adds it to the comparison data And doing, thereby generating a restoration data, the image processing apparatus and performs a process of repeating the same process by replacing the restored data to the predetermined image. In an image processing apparatus having a processing unit for processing an image, the processing unit generates data for comparison from data of a predetermined image using data of change factor information that is a factor of image change. The comparison image is compared with the original image data in which the image is changed, and if the obtained difference data is less than or equal to a predetermined value or smaller than the predetermined value, the process is stopped, and the above-mentioned data that is the source of the comparison data A value obtained by treating a predetermined image as an image before the change of the original image or an image similar thereto, and when the difference is larger than a predetermined value or larger than a predetermined value, the difference data as it is or multiplied by a value less than 1 the divided by the value of the data in a by and less than 1 of the variation factor information and returning amount to be added to the comparative data, defined as the value of the recovered data for a particular pixel, the value of the recovered data is calculated By repeating the process of removing the influence of the image change from the specific pixel whose restoration data value has already been calculated for all the pixels according to the change factor information data for all the pixels, An image processing apparatus for generating restored data, performing a process of repeating the same process by replacing the restored data with the predetermined image.   The image processing apparatus according to claim 4, wherein the processing unit performs a process of stopping when the number of repetitions reaches a predetermined number during the repetition processing.   The image processing apparatus according to claim 1, further comprising: a detection unit that detects the change factor information; and a factor information storage unit that stores known change factor information.   8. The image according to claim 1, wherein a value used when dividing by the data of the change factor information is a value having the largest weight among the data of the change factor information. Processing equipment.   9. The image processing apparatus according to claim 1, wherein an order of calculating the return amount depends on data characteristics of the change factor information.

Images