WO2007032148A1 - Image processor - Google Patents

Abstract

An image resembling a completely restored image is displayed in a short time after the imaging, and the contents of the image to be processed, e.g., recorded are confirmed in a short time. An image processor comprises a processing section for restoring an image before a change, or an image to be intentionally captured, or a resembling image (base image) resembling them. The processor executes the steps of acquiring an original image reduction data by reducing the number of pixels of the original image (step S201), restoring reduction restoration data reassembling the reduced base image before a change from the original image reduction data (step S202), and displaying the reduction restoration data on the display unit. It is preferable that the steps (step S203, step S204, step S205) of restoring the base image by using the data acquired by the restoration. The display section is e.g., a monitor of a camera.

Description

 Specification

 Image processing device

 Technical field

 [0001] The present invention relates to an image processing apparatus.

 Background art

 Conventionally, it is known that when a subject is photographed by an image processing apparatus such as a camera, the recorded image sometimes deteriorates. The cause of image degradation is camera shake during shooting. The first method for correcting image degradation due to camera shake is a circuit processing method. This circuit processing method acquires a transfer function that represents the degradation state of a captured image, performs inverse transformation of the acquired transfer function on the captured image, and corrects the captured image (see Patent Document 1). .

 [0003] As a second method for correcting image deterioration due to camera shake, there is a method of moving a lens. For example, as a method of moving a lens, there is known a method in which camera shake is detected and corrected by moving a predetermined lens in accordance with the detected camera shake (see Patent Document 2).

 [0004] Patent Document 1: JP-A-11 24122 (refer to abstract)

 Patent Document 2: JP-A-6-317824 (see abstract)

 Disclosure of the invention

 Problems to be solved by the invention

[0005] Adopting the first method has the advantage of shortening the time required for correction, but also has the following disadvantages. In other words, the transfer function to be acquired is very vulnerable to noise and blur information errors. For this reason, the corrected image obtained by the inverse transformation is far from an image photographed with no camera shake, and cannot be used in practice. In addition, when performing inverse transformation considering noise etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size, There is a high risk that it will not be solved in practice. Cameras that use the second method require a large amount of hardware, such as a motor, that drives the lens. It will become. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

 [0006] On the other hand, a person who operates a camera or the like may want to check immediately after shooting how a recorded image is recorded on a display unit (motor) of the digital camera or the like immediately after shooting. The The reason is that if the image that can be confirmed on the monitor is inferior immediately after photographing, there is a high possibility that the retaking can be performed in consideration of correction of the inferiority.

 Therefore, the problem to be solved by the present invention is to display an image that approximates an image that has been completely corrected in a short time after shooting, and to confirm the content of the image that is subsequently processed such as recording. An image processing apparatus configured as described above is provided.

 Means for solving the problem

 [0008] In order to solve the above-described problem, the image processing apparatus of the present invention provides an original image power to be processed, an image before a change, an image that should have been originally taken, or a similar image thereof (hereinafter referred to as an image). In the image processing apparatus having a processing unit for performing processing for restoring the original image and i), the processing unit obtains original image reduced data obtained by subtracting the number of pixels from the original image, and based on the original image reduced data. The reduced restoration data approximated to the original reduced image before the change is restored, and the reduced restoration data is displayed on the display unit.

 According to the present invention, the amount of image data to be corrected is reduced by reducing the number of original image power pixels. Therefore, the time required for restoration such as camera shake correction can be shortened, and restoration can be completed in a short time after photographing. Then, by displaying the reduced and restored data after the restoration on the display unit, it is possible to confirm the contents of the image to be processed thereafter such as recording in a short time after the photographing.

 [0010] It is very preferable that the display unit is a monitor. This is because the reduced / restored data after restoration displayed on the display unit is smaller than the original image by the number of pixels, and becomes an image. However, for example, a monitor included in a digital camera, a digital video camera, or the like usually has a display area with a smaller number of pixels than a captured image that is actually recorded. Therefore, it is preferable when checking what kind of image is taken and how it is recorded on the monitor of each kind of camera.

In another invention, in addition to the above-described invention, the processing unit is obtained when restoring the reduced restoration data. The original data is restored using the obtained data. In the process of restoring from the original image to the original image, it is possible to efficiently restore the original image to the original image by using the data obtained in the process of restoring the original image reduced data to the reduced restored data. It will be possible.

 [0012] Further, another invention is based on the above-described invention, and the processing unit uses the data of the change factor information that causes the image change as a process of restoring the reduced restoration data, and performs predetermined processing. Image data power Comparison data is generated, the comparison data is compared with the original image reduced data, and reduced restoration data is generated using the obtained difference data. Data is used instead of predetermined image data, and thereafter, the obtained reduced / restored data is replaced with the previous reduced / restored data, and the same processing is repeated.

 [0013] According to the present invention, the reduced restoration data approximated to the original image is generated only by generating predetermined data using the image change factor information. There is almost no equipment that does not increase in size. Also, comparison data is created from the reduced and restored data, and the comparison data is compared with the original image reduced data to be processed, and the reduced and restored data close to the original image is gradually obtained. Restoration work. For this reason, an image processing apparatus having a realistic circuit processing method can be provided for image restoration.

 [0014] Note that the processing unit may perform a process of stopping if the difference data becomes equal to or smaller than a predetermined value or smaller than a predetermined value during the repeated processing. When this configuration is adopted, the processing is stopped even if the difference does not become “0”, so that it is possible to prevent a long processing time. In addition, since the value is less than the predetermined value, the approximated reduced / restored data is closer to the original reduced image before the change (before deterioration, etc.) that is the original of the original image. In addition, when there is noise, etc., there is a tendency that a situation where the difference cannot be “0” in reality is likely to occur. Don't be.

[0015] Furthermore, the processing unit may perform a process of stopping when the number of repetitions reaches a predetermined number during the repetition process. When this configuration is adopted, the processing is stopped regardless of whether the difference becomes “0”, so that it is possible to prevent the processing from taking a long time. In addition, since the processing is continued up to a predetermined number of times! / It becomes closer to the reduced original image before deterioration. In addition, when there is noise, etc., a force that tends to cause a situation where the difference does not become “0” in reality. Even in such a case, the process is terminated a predetermined number of times, so the process is repeated infinitely. It doesn't matter.

 [0016] Further, the processing unit, in the case of repeated processing, stops if the difference data when the number of repetitions reaches a predetermined number is less than a predetermined value or smaller than a predetermined value, exceeds the predetermined value or If the value is equal to or greater than the predetermined value, the process may be repeated a predetermined number of times. If this configuration is adopted, the number of processes and the difference value are combined, so the image quality is better than when the number of processes is simply limited or when the difference value is limited. And a process that balances the shortness of the processing time.

 [0017] Further, in another invention, in addition to the image processing apparatus of the above invention, the processing unit uses data of change factor information that causes an image change as a process of restoring the reduced restoration data, Data power of a predetermined image Comparison data is generated, the comparison data is compared with the original image reduced data, and if the obtained difference data is greater than or equal to a predetermined value, the difference data Reduced restoration data is generated using the data, the reduced restoration data is replaced with a predetermined image, and thereafter, the obtained reduced restoration data is replaced with the previous reduced restoration data and the same process is repeated! ヽProcessing to generate reduced and restored data that approximates the original reduced image before changing to the original image reduced data, and if the difference data is less than a predetermined value or smaller than a predetermined value, the processing is stopped. To have.

 [0018] According to the present invention, the reduced restoration data approximate to the original image is generated only by generating predetermined data using the image change factor information. There is almost no equipment that does not increase in size. Also, comparison data is created from the reduced and restored data, and the comparison data is compared with the original image reduced data to be processed, and the reduced and restored data close to the original image is gradually obtained. Restoration work. For this reason, an image processing apparatus having a realistic circuit processing method can be provided for image restoration. In addition, since the processing is stopped when the difference data becomes small, the processing can be stopped with a certain number of processing times.

[0019] Note that the processing unit may perform a process of stopping when the number of repetitions reaches a predetermined number during the repetition processing. When this configuration is adopted, even if the difference is "0" Since the processing is stopped even if it does not occur, it is possible to prevent the processing from taking a long time. Further, since the processing is continued up to a predetermined number of times! /, The approximated reduced / restored data is closer to the original reduced image that is the original original reduced data. Furthermore, when there is noise, etc., the situation in which the difference does not become “0” tends to occur in reality, but in such a case, the processing is repeated indefinitely, but this configuration is adopted. Then, such a problem does not arise.

 [0020] Further, another invention is based on the above-described invention, and acquires a transfer function indicating the degradation state of the original image when restoring the reduced restoration data, and the acquired transfer function is obtained for the original image. Inverse conversion is performed. According to the present invention, it is possible to speed up restoration of an original image based on an appropriate transfer function. In addition, since iterative processing to obtain reduced restoration data and deconvolution processing using a transfer function are combined, the apparatus does not increase in size and becomes a realistic restoration work. Processing is speeded up.

 In another invention, in addition to the above-described invention, a transfer function is obtained from the original image reduced data and the reduced and restored data, and an original image before changing to the original image is generated using the transfer function. It is carried out. By adopting this configuration, it is possible to speed up the restoration of the original image based on an appropriate transfer function. In addition, since the iterative process for obtaining the reduced restoration data and the deconvolution process using the transfer function are combined, the apparatus does not increase in size and becomes a realistic restoration work. Is faster.

 [0022] Still another invention is based on the above-described invention, and the original image reduced data is formed by thinning out the data of the original image, and the processing unit uses the transfer function as the original image reduced data. The original image power of the image is reduced to a reciprocal of the reduction ratio, and the enlarged image is interpolated to obtain a new transfer function, and the new transfer function is used to generate restored data that approximates the original image. . If this configuration is adopted, a transfer function corresponding to the whole picture can be obtained.

 [0023] In addition to the above-described invention, another invention assumes that the original image reduced data is formed by extracting a part of the area from the original image data as it is. If this configuration is adopted, a transfer function corresponding to a partial region and applicable to the entire image can be obtained.

[0024] In addition to the above-mentioned inventions, the invention further reduces the original image restoration processing to (1) the display unit. In parallel with the process of displaying the restored data, this is done when (2) the imaging power is turned off or (3) after the process of displaying the reduced restored data on the display.

[0025] According to the above invention (1), it is preferable when the original image restoration processing is executed as soon as possible. In addition, according to the inventions of (2) and (3) described above, even if a long time is required for the restoration process of the original image, the original image is copied during a period other than the restoration process period of the original image reduced data. By executing the restoration process, the burden on the image processing apparatus can be reduced. Even if processing for delaying the execution time of restoration processing of the original image is performed in this way, the image displayed on the display unit can be displayed as reduced restoration data in a short time after shooting. There is no inconvenience or harmful effect for those who operate an image processing apparatus having a photographing function.

 [0026] Further, in addition to the above-described invention, in another aspect of the invention, the processing unit displays the reduced / restored data on the display unit according to the value of the change factor information that causes the image change. The processing is divided into the case where the data is displayed as it is. According to the present invention, in the case of an image that has undergone remarkable changes such as deterioration, the restored image is displayed on the display unit, and in the case of an image that has no deterioration, the restoration operation is not performed and the reduced image is displayed as it is. The For this reason, restoration work is performed only when necessary, and the processing load can be reduced.

 The invention's effect

 [0027] According to the present invention, an image processing that displays an image that approximates an image that has been completely corrected in a short time after shooting, and that allows the content of an image to be processed thereafter, such as recording, to be confirmed in a short time. A physical device can be obtained.

 Brief Description of Drawings

 FIG. 1 is a block diagram showing a main configuration of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is an external perspective view showing an outline of the image processing apparatus shown in FIG. 1, and is a view for explaining an arrangement position of angular velocity sensors.

3 is a process flow diagram for illustrating a processing method performed in the processing unit in the image processing apparatus (processing routine) shown in FIG. 1 _n 4 is a diagram for explaining the concept of the processing method shown in FIG.

 FIG. 5 is a diagram for specifically explaining the processing method shown in FIG. 3 using hand shake as an example, and a table showing energy concentration when there is no hand shake.

 FIG. 6 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram showing image data when there is no camera shake.

 FIG. 7 is a diagram for specifically explaining the processing method shown in FIG. 3 with an example of camera shake, and is a diagram showing energy dispersion when camera shake occurs.

 FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which data for comparison is generated with any image force.

[FIG. 9] A diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. Comparison data is compared with the blurred original image to be processed, and difference data is obtained. It is a figure for demonstrating the condition to produce | generate.

 FIG. 10 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, and explains the situation in which restored data is generated by allocating the difference data and adding it to an arbitrary image. FIG.

 [FIG. 11] A diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake. New comparison data is generated from the generated restored data, and the data and processing target are generated. It is a figure for demonstrating the condition which compares the blurred original image and produces | generates the data of a difference

[Fig. 12] A diagram for specifically explaining the processing method shown in Fig. 3 by taking an example of camera shake, and explaining the situation in which newly generated difference data is allocated and new restoration data is generated. FIG.

圆 13] This is a diagram for explaining processing using the center of gravity of the change factor, which is the first processing method using the processing method shown in Fig. 3. (A) focuses on one pixel in the correct image data. (B) is a diagram showing a state in which the data of the pixel of interest is expanded in the diagram showing the data of the original image.

[14] FIG. 14 is a diagram for specifically explaining the processing using the center of gravity of the change factor, which is the first processing method shown in FIG. FIG. 15 is a diagram for explaining the first of the second processing methods using the processing method shown in FIG. 3 and achieving high speed. FIG. 15 (A) shows the original image data to be processed. (B) is a figure which shows the data which thinned out the data of (A).

FIG. 16 is a flowchart of the second processing method shown in FIG.

FIG. 17 is a diagram for explaining a second method of speeding up the third processing method using the processing method shown in FIG. 3, and (A) shows data of the original image to be processed. (B) is a figure which shows the data which extracted some data of (A).

FIG. 18 is a flowchart of the third processing method shown in FIG.

FIG. 19 is a diagram for explaining a modification of the third processing method shown in FIG. 17 and FIG. 18, in which the original image data is divided into four parts, and a part of the areas for iterative processing from each divided area. It is a figure which shows taking out.

Explanation of symbols

 1 Image processing device

 2 Shooting section

 3 Control system

 4 Processing section

 5 Recording section

 6 Detector

 7 Factor information storage

 8 Display section

 Io Initial image data (any image data)

 Ιο 'Comparison data

 G Change factor information data (degradation factor information data)

 GS Reduced change factor information data

 Img 'Original image data (captured image)

 ISmg 'original image reduction data

 δ Difference data

k Allocation ratio Io + n Restored data (Restored image data)

 ISo + n reduced restoration data

 Img No deterioration! Original correctness! / Image data (original image)

 ISmg Reduced original image

 g (x), g '(x), g2 (x) transfer function (transfer function to restore a large image) gl (x), gl' (x) transfer function (transfer obtained from reduced data) Function) BEST MODE FOR CARRYING OUT THE INVENTION

 Hereinafter, the image processing apparatus 1 according to the first embodiment of the present invention will be described with reference to the drawings. Although this image processing apparatus 1 is a consumer camera, it may be a camera for other uses such as a surveillance camera, a television camera, an endoscopic camera, a microscope, binoculars, and NMR imaging. The present invention can also be applied to devices other than cameras, such as diagnostic imaging devices.

 [0031] The image processing apparatus 1 includes a photographing unit 2 that captures images of a person and the like, a control system unit 3 that drives the photographing unit 2, a processing unit 4 that processes images captured by the photographing unit 2, have. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records the image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a display unit 8 serving as a monitor. The detection unit 6 includes a factor information storage unit 7 that stores known change factor information that causes image degradation.

 [0032] The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD (Charge Coupled Devices) ^ C—MOS (Complementary Metal Oxide Beam Semiconductor) that converts light passing through the lens into an electrical signal. It is. The control system unit 3 controls each unit in the image processing apparatus 1 such as the photographing unit 2, the processing unit 4, the recording unit 5, the detection unit 6, the factor information storage unit 7, and the display unit 8.

The processing unit 4 is composed of an image processing processor, and is composed of hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 may store an image serving as a base when generating comparison data to be described later. The processing unit 4 may be configured to process with software rather than configured as hardware such as an ASIC. The recording unit 5 is composed of a semiconductor memory, but adopts magnetic recording means such as a hard disk drive, optical recording means using a DVD (Digital Versatile Disk), etc. You may do it.

 As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X and Y axes that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. Is provided. By the way, camera shake when shooting with the camera is the force that also causes movement in the X, Y, and Z directions and rotation around the Z axis. Rotation and rotation around the X axis. These two variations are only a slight variation, and the captured image is greatly blurred. Therefore, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. For the sake of completeness, an additional angular velocity sensor around the Z axis or a sensor that detects movement in the X or Y direction may be added. In addition, the sensor used may be an angular acceleration sensor that is not an angular velocity sensor.

 The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the optical system and lens distortion. The information is used when restoring blurring of camera shake described later. Not done. The display unit 8 obtains the reduced / restored data, which is image data after correction (after restoration) of the image obtained by subtracting the number of pixels from the captured image, from the recording unit 5, and displays the reduced / restored data as a reduced original image. To do. As shown in FIG. 2, the display unit 8 is often disposed on the surface opposite to the image capturing unit 2, but it may be disposed on the same side as the image capturing unit 2 or on the side surface. good.

 Next, an outline of the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described with reference to FIG.

 In FIG. 3, “Io” is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4. “Γΐο '” indicates the data of the degraded image of 初期 ο of the initial image data, and is comparative data for comparison. “G” is data of change factor information (= deterioration factor information (point spread function)) detected by the detection unit 6, and is stored in the recording unit of the processing unit 4. “Img ′” indicates captured image data, that is, data of a degraded image, and is data of an original image to be processed.

“Δ” is difference data between the original image data Img ′ and the comparison data Io ′. "K "Is a distribution ratio based on data of change factor information. “Io + n” is restored image data (restored data) newly generated by allocating the difference data δ to the initial image data Io based on the data of the change factor information. rimgj is the original correct image data with no deterioration, which is the basis of the original image data Img ^, which is the deteriorated image taken. Here, the relationship between Img and Img 'is expressed by the following equation (1).

[0039] Img '= Img X G ... hi)

 [0040] Note that the difference data δ may be a simple difference between corresponding pixels, but generally, the difference data δ differs depending on the data G of the change factor information, and is expressed by the following equation (2).

[0041] δ = f (Img ', Img, G) --- (2)

 [0042] The processing routine of the processing unit 4 starts by preparing arbitrary image data Io (step S101). As the initial image data Io, it is possible to use the image Img 'of the deteriorated image that has been taken, or any image data such as a black solid, a white solid, a gray solid, or a pine pattern. . In step S102, the data Io of an arbitrary image to be an initial image is input instead of Img in the equation (1), and comparison data Io ′ that is a degraded image is obtained. Next, the data Img ′ of the original image, which is the captured degraded image, is compared with the comparison data I, and difference data δ is calculated (step S103).

 Next, in step S 104, it is determined whether or not the difference data δ is equal to or greater than a predetermined value. If the difference data δ is equal to or greater than the predetermined value, a new restored image data (= restored data) is determined in step S 105. ) Process to generate. That is, the difference data δ is distributed to arbitrary image data Io based on the change factor information data G to generate new restored data Io + n. Thereafter, steps S102, S103, and S104 are repeated.

If the difference data δ is smaller than the predetermined value in step S104, the process is terminated (step S106). Then, the restored data Io + n at the end of the processing is estimated as the correct image, that is, the data Img of the image without deterioration, and the data is recorded in the recording unit 5. The recording unit 5 may record the initial image data Io and the change factor information data G, and pass them to the processing unit 4 as necessary. Here, the original image, the correct image that should have been originally taken, or an approximate image thereof is referred to as an “original image”. Therefore, the data Img and the restored data Io + n are “original image” data. [0045] The concept of the above processing method is summarized as follows. In other words, in this processing method, the processing solution is not solved as an inverse problem, but as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

 [0046] Solving as an optimization problem is based on the following conditions.

 That is,

 (1) The output corresponding to the input is uniquely determined.

 (2) If the output is the same, the input is the same.

 (3) The solution is converged by iteratively updating the input so that the output is the same.

 In other words, as shown in FIGS. 4 (A) and 4 (B), the comparison data Ιο ^ (Ιο + η ') is approximate to the data Img' of the original image that was taken. Can be generated, the initial image data Ιο or restored data Ιο + η, which is the original data for the generation, is approximate to the positive U, the image data Img, which is the original image data I mg '.

 [0048] In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 seconds. Also, the value used as the determination criterion for the difference data δ is “6” in this embodiment when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. In addition, the shake data detected by the angular velocity detection sensor does not correspond to actual shake when the sensor itself is not calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, a correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

 Next, the details of the processing method shown in FIGS. 3 and 4 will be described with reference to FIGS. 5, 6, 7, 8, 8, 9, 10, 11 and 12. FIG.

 [0050] (Image restoration algorithm)

When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed in the blurred pixels during the exposure time. Furthermore, if the blur during the exposure time is known, the manner in which the energy is dispersed during the exposure time can be clearly understood, so that it is possible to produce an image with no blurring of image blur. [0051] Hereinafter, for simplicity, the description will be made in one horizontal dimension. From the left, the pixels are n-1, n, n + 1, n + 2, n + 3,. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in Fig. 5. The table of Fig. 6 shows the shooting results at this time. The power shown in Fig. 6 is the correct image data Img when there is no deterioration. Each data is expressed as 8-bit (0 to 255) data.

 [0052] There is a blur during the exposure time, 50% of the exposure time is at the nth pixel, 30% is at the n + 1st pixel, 20% is at the n + 2th pixel, Suppose that each was blurred. The way of energy distribution is shown in the table shown in Fig. 7. This becomes the data G of the change factor information.

 [0053] Since blurring is uniform for all pixels, assuming that there is no top blur (vertical blurring), the blurring situation is as shown in the table in FIG. The data force Img of the correct image data shown as “shooting result” in FIG. 8 becomes the data force Img ′ of the deteriorated image taken as the data force “blurred image”. Specifically, for example, “120” of the pixel “n−3” is determined according to the distribution ratio of “0.5J”, “0.3”, “0.2” in the data G of the change factor information that is the blur information. It is distributed as “60” to n-3 pixels, “36” to “n-2” pixels, and “24” to “n-1” pixels. Similarly, “60”, which is pixel data of “n−2”, is distributed as “30” in “n−2”, “1 8” in “n−1”, and “12” in “n”. . This deteriorated image data Im and the change factor information data G shown in FIG. 7 are calculated.

[0054] Any image data Io shown in step S101 can be used. For this description, the photographed original image data Img 'is used. That is, the process is started as I o = Img ′. In the table in Fig. 9, “input” corresponds to the data Io of the initial image. This data Io, ie, Img ', is multiplied by the change factor information data G in step S102. That is, for example, “60” of the “n−3” pixel of the initial image data Io is “30” for the n−3 pixel, “18” for the “n−2” pixel, “12” is assigned to each “1” pixel. The other pixels are similarly allocated to generate comparison data Io ′ shown as “output Io ′”. Therefore, the difference data δ in step S103 is as shown in the bottom column of FIG. [0055] Thereafter, the size of the difference data δ is determined in step S104. Specifically, the power to end the processing when all the difference data δ becomes 5 or less in absolute value. Since the difference data δ shown in FIG. 9 does not meet this condition, the process proceeds to step S105. In other words, the difference data δ is distributed to arbitrary image data Ιο using the change factor information data G, and the restored data Ιο + η shown as “next input” in FIG. 10 is generated. In this case, since this is the first time, Io + l is shown in FIG.

 For example, the difference data δ is distributed by multiplying the data “30” of the pixel “η−3” by 0.5, which is the distribution ratio of its place (= the pixel “η−3”). 15 ”is distributed to the pixel“ η−3 ”, and the data“ 15 ”of the pixel“ η 2 ”is assigned the distribution ratio 0.3 that should have come to the pixel“ η−2 ”. “4.5” is distributed, and the data “9.2” of the pixel “η−1” is allocated to the distribution ratio 0.2 that should have come to the pixel “η−1”. Allocate “1.84”. The total amount allocated to the “η-3” pixel is “21. 34”, and this value is added to the initial image data Ιο (here, the original image data Img 'was used). The restoration data Io + 1 is generated.

 [0057] As shown in FIG. 11, the restored data Io + l is the input image data (step S102).

 = Initial image data Io), step S102 is executed, and the process proceeds to step S103 to obtain new difference data δ. The size of the new difference data δ is determined in step SI 04, and if it is larger than the predetermined value, in step S 105, the new difference data δ is allocated to the previous restoration data Io + l, and the new restoration data Io + 2 (See Figure 12). After that, by performing step S102, new comparison data Io + 2 ′ is generated from the restored data Io + 2. As described above, after steps S102 and S103 are executed, the process goes to step S104, whereupon the process proceeds to step S105 or the process proceeds to step S106. Such a process is repeated.

In this image processing apparatus 1, before processing, in step S 104, either or both of the number of processes and the determination reference value of the difference data δ can be set in advance. For example, the number of processing can be set to any number such as 20 or 50 times. Also, set the value of the difference data δ to stop processing to “5” in 8 bits (0 to 255). When it becomes 5 or less, the processing is terminated or set to “0.5”. The process can be terminated when the value falls below "0.5". wear. This set value can be set arbitrarily. If both the number of processing times and the criterion value are entered, the processing is stopped when either one is satisfied. When both of these settings are possible, the determination reference value may be prioritized, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

 In the description of this embodiment, the force that did not use the information stored in the factor information storage unit 7, such as known deterioration factors stored here, such as optical aberrations and lens distortions, etc. Data may be used. In this case, for example, in the processing method in the previous example (Fig. 3), it is preferable to perform processing by combining blur information and optical aberration information as one deterioration factor! After the processing in step S5 is completed, correction may be performed using information on optical aberration. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors during shooting, for example, only blurring.

 [0060] As described above, the basic configuration of the image processing apparatus 1 according to the embodiment of the present invention and the power that explains the basic routine of the restoration process. In the following, an example or a modification using the image processing apparatus 1 will be described.

 [0061] For example, when generating the restored data Io + n, the distribution ratio k is not used, and the difference data δ of the corresponding pixel is directly added to the corresponding pixel of the previous restored data Io + n-1; The data k S (value indicated as “update amount” in FIGS. 10 and 12) after adding the difference data δ of the corresponding pixel after scaling or adding the difference data δ is added. You can zoom in and add it to the previous restored data Io + n— 1. When these processing methods are used well, the processing speed increases.

 [0062] When the restoration data Io + n is generated, the center of gravity of the change factor such as deterioration is calculated, and the difference of only the center of gravity or the scaling of the difference is added to the previous restoration data Io + n-1. You may do it. This concept, that is, the processing method using the center of gravity of the change factor will be described below as the first processing method using the processing method shown in FIG. 3 with reference to FIGS.

As shown in FIG. 13, when the correct image data Img is composed of pixels 11 to 15, 21 to 25, 31 to 35, 41 to 45, 51 to 55, it is shown in FIG. As you look at pixel 33 The When moving to the position of pixel 33 force pixel 33, 43, 53, 52 due to camera shake etc., the original image data Img 'which is a poor image, as shown in Fig. 13 (B), The pixels 33, 43, 52 and 53 are affected by the first pixel 33.

 [0064] In the case of such deterioration, if the pixel 33 moves and is located at the position of the pixel 43 for the longest time, the center of deterioration, that is, the cause of the change is the pixel 33 in the correct image data Img. In the original image data Img ', pixel 43 is located. As a result, the difference data δ is calculated as the difference between the pixels 43 of the original image data Img ′ and the comparison data Io ′ as shown in FIG. The difference data δ is added to the pixel 33 of the initial image data Io and the restoration data Ιο + η.

 Further, in the above example, the three centroids of “0.5”, “0.3”, and “0.2” have the largest value “0.

 5 ”position and will be my position. Therefore, the allocation of “0.5” or “0.5” of the difference data δ is allocated to its own position without considering the allocation of “0.3” or “0.2”. Such a process is suitable when the blur energy is concentrated.

 [0066] Furthermore, each processing method described above can be automatically selected according to the contents of the data G of the change factor information. That is, the processing unit 4 can classify the data G of the change factor information into one of a plurality of types, and perform different processing for each classification. For example, as a processing method, (1) a method of allocating difference data δ using an allocation ratio k as shown in FIGS. 5 to 12 (embodiment method), (2) a corresponding pixel difference, Alternatively, a program that can execute three methods: scaling the difference data δ (corresponding pixel method), (3) detecting the centroid of the deterioration factor, and using the data of the centroid (centroid method). Save it in the processing unit 4, analyze the situation of the deterioration factor, and select one of the three methods based on the analysis result. Also, you can select any one of the three methods and use them alternately for each routine, or process the first few times using a certain method and then use another method. .

In this image processing apparatus 1, the restored image is displayed on the display unit 8. However, if the captured degraded image is restored as it is, the processing time becomes long, and it cannot be displayed on the display unit 8 immediately after shooting. Therefore, the restoration process is speeded up. Hereinafter, this high speed key will be described. There is a method combined with the inverse problem in order to increase the speed of the restoration process. In other words, iterative processing is performed on the reduced data, and a transfer function from the reduced original image to the reduced restored data is calculated. Then, the calculated transfer function is enlarged and interpolated, and the restored data of the original image is obtained using the enlarged and interpolated transfer function. This processing method is advantageous for processing large images.

 [0069] The basic concept of high-speed processing that is advantageous for restoring a large image will be described below.

 [0070] With only iterative processing, it takes time to converge. This disadvantage is noticeable for large images. On the other hand, deconvolution in the frequency space is very attractive because it can perform high-speed calculations using Fast Fourier Transform (FFT). Optical deconvolution as used herein refers to restoring the original image by removing the image power that has deteriorated due to distortion, blurring, etc., and removing the distortion.

[0071] In the case of an image, when the input is in (x), the output is ou (x), and the transfer function is g (x), in the ideal state, the output ou (x) is convolution integration,

 ou (x) = J in (t) g (x-t) dt --- (3)

 It becomes. Note that “ί” is an integration symbol. This equation (3) is 0 (u) = l (u) G (u)-(4)

 It becomes. From this known output ou (x), the transfer function g (x) or unknown input in (X) is deconvolution, and for this purpose, in frequency space, I (u) = 0 If (u) / G (u) is obtained, the unknown input in (X) can be obtained by returning it to the real space.

However, in actuality, due to noise or the like, Equation (3) becomes “ou (x) + a (x) = J in (t) g (x−t) dt + a (x)”. Here, “ou (x) + a (x) J is known, but each of ou (x) and a (x) is unknown. Even if this is solved approximately as an inverse problem, it is sufficient. It is practically difficult to obtain a satisfactory solution, so ou (x) + a (x) = J in (t) g (x—t) dt + a (x) ^ J jn (t) g ( xt) dt, jn (x) is converged using an iterative method! /, and the process flow of FIG. 3 described above is obtained. Here, if “α (χ) << ou (x)”, it is considered that jn (x) in (χ).

However, since this method repeatedly and converges the calculation in the entire data area, a sufficiently satisfactory solution can be obtained, but the disadvantage is that time increases as the number of data increases. On the other hand, in an ideal state without noise, a solution can be obtained at high speed by deconvolution calculation in the frequency space. Therefore, by combining these two processes, a sufficiently satisfactory solution can be obtained at high speed.

[0074] As such a processing method, two methods are conceivable. The first method is to reduce the data by thinning out the data. This method will be described as a second processing method using the processing method shown in FIG. When thinning out data, for example, as shown in FIG. 15, the original image data ImgZ force Pixel 1 1-16, 21-26, 31-36, 41-46, 51-56, 6 1-66 When every other pixel is thinned out, the original image reduced data ISmg ^ with a quarter size consisting of pixels 11, 13, 15, 31, 33, 35, 51, 53, 55 is generated. It is.

 [0075] In this way, the original image data Img 'and the change factor information data G are thinned out, the thinned original image reduced data ISmg' and the reduced change factor information data GS are generated, and the original image reduced data is generated. ISmg 'and reduced change factor information data GS is used to perform the iterative processing shown in Fig. 3 and thin out sufficiently satisfactorily to approximate the original image ISmg before changing to the original image reduced data ISmg'. Obtain approximate reduced restoration data ISo + n.

 [0076] It is estimated that this reduced approximate restored data ISo + n is a reduced original image ISmg before being converted into the original image reduced data ISmg ', that is, a reduced image of the image Img. The original image reduced data ISmg 'is considered to be a convolution integral of the reduced and restored data ISo + n and the transfer function g (x), and the obtained reduced and restored data ISo + n and the known original image reduced data ISmg' An unknown transfer function gl (X) can be obtained.

[0077] Reduced restoration data ISo + n is a sufficiently satisfactory data and is only an approximation. Therefore, the transfer function g (X) of the original restoration data Io + n and the original image data Img 'is not the transfer function gl (x) obtained by iterative processing with the reduced data. Therefore, the transfer function gl (x) is calculated from the reduced restoration data ISo + n and the original image reduced data ISmg ', which is the reduced original image data. Interpolate and fix The new transfer function g2 (x) obtained by correction is the transfer function g (X) for the original image data Img '. The new transfer function g2 (x) is the inverse of the reduction rate of the original image reduction data with respect to the obtained transfer function gl (X), and then the value between the enlargement is interpolated by linear interpolation, spline interpolation, etc. It is obtained by processing. For example, as shown in Fig. 15, when both vertical and horizontal are thinned to 1Z2, the reduction ratio is 1Z4, so the inverse number is 4 times.

 [0078] Then, using the modified new transfer function g2 (X) (= g (X)), perform deconvolution calculation in frequency space (calculation to remove blur by image group force calculation including blur), The complete restoration data Io + n of the entire image is obtained, and it is estimated as the original correct image Img (original image) that has not deteriorated.

 The flow of the above processing is shown in the flowchart diagram shown in FIG.

 [0080] In step S201, the original image data Img 'and the change factor information data G are reduced to IZM. In the example of Fig. 15, it is reduced to 1Z4. Using the obtained original image reduction data ISmg ′, reduction change factor information data GS, and arbitrary image (predetermined image) data Io, steps S102 to S105 shown in FIG. 3 are repeated. Then, the reduced restoration data ISo + n approximate to the reduced original image I Smg before changing to the original image reduced data ISmg ′ is obtained (step S202). At this time, “G, Img ′, Ιο + η” shown in FIG. 3 is replaced with “GS, ISmg ', ISo + n”.

 [0081] The transfer function gl (x) from the original image reduction data ISmg 'to the reduction / restoration data ISo + n is calculated from the obtained reduction / restoration data ISo + n and the known original image reduction data ISmg' (step S203). After that, in step S204, the obtained transfer function gl (X) is enlarged by M times (4 times in the example of Fig. 15), and the enlarged portion is interpolated by an interpolation method such as linear interpolation. Get the transfer function g2 (x). This new transfer function g2 (x) is estimated as the transfer function g (X) for the original image.

[0082] Next, the calculated new transfer function g2 (x) and the original image data Img 'are also deconvolved to obtain restored data Io + n. This restored data Io + n is used as the original image (step S205). As described above, i) iterative processing and mouth) transfer function gl (X), g2 (x) are obtained, and the process using the obtained new transfer function g2 (x) is used together. Therefore, the restoration process can be speeded up. [0083] In the image processing apparatus 1, the reduced restoration data ISo + n obtained by the iterative processing of (i) is displayed on the display unit 8 immediately after shooting, and the subsequent processing using the transfer function is performed by the image processing apparatus 1. When the operation is performed, for example, when the photographing unit 2 is turned off, the operation is performed. Then, the restored original image is recorded in the recording unit 5.

 In this process, the obtained normal U 正 image and the estimated restored data Io + n are used as the initial image data Io of the process shown in FIG. Using the original image data Img ′, iterative processing may be further performed, and the data obtained thereby may be stored in the recording unit 5.

 [0085] A second method of using the reduced data is a method for obtaining original image reduced data ISmg 'by taking out data of a partial area of original image data Img'. This method will be described as a third processing method using the processing method shown in FIG. For example, as shown in Fig. 17, the original image data is composed of Img 'force S, pixels 11-16, 21-26, 31-36, 41-46, 5 1-56, 61-66! This is a method of extracting the area consisting of the pixels 32, 33, 34, 4 2, 43, 44, which is the central area, and generating the original image reduced data ISmg ′.

 This second method will be described in detail with reference to the flowchart of FIG.

In the second method, first, in step S301, the original image reduced data ISmg ′ is obtained as described above. Next, the original image reduction data ISmg ', the change factor information data G, and the initial image data Io of the same size (= the same number of pixels) as the original image reduction data ISmg' are used. Then, the processing from step S102 to step S105 shown in FIG. 3 is repeated to obtain reduced restoration data ISo + n (step S302). In this process, “Img ′” in FIG. 3 can be replaced with “ISmg ′”, and “Io + n” can be replaced with “ISo + n”.

[0088] A transfer function g (x) from the reduced restoration data ISo + n to the original image reduction data ISmg 'is calculated from the obtained reduction restoration data ISo + n and the known original image reduction data ISmg' ( Step S303). Next, the calculated transfer function gl '(X) is the transfer function g' (X) for the original image Img, and this transfer function gl (x) (= g '(x)) and the known original image data Img Use 'to find the original image Img by inverse calculation. Note that the obtained data is actually the original image data that approximates the original image Img. [0089] As described above, i) iterative processing and mouth) transfer function g (X) is obtained, and the process using the obtained transfer function (X) is used in combination. High speed can be achieved. The obtained transfer function gl ′ (X) may be corrected using the change factor information data G instead of the entire transfer function g ′ (X) as it is.

 [0090] In the processing unit 4 of the image processing apparatus 1, the reduced and restored data ISo + n obtained by the iterative process of (i) is displayed on the display unit 8 immediately after photographing, and the original is obtained by inverse calculation using a transfer function. Perform the image restoration process in parallel with the display process! The restored original image is stored in the recording unit 5. It should be noted that the process using the transfer function following the repeated process may be performed after the display process, or may be performed when the image processing apparatus 1 is operated!

 As described above, in the second method for high-speed processing described above, the entire image area is not restored by iterative processing, but a part of the area is iteratively processed to obtain a good restored image, which is used. The transfer function gl '(X) for the part is obtained, and the entire image is restored using the transfer function gl' (X) itself or a modified version (such as enlargement). However, the area to be extracted must be sufficiently larger than the fluctuation area. In the previous example shown in Fig. 5, etc., it fluctuates over 3 pixels, so it is necessary to extract an area of 3 pixels or more.

 [0092] Note that in the case of the method of extracting the reduced area shown in FIGS. 17 and 18, the original image data I mg ′ is divided into four parts as shown in FIG. By extracting the area, iteratively processing each of the four original image reduced data ISmg ', which is a small area, restoring the divided areas divided into four, and combining the four restored divided images into one The original whole image may be used. In addition, when dividing into a plurality of areas, it is preferable to always have an overlapping area (overlapping area) over a plurality of areas. In addition, it is preferable to use an average value for the overlap area of each restored image, or to perform processing such as smooth connection in the overlap area.

[0093] Further, when the processing method of Fig. 3 is actually adopted, it has been found that an image with a sharp contrast change converges slowly to a good approximate restored image. Thus, depending on the nature of the subject that is the original image, it may be necessary to increase the number of iterations at which the convergence speed of the iteration process is slow. For these subjects, the following processing method It is presumed that this problem can be solved by adopting the law.

 [0094] The method is as follows. In other words, a subject with an abrupt change in contrast uses an iterative process of restoration by the processing method shown in Fig. 3, and the number of iterations becomes very large when trying to obtain an approximation of the original image. In addition, even after many processes, restored data Io + n that approximates the original subject cannot be generated. Therefore, the original image (blurred image) data Img '〖こ, the data B of the change factor information at the time of shooting is generated from the data B of the known image, and the data B' of the blurred image is generated. Overlay B 'to make "Img' + B '". After that, the superimposed image is restored by the process shown in Fig. 3, and the result data C force that becomes the restored data Io + n is removed. The already added field image data B is removed and the desired restored image data Img Take out.

 [0095] This method is applied to the above-described reduced image, and the restored reduced image is displayed on the display unit 8 immediately after shooting even for a subject with a sharp change in contrast. On the other hand, in the process of a large original image, the restoration process is executed by the processing unit 4 using the transfer function obtained from this process after the display process. The restored original image is then stored in the recording unit 5. In this method, the correct image data Img includes a sharp contrast change. By capturing the known image data B, this rapid contrast change can be reduced, and restoration processing can be performed. The number of iterations can be reduced.

 [0096] Also, other processing methods can be adopted as a processing method for a subject that is difficult to restore and a high-speed processing method. For example, if the number of iterations of restoration processing is increased, it takes time to perform force processing that can be brought closer to a good restored image. Therefore, by using the image obtained with a certain number of iterations, the error component included in the image is calculated, and the restored error including the error is removed from the restored image. n can be obtained.

[0097] This method will be specifically described below. The obtained positive image is A, the captured original image is A ′, the image restored from the original image A ′ is A + γ, and the restored data force is generated. To do. If the original image "Α '" is added to this "Α' + γ '" and restored, it becomes ΓΑ + y + A + y + γ ”, which is“ 2Α + 3γ ” And “2 (Α + γ) +”. Since “Α + γ” is obtained in the previous restoration process, “2 (Α + γ) + γ−2 (Α + γ))” can be calculated, and “γ” is obtained. Therefore, by removing “γ” from “Α + γ”, the correct image た い to be obtained can be obtained. This method is applied to the reduced image in the same manner as described above, the restored original image reduced data is displayed on the display unit 8, the large original image is restored using the obtained transfer function, and stored in the recording unit 5. To do. The control unit 4 executes these controls and processes.

 [0098] The power described in the preferred embodiments of the present invention has been described above. The present invention can be variously modified without changing the gist thereof. For example, the processing performed by the processing unit 4 is configured by software, but each may be configured by a hard- ware composed of parts that are performed by sharing a part of the processing.

 In addition to the captured image, the original image to be processed may be processed such as color-corrected or Fourier-transformed. Furthermore, as comparison data, in addition to the data generated using the data G of the change factor information, color correction is added to the data generated using the data G of the change factor information, or Fourier transform is performed. It is also possible to use such data. In addition, the change factor information data includes not only the degradation factor information data but also information that simply changes the image, and information that improves the image contrary to degradation.

 [0100] When the number of processing iterations is automatically or fixedly set on the image processing apparatus 1, the set number of times may be changed by the data G of the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.

 [0101] Furthermore, during the iterative process, if the difference data δ diverges, that is, if it becomes larger, the process may be stopped. To determine whether or not the force is diverging, for example, a method can be adopted in which the average value of the difference data δ is observed and if the average value becomes larger than the previous value, it is judged that the force is diverging. In addition, if the divergence occurs once, the processing may be stopped immediately, but if the divergence occurs twice, the method may be stopped, or the processing may be stopped if the divergence continues for a predetermined number of times. good.

[0102] Also, during an iterative process, if an attempt is made to change the input to an abnormal value, the process may be stopped. For example, in the case of 8 bits, the value to be changed exceeds 255 If it is, the process is stopped. In addition, during an iterative process, when an attempt is made to change the input, which is new data, to an abnormal value, that value may be used instead of the normal value. For example, if you try to use more than 255 values as input data in 0 to 255 of 8 bits, it will be processed as 255, which is the maximum value. In other words, if an abnormal value (value exceeding 255 in the above example) other than the allowable value (0 to 255 in the above example) is included in the restored data, the processing is stopped or the restored data If an abnormal numerical value other than the allowable numerical value is included, the abnormal numerical value can be changed to an allowable numerical value and the processing can be continued.

[0103] Further, when generating the restoration data to be an output image, depending on the data G of the change factor information, there may occur data that goes out of the area of the image to be restored. In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable to bring that data from the opposite side. For example, if the data assigned to the lower pixel is generated from the data of the pixel XN1 (N rows and 1 column) located at the bottom in the area, the position is outside the area. Therefore, the data is assigned to the pixel XI I (1 row and 1 column) located at the top right above the pixel XN1. Similarly, the pixel XN2 (N rows and 2 columns) adjacent to the pixel XN1 is allocated to the pixel X12 in the uppermost column directly above (= 1 row and 2 columns adjacent to the pixel XI I). In this way, when data that is outside the restoration target area is generated when the restoration data is generated, restoration of the position on the opposite side of one of the vertical, horizontal, and diagonal directions of the data generation position is performed. If it is arranged within the target area, it is possible to reliably restore the target area to be restored.

In addition, each processing method described above, that is, (1) a method of allocating difference data δ using the distribution ratio k (example method), (2) corresponding pixel difference, or difference data Method of scaling δ (corresponding pixel method), (3) Method of detecting centroid of deterioration factor and using data of centroid part (centroid method), (4) Thinning out data, combining with inverse problem Method (inverse problem decimation method), (5) Extraction of reduced area and combination with inverse problem (inverse problem area extraction method), (6) Iterates over a predetermined image, then the predetermined image (7) Restored images containing errors The processing method program for removing the calculated error (error extraction method) is stored in the processing unit 4 so that the processing method can be automatically selected according to the user's selection or image type. Anyway, okay.

[0105] For example, in the iterative processing of reduced images, not only (4) (5) (use (1)) but also (2) (3) (6) (7) Then, it is displayed on the display unit 8, and the transfer function processing in (4) (5) and the reduced image obtained in (2) (3) (6) (7) are used for the restoration process from the original image to the original image. A method can be employed in which a transfer function is obtained from the image and processing using the transfer function or an image obtained using the transfer function is repeatedly processed again. In addition, the restoration from the original image to the original image may be performed slowly with or without using the transfer function. In other words, apart from the display process, the original image is restored by directly obtaining the original image from the original image using any of the above (1) (2) (3) (6) (7). Good.

 It should be noted that the original image restoration processing is performed in parallel with the processing for displaying the reduced restoration data on the display unit 8, or when the photographing unit 2 is turned off and no photographing is performed, or the display unit 8 Various methods can be employed, such as after the display processing is performed. The restored original image is stored in the recording unit 5, but may be transmitted to another server or the like using a communication line such as the Internet together with or instead of the storage. good.

 [0107] In addition, the processing unit 4 classifies the change factor information data G into one of a plurality of types, and performs different processing for each of the classifications (one of the above methods! Alternatively, the number of repetitions may be different for each classification. Further, in the processing of the processing unit 4, it is preferable to divide whether or not the display image is restored by the value of the data G of the change factor information. For example, only when it is determined that the deterioration is significant, the image displayed on the display unit 8 is restored. When the deterioration is not so much, a simple reduced image is displayed as it is or a slightly corrected image is displayed. It is preferable. This reduces the load on the processing unit 4.

[0108] Further, any one of (1) to (7) is stored in the processing unit 4 so that the processing method can be automatically selected according to the user's selection or the type of image. Also good. In addition, select any one of these seven methods, and alternately or in sequence for each routine It may be used for the first time or processed in a certain method for the first few times, and then processed in another method. Note that the image processing apparatus 1 may have a different processing method in addition to any one or more of (1) to (7) described above.

 Moreover, each processing method mentioned above may be programmed. The program may be stored in a storage medium such as a CD (Compact Disc), a DVD, or a USB (Universal Serial Bus) memory so that it can be read by a computer. In this case, the image processing apparatus 1 has reading means for reading the program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading the program in the storage medium.

Claims

The scope of the claims

 [1] A processing unit that performs processing to restore the image before the change from the original image to be processed, the image that should have been taken, or the approximate image (hereinafter referred to as the original image!) In an image processing apparatus having

 The processing unit obtains original image reduced data obtained by subtracting the number of pixels from the original image, restores the reduced / restored data approximated to the original reduced image before the change based on the original image reduced data, and displays it on the display unit. An image processing apparatus that performs processing for displaying the reduced and restored data.

 2. The image processing apparatus according to claim 1, wherein the processing unit performs a process of restoring the original image using data obtained when the reduced restoration data is restored.

 [3] As the process of restoring the reduced restoration data, the processing unit generates data for comparing the data force of a predetermined image using data of change factor information that causes an image change, and compares the data. Data is compared with the original image reduced data, and reduced restoration data is generated using the obtained difference data, and the reduced restoration data is used in place of the predetermined image data. 3. The image processing apparatus according to claim 1, wherein thereafter, the obtained reduced / restored data is replaced with the previous reduced / restored data and the same processing is repeated.

 [4] As the process of restoring the reduced restoration data, the processing unit uses the data of the change factor information that causes the image change to generate data force comparison data of a predetermined image, and this comparison If the obtained difference data is greater than or equal to a predetermined value, the reduced data is generated using the difference data. The reduced and restored data is replaced with the predetermined image, and then the obtained reduced and restored data is replaced with the previous reduced and restored data, and the same processing is repeated! Performing the process of generating the reduced and restored data similar to the previous reduced original image, and performing the process of stopping the process when the difference data is less than or equal to a predetermined value Contract The image processing apparatus of claim 1 or 2 wherein.

[5] When restoring the reduced restoration data, a transfer function representing a degradation state of the original image is obtained. The obtained transfer function is inversely transformed with respect to the original image.

The image processing apparatus according to 3 or 4.

[6] It is characterized in that a transfer function is obtained from the original image reduced data and the reduced / restored data, and processing for generating the original image before changing to the original image is performed using the transfer function. The image processing apparatus according to claim 3 or 4.

[7] The original image reduced data is formed by thinning out the data of the original image, and the processing unit reduces the transfer function to reduce the original image reduced data from the original image. 7. The image processing according to claim 5 or 6, wherein a new transfer function is obtained by interpolating between the reciprocal of the rate and the enlargement, and the original image is generated using the new transfer function. apparatus.

 8. The image processing apparatus according to claim 5, wherein the original image reduced data is formed by extracting a partial area from the original image data as it is.

 9. The image processing apparatus according to any one of claims 1 to 8, wherein the restoration process of the original image is performed in parallel with a process of displaying the reduced restoration data on the display unit.

 10. The image processing apparatus according to claim 1, wherein the original image restoration process is performed when a photographing power is turned off.

 11. The image processing device according to claim 1, wherein the restoration process of the original image is performed after a process of displaying the reduced restoration data on the display unit.

[12] The processing unit is divided into a case where the reduced / restored data is displayed on the display unit and a case where the original image reduced data is displayed as it is, depending on the value of change factor information which causes an image change. The image processing device according to claim 1, wherein the image processing device performs processing.