JP2007081905A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of displaying images approximating completely restored images, in a short time after photographing, and then confirming contents of the image to be processed, such as recording, thereafter in a short period of time. <P>SOLUTION: The image processing apparatus, including a processing section for applying restoration processing to an image before being changed from an original image being a processing object, or an image having to be substantially photographed, or an approximate image (original image) to them, obtains original image reduction data resulting from reducing number of pixels from the original image (step S201), restores reduced restoration data approximate to a reduced original image, before being changed on the basis of the original image reproduction data (step S202), and displays the reproduction restoration data to a display section. Furthermore, the image processing apparatus preferably carries out processes (steps S203, S204, S205) of restoring the original image, by utilizing the data obtained at restoration. Further, the display section is, e.g. a monitor or the like of a camera. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus.

  Conventionally, it is known that when a subject is photographed by an image processing apparatus such as a camera, the recorded image sometimes deteriorates. As a factor of image deterioration, there is camera shake at the time of photographing. There is a circuit processing method as a first method for correcting image deterioration due to camera shake. In this circuit processing method, a transfer function representing a deterioration state of a captured image is acquired, and the acquired transfer function is inversely transformed for the captured image to correct the captured image (see Patent Document 1).

  As a second method for correcting image degradation due to camera shake, there is a method of moving a lens. For example, as a method for moving a lens, a method is known in which camera shake is detected and a predetermined lens is moved in accordance with the detected camera shake (see Patent Document 2).

Japanese Patent Laid-Open No. 11-24122 (see abstract) JP-A-6-317824 (see abstract)

  Employing the first method has the advantage that the time required for correction can be shortened, but has the following disadvantages. That is, the transfer function to be acquired is very weak against noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. For this reason, the corrected image obtained by the inverse transformation is far from an image photographed with no camera shake, and cannot be used in practice. In addition, when performing inverse transformation considering noise etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can also be adopted, but the calculated value for the estimation becomes astronomical size and actually There is a high risk that it will not be solved. Further, the camera adopting the second method requires a hardware space for driving a lens such as a motor, and thus becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  On the other hand, there are cases where a person who operates a camera or the like wants to check immediately after shooting how a shot image is recorded on a display unit (monitor) of the digital camera or the like immediately after shooting. The reason is that, if the image that can be confirmed on the monitor is inferior immediately after photographing, there is a high possibility that re-taking can be performed in consideration of correction of the inferiority.

  Therefore, the problem to be solved by the present invention is to display an image that approximates an image that has been completely corrected in a short time after shooting, and to be able to confirm the content of the image that is processed later, such as recording, in a short time. An image processing apparatus is provided.

  In order to solve the above-described problem, the image processing apparatus of the present invention is an image before changing from an original image to be processed, an image that should have been originally taken, or an approximate image thereof (hereinafter referred to as an original image). In the image processing apparatus having a processing unit that performs processing for restoring the original image, the processing unit obtains original image reduced data obtained by subtracting the number of pixels from the original image, and a reduced original image before changing based on the original image reduced data It is assumed that the reduced restoration data approximate to is restored, and the reduced restoration data is displayed on the display unit.

  According to the present invention, the amount of image data to be corrected is reduced by subtracting the number of pixels from the original image. Therefore, the time required for restoration such as camera shake correction can be shortened, and restoration can be completed in a short time after photographing. By displaying the reduced and restored data after the restoration on the display unit, it is possible to confirm the contents of the image processed thereafter such as recording in a short time after photographing.

In addition, it will become very suitable when a display part is a monitor. This is because the restored reduced / restored data displayed on the display unit is smaller than the original image by the number of pixels. However, for example, a monitor included in a digital camera, a digital video camera, or the like usually has a smaller number of pixels and a smaller display area than a captured image that is actually recorded. Therefore, it is preferable when checking what kind of captured images are recorded on various camera monitors.

  In another invention, in addition to the above-described invention, the processing unit performs processing to restore the original image using data obtained when restoring the reduced restoration data. In the process of restoring from the original image to the original image, if the data obtained in the process of restoring from the original image reduced data to the reduced restored data is used, the restoration from the original image to the original image can be performed efficiently. .

  Further, according to another invention, in addition to the above-described invention, the processing unit uses the data of the change factor information that causes the image change as a process of restoring the reduced restoration data, and uses the data of the predetermined image for comparison. Data is generated, the comparison data is compared with the original image reduced data, and reduced restoration data is generated using the obtained difference data. The reduced restoration data is used instead of the data of a predetermined image. Thereafter, the obtained reduced / restored data is replaced with the previous reduced / restored data, and the same processing is repeated.

  According to the present invention, the reduced and restored data that approximates the original image is generated only by generating predetermined data using the factor information of the image change, so that there is almost no increase in hardware, and the device Does not increase in size. Also, comparison data is created from the reduced and restored data, and the process of comparing the comparison data and the original image reduced data to be processed is repeated to gradually obtain reduced and restored data that is close to the reduced original image. It becomes work. For this reason, when restoring an image, an image processing apparatus having a realistic circuit processing method can be obtained.

  Note that the processing unit may perform a process of stopping when the difference data becomes equal to or smaller than a predetermined value or smaller than a predetermined value during the repeated processing. When this configuration is adopted, the processing is stopped even if the difference does not become “0”, so that it is possible to prevent a long processing time. Further, since it is set to a predetermined value or less, the approximated reduced / restored data is closer to the reduced original image before the change (before deterioration or the like) that is the original of the original image. Furthermore, when there is noise or the like, there is a tendency that the difference cannot be “0” in reality. However, even in such a case, the processing is not repeated infinitely. .

  Furthermore, the processing unit may perform a process of stopping when the number of repetitions reaches a predetermined number during the repetition process. When this configuration is adopted, the processing is stopped regardless of whether the difference becomes “0”, so that it is possible to prevent the processing from taking a long time. Further, since the processing is continued up to a predetermined number of times, the approximated reduced / restored data is closer to the original reduced image before the deterioration of the original image. Furthermore, when there is noise or the like, a situation in which the difference does not become “0” tends to occur in reality, but even in such a case, since the process is terminated a predetermined number of times, the process is repeated infinitely. It doesn't matter.

  Furthermore, the processing unit stops when the number of repetitions reaches a predetermined number, and stops when the difference data is less than or equal to a predetermined value or less than a predetermined value. May be further repeated a predetermined number of times. When this configuration is adopted, the number of processes and the difference value are combined, so the image quality is better than when the number of processes is simply limited or when the difference value is limited. And a process that balances the shortness of the processing time.

  In another aspect of the invention, in addition to the image processing apparatus of the above-described invention, the processing unit uses the data of the change factor information that causes the change of the image as the process of restoring the reduced restoration data. Data for comparison is generated from the data, the comparison data is compared with the original image reduced data, and if the obtained difference data is greater than or equal to a predetermined value, the difference data is used. Reduced restoration data is generated, and the reduced restoration data is replaced with a predetermined image. Thereafter, the obtained reduced restoration data is replaced with the previous reduced restoration data, and the same processing is repeated to obtain the original image reduced data. A process for generating reduced restoration data that approximates the original reduced image before the change is performed, and when the difference data is equal to or smaller than a predetermined value or smaller than a predetermined value, a process for stopping the process is performed.

  According to the present invention, the reduced and restored data that approximates the original image is generated only by generating predetermined data using the factor information of the image change, so that there is almost no increase in hardware, and the device Does not increase in size. Also, comparison data is created from the reduced and restored data, and the process of comparing the comparison data and the original image reduced data to be processed is repeated to gradually obtain reduced and restored data that is close to the reduced original image. It becomes work. For this reason, when restoring an image, an image processing apparatus having a realistic circuit processing method can be obtained. Further, since the processing is stopped when the difference data becomes small, the processing can be stopped after a certain number of processing times.

  Note that the processing unit may perform a process of stopping when the number of repetitions reaches a predetermined number during the repetition process. When this configuration is adopted, the processing is stopped regardless of whether the difference becomes “0”, so that it is possible to prevent the processing from taking a long time. Further, since the processing is continued up to a predetermined number of times, the approximated reduced / restored data is closer to the reduced original image before the change, which is the original of the original image reduced data. Furthermore, when there is noise or the like, a situation where the difference does not become “0” tends to occur in reality, but in such a case, the process is repeated infinitely, but if this configuration is adopted , No such problem occurs.

  Furthermore, in addition to the above-described invention, another invention obtains a transfer function representing the degradation state of the original image when restoring the reduced restoration data, and performs inverse transformation of the obtained transfer function on the original image. Yes. According to the present invention, it is possible to speed up restoration of an original image based on an appropriate transfer function. In addition, since the iterative process for obtaining the reduced restoration data and the deconvolution process using the transfer function are combined, the apparatus does not increase in size and becomes a realistic restoration work. Speeded up.

  In another invention, in addition to the above-described invention, a transfer function is obtained from the original image reduced data and the reduced / restored data, and processing for generating an original image before changing to the original image is performed using the transfer function. . By adopting this configuration, it is possible to speed up restoration of the original image based on an appropriate transfer function. In addition, since the iterative process for obtaining the reduced restoration data and the deconvolution process using the transfer function are combined, the apparatus does not increase in size and becomes a realistic restoration work. Speeded up.

  Furthermore, in addition to the above-mentioned invention, the original image reduced data is formed by thinning out the original image data, and the processing unit determines the transfer function from the original image of the original image reduced data. A new transfer function is obtained by reciprocal times the reduction ratio and interpolated between the enlarged images, and restored data that approximates the original image is generated using the new transfer function. When this configuration is adopted, a transfer function corresponding to the whole image can be obtained.

  In another invention, in addition to the above-described invention, the original image reduced data is formed by extracting a part of the area from the original image data as it is. When this configuration is adopted, a transfer function corresponding to a partial region and applicable to the entire image can be obtained.

  Furthermore, in addition to the above-described invention, in another aspect of the invention, the restoration process of the original image is performed in parallel with (1) the process of displaying the reduced restoration data on the display unit, and (2) when the imaging power is turned off. Or (3) after the process of displaying the reduced and restored data on the display unit.

  According to the above invention (1), it is preferable when the original image restoration process is executed as soon as possible. Further, according to the inventions of (2) and (3) described above, even when the restoration process of the original image takes a long time, the restoration process of the original image is performed in a period other than the restoration process period of the original image reduced data. By executing this, it is possible to reduce the burden on the image processing apparatus. Even if processing for delaying the execution timing of the original image restoration processing is performed in this way, the image displayed on the display unit can be displayed as reduced restoration data in a short time after shooting, so that an image processing apparatus, particularly a camera, etc. There is no inconvenience or adverse effect for those who operate the image processing apparatus having the above photographing function.

  Furthermore, in addition to the above-described invention, the processing unit displays the reduced / restored data on the display unit and the original image reduced data as it is based on the value of the change factor information that causes the image change. Processing is divided into cases. According to the present invention, in the case of an image having a remarkable change such as deterioration, the restored image is displayed on the display unit, and in the case of an image having no deterioration, the restoration work is not performed and the reduced image is displayed as it is. The Therefore, the restoration work is performed only when necessary, and the processing load can be reduced.

  According to the present invention, an image processing apparatus is provided that displays an image that approximates an image that has been completely corrected in a short time after shooting, and that allows the content of an image to be processed thereafter, such as recording, to be confirmed in a short time. be able to.

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a consumer camera. However, the image processing apparatus 1 may be a camera for other uses such as a monitoring camera, a television camera, an endoscopic camera, or an image such as a microscope, binoculars, or NMR imaging. It can also be applied to devices other than cameras, such as diagnostic devices.

  The image processing apparatus 1 includes a photographing unit 2 that shoots a video of a person, the like, a control system unit 3 that drives the photographing unit 2, and a processing unit 4 that processes an image photographed by the photographing unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a display unit 8 serving as a monitor. The detection unit 6 includes a factor information storage unit 7 that stores known change factor information that causes image degradation and the like.

  The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD (Charge Coupled Device) or a C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electric signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the photographing unit 2, the processing unit 4, the recording unit 5, the detection unit 6, the factor information storage unit 7, and the display unit 8.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 may store an image serving as a base when generating comparison data described later. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to process with software. The recording unit 5 is composed of a semiconductor memory, but may employ magnetic recording means such as a hard disk drive, optical recording means using a DVD (Digital Versatile Disk), or the like.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the optical system and lens distortion. However, when restoring blurring of camera shake described later, the information is used. Not done. The display unit 8 acquires the reduced / restored data, which is image data after correction (after restoration) of the image obtained by subtracting the number of pixels from the captured image, from the recording unit 5 and displays the reduced / restored data as a reduced original image. . As shown in FIG. 2, the display unit 8 is often arranged on the surface opposite to the imaging unit 2, but may be arranged on the same side as the imaging unit 2 or on the side surface. .

  Next, an outline of the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described with reference to FIG.

  In FIG. 3, “Io” is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4. “Io ′” indicates data of a degraded image of Io of the initial image data, and is comparison data for comparison. “G” is data of change factor information (= deterioration factor information (point image function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “Img ′” refers to captured image data, that is, data of a degraded image, and is data of an original image to be processed in this processing.

  “Δ” is difference data between the original image data Img ′ and the comparison data Io ′. “K” is an allocation ratio based on the data of the change factor information. “Io + n” is restored image data (restored data) newly generated by allocating the difference data δ to the initial image data Io based on the data of the change factor information. “Img” is the original correct image data having no deterioration, which is the basis of the original image data Img ′, which is a captured deteriorated image. Here, it is assumed that the relationship between Img and Img ′ is expressed by the following equation (1).

    Img ′ = Img × G (1)

  The difference data δ may be a simple difference between the corresponding pixels, but generally differs depending on the data G of the change factor information and is expressed by the following equation (2).

    δ = f (Img ′, Img, G) (2)

  The processing routine of the processing unit 4 starts by preparing arbitrary image data Io (step S101). As the initial image data Io, the photographed degraded image data Img ′ may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. In step S102, data Io of an arbitrary image that is an initial image is inserted instead of Img in the equation (1), and comparison data Io ′ that is a degraded image is obtained. Next, the data Img ′ of the original image, which is a captured degraded image, is compared with the comparison data Io ′, and difference data δ is calculated (step S103).

  Next, in step S104, it is determined whether or not the difference data δ is equal to or larger than a predetermined value. If the difference data δ is equal to or larger than the predetermined value, a process for generating new restored image data (= restored data) in step S105. I do. That is, the difference data δ is distributed to arbitrary image data Io based on the change factor information data G to generate new restored data Io + n. Thereafter, steps S102, S103, and S104 are repeated.

  If the difference data δ is smaller than the predetermined value in step S104, the process is terminated (step S106). Then, the restored data Io + n at the time when the processing is completed is estimated as data Img of a correct image, that is, an image without deterioration, and the data is recorded in the recording unit 5. The initial image data Io and change factor information data G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary. Here, an original image, a correct image that should have been originally taken, or an approximate image thereof is referred to as an “original image”. Therefore, the data Img and the restored data Io + n are “original image” data.

  The concept of the above processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

Solving as an optimization problem assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same.

  In other words, as shown in FIGS. 4A and 4B, if comparison data Io ′ (Io + n ′) that is approximate to data Img ′ of the original image that is the captured image can be generated, The initial image data Io or the restored data Io + n, which is the original data for generation, is approximate to the correct image data Img that is the original of the original image data Img ′.

  In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 μsec. In addition, in this embodiment, the value serving as a determination criterion for the difference data δ is “6” when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. In addition, the raw shake data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

  Next, details of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. From the left, the pixels are set to n-1, n, n + 1, n + 2, n + 3,. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the correct image data Img when no deterioration occurs. Each data is represented by 8 bits (0 to 255).

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the n + 1th pixel, and 20% of time is shifted to the n + 2th pixel. . The way of energy dispersion is as shown in the table of FIG. This becomes the data G of the change factor information.

  Since blurring is uniform for all pixels, assuming that there is no upper blur (vertical blurring), the blurring situation is as shown in the table of FIG. The data shown as “imaging result” in FIG. 8 is the original correct image data Img, and the data shown as “blurred image” is the taken degraded image data Img ′. Specifically, for example, “120” of the pixel “n−3” is in accordance with the distribution ratio of “0.5”, “0.3”, and “0.2” of the data G of the change factor information that is the blur information. It is distributed such that “60” is distributed to “n-3” pixels, “36” is distributed to “n-2” pixels, and “24” is distributed to “n-1” pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this deteriorated image data Img ′ and the change factor information data G shown in FIG.

  Any data Io of the arbitrary image shown in step S101 can be adopted, but in this description, the photographed original image data Img ′ is used. That is, the process starts with Io = Img ′. In the table of FIG. 9, “input” corresponds to the initial image data Io. This data Io, ie, Img ′ is multiplied by the change factor information data G in step S102. That is, for example, “60” of the “n-3” pixel of the initial image data Io is “30” for the n-3 pixel, “18” for the “n-2” pixel, “12” is allocated to each pixel of “1”. The other pixels are similarly distributed, and comparison data Io ′ shown as “output Io ′” is generated. Therefore, the difference data δ in step S103 is as shown in the bottom column of FIG.

  Thereafter, the size of the difference data δ is determined in step S104. Specifically, the processing is terminated when all the difference data δ is 5 or less in absolute value, but the difference data δ shown in FIG. 9 does not meet this condition, so the process proceeds to step S105. That is, the difference data δ is distributed to arbitrary image data Io using the change factor information data G to generate restored data Io + n shown as “next input” in FIG. In this case, since this is the first time, it is represented as Io + 1 in FIG.

  The distribution of the difference data δ is, for example, “15” obtained by multiplying the data “30” of the pixel “n−3” by 0.5 which is the distribution ratio of the place (= “n−3” pixel). “N-3” pixels are distributed, and “n-2” pixel data “15” is multiplied by 0.3 which is the distribution ratio that should have come to the “n-2” pixel. 4.5 ”is allocated, and the data“ 9.2 ”of the pixel“ n−1 ”is multiplied by 0.2 which is the distribution ratio that should have come to the pixel“ n−1 ”. Allocate “1.84”. The total amount allocated to the pixels of “n-3” is “21.34”, and this value is added to the initial image data Io (here, the captured original image data Img ′ is used) for restoration. Data Io + 1 is generated.

  As shown in FIG. 11, the restored data Io + 1 becomes the input image data (= initial image data Io) in step S102, step S102 is executed, and the process proceeds to step S103 to obtain new difference data δ. . The size of the new difference data δ is determined in step S104. If it is larger than the predetermined value, the new difference data δ is distributed to the previous restored data Io + 1 in step S105 to generate new restored data Io + 2 (FIG. 12). reference). Thereafter, new comparison data Io + 2 ′ is generated from the restored data Io + 2 by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, whereupon the process proceeds to step S105 or the process proceeds to step S106. Such a process is repeated.

  In this image processing apparatus 1, in processing, in step S <b> 104, one or both of the number of processes and the determination reference value of the difference data δ can be set in advance. For example, an arbitrary number such as 20 times or 50 times can be set as the number of times of processing. Further, the value of the difference data δ for stopping the processing is set to “5” in 8 bits (0 to 255), and when it becomes 5 or less, the processing is ended, or “0.5” is set to “0”. .5 "or less, the process can be terminated. This set value can be set arbitrarily. When both the number of processing times and the judgment reference value are input, the processing is stopped when either one is satisfied. When both settings are possible, the determination reference value may be given priority, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

  In the description of this embodiment, the information stored in the factor information storage unit 7 is not used, but known deterioration factors stored here, such as optical aberrations and lens distortions, are used. It may be used. In this case, for example, in the processing method of the previous example (FIG. 3), it is preferable to perform processing by combining blur information and optical aberration information as one deterioration factor. You may make it correct | amend with the information of an optical aberration after it complete | finishes. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors at the time of shooting, for example, only blurring.

  The basic configuration of the image processing apparatus 1 and the basic routine of the restoration process according to the embodiment of the present invention have been described above. Hereinafter, an example or a modification using the image processing apparatus 1 will be described.

  For example, when generating the restoration data Io + n, the distribution ratio k is not used, and the corresponding pixel difference data δ is directly added to the corresponding pixel of the previous restoration data Io + n−1, or the corresponding pixel difference data is generated. δ is added after scaling, or the data kδ (value shown as “update amount” in FIGS. 10 and 12) after the difference data δ is allocated is scaled, and the previous restored data Io + n You may make it add to -1. If these processing methods are used well, the processing speed is increased.

  Further, when the restoration data Io + n is generated, the center of gravity of a change factor such as deterioration may be calculated, and the difference of only the center of gravity or the scaling of the difference may be added to the previous restoration data Io + n-1. This concept, that is, the processing method using the center of gravity of the change factor will be described below as a first processing method using the processing method shown in FIG. 3 with reference to FIGS.

  As shown in FIG. 13, when correct image data Img is composed of pixels 11 to 15, 21 to 25, 31 to 35, 41 to 45, and 51 to 55, as shown in FIG. Attention is paid to the pixel 33. When the pixel 33 moves to the positions of the pixels 33, 43, 53, and 52 due to camera shake or the like, in the original image data Img ′ that is a deteriorated image, as shown in FIG. 43, 52, and 53 are affected by the first pixel 33.

  In the case of such deterioration, if the pixel 33 moves and is located at the position of the pixel 43 for the longest time, the center of deterioration, that is, the change factor is the original pixel 33 in the correct image data Img. In the image data Img ′, the pixel 43 is positioned. As a result, the difference data δ is calculated as the difference between the respective pixels 43 of the original image data Img ′ and the comparison data Io ′, as shown in FIG. The difference data δ is added to the pixels 33 of the initial image data Io and the restored data Io + n.

  In the above example, the three centroids of “0.5”, “0.3”, and “0.2” are the positions of “0.5” having the largest value, and are their own positions. Therefore, the allocation of “0.3” or “0.2” is not considered, and only “0.5” or 0.5 magnification of the difference data δ is allocated to its own position. Such a process is suitable when the blur energy is concentrated.

  Furthermore, each processing method described above can be automatically selected according to the contents of the data G of the change factor information. That is, the processing unit 4 can classify the data G of the change factor information into any one of a plurality of types, and perform different processing for each classification. For example, as a processing method, (1) a method of allocating difference data δ using an allocation ratio k as shown in FIGS. 5 to 12 (example method), (2) a difference between corresponding pixels, Alternatively, a program that can execute three methods: a method of scaling the difference data δ (corresponding pixel method) and (3) a method of detecting the centroid of the deterioration factor and using the data of the centroid (centroid method) 4 is stored, the state of the deterioration factor is analyzed, and one of the three methods is selected based on the analysis result. Alternatively, any one of the three methods may be selected and used alternately every routine, or may be processed by a certain method for the first several times, and then processed by another method.

  In the image processing apparatus 1, the restored image is displayed on the display unit 8. However, if it is attempted to restore the captured degraded image as it is, the processing time becomes long, and the display unit 8 cannot display the image immediately after shooting. Therefore, the restoration process is speeded up. Hereinafter, this speeding up will be described.

  In order to speed up the restoration process, there is a method combined with the inverse problem. That is, iterative processing is performed on the reduced data, and a transfer function from the reduced original image to the reduced restored data is calculated. Then, the calculated transfer function is enlarged and interpolated, and the restored data of the original image is obtained using the enlarged and interpolated transfer function. This processing method is advantageous for processing a large image.

  The basic concept of high-speed processing that is advantageous for restoring a large image will be described below.

  Iterative processing alone will inevitably take time to converge. This drawback becomes noticeable for large images. On the other hand, deconvolution in the frequency space is very attractive because high-speed calculation can be performed using Fast Fourier Transform (FFT). The optical deconvolution referred to here is to restore the original image which is not deteriorated by removing the distortion or the like from the image deteriorated due to distortion or blur.

In the case of an image, when the input is in (x), the output is ou (x), and the transfer function is g (x), in the ideal state, the output ou (x) is a convolution integral,
ou (x) = ∫in (t) g (x−t) dt (3)
It becomes. “∫” is an integral symbol. This equation (3) is a frequency space,
0 (u) = I (u) G (u) (4)
It becomes. Deconvolution is to determine the transfer function g (x) or the unknown input in (x) from this known output ou (x). For this purpose, I (u) = O (u ) / G (u) is obtained, the unknown input in (x) can be obtained by returning it to the real space.

However, in reality, due to noise or the like, Equation (3) becomes “ou (x) + α (x) = ∫in (t) g (x−t) dt + α (x)”. Here, “ou (x) + α (x)” is known, but ou (x) and α (x) are unknown. Even if this is approximated as an inverse problem, it is practically difficult to obtain a sufficiently satisfactory solution. Therefore, jn (x) where ou (x) + α (x) = ∫in (t) g (x−t) dt + α (x) ≈∫jn (t) g (x−t) dt is used as an iterative method. The process flow of FIG. 3 described above is obtained by converging using.
Here, if “α (x) << ou (x)”, it is considered that jn (x) ≈in (x).

  However, since this method iterates and converges the calculation in the entire data area, a sufficiently satisfactory solution can be obtained, but it takes a long time if the number of data increases. On the other hand, in an ideal state without noise, a solution can be obtained at high speed by deconvolution calculation in a frequency space. Thus, by combining these two processes, a sufficiently satisfactory solution can be obtained at high speed.

  As such a processing method, two methods are conceivable. The first is a method of reducing data by thinning out data. This method will be described as a second processing method using the processing method shown in FIG. When thinning out data, for example, as shown in FIG. 15, the original image data Img ′ is composed of pixels 11-16, 21-26, 31-36, 41-46, 51-56, 61-66. When every other pixel is thinned out, the original image reduced data ISmg ′ having a quarter size composed of the pixels 11, 13, 15, 31, 33, 35, 51, 53, and 55 is generated. is there.

  In this way, the original image data Img ′ and the change factor information data G are thinned out, and the thinned original image reduced data ISmg ′ and the reduced change factor information data GS are generated, and the original image reduced data ISmg ′ and Using the reduced change factor information data GS, the iterative process shown in FIG. 3 is performed, and a sufficiently satisfactory thinned approximate reduction that approximates the original image ISmg before changing to the original image reduction data ISmg ′. Restored data ISo + n is obtained.

  The reduced approximate restoration data ISo + n is estimated as a reduced original image ISmg before changing to the original image reduced data ISmg ′, that is, a reduced image of the correct image Img. The original image reduced data ISmg ′ is considered as a convolution integral of the reduced restoration data ISo + n and the transfer function g (x), and an unknown transfer function g1 is obtained from the obtained reduced restoration data ISo + n and the known original image reduced data ISmg ′. (X) can be obtained.

  The reduced and restored data ISo + n is sufficiently satisfactory, but is only an approximation. Therefore, the transfer function g (x) of the original restoration data Io + n and the original image data Img ′ is not the transfer function g1 (x) obtained by the iterative processing with the reduced data. Therefore, the transfer function g1 (x) is calculated from the reduced restoration data ISo + n and the original image reduced data ISmg ′ which is the reduced original image data, the calculated transfer function g1 (x) is enlarged, and the enlarged portion is interpolated. Then, the new transfer function g2 (x) obtained by the correction is set as a transfer function g (x) for the original image data Img ′ as the original data. The new transfer function g2 (x) is obtained by multiplying the obtained transfer function g1 (x) by the reciprocal of the reduction ratio of the original image reduction data, and thereafter, the enlarged value is interpolated such as linear interpolation or spline interpolation. It is obtained by processing. For example, as shown in FIG. 15, when the length and width are both halved, the reduction ratio is ¼, so that the reciprocal number is four times.

  Then, using the modified new transfer function g2 (x) (= g (x)), deconvolution calculation (calculation for removing blur by calculation from a group of images including blur) is performed in the frequency space, Obtain complete restoration data Io + n and estimate it as the original correct image Img (original image) that has not been degraded.

  The above processing flow is shown in the flowchart of FIG.

  In step S201, the original image data Img ′ and the change factor information data G are reduced to 1 / M. In the example of FIG. 15, it is reduced to ¼. Steps S102 to S105 shown in FIG. 3 are repeated using the obtained original image reduction data ISmg ′, reduction change factor information data GS, and arbitrary image (predetermined image) data Io. Then, the reduced restoration data ISo + n approximate to the reduced original image ISmg before changing to the original image reduced data ISmg ′ is obtained (step S202). At this time, “G, Img ′, Io + n” shown in FIG. 3 is replaced with “GS, ISmg ′, ISo + n”.

  A transfer function g1 (x) from the original image reduced data ISmg ′ to the reduced restored data ISo + n is calculated from the obtained reduced / restored data ISo + n and the known original image reduced data ISmg ′ (step S203). After that, in step S204, the obtained transfer function g1 (x) is enlarged by M times (4 times in the example of FIG. 15), and the enlarged portion is interpolated by an interpolation method such as linear interpolation to obtain a new transfer. The function g2 (x) is obtained. The new transfer function g2 (x) is estimated as the transfer function g (x) for the original image.

  Next, deconvolution is performed from the calculated new transfer function g2 (x) and original image data Img ′ to obtain restored data Io + n. The restored data Io + n is used as an original image (step S205). As described above, a) iterative processing and b) determining transfer functions g1 (x) and g2 (x) and using the calculated new transfer function g2 (x) together, The restoration process can be speeded up.

  In the image processing apparatus 1, the reduced and restored data ISo + n obtained by the iterative process of (a) is displayed on the display unit 8 immediately after shooting, and the subsequent processing using the transfer function is performed when the image processing apparatus 1 is not operated. For example, it is executed when the power of the photographing unit 2 is turned off. The restored original image is recorded in the recording unit 5.

  In the case of this processing, the obtained correct image and the restored data Io + n estimated are used as the initial image data Io of the processing shown in FIG. 3, and the change factor information data G and the deteriorated original image data Img ′ are used. Further, iterative processing may be executed and data obtained thereby may be stored in the recording unit 5.

  A second method of using the reduced data is a method for obtaining original image reduced data ISmg ′ by extracting data of a part of the original image data Img ′. This method will be described as a third processing method using the processing method shown in FIG. For example, as shown in FIG. 17, when the original image data Img ′ is composed of pixels 11 to 16, 21 to 26, 31 to 36, 41 to 46, 51 to 56, 61 to 66, This is a method of taking out an area composed of pixels 32, 33, 34, 42, 43, and 44, and generating original image reduced data ISmg ′.

  This second method will be described in detail with reference to the flowchart of FIG.

  In the second method, first, in step S301, the original image reduced data ISmg ′ is obtained as described above. Next, the original image reduced data ISmg ′, the change factor information data G, and the initial image data Io having the same size (= the same number of pixels) as the original image reduced data ISmg ′ in any image data are used. The processing in steps S102 to S105 shown in FIG. 3 is repeated to obtain reduced restoration data ISo + n (step S302). In this process, “Img ′” in FIG. 3 is replaced with “ISmg ′”, and “Io + n” is replaced with “ISo + n”.

  A transfer function g1 ′ (x) from the reduced / restored data ISo + n to the original image reduced data ISmg ′ is calculated from the obtained reduced / restored data ISo + n and the known original image reduced data ISmg ′ (step S303). Next, the calculated transfer function g1 ′ (x) is used as a transfer function g ′ (x) for the original image Img, and this transfer function g1 ′ (x) (= g ′ (x)) and known original image data are used. Using Img ′, the original image Img is obtained by inverse calculation. Note that the obtained data is actually original image data that approximates the original image Img.

  As described above, it is possible to speed up the restoration process by using a) iterative processing and b) obtaining the transfer function g1 ′ (x) and using the obtained transfer function g1 ′ (x). Can be achieved. The obtained transfer function g1 ′ (x) may be corrected using the change factor information data G without using the transfer function g ′ (x) as it is.

  The processing unit 4 of the image processing apparatus 1 displays the reduced restoration data ISo + n obtained by the iterative process (a) on the display unit 8 immediately after shooting, and restores the original image by inverse calculation using a transfer function. It is performed in parallel with the display process. The restored original image is stored in the recording unit 5. Note that the process using the transfer function following the repeated process may be performed after the display process or when the image processing apparatus 1 is not operated.

  Thus, in the second method for speeding up described above, the entire image area is not restored by iterative processing, but a part of the area is iteratively processed to obtain a good restored image, which is used to transmit to that part. The function g1 ′ (x) is obtained, and the entire image is restored by using the transfer function g1 ′ (x) itself or a modification (enlargement etc.) of the function g1 ′ (x). However, the area to be extracted needs to be an area sufficiently larger than the fluctuation area. In the previous example shown in FIG. 5 and the like, since it fluctuates over 3 pixels, it is necessary to extract an area of 3 pixels or more.

  In the method of extracting the reduced area shown in FIGS. 17 and 18, the original image data Img ′ is divided into four parts as shown in FIG. 19, for example, and a part of the area is extracted from each divided area. The original original reduced image may be obtained by repeatedly processing each of the four original image reduced data ISmg ′, which is a small area, restoring each of the four divided areas, and combining the four restored divided images into one. . In addition, when dividing into a plurality of areas, it is preferable to always have an overlapping area (overlapping area) over a plurality of areas. Further, it is preferable to perform processing such as using an average value or smoothly connecting the overlap regions of the restored images in the overlap regions.

  Further, when the processing method of FIG. 3 is actually adopted, it has been found that convergence of an image with a sharp change in contrast to a good approximate restored image is slow. Thus, depending on the nature of the subject that is the original image, the convergence speed of the iterative process may be slow and the number of iterations may have to be increased. In the case of such a subject, it is estimated that this problem can be solved by adopting the following processing method.

  The method is as follows. That is, for an object with a sharp change in contrast, if the iterative process of restoration by the processing method shown in FIG. 3 is used to obtain an object that approximates the original image, the number of iterations becomes very large and many Even after the number of processings, the restoration data Io + n that approximates the original subject cannot be generated. Therefore, blur data B ′ is generated using data G of the change factor information at the time of shooting from data B of the known image as data Img ′ of the shot original image (blur image). And “Img ′ + B ′” is created. After that, the superimposed image is restored by the process shown in FIG. 3, and the known added field image data B is removed from the result data C as the restored data Io + n, and the restored image data Img to be obtained is taken out.

  This method is applied to the above-mentioned reduced image, and the restored reduced image is displayed on the display unit 8 immediately after shooting even for a subject with a sharp change in contrast. On the other hand, in the process of a large original image, a restoration process is performed by using the transfer function obtained from this process by the processing unit 4 after the display process. The restored original image is then stored in the recording unit 5. In this method, the correct image data Img includes an abrupt contrast change, but by adding the known image data B, the abrupt contrast change can be reduced, and the number of iterations of the restoration process Can be reduced.

  Also, other processing methods can be adopted as a processing method for a subject that is difficult to restore and a high-speed processing method. For example, if the number of iterations of the restoration process is increased, it can be brought closer to a good restored image, but the process takes time. Therefore, by using an image obtained by a certain number of iterations, an error component included therein is calculated, and a good restored image, that is, restored data Io + n is obtained by removing the calculated error from the restored image including the error. be able to.

  This method will be specifically described below. The correct image to be obtained is A, the captured original image is A ′, the image restored from the original image A ′ is A + γ, and the blurred comparison data generated from the restored data is A ′ + γ ′. When the captured original image “A ′” is added to “A ′ + γ ′” and restored, it becomes “A + γ + A + γ + γ”, which is “2A + 3γ”, and “2 (A + γ) +” is there. Since “A + γ” is obtained in the previous restoration process, “2 (A + γ) + γ−2 (A + γ)” can be calculated, and “γ” is obtained. Therefore, by removing “γ” from “A + γ”, the correct image A to be obtained can be obtained. This method is applied to the reduced image in the same manner as described above, the restored original image reduced data is displayed on the display unit 8, the large original image is restored using the obtained transfer function, and stored in the recording unit 5. . These controls and processes are executed by the processing unit 4.

  Although the preferred embodiments of the present invention have been described above, the present invention can be variously modified without changing the gist thereof. For example, the processing performed by the processing unit 4 is configured by software, but may be configured by hardware composed of parts that perform part of the processing.

  In addition to the photographed image, the original image to be processed may be a photographed image subjected to processing such as color correction or Fourier transform. In addition to the data generated using the change factor information data G, the comparison data includes data generated using the change factor information data G, color correction, or Fourier transform. It is also good. Further, the data of the change factor information includes not only the data of the deterioration factor information but also information that simply changes the image and information that improves the image contrary to the deterioration.

  When the number of processing iterations is set automatically or fixedly on the image processing apparatus 1 side, the set number of times may be changed by the data G of the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.

  Furthermore, when the difference data δ diverges during the iterative process, that is, when the difference data δ becomes larger, the process may be stopped. For example, it is possible to adopt a method of determining whether or not the light is diverging by observing the average value of the difference data δ and determining that the light is diverging if the average value is larger than the previous value. In addition, if the divergence occurs once, the process may be stopped immediately. However, if the divergence occurs twice, the method may be stopped, or the process may be stopped if the divergence continues for a predetermined number of times. .

  In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed exceeds 255, the process is stopped. Further, during an iterative process, when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 within the 8-bit range of 0 to 255 is used as input data, it is processed as a maximum value of 255. That is, if the restored data includes an abnormal value (a value exceeding 255 in the above example) other than an allowable value (0 to 255 in the above example), the process is stopped or the restored data When an abnormal numerical value other than the allowable numerical value is included, the abnormal numerical value can be changed to an allowable numerical value and the processing can be continued.

  In addition, when generating restoration data to be an output image, depending on the data G of the change factor information, there may be data that goes out of the area of the image to be restored. In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable to bring that data from the opposite side. For example, when data allocated to a lower pixel is generated from the data of the pixel XN1 (N rows and 1 column) located at the bottom in the area, the position is outside the area. Therefore, the data is processed to be allocated to the pixel X11 (one row and one column) located directly above the pixel XN1. Similarly, the pixel XN2 (N rows and 2 columns) adjacent to the pixel XN1 is assigned to the pixel X12 in the uppermost column directly above (= 1 row and 2 columns adjacent to the pixel X11). In this way, when data that is outside the restoration target area is generated when the restoration data is generated, the restoration target at a position opposite to one of the vertical, horizontal, and diagonal directions of the data generation position is generated. If it is arranged within the area, it is possible to reliably restore the target area to be restored.

  In addition, each processing method described above, that is, (1) a method of allocating difference data δ using the distribution ratio k (method according to the embodiment), (2) a corresponding pixel difference, or difference data δ is changed. Method of doubling (corresponding pixel method), (3) Method of detecting centroid of deterioration factor and using data of centroid part (centroid method), (4) Method of thinning out data and combining with inverse problem (inverse problem culling) (Method), (5) Extraction of reduced area and combination with inverse problem (extratitle area extraction method), (6) Method of repeatedly processing predetermined images superimposed and then removing the predetermined image (Image countermeasure overlay method), (7) A program for each processing method of removing a calculated error (error extracting method) from a restored image including an error (error extracting method) is stored in the processing unit 4 and selected by a user or an image of A processing method may be automatically selected according to the type.

  For example, in the repetitive processing of a reduced image, not only (4) (5) (using (1)) but also (2) (3) (6) (7) is used for quick restoration and display. In the restoration process from the original image to the original image, the transfer function processing of (4) (5) and the transfer function from the reduced images obtained in (2) (3) (6) (7) Thus, a process using the transfer function or a method of repeatedly processing an image obtained using the transfer function again may be employed. Further, the restoration from the original image to the original image may be performed slowly with or without using the transfer function. That is, apart from the display processing, the original image is restored by directly obtaining the original image from the original image using any one of the above (1), (2), (3), (6), and (7). good.

  The restoration process of the original image is performed in parallel with the process of displaying the reduced restoration data on the display unit 8 or when the photographing unit 2 is turned off and no photographing is performed, or displayed on the display unit 8. Various methods can be employed, for example, after the processing is performed. The restored original image is stored in the recording unit 5, but may be transmitted to another server or the like using a communication line such as the Internet together with or instead of the storage.

  Further, the processing unit 4 classifies the data G of the change factor information into any of a plurality of types, and performs different processing (any one of the above-described methods) for each classification, Further, the number of repetitions may be varied for each classification. Further, in the processing of the processing unit 4, it is preferable to determine whether or not to restore the display image according to the value of the data G of the change factor information. For example, an image displayed on the display unit 8 is restored only when it is determined that the deterioration is significant, and when there is not much deterioration, a mere reduced image is displayed as it is or a slightly corrected image is displayed. Is preferred. As a result, the load on the processing unit 4 is reduced.

  Alternatively, any one of (1) to (7) may be stored in the processing unit 4 so that the processing method can be automatically selected according to the user's selection or the type of image. Also, select any one of these seven methods and use them alternately or in sequence for each routine, or process them in one method for the first few times, and then process them in another method. Also good. Note that the image processing apparatus 1 may have a different processing method in addition to any one or a plurality of (1) to (7) described above.

  Moreover, each processing method mentioned above may be programmed. Alternatively, the program may be stored in a storage medium, such as a CD (Compact Disc), a DVD, or a USB (Universal Serial Bus) memory, and read by a computer. In this case, the image processing apparatus 1 has reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading a program in the storage medium.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. FIG. 2 is a processing flowchart for explaining a processing method (processing routine) performed by a processing unit of the image processing apparatus shown in FIG. 1. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which comparison data is generated from an arbitrary image. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. The situation in which the comparison data is compared with the blurred original image to be processed to generate difference data. It is a figure for demonstrating. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, in which new comparison data is generated from the generated restored data, the data and the blurred original image to be processed, It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and a diagram for explaining a situation in which newly generated difference data is allocated and new restoration data is generated. is there. FIG. 4 is a diagram for explaining processing using the center of gravity of a change factor, which is a first processing method using the processing method shown in FIG. 3, and (A) shows a state in which attention is paid to one pixel in correct image data. FIG. 4B is a diagram illustrating a state in which the data of the pixel of interest is expanded in the diagram illustrating the data of the original image. It is a figure for demonstrating concretely the process using the gravity center of the change factor which is the 1st processing method shown in FIG. FIG. 4 is a diagram for explaining a first of the methods for speeding up the second processing method using the processing method shown in FIG. 3, wherein (A) shows data of an original image to be processed, and (B) FIG. 4 is a diagram showing data obtained by thinning out the data of (A). It is a flowchart figure of the 2nd processing method shown in FIG. FIG. 6 is a diagram for explaining a second processing method for speeding up the third processing method using the processing method shown in FIG. 3, in which (A) shows data of an original image to be processed, and (B) FIG. 4 is a diagram showing data obtained by extracting a part of the data of (A). It is a flowchart figure of the 3rd processing method shown in FIG. FIG. 17 is a diagram for explaining a modification of the third processing method shown in FIGS. 17 and 18 in which the original image data is divided into four parts, and a part of the area for repeated processing is extracted from each divided area. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image pick-up part 3 Control system part 4 Processing part 5 Recording part 6 Detection part 7 Factor information preservation | save part 8 Display part Io Initial image data (data of arbitrary images)
Io 'comparison data G Change factor information data (degradation factor information data)
GS Reduced change factor information data Img 'Original image data (captured image)
ISmg ′ Original image reduced data δ Difference data k Distribution ratio Io + n Restored data (Restored image data)
ISo + n Reduced restoration data Img Original correct image data without deterioration (original image)
ISmg Reduced original image
g (x), g ′ (x), g2 (x) transfer function (transfer function for restoring a large image)
g1 (x), g1 ′ (x) transfer function (transfer function obtained from reduced data)

Claims (12)

In an image processing apparatus having a processing unit that performs processing for restoring an image before change from an original image to be processed, an image that should have been originally taken, or an approximate image thereof (hereinafter referred to as an original image),
The processing unit obtains original image reduced data obtained by subtracting the number of pixels from the original image, restores reduced restoration data that approximates the original reduced image before the change based on the original image reduced data, and An image processing apparatus that performs processing for displaying reduced restoration data.   The image processing apparatus according to claim 1, wherein the processing unit performs processing to restore the original image using data obtained when restoring the reduced restoration data.   The processing unit generates data for comparison from data of a predetermined image using data of change factor information that causes image change as a process of restoring the reduced restoration data, and the comparison data, The original image reduced data is compared with each other, and reduced restoration data is generated using the obtained difference data. The reduced / restored data is used instead of the predetermined image data. 3. The image processing apparatus according to claim 1, wherein the reduced restoration data is replaced with the previous reduced restoration data, and the same processing is repeated.   The processing unit generates data for comparison from data of a predetermined image using data of change factor information that causes image change as a process of restoring the reduced restoration data, and the comparison data, The original image reduced data is compared, and if the obtained difference data is larger than a predetermined value or greater than a predetermined value, reduced restoration data is generated using the difference data, and the reduced restoration data is Replaced with the predetermined image, and thereafter, the obtained reduced / restored data is replaced with the previous reduced / restored data and the same processing is repeated, approximating the reduced original image before changing to the original image reduced data. 2. A process for generating the reduced restoration data to be performed, and a process for stopping the process when the difference data is equal to or smaller than a predetermined value or smaller than a predetermined value. The image processing apparatus 2 described.   5. The transfer function representing a degradation state of the original image is acquired when the reduced restoration data is restored, and the acquired transfer function is inversely transformed with respect to the original image. Image processing apparatus.   4. A process for obtaining a transfer function from the original image reduced data and the reduced and restored data and generating the original image before changing to the original image using the transfer function. 4. The image processing apparatus according to 4.   The original image reduction data is formed by thinning out the data of the original image, and the processing unit sets the transfer function to an inverse multiple of the reduction rate of the original image reduction data from the original image. 7. The image processing apparatus according to claim 5, wherein a new transfer function is obtained by interpolating between the enlarged portions and the original image is generated using the new transfer function.   7. The image processing apparatus according to claim 5, wherein the original image reduced data is formed by extracting a partial area from the original image data as it is.   The image processing apparatus according to claim 1, wherein the restoration process of the original image is performed in parallel with a process of displaying the reduced restoration data on the display unit.   The image processing apparatus according to claim 1, wherein the restoration process of the original image is performed when a photographing power is turned off.   The image processing apparatus according to claim 1, wherein the restoration process of the original image is performed after performing the process of displaying the reduced restoration data on the display unit.   The processing unit performs a process of dividing the case where the reduced / restored data is displayed on the display unit and the case where the original image reduced data is displayed as it is based on the value of the change factor information that causes the image change. The image processing apparatus according to claim 1, wherein:

Images