JPWO2008090858A1 - Image processing apparatus and image processing method - Google Patents

Abstract

In order to prevent the enlargement of the apparatus and to propose a circuit processing method for realistic image restoration, in this image processing apparatus, by using the change factor data G that causes the image change, the arbitrary image luminance Comparison image luminance data YI0 ′ is generated from the data YI0, and then the original image luminance data YImg ′ to be processed is compared with the comparison image luminance data YI0 ′. The obtained difference luminance data Yδ is used as the change factor data G. Based on this, the restored image luminance data YI0 + n is generated by allocating to the arbitrary image data YI0. Thereafter, the restored image luminance data YI0 + n is used in place of the arbitrary image luminance data YI0, and the same process is repeated before the change ( A process of generating luminance data of the base image before the deterioration or the like, or restored image luminance data YI0 + n that approximates the luminance data is performed.

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Conventionally, it is known that image degradation occurs in a photographed image photographed with a camera or the like. Factors for this image degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, and the like.

  In order to correct camera shake at the time of photographing, a method of moving a predetermined lens among photographing lenses and a method of circuit processing are known. For example, as a method of moving the lens, a method of correcting by detecting camera shake at the time of shooting and moving a predetermined lens in accordance with the detected camera shake is known (see Patent Document 1).

  In addition, as a circuit processing method, a change in the optical axis of the camera is detected by an angular acceleration sensor, a transfer function indicating a blurring state at the time of shooting is acquired from the detected angular velocity, etc., and the acquired transfer function is obtained for a shot image. A method is known in which an image is restored by performing an inverse transformation of (see Patent Document 2).

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens, such as a motor, and becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs.

  In addition, in the case of camera shake correction described in Patent Document 2, the above-described problems are eliminated, but the following problems occur. That is, it is theoretically possible to perform image restoration by inverse transformation of the acquired transfer function, but as a practical problem, image restoration is difficult for the following two reasons.

  First, the transfer function to be acquired is very weak against noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice. Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In practice, there is a high risk of being unable to solve.

  As described above, an object of the present invention is to provide an image processing apparatus and an image processing method having a realistic circuit processing method as well as preventing an increase in size of the apparatus when restoring an image.

  In order to solve the above problems, an image processing apparatus according to the present invention includes an image processing apparatus having a processing unit that processes a color image. The image luminance data and the original image color difference data are separated, and each of the original image luminance data and the original image color difference data is set as processing target data. The comparison image luminance data for comparison in which the arbitrary image luminance data is changed is generated from the arbitrary image luminance data which is arbitrary image luminance data, and the original image luminance data and the comparative image luminance data are compared. Difference luminance data, which is difference data, is obtained, and restored image luminance data is generated by distributing the difference luminance data to arbitrary image luminance data based on the change factor data. Then, by using the restored image luminance data instead of the arbitrary image luminance data, and repeating the same process, the base image luminance data of the base image which is the image before the original image is changed or approximated to it The restored image brightness data is generated, and the original image color difference data is changed from the arbitrary image color difference data, which is arbitrary image color difference data, using the change factor data. Generating comparison image color difference data for comparison, obtaining difference color difference data which is a difference data obtained by comparing the original image color difference data and the comparison image color difference data, and calculating the difference color difference data based on the change factor data. The restored image color difference data distributed to the data is generated, the restored image color difference data is used in place of the arbitrary image color difference data, and the same processing is repeated. Be iterative process by performing, and performing the process of generating the reconstructed restored image chrominance data was approximating that based on image color difference data or the original image is an image before the original image is changed.

  When the image processing apparatus is configured as described above, the restored image luminance data and the restored image that approximate the luminance data and the color difference data of the original image simply by generating predetermined data using the image change factor information. Since the color difference data is generated, there is almost no increase in hardware, and the apparatus does not increase in size. Also, comparison luminance data and comparison color difference data are created from the restored image luminance data and the restored image color difference data, respectively, and the comparison luminance data is compared with the luminance data of the original image to be processed. Is repeated, and the restored image luminance data and restored image color difference data close to the original image that is the basis of the original image are gradually obtained. It becomes work. For this reason, an image processing apparatus having a realistic circuit processing method can be provided in the image restoration process.

  In addition to the above-described invention, another invention performs a process of combining the restored image luminance data and the restored image color difference data to generate base image data of the base image or restored restored image data approximated thereto. And

  When the image processing apparatus is configured as described above, an image processing apparatus having a realistic circuit processing method can be obtained.

  In addition to the above-described invention, another invention uses the original image color difference data as two data for different colors, and repeats each of them.

  When the image processing apparatus is configured as described above, the restored image color difference data is generated for each different color, so that the restoration accuracy can be increased.

  According to another invention, in addition to the above-described invention, the original image color difference data is subjected to compression processing, and the compressed compressed image color difference data is repeatedly processed to generate restored image color difference data.

  When the image processing apparatus is configured in this way, the restoration process can be speeded up.

  According to another invention, in addition to the above-described invention, the compression of the original image color difference data is a thinning process for thinning out the data of the original image color difference data.

  When the image processing apparatus is configured in this way, the restoration process can be speeded up.

  In addition to the above-described invention, in another aspect of the invention, the compression of the image color difference data is a process of generating the restored image color difference data with a smaller number of bits than the number of bits for performing the process of generating the restored image luminance data. I will do it.

  When the image processing apparatus is configured in this way, the restoration process can be speeded up.

  In order to solve the above-described problems, an image processing apparatus according to the present invention is an image processing method for processing a color image, wherein original image luminance data and original image color difference data are obtained from original image data as original image data to be processed. The original image luminance data and the original image color difference data as processing target data, and for the original image luminance data, using the change factor data that is the image change factor data, From the arbitrary image luminance data which is image luminance data, a step of generating comparative image luminance data for comparison in which the arbitrary image luminance data is changed, and difference data obtained by comparing the original image luminance data and the comparative image luminance data. A step of obtaining a certain difference luminance data, and a restored image brightness obtained by allocating the difference luminance data to arbitrary image luminance data based on the change factor data. A step of generating data, using the restored image luminance data instead of the arbitrary image luminance data, and repeating the same process repeatedly, so that the original image before the original image is changed The base image luminance data or a restored restored image luminance data approximated to it is processed, and for the image color difference image data, using the change factor data, from the arbitrary image color difference data which is arbitrary image color difference data, A step of generating comparison image color difference data for comparison in which the arbitrary image color difference data is changed, a step of obtaining difference color difference data which is difference data obtained by comparing the original image color difference data and the comparison image color difference data, and difference color difference Generating restored image color difference data in which data is distributed to arbitrary image color difference data based on the change factor data. By using the restored image color difference data instead of the arbitrary image color difference data and repeating the same process, the original image color difference data of the base image, which is the image before the original image is changed, or the approximated image is restored. The restored image color difference data is generated.

  In this way, in the case of the image processing method, the restored image luminance data and restored image color difference data that approximate the luminance data and color difference data of the original image simply by generating predetermined data using the factor information of the image change. Therefore, there is almost no increase in hardware, and the apparatus adopting this method does not increase in size. Also, comparison luminance data and comparison color difference data are created from the restored image luminance data and the restored image color difference data, respectively, and the comparison luminance data is compared with the luminance data of the original image to be processed. Is repeated, and the restored image luminance data and restored image color difference data close to the original image that is the basis of the original image are gradually obtained. Become a method. Therefore, a realistic image processing method can be used in the image restoration process.

  According to the present invention, when restoring a deteriorated image, it is possible to prevent an increase in size of the apparatus and to provide an image processing apparatus and an image processing method having a realistic circuit processing method.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. It is a processing method (processing routine) performed by the processing unit of the image processing apparatus shown in FIG. 1 and is a processing flowchart for explaining the basic concept. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows distribution of the light energy amount about luminance information when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 as an example of camera shake, and is a figure which shows dispersion | distribution of the energy of the amount of light energy about the luminance information when camera shake occurs. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which comparison image luminance data is generated from luminance data of an arbitrary image. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example. Comparison image brightness data is compared with brightness data of a blurred original image to be processed, and difference brightness data is obtained. It is a figure for demonstrating the condition to produce | generate. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 with an example of camera shake, and illustrates a situation in which restored image luminance data is generated by allocating difference luminance data and adding it to luminance data of an arbitrary image. FIG. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example. New comparison image luminance data is generated from the generated restored image luminance data, and the data and the blur to be processed It is a figure for demonstrating the condition which compares with the luminance data of an original image, and produces | generates difference luminance data. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and for explaining a situation in which newly generated differential luminance data is allocated and new restored image luminance data is generated. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 4 ... Processing part Img '... Original image data YImg' ... Original image luminance data UImg ', VImg' ... Original image color difference data G ... Change factor data YI ₀ ... Arbitrary image brightness data YI ₀ '... Comparison image brightness data Yδ ... Difference brightness data YI _{0 + n} ... Restored image brightness data YImg ... Base image brightness data UI ₀ , VI ₀ ... Arbitrary image color difference data UI ₀ ', VI ₀ '... Comparison image color difference data Uδ, Vδ... Difference color difference data UI _{0 + n} , VI _{0 + n} ... Restored image color difference data UImg, VImg.

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a so-called consumer digital camera using a CCD (Charge Coupled Device) as an imaging unit. However, the image processing apparatus 1 is used for a monitoring camera, a television camera, and the like that uses an imaging element such as a CCD as the imaging unit. The present invention can also be applied to devices other than cameras, such as endoscope cameras and other imaging cameras, microscopes, binoculars, and diagnostic imaging apparatuses for nuclear magnetic resonance imaging. The image processing method will be described in conjunction with the description of the image processing apparatus 1.

  The image processing apparatus 1 includes an imaging unit 2 that captures an image of a person or the like, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. Yes. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4, an angular velocity sensor, and the like, and causes a change (deterioration) of a captured image. It has a detection unit 6 that detects change factor information, and a factor information storage unit 7 that stores known change factor information that causes an image change or the like.

  The imaging unit 2 includes an imaging optical system having a lens and an imaging element such as a CCD or a CMOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The imaging unit 2 generates and outputs three color image signals of R (red), G (green), and B (blue). The imaging unit 2 is a so-called single-plate type that uses one imaging element in which RGB-specific filters are arranged in layers on the imaging surface, or three colors of RGB using a dichroic prism for imaging light that has passed through the imaging optical system, A so-called three-plate type, which is separated into separate color lights and includes an image sensor for each color light, may be any (single-plate type / 3-plate type). The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 includes an image that is a source for generating image luminance data of a comparative image (hereinafter referred to as comparative image luminance data) and image color difference data of a comparative image (hereinafter referred to as comparative image color difference data). Sometimes stored. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 is composed of a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD, or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 stores information about change factors that change the captured image and known change factor data that is known before the start of shooting. The information on the known change factor is, for example, aberration of the optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the optical system and lens distortion. However, when restoring blurring of camera shake described later, the information is used. Not done.

  Next, an outline of the processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described.

  The processing unit 4 first separates the YUV signal from the original image data Img ′, which is the captured image data to be processed. That is, from the original image data Img ′, the original image luminance data YImg ′ that is the luminance information of the photographed image, the original image color difference data UImg ′ that is the color difference information about the blue color of the photographed image, and the original that is the color difference information about the red color. The image color difference data VImg ′ is separated. After the processing step for performing such separation, the restoration process shown in FIG. 3 is performed for each of the original image luminance data YImg ′, the original image color difference data UImg ′, and the original image color difference data VImg ′. FIG. 3 is a flowchart illustrating the processing (restoration processing) of the original image luminance data YImg ′. The same restoration processing is also performed on the original image color difference data UImg ′ and the original image color difference data VImg ′.

In FIG. 3, “YI ₀ ” is arbitrary image luminance data that is luminance data of an arbitrary image, and this arbitrary image luminance data YI ₀ is stored in advance in the recording unit of the processing unit 4. “YI ₀ ′” indicates data of a change image of the arbitrary image luminance data YI ₀ and is image luminance data for comparison (hereinafter referred to as comparative image luminance data). “G” is change factor data serving as change factor information (= deterioration factor information (point spread function)) detected by the detection unit 6, and is stored in the recording unit of the processing unit 4. . The original image luminance data YImg ′ is separated from the captured image data. That is, the luminance data is separated from the original image data Img ′ (photographed image data) that is changed by a change factor due to camera shake or the like at the time of shooting, and the original image luminance data YImg ′ is also changed by the change factor. ing.

“Yδ” is difference luminance data which is difference data between the original image luminance data YImg ′ and the comparison image luminance data YI ₀ ′. “K” is a distribution ratio based on the change factor data G. “YI _{0 + n} ” is restored image luminance data newly generated by distributing the difference luminance data Yδ to the arbitrary image luminance data YI ₀ based on the change factor data. “YImg” is luminance data (hereinafter, referred to as base image luminance data) regarding the original image data (hereinafter referred to as base image data) before the change, which is the basis of the original image data Img ′. . Here, it is assumed that the relationship between YImg and YImg ′ is expressed by the following equation (1).
YImg ′ = YImg * G (1)
“*” Is an operator representing a superposition integral.
The difference luminance data Yδ may be a simple difference between corresponding pixels, but generally differs depending on the change factor data G and is expressed by the following equation (2).
Yδ = f (YImg ′, YI ₀ , G) (2)

The processing routine of the processing unit 4 starts by preparing arbitrary image luminance data YI ₀ as initial data (step S101). As the initial image arbitrary data luminance data YI ₀ , the luminance data of the original image data Img ′ that is the imaged image data may be used, and any of solid black, white solid, gray solid, checkered pattern, etc. Such image luminance data may be used. In step S102, arbitrary image luminance data YI ₀ is input instead of YImg in the equation (1), and comparison image luminance data YI ₀ ′ in which the arbitrary image luminance data YI ₀ is changed by the change factor data G is obtained. Next, the original image luminance data YImg ′ and the comparative image luminance data YI ₀ ′ are compared to calculate difference luminance data Yδ (step S103).

Next, in step S104, it is determined whether or not the difference luminance data Yδ is equal to or greater than a predetermined value. If it is equal to or greater than the predetermined value, new arbitrary image luminance data (= restored image luminance data) YI _{0 + n} is determined in step S105. Process to generate. That is, the difference luminance data Yδ is distributed to the arbitrary image luminance data YI ₀ based on the change factor data G to generate restored image luminance data YI _{0 + n} . Thereafter, steps S102, S103, and S104 are repeated for the restored image luminance data YI _{0 + n} instead of the arbitrary image luminance data YI ₀ in step S102.

If the difference luminance data Yδ is smaller than the predetermined value in step S104, the process is terminated (step S106). Then, the restored image luminance data YI _{0 + n} at the time when the processing is completed is estimated as correct luminance data, that is, luminance data of the base image data Img without deterioration, and the luminance data is recorded in the recording unit 5. Note that arbitrary image luminance data YI ₀ and change factor data G may be recorded in the recording unit 5 and passed to the processing unit 4 as necessary.

  The concept of the above processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 2, but it is difficult as a real problem.

Solving as an optimization problem assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same.

In other words, as shown in FIGS. 4A and 4B, if comparative image luminance data YI ₀ ′ (YI _{0 + n} ′) that is approximate to the original image luminance data YImg ′ can be generated, The arbitrary image luminance data YI ₀ or the restored image luminance data YI _{0 + n as} data is approximate to the base image luminance data YImg that is the basis of the original image luminance data YImg ′.

  In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 μsec. In addition, in this embodiment, the value serving as a determination criterion for the difference luminance data Yδ is “6” when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. In addition, raw camera shake data detected by the angular velocity detection sensor does not correspond to actual camera shake when the sensor itself is insufficiently calibrated. Therefore, in order to deal with an actual camera shake, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

  Next, details of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. Further, when the image processing apparatus 1 is shaken due to camera shake during the exposure time at the time of photographing, the light energy is dispersed to the pixels that are shaken during the exposure time. Furthermore, if the blur during the exposure time is known, it is possible to know how to disperse the light energy during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. .., N−1, n, n + 1, n + 2, n + 3,..., And attention is paid to a certain pixel n. When there is no blur, the light energy during the exposure time is concentrated on the pixel, so the light energy concentration is “1.0”. This state is shown in FIG. The light energy amount distribution with respect to the luminance information for each pixel at this time is shown in the table of FIG. What is shown in FIG. 6 is the base image luminance data YImg for the luminance information about the base image data Img that is not deteriorated by camera shake. Each data is represented by 8-bit (0 to 255) data.

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the (n + 1) th pixel, and 20% of time is shifted to the (n + 2) th pixel. . The method of dispersing light energy is as shown in the table of FIG. This becomes the change factor data G.

  Since blurring is uniform for all pixels, assuming that there is no top blur (vertical blur), the blur situation is as shown in the table of FIG. The data shown as “ideal image” in FIG. 8 is the base image luminance data YImg of the base image data Img, and the data shown as “blurred image” is the original image of the original image data Img ′, which is the captured image. It becomes luminance data YImg ′. Specifically, for example, “120” of the pixel “n−3” is an allocation of “0.5”, “0.3”, and “0.2” of the change factor data G for blurring that is a factor of image change. According to the ratio, “60” is distributed to “n-3” pixels, “36” is distributed to “n-2” pixels, and “24” is distributed to “n−1” pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this original image luminance data YImg ′ and the change factor data G shown in FIG.

Any arbitrary image luminance data YI ₀ shown in step S101 can be used. However, in this description, original image luminance data YImg ′ is used. That is, the process starts with YI ₀ = YImg ′. In the table of FIG. 9, “input” corresponds to the arbitrary image luminance data YI ₀ . The change factor data G is applied to the arbitrary image luminance data YI ₀ , that is, the original image luminance data YImg ′ in step S102. That is, for example, “60” of the “n-3” pixel of the arbitrary image data YI ₀ is “30” for the n-3 pixel, “18” for the “n-2” pixel, “12” is allocated to each pixel of “1”. The other pixels are similarly distributed and the comparison image luminance data YI ₀ ′ shown as “output” is generated. As a result, the difference luminance data Yδ in step S103 is as shown in the bottom column of FIG.

Thereafter, in step S104, the size of the difference luminance data Yδ is determined. Specifically, the process is terminated when all the difference luminance data Yδ is less than 5 in absolute value, but the difference luminance data Yδ shown in FIG. 9 does not meet this condition, so the process proceeds to step S105. That is, the difference luminance data Yδ is distributed to the arbitrary image luminance data YI ₀ using the change factor data G, and the restored image luminance data YI _{0 + n} shown as “next input” in FIG. 10 is generated. In this case, since this is the first time, in FIG. 10, n = ₁ is represented as YI _{0 + 1} .

The distribution of the difference luminance data Yδ is, for example, “15” which is obtained by multiplying the data “30” of the pixel “n-3” by 0.5 which is the distribution ratio of its place (= “n-3” pixel). “N-3” pixels are distributed, and “n-2” pixel data “15” is multiplied by 0.3 which is the distribution ratio that should have come to the “n-2” pixel. 4.5 ”is allocated, and the data“ 9.2 ”of the pixel“ n−1 ”is multiplied by 0.2 which is the distribution ratio that should have come to the pixel“ n−1 ”. Allocate “1.84”. The total amount allocated to the “n-3” pixels is “21.34”, and this value is used as the arbitrary image luminance data YI ₀ (here, the original image luminance data YImg ′ which is the luminance data of the photographed image is used). In addition, restored image luminance data YI _{0 + 1} is generated. The restored image luminance data YI _{0 + 1} corresponds to “next input” in the table of FIG.

As shown in FIG. 11, the restored image luminance data YI _{0 + 1} becomes the input image data (= arbitrary image luminance data YI ₀ ) in step S102, and step S102 is executed. “Input” in the table of FIG. 11 corresponds to the restored image luminance data YI _{0 + 1} . Then, the process proceeds to step S103 to obtain new difference luminance data Yδ. The size of the new difference luminance data Yδ is determined in step S104, and if it is greater than or equal to a predetermined value, the new difference luminance data Yδ is distributed to the previous restored image luminance data YI _{0 + 1} in step S105, and the new restored image luminance data YI _{0 + 2} Is generated. The restored image luminance data YI _{0 + 2} corresponds to “next input” in the table of FIG. Thereafter, new comparison image luminance data YI _{0 + 2} ′ is generated from the restored image luminance data YI _{0 + 2} by performing step S102. As described above, after steps S102 and S103 are executed, the process goes to step S104, and the process proceeds to step S105 or shifts to step S106 depending on the determination there. Such a process is repeated.

  In this image processing apparatus 1, in processing, in step S <b> 104, either one or both of the number of processes and the determination reference value of the difference luminance data Yδ can be set in advance. For example, an arbitrary number such as 20 times or 50 times can be set as the number of times of processing. Further, the value of the difference luminance data Yδ for stopping the processing is set to “5” in 8 bits (0 to 255), and when it becomes 5 or less, the processing is ended, or “0.5” is set to “0”. .5 "or less, the process can be terminated. This set value can be set arbitrarily. When both the number of processing times and the judgment reference value are input, the processing is stopped when either one is satisfied. When both settings are possible, the determination reference value may be given priority, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

  The above-described process shown in FIG. 3 is also performed on the original image color difference data UImg ′ and the original image color difference data VImg ′. In this case, when processing is performed on the original image color difference data UImg ′, the contents of steps S101 to S105 shown in FIG. 3 are as follows.

First, in step S101, arbitrary image color difference data UI ₀ is prepared instead of arbitrary image luminance data YI ₀ . The arbitrary image color difference data UI ₀ may be stored in the recording unit of the processing unit 4 in advance, but here, original image color difference data UImg ′ which is color difference data of the original image data Img ′ is used.

In step S102, the arbitrary image chrominance data UI _0, by the action of the change factor data G, determine the comparative image chrominance data UI ₀ '. In step S103, the original image color difference data UImg ′ and the comparison image color difference data UI ₀ ′ are compared to calculate difference color difference data Uδ.

In step S104, it is determined whether or not the difference color difference data Uδ is equal to or greater than a predetermined value. If the difference color difference data Uδ is equal to or greater than the predetermined value, new arbitrary image color difference data (= restored image color difference data) UI _{0 + 1} is generated in step S105. Processing is performed. Thereafter, the restored image color difference data UI _{0 + 1} is set as the arbitrary image color difference data UI ₀ in step S102, and step 102, step 103, and step 104 are repeated.

If the difference color difference data Uδ is smaller than the predetermined value in step S104, the process is terminated (S106). The restored image color difference data UI _{0 + n} ′ at the end of this process is estimated as the correct color difference data, that is, the color difference information data for the blue color of the base image data Img without deterioration, and the restored image color difference data UI _{0 + n} ′ is Record in the recording unit 5.

  Next, the case where the processing from step S101 to step S105 shown in FIG. 3 is performed on the original image color difference data VImg ′ will be described.

First, in step S101, arbitrary image color difference data VI ₀ is prepared instead of arbitrary image luminance data YI ₀ . The arbitrary image color difference data VI ₀ may be stored in the recording unit of the processing unit 4 in advance, but here, original image color difference data VImg ′ which is color difference data of the original image data Img ′ is used.

In step S102, change factor data G is applied to the arbitrary image color difference data VI ₀ to obtain comparison image color difference data VI ₀ ′. In step S103, the original image color difference data VImg ′ is compared with the comparison image data VI ₀ ′ to calculate difference color difference data Vδ.

In step S104, it is determined whether or not the difference color difference data Vδ is equal to or greater than a predetermined value. If the difference color difference data Vδ is equal to or greater than the predetermined value, new arbitrary image color difference data (= restored image color difference data) VI _{0 + 1} is generated in step S105. Processing is performed. Thereafter, the restored image color difference data VI _{0 + 1} is set as the arbitrary image color difference data VI ₀ in step S102, and step 102, step 103, and step 104 are repeated.

If the difference color difference data Vδ is smaller than the predetermined value in step S104, the process ends (S106). The restored image color difference data VI _{0 + n} ′ at the end of this processing is estimated as correct color difference data, that is, color difference information data for red of the base image data Img without deterioration, and this restored image color difference data VI _{0 + n} ′. Is recorded in the recording unit 5.

  Note that the modified example of the processing shown in FIG. 3 and the processing of the original image luminance data described with reference to FIGS. 4 to 12 are applied to the processing of the original image color difference data UImg ′ and the original image color difference data VImg ′ as they are. can do.

The restored image luminance data YI _{0 + n} ', the restored image color difference data UI _{0 + n} ', and the restored image color difference data VI _{0 + n} 'obtained as described above and recorded in the recording unit 5 are converted into RGB signals and synthesized. Thus, restored image data I _{0 + n} ′ in which the blur is eliminated or almost eliminated is generated. The restored image data I _{0 + n} ′ is estimated as basic image data Img and displayed on a display unit such as a display (not shown) or recorded in the recording unit 5.

By the way, the separation of the YUV signal (that is, the original image luminance data YImg ′, the original image color difference data UImg ′ and the original image color difference data VImg ′) from the original image data Img ′ is performed by the following equation, for example.

YImg '= 0.29891 x R + 0.58661 x G + 0.11448 x B (3)
UImg ′ = − 0.16974 × R−0.33126 × G + 0.50000 × B (4)
VImg ′ = 0.50000 × R−0.41869 × G + 0.08131 × B (5)

Here, R, G, and B are light energy amount (luminance) data for the R (red), G (green), and B (blue) color components of the original image data Img ′. Then, the restored image luminance data YI _{0 + n} ′, the restored image color difference data UI _{0 + n} ′, and the restored image luminance data VI _{0 + n} ′, which are the results restored by the above-described processing of FIG. Convert to

R = YI _{0 + n} ′ + 1.40200 × VI _{0 + n} ′ (6)
G = YI _{0 + n′−} 0.34414 × UI _{0 + n′−} 0.71414 × VI _{0 + n} ′ (7)
B = YI _{0 + n} ′ + 1.77200 × UI _{0 + n} ′ (8)

Then, the restored image I _{0 + n} ′ is obtained by synthesizing the RGB signals obtained by the above formula. The above formulas (3) to (8) are examples, and other conversion formulas may be adopted.

Here, in the equations (4) and (5), which are conversion equations from RGB signals to “UImg ′” and “VImg ′”, there is a term for performing a subtraction operation. Therefore, “UImg ′” and “VImg ′” may be negative values. In this case, when the 8 bits are set from 0 to 255, the processing of FIG. 3 is hindered. Therefore, by setting the 8-bit value from −128 to 127, it is possible to cope with negative values of “UImg ′” and “VImg ′”. In the restoration process of the original image color difference data UImg ′ and the original image color difference data VImg ′, “UI _{0 + n} ′” or “VI _{0 + n} ′” exceeds the range of −128 to 127 during the repetition process. The amount exceeding the negative side (minus side) is processed as -128, and conversely, the amount exceeding the positive side (plus side) is processed as 127. For “YI _{0 + n} ′”, when a value exceeding 255 is restored, a process of limiting to 255 is performed. By doing so, it is possible to prevent abnormal processing and restoration to strange colors.

In the equations (6), (7), and (8), depending on the values of “UI _{0 + n} ′” and “VI _{0 + n} ′”, the RGB value may be negative in the calculation formula. In RGB, RGB values never become negative. Also in this case, with respect to the portion exceeding the range from 0 to 255, the portion exceeding the negative side (minus side) is corrected to 0, and conversely, the portion exceeding the positive side (plus side) is corrected to 255. The value is an RGB value, and this RGB is processed. Alternatively, “YImg ′”, “UImg ′”, “VImg ′” are calculated from the corrected RGB values by the equations (3), (4), and (5), and these “YImg ′”, “UImg ′” The restoration process shown in FIG. 3 is performed by “′” and “VImg ′”, and RGB is again calculated by the equations (6), (7), and (8). Then, until the RGB value becomes a positive value, the correction of the RGB value and “YImg ′”, “UImg ′”, “VImg ′” based on the corrected RGB values are calculated and restored as shown in FIG. The process and the process of calculating RGB may be repeated.

By the way, the human eye is less sensitive to color changes than it is to brightness changes. In other words, the human sensation of viewing the captured image is less sensitive to color changes than the sensitivity to luminance changes. Therefore, even if the amount of information about color difference is lowered, it may not be discernible to human vision. Therefore, the original image color difference data UImg ′ and the original image color difference data VImg ′, which are data relating to color differences, are subjected to compression processing to reduce the data amount, and the compressed original image color difference data UImg ′ and original image color difference data VImg ′. By performing the restoration process described above for, the processing speed of the restoration process can be increased. Further, since the amount of data is reduced by performing the compression process, the recording capacity of the recording unit 5 or the recording unit of the processing unit 4 can be reduced. The restored image color difference data UI _{0 + n} ′ and restored image luminance data VI _{0 + n} ′ obtained as a result of the restoration process based on the compressed data have low restoration accuracy.

However, the original image luminance data YImg ′ relating to brightness sensitive to human eyes is restored without being compressed. That is, the restored image data I _{0 + n} ′ is restored with high accuracy for the luminance component. Therefore, the video displayed based on the restored image data I _{0 + n} ′ can be recognized as a well restored image by human eyes.

  As the compression method, the original image data luminance data YImg ′ is processed with 8 bits, while the original image color difference data UImg ′ and the original image color difference data VImg ′ are processed with 6 bits. In addition to a method of processing with a reduced number of data bits, there is a method of thinning out processing pixels (for example, every other processing pixel or every other processing pixel). When processing pixels are thinned out, after converting into RGB signals, the processed pixels are used to complement the RGB signals of pixels that have not been processed.

Further, in step S104, when the determination reference value is set for the difference luminance data Yδ and the difference color difference data Uδ, Vδ, the restoration processing of the original image color difference data UImg ′ and the original image color difference data VImg ′ Even when the difference reference data for the color difference data Uδ and Vδ is larger than the determination reference value for the difference luminance data Yδ, the restoration process is terminated (step S106), so that the restoration process is completed. The processing time may be shortened. Since the original image color difference data UImg ′ and the original image color difference data VImg ′ are data relating to the color difference, they are displayed based on the restored image data I _{0 + n} ′ without being restored as accurately as the original image luminance data YImg ′. The video can be recognized by the human eye as a well-restored image. Therefore, for example, in the above description, when the difference luminance data Yδ is less than 5, the process is terminated (step S106). However, the original image color difference data UImg ′ and the original image color difference data VImg ′ are restored. As for the processing, when the difference color difference data Uδ, Vδ is less than 10, the processing is ended (step S106), so that the original image color difference data UImg ′ and the original image color difference data VImg ′ are restored. Time may be achieved.

  In step S104, when the number of times of processing is set in advance, the original image luminance data YImg ′ is set to, for example, 20 times as the number of times of processing, whereas the original image color difference data UImg ′ and the original image are set. Regarding the restoration process of the color difference data VImg ′, the number of processes may be set to 10 times. By doing so, it is possible to shorten the restoration process of the original image color difference data UImg ′ and the original image color difference data VImg ′, and to shorten the restoration process time of the original image data Img ′. it can. Note that the number of repetitions of the two original image color difference data UImg ′ and VImg ′ may be made different.

  In the description of the embodiment described above, the information stored in the factor information storage unit 7 is not used, but the known change factors stored here, such as optical aberrations and lens distortions, etc. The data may be used. In this case, for example, in the processing method of the previous example (FIG. 3), it is preferable to perform processing by combining blur information and optical aberration information as one change factor. You may make it correct | amend with the information of an optical aberration after it complete | finishes. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors at the time of shooting, for example, shake.

  The image processing apparatus 1 according to the embodiment of the present invention has been described above, but various modifications can be made without departing from the gist of the present invention. For example, the processing performed by the processing unit 4 is configured by software, but may be configured by hardware composed of parts that perform part of the processing.

  In addition to the photographed image, the original image to be processed may be a photographed image subjected to processing such as color correction or Fourier transform. Further, as the comparison image data, in addition to the data generated using the change factor data G, data generated using the change factor data G may be data obtained by performing color correction or Fourier transform. . Further, the change factor data may include, in addition to data (information) that causes image degradation due to camera shake, information that simply changes the image, and information that improves the image contrary to degradation. .

  Moreover, each processing method mentioned above may be programmed. Alternatively, the program may be stored in a storage medium, such as a CD, DVD, or USB memory, and read by a computer. In this case, the image processing apparatus 1 has reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading a program in the storage medium.

Claims (7)

In an image processing apparatus having a processing unit for processing a color image,
The processing unit
The original image luminance data and the original image color difference data are separated from the original image data, which is the original image data to be processed,
Each of the original image luminance data and the original image color difference data is processed data,
For the original image luminance data,
Using the change factor data which is the data of the factor of image change, from the arbitrary image luminance data which is arbitrary image luminance data, generate comparative image luminance data for comparison in which the arbitrary image luminance data is changed,
Obtaining difference luminance data which is difference data comparing the original image luminance data and the comparative image luminance data;
Generating restored image luminance data in which the difference luminance data is distributed to the arbitrary image luminance data based on the change factor data;
By using the restored image luminance data instead of the arbitrary image luminance data and repeating the same process,
A process of generating base image luminance data of a base image that is an image before the original image is changed or restored restored image luminance data approximated thereto,
For the original image color difference data,
Using the change factor data, from the arbitrary image color difference data which is arbitrary image color difference data, to generate comparative image color difference data for comparison, which is a change of the arbitrary image color difference data,
Obtaining difference color difference data which is difference data comparing the original image color difference data and the comparison image color difference data;
Generating restored image color difference data in which the difference color difference data is distributed to the arbitrary image color difference data based on the change factor data;
By using the restored image color difference data instead of the arbitrary image color difference data, and repeating the same process,
A process of generating base image color difference data of a base image that is an image before the original image is changed or restored restored image color difference data approximated thereto,
An image processing apparatus.   2. The process according to claim 1, wherein the restored image luminance data and the restored image color difference data are combined to generate base image data of the base image or restored restored image data that approximates the base image data. Image processing device.   The image processing apparatus according to claim 1, wherein the original image color difference data is set to two, and the repetition processing is performed for each.   4. The original image color difference data is subjected to compression processing, and the repetitive processing is performed on compressed compressed image color difference data to generate the restored image color difference data. The image processing apparatus described.   The image processing apparatus according to claim 4, wherein the compression of the original image color difference data is a thinning process for thinning out the data of the original image color difference data.   5. The compression of the image color difference data is performed by processing the generation of the restored image color difference data with a bit number smaller than the number of bits for performing the process of generating the restored image luminance data. Image processing apparatus. In an image processing method for processing a color image,
Separating the original image luminance data and the original image color difference data from the original image data that is the original image data to be processed;
Each of the original image luminance data and the original image color difference data is processed data,
For the original image luminance data,
Generating comparison image luminance data for comparison in which the arbitrary image luminance data is changed from arbitrary image luminance data, which is arbitrary image luminance data, by using the change factor data, which is image change factor data; ,
Obtaining difference luminance data which is difference data comparing the original image luminance data and the comparative image luminance data;
Generating restored image luminance data in which the difference luminance data is distributed to the arbitrary image luminance data based on the change factor data;
Have
By using the restored image luminance data instead of the arbitrary image luminance data and repeating the same process,
A process of generating base image luminance data of a base image that is an image before the original image is changed or restored restored image luminance data approximated thereto,
For the image color difference image data,
Using the change factor data to generate comparison image color difference data for comparison in which the arbitrary image color difference data is changed from arbitrary image color difference data which is arbitrary image color difference data;
Obtaining difference color difference data, which is difference data obtained by comparing the original image color difference data and the comparison image color difference data;
Generating restored image color difference data in which the difference color difference data is distributed to the arbitrary image color difference data based on the change factor data;
Have
By using the restored image color difference data instead of the arbitrary image color difference data, and repeating the same process,
A process of generating base image color difference data of a base image that is an image before the original image is changed or restored restored image color difference data approximated thereto,
An image processing method.

Images