JP2006121138A - Image processor, image processing method and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and a processor for realizing efficient image processing by applying a reduced image to perform false color correction such as purple fringe to reduce a calculation cost. <P>SOLUTION: Reducing processing of image data to be correction-processed is executed, image processing including pixel value correction of false pixels around halation for a reduced image by stopgap interpolation processing and each correction processing to which the degree p of a false color (PF) is applied to generate two correction images; and these different result images are enlarged and blended, thereby generating a final correction image. This configuration enables correction for the reduced image, and image correction can be performed with less calculation cost. Also, correction processing in which image corrections are merged in different processing modes can be realized, and harmonized pixel value correction in which advantage of each of the processing modes are merged can be realized. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program. More specifically, the present invention relates to an image processing apparatus, an image processing method, and a computer program that correct image data in which false color due to lens aberration occurs and generate high-quality image data.

  When photographing images with a camera, various problems are caused by the aberration of the lens. Examples of typical aberrations include Seidel's five aberrations, which are monochromatic aberrations. This is five aberrations based on the spherical surface of the lens analyzed by Seidel in Germany, and is a collective term for each aberration of spherical aberration, coma aberration, astigmatism, distortion aberration, and field distortion. In addition to these aberrations, chromatic aberration is also known to cause significant problems. Chromatic aberration is caused by the fact that the refractive index of light with respect to the lens material differs depending on the wavelength, and a false color is generated on the imaging surface.

  As typical examples of chromatic aberration, axial chromatic aberration in which color blur occurs because the focal position on the optical axis differs depending on wavelength, and lateral chromatic aberration in which color shift occurs because image magnification varies depending on wavelength are well known. A phenomenon generally called “Purple Fringe” in English is also an important cause of image deterioration. This is a phenomenon in which a false color is generated at an edge portion in an image because the point image distribution differs depending on the wavelength of light. Even if it is not so conspicuous with normal pixels, if there is a high-contrast edge part where whiteout occurs as a saturated state at the luminance level, a purple false color occurs around it and an unnatural image is formed Will be. In general, false colors that occur in the vicinity of whiteout are called purple fringes because many purple colors are generated. However, the false color varies depending on the lens, shooting conditions, and the like, such as when the color becomes greenish. Hereinafter, the purple fringe refers to a false color generation phenomenon at an edge portion of high contrast where whiteout occurs regardless of the generated color.

  As a technique for reducing chromatic aberration, there is a lens using glass made of a special material such as fluorite. However, manufacturing such a lens is expensive and expensive, and is used in some high-end cameras such as interchangeable lens cameras. However, it cannot be said that it is generally widely used.

  Patent Document 1 discloses a method for reducing false colors caused by chromatic aberration by image processing. This is a process for suppressing the color, that is, reducing the saturation, in a portion where the high frequency component of the green channel is high. Also, as a countermeasure against overexposure, a configuration is disclosed in which two images are captured with different exposures, the original brightness of the overexposed portion is estimated, and the saturation is reduced.

However, since the processing described in Patent Document 1 reduces false colors due to a decrease in saturation, the saturation of the original subject color also decreases, making it impossible to reproduce the original true color of the subject. As a result, there is a problem that an unnatural image is output. In addition, it is necessary to shoot twice in order to estimate the brightness of the overexposed part. If camera shake or subject shake occurs between these two shoots, it is difficult to obtain a correct result. There is.
JP 2003-60983 A

  The present invention has been made in view of the above-described problems, and pays attention to false colors such as purple fringes that occur in the vicinity of overexposure, efficiently detects this false color area, and performs partial correction. Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a computer program that can generate and output high-quality image data without causing an influence on the entire image.

  In particular, the present invention executes a reduction process of image data in a special manner before executing false color correction. Specifically, in the reduction of the luminance data, the reduction is performed so that the overexposed pixel data is not lost, and in the reduction of the color information, the reduction is performed so that the false color data such as purple fringe is not lost. Image processing apparatus that realizes reduction in calculation cost of image processing in correction and enables high-quality pixel correction by executing correction of false color pixels or the like by image processing on the reduced data obtained by the reduction processing And an image processing method and a computer program.

  Further, the present invention calculates a false color degree as a parameter indicating the possibility of a false color for a target pixel selected from the constituent pixels of the image data, and determines the original pixel value of the target pixel according to the false color degree. The correction pixel value of the target pixel is calculated by changing the contribution ratio with the interpolation value generated based on the pixel values of the surrounding pixels, and the correction pixel value acquired as a result of the interpolation processing for filling the hole Image processing apparatus and image processing method capable of realizing a correction process with a higher degree of freedom and generating high-quality image data by calculating and outputting an optimum pixel value by blending processing of And a computer program.

The first aspect of the present invention is:
An image processing device,
A reduction processing unit that executes a reduction process of the correction target image data;
An image correction processing unit that performs image processing including pixel value correction of a false color pixel around a whiteout on the reduced image generated in the reduction processing unit;
An enlargement processing unit for executing an enlargement process of the corrected image generated in the image correction processing unit;
An image processing apparatus characterized by comprising:

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the reduction processing unit is configured to individually execute luminance information reduction processing and color information reduction processing on the original correction processing target image data. In the reduction process of luminance information, a reduction process that does not erase overexposed pixel information included in the correction process target image data is performed. In the reduction process of color information, the false color included in the correction process target image data is processed. The present invention is characterized in that a reduction process that does not lose color information is executed.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the reduction processing unit is configured to reduce the luminance corresponding to the constituent pixels of the reduced image in the luminance information reduction processing for the correction processing target image data. The present invention is characterized in that a process for setting the maximum luminance included in the corresponding image area of the image data is executed.

  Furthermore, in one embodiment of the image processing apparatus of the present invention, the reduction processing unit converts the color information corresponding to the constituent pixels of the reduced image into the correction process in the color information reduction process for the correction processing target image data. When a false color pixel is included in the corresponding image area of the target image data, it is set as an average value of the false color pixels included in the corresponding image area, and the false color pixel is included in the corresponding image area of the correction processing target image data If not, the processing is performed to set the average value of all the pixels included in the corresponding image area.

Furthermore, in an embodiment of the image processing apparatus of the present invention, the image correction processing unit calculates a false color degree p of the constituent pixels of the reduced image generated by the reduction processing unit, and sets the calculated false color degree p to It is a configuration that executes correction processing based on an averaging method that executes pixel value correction according to
For a target pixel selected from the constituent pixels of the reduced image, based on a false color degree calculation unit that calculates a false color degree p as a parameter indicating the possibility of a false color, and a pixel value in a region near the target pixel, An interpolation value calculation unit that calculates an interpolation value corresponding to the pixel of interest; an original pixel value of the pixel of interest and the interpolation value; and the original pixel value and the interpolation value according to the false color degree p of the pixel of interest And a correction pixel value calculation unit that calculates a correction pixel value of the target pixel by changing the contribution ratio of the target pixel.

Furthermore, in one embodiment of the image processing apparatus of the present invention, the false color degree calculation unit is configured to obtain a pixel of interest
(A) White jump distance evaluation value P _d indicating the proximity to white jump (high luminance pixel),
(B) a saturation evaluation value P _s indicating the magnitude of saturation;
(C) Hue evaluation value P _θ indicating the proximity to the specific hue corresponding to the false color,
The false color degree p of the target pixel is calculated by applying at least one of the above.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the image correction processing unit detects a skipped pixel from constituent pixels of the reduced image generated by the reduction processing unit, and false color pixels around the skipped pixel Correction processing based on the hole filling method for correcting the pixel value of the pixel based on the pixel values of the surrounding pixels, the overexposure detection unit that detects overexposed pixels from the image data, and the overexposure detection unit detects A false color pixel detection region setting unit that sets a false color pixel detection region around a whiteout pixel, and a pixel having a color corresponding to the false color within the region set by the false color pixel detection region setting unit is false color A false color detection unit that is specified as a pixel, and a pixel value correction unit that performs pixel value correction processing based on surrounding pixel values for the false color pixel detected by the false color detection unit, To do.

  Furthermore, in one embodiment of the image processing device of the present invention, the pixel value correction unit includes a hole-filling interpolation unit that performs hole-filling interpolation processing based on surrounding pixel values for false color pixels, and color blurring processing for false color pixels. And a color blur processing unit for executing the above.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the pixel value correction unit selects other pixels excluding the false color pixel and the overexposed pixel from the pixels existing around the false color pixel, and sets the selected pixel as the selected pixel. It is the structure which performs the hole-filling interpolation process based on.

  Furthermore, in one embodiment of the image processing apparatus of the present invention, the enlargement processing unit is configured to enlarge the corrected image by blending the enlarged data of the correction image generated by the image correction processing unit and the original correction target image data. It is the structure which produces | generates.

  Furthermore, in one embodiment of the image processing device of the present invention, the image correction processing unit applies a false color around a whiteout in a first image correction processing mode with respect to the reduced image generated in the reduction processing unit. A first image correction processing unit that performs image processing including pixel value correction of a pixel, and a second image correction process different from the first image correction processing mode for the reduced image generated in the reduction processing unit And a second image correction processing unit that performs image processing including pixel value correction of the surrounding false color pixels, and the enlargement processing unit generates the first image generated in the first image correction processing unit. It is the structure which performs the expansion process of a correction image and the 2nd correction image produced | generated in the said 2nd image correction process part, Furthermore, the enlarged 1st correction image and the enlarged 2nd correction which were expanded in the said expansion process part Bren with images It characterized by having a blend processing unit for generating the final corrected image by processing.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the enlargement processing unit performs a blending process of the enlarged data of the first correction image generated in the first image correction processing unit and the original correction processing target image data. A configuration for generating an enlarged first corrected image and generating an enlarged second corrected image by blending the enlarged data of the second corrected image generated by the second image correction processing unit and the original correction processing target image data It is characterized by being.

Furthermore, in an embodiment of the image processing apparatus of the present invention, the blend processing unit generates a final corrected image by blending the enlarged first corrected image and the enlarged second corrected image enlarged in the enlargement processing unit. The pixel value of the constituent pixel (x, y) of the enlarged first corrected image is C _avgnew (x, y), and the pixel of the constituent pixel (x, y) of the enlarged second corrected image When the value is C _holenew (x, y), the pixel value C _final (x, y) of the constituent pixel (x, y) of the final corrected image is expressed by the following equation:
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
However, k = 0-1
It is the structure which performs the process calculated according to this.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the false color is a purple fringe, and the image correction processing unit performs overexpanded purple fringe on the reduced image generated by the reduction processing unit. The present invention is characterized in that image processing including pixel value correction of pixels is executed.

Furthermore, the second aspect of the present invention provides
An image processing method,
A reduction processing step for executing the reduction processing of the correction target image data;
An image correction processing step for performing image processing including pixel value correction of a false color pixel around the image on the reduced image generated in the reduction processing step;
An enlargement process step for executing an enlargement process of the corrected image generated in the image correction process step;
An image processing method characterized by comprising:

  Further, in one embodiment of the image processing method of the present invention, the reduction processing step separately executes luminance information reduction processing and color information reduction processing on the original correction processing target image data, In the reduction process, a reduction process is performed so as not to lose overexposed pixel information included in the correction process target image data. In the color information reduction process, false color color information included in the correction process target image data is processed. A reduction process that does not disappear is executed.

  Furthermore, in one embodiment of the image processing method of the present invention, the reduction processing step is configured to reduce the luminance corresponding to the constituent pixels of the reduced image in the luminance information reduction processing for the correction processing target image data. A process for setting the maximum luminance included in the corresponding image area of the image data is executed.

  Furthermore, in one embodiment of the image processing method of the present invention, the reduction processing step is configured to reduce the color information corresponding to the constituent pixels of the reduced image in the color information reduction processing for the correction processing target image data. When a false color pixel is included in the corresponding image area of the target image data, it is set as an average value of the false color pixels included in the corresponding image area, and the false color pixel is included in the corresponding image area of the correction processing target image data If not, a process of setting as an average value of all the pixels included in the corresponding image area is executed.

  Furthermore, in an embodiment of the image processing method of the present invention, the image correction processing step calculates a false color degree p of the constituent pixels of the reduced image generated in the reduction processing step, and sets the calculated false color degree p to A false color degree p as a parameter indicating a possibility of false color for a target pixel selected from the constituent pixels of the reduced image. Applying a false color degree calculating step for calculating the interpolation value, an interpolation value calculating step for calculating an interpolation value corresponding to the pixel of interest based on a pixel value in the vicinity region of the pixel of interest, and the original pixel value of the pixel of interest and the interpolation value And a correction pixel value calculation step of calculating a correction pixel value of the target pixel by changing a contribution ratio between the original pixel value and the interpolation value according to the false color degree p of the target pixel. And wherein the Mukoto.

Furthermore, in one embodiment of the image processing method of the present invention, the false color degree calculation step includes:
(A) White jump distance evaluation value P _d indicating the proximity to white jump (high luminance pixel),
(B) a saturation evaluation value P _s indicating the magnitude of saturation;
(C) Hue evaluation value P _θ indicating the proximity to the specific hue corresponding to the false color,
This is a step of calculating the false color degree p of the target pixel by applying at least one of the above.

  Furthermore, in one embodiment of the image processing method of the present invention, the image correction processing step detects a whiteout pixel from constituent pixels of the reduced image generated in the reduction processing step, and a false color pixel around the whiteout pixel. Is a step of executing correction processing based on a hole-filling method for correcting the pixel value of the pixel based on the pixel values of surrounding pixels, the step of detecting overexposure from image data, and the detection of the overexposure detection step A false color pixel detection area setting step for setting a false color pixel detection area around a whiteout pixel, and a pixel having a color corresponding to the false color within the area set by the false color pixel detection area setting step A pixel value for executing a pixel value correction process based on the surrounding pixel value for the false color detection step specified as a pixel and the false color pixel detected by the false color detection step Characterized in that it comprises a positive step.

  Furthermore, in an embodiment of the image processing method of the present invention, the pixel value correcting step includes a hole filling interpolation step for performing hole filling interpolation processing based on surrounding pixel values for false color pixels, and a color blurring process for false color pixels. And a color blur processing step for executing the above.

  Furthermore, in an embodiment of the image processing method of the present invention, the pixel value correcting step selects other pixels excluding the false color pixel and the overexposed pixel from the pixels existing around the false color pixel, and sets the selected pixel as the selected pixel. The method includes a step of executing a hole filling interpolation process based on the above.

  Furthermore, in an embodiment of the image processing method of the present invention, the enlargement processing step includes an enlarged corrected image obtained by blending the enlarged data of the correction image generated in the image correction processing step and the original correction target image data. It is the step which produces | generates.

  Furthermore, in an embodiment of the image processing method of the present invention, the image correction processing step is a first image correction processing mode for the reduced image generated in the reduction processing step, and the false color surrounding the overexposure is obtained. A first image correction processing step for performing image processing including pixel value correction of a pixel, and a second image correction processing different from the first image correction processing mode for the reduced image generated in the reduction processing step. And a second image correction processing step for executing image processing including pixel value correction of the surrounding false color pixels, wherein the enlargement processing step is a first image generated in the first image correction processing step. A step of enlarging the corrected image and the second corrected image generated in the second image correction processing step; It characterized by having a blend processing step of generating a final corrected image and the enlarged been first corrected image by blending processing of the enlargement already second corrected image.

  Furthermore, in an embodiment of the image processing method of the present invention, the enlargement processing step is performed by blending the enlarged data of the first correction image generated in the first image correction processing step and the original correction processing target image data. An enlarged second corrected image is generated by blending the enlarged data of the second corrected image generated in the step of generating the enlarged first corrected image and the second corrected image generated in the second image correction processing step and the original correction processing target image data. And a step of performing.

Furthermore, in one embodiment of the image processing method of the present invention, the blending step generates a final corrected image by blending the enlarged first corrected image and the enlarged second corrected image enlarged in the enlarging step. The pixel value of the constituent pixel (x, y) of the enlarged first corrected image is C _avgnew (x, y), and the pixel of the constituent pixel (x, y) of the enlarged second corrected image When the value is C _holenew (x, y), the pixel value C _final (x, y) of the constituent pixel (x, y) of the final corrected image is expressed by the following equation:
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
However, k = 0-1
The process of calculating according to is performed.

  Furthermore, in an embodiment of the image processing method of the present invention, the false color is a purple fringe, and the image correction processing step is performed by using a purple fringe surrounding the reduced image generated in the reduction processing step. Image processing including pixel value correction of pixels is executed.

Furthermore, the third aspect of the present invention provides
A computer program for executing image processing on a computer;
A reduction processing step for executing the reduction processing of the correction target image data;
An image correction processing step for performing image processing including pixel value correction of a false color pixel around the image on the reduced image generated in the reduction processing step;
An enlargement process step for executing an enlargement process of the corrected image generated in the image correction process step;
There is a computer program characterized by comprising:

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the reduction processing of the correction processing target image data is executed, and the reduced image is processed in the first image correction processing mode and includes the pixel value correction of the surrounding false color pixels. To generate a first corrected image, and further, for the reduced image, perform pixel value correction of the surrounding false color pixels in a second image correction processing mode different from the first image correction processing mode. In this configuration, the second corrected image is generated by executing the image processing, and the final corrected image is generated by magnifying and blending these different processed result images. Therefore, the image can be corrected at a low calculation cost. In addition, a correction process that combines image corrections in different processing modes is realized, and harmonized pixel value correction that combines the respective advantages is possible.

  Further, according to the configuration of the present invention, it is possible to perform harmonized pixel value correction that combines the advantages of the hole-filling interpolation processing and the correction processing to which the false color (PF) degree p is applied. It is possible to obtain a corrected image that reduces the color and does not fail. Specifically, it is possible to appropriately correct false colors such as purple fringes due to chromatic aberration occurring in an image photographed by a camera, and it is possible to realize generation and output of high-quality image data.

  Furthermore, according to the configuration of the present invention, it is possible to generate an image without failure even when all image correction processing is performed as automatic processing. By applying the image processing of the present invention, it is possible to correct the captured image so that it looks more natural. Therefore, it is not necessary to pay attention to the lens aperture and focal length so that purple fringing does not occur during shooting, and shooting can be performed with a higher degree of freedom.

  Furthermore, according to the configuration of the present invention, the false color correction process is performed after the reduction process, and the enlargement process is performed. However, the part where the pixel value is updated in the original image is only the part where the false color is generated, and only the color information is corrected. Therefore, the portion other than the false color is not deteriorated at all. Since a special reduction process is performed on the portion where the false color is generated, the false color pixel can be reliably recognized. As described above, although the calculation cost is reduced by the reduction process, the false color correction quality is hardly deteriorated.

  Hereinafter, an image processing apparatus, an image processing method, and a computer program according to the present invention will be described in detail with reference to the drawings.

  First, a configuration example of the image processing apparatus will be described with reference to FIG. The image processing apparatus illustrated in FIG. 1 includes an imaging unit and is illustrated as an example of an apparatus that performs correction processing on image data captured by the imaging unit. However, the image processing apparatus of the present invention is a storage device such as a hard disk. It is also possible to execute correction of the input image by inputting the image data stored in the unit, and the image data to be corrected is not only data input via the imaging unit, but also storage means or network It is possible to cope with any input image data such as image data input via the. FIG. 1 shows a configuration example of an image processing apparatus according to the present invention.

  A detailed configuration of the image processing apparatus shown in FIG. 1 will be described. As shown in FIG. 1, the image processing apparatus includes a lens 101, a diaphragm 102, a solid-state image sensor 103, a correlated double sampling circuit 104, an A / D converter 105, a DSP block 106, a timing generator 107, a D / A converter 108, A video encoder 109, a video monitor 110, a codec (CODEC) 111, a memory 112, a CPU 113, an input device 114, a flash control device 115, and a flash light emitting device 116 are included.

  Here, the input device 114 refers to operation buttons such as a recording button on the camera body. The DSP block 106 is a block having a signal processing processor and an image RAM. The signal processing processor can perform pre-programmed image processing on image data stored in the image RAM. ing. Hereinafter, the DSP block is simply referred to as a DSP.

The overall operation of this embodiment will be described below.
Incident light that has passed through the optical system and reached the solid-state image sensor 103 first reaches each light receiving element on the imaging surface, is converted into an electrical signal by photoelectric conversion at the light receiving element, and is subjected to noise by the correlated double sampling circuit 104. Removed and digitized by A / D converter 105. After being converted into a signal, it is temporarily stored in an image memory in the digital signal processing unit (DSP) 106. If necessary, the flash light emitting device 116 can be caused to emit light via the flash control device 115 during photographing.

  In the state during imaging, the timing generator 107 controls the signal processing system so as to maintain image capture at a constant frame rate. A stream of pixels is also sent to the digital signal processing unit (DSP) 106 at a constant rate, and after appropriate image processing is performed there, the image data is sent to the D / A converter 108 and / or the codec (CODEC) 111 or both. Sent. The D / A converter 108 converts the image data sent from the digital signal processing unit (DSP) 106 into an analog signal, which is converted into a video signal by the video encoder 109 so that the video signal can be monitored by the video monitor 110. In this embodiment, the video monitor 110 serves as a camera finder. The codec (CODEC) 111 encodes image data sent from the digital signal processing unit (DSP) 106, and the encoded image data is recorded in the memory 112. Here, the memory 112 may be a recording device using a semiconductor, a magnetic recording medium, a magneto-optical recording medium, an optical recording medium, or the like.

  The above is the description of the entire system of the digital video camera as an example of the image processing apparatus of the present embodiment. Image processing according to the present invention, that is, image data correction processing is executed by the digital signal processing unit (DSP) 106. The Details of the image processing will be described below. Note that, as the color space of the image handled in the present embodiment, one in which luminance information and color information are defined independently is used. For example, a YCbCr space or a CIEL * a * b * space may be used.

  Hereinafter, in the image processing apparatus according to the present invention, attention is paid to false colors such as purple fringes that appear in the vicinity of the overexposure that appear as pixels whose luminance level is substantially equal to or higher than the saturation level. It has a configuration that makes it possible to generate and output quality image data. The false color (purple fringe) is a phenomenon that appears conspicuously around the overexposed portion, and is characterized in that, for example, a purple false color is generated due to a color shift due to chromatic aberration. The size of the area around the whiteout where false color occurs is linked to the aperture and focal length of the optical system, and also changes depending on the distance from the optical center. In addition, the direction in which the false color is generated also occurs in the direction from the optical center toward the outside of the pixel and from the highlight pixel to the optical center, and the tendency varies depending on the optical system to be imaged.

  Hereinafter, details of the image data correction processing executed in the digital signal processing unit (DSP) 106 will be described. FIG. 2 is an overall processing flow as a flowchart showing the entire sequence of image processing according to the present invention. First, the outline of the image processing of the present invention will be described according to this flow, and then the details of the processing in each step will be described sequentially.

  First, in step S11, a reduction process is performed on an input image that is image data on which correction processing is executed. In step S12, an image correction process based on an average method is executed on the reduced image data generated by the reduction process. The details of the averaging method will be described later. For the target pixel selected from the constituent pixels of the image data, a false color degree p as a parameter indicating the possibility of false color is calculated, and Based on the pixel value, an interpolation value corresponding to the target pixel is calculated, and a correction pixel value of the target pixel is calculated by changing a contribution ratio between the original pixel value and the interpolation value according to the false color degree p of the target pixel. It is processing to do. Next, in step S13, a correction image enlargement process is executed by the averaging method in step S12.

  In step S14, image correction processing based on the hole filling method is executed on the reduced image generated by the reduction processing in step S11. The hole filling method will be described in detail later, but pixels that are determined to be false colors such as purple fringe are extracted from the vicinity of overexposed pixels included in the constituent pixels of the image data, and are determined to be false colors. This is a process of correcting image data by calculating pixel values for pixels based on pixel values of surrounding pixels. In step S15, a correction image enlargement process is executed by the hole filling method in step S14. Finally, in step S16, image data obtained by executing image processing based on the average method generated in the processing in steps S12 and S13, and image processing based on the hole filling method generated in the processing in steps S14 and S15. A blend process with the image data obtained by execution is performed, and the corrected image data composed of the final corrected pixel values is assigned, and all the processes are completed. Details of each process will be described later.

  In the image processing according to the present invention, first, a special reduction process is performed on the input image in step S11, so that correction of false color pixels such as purple fringe (PF) is reliably executed. The calculation cost is reduced without impairing the effect. In the reduction process of this embodiment, the luminance information and the color information are reduced by different methods. In order to prevent overexposure in the reduction processing of luminance information, in the reduction processing of color information, reduction processing is performed on the input image in consideration of not losing false color data such as purple fringe (PF). Apply.

  By performing such a special reduction process, when the original image due to the reduction is not lost, and pixels of a color corresponding to a false color such as purple fringe (PF) exist in the original image data. Further, the false color pixels are not lost in the reduced image data, and can be prevented from being overlooked even when reduced. Further, the image of the part where the purple fringe (PF) is generated is blurred due to the influence of the aberration, and the purple fringe (PF) can be effectively corrected even when the input image is reduced to some extent. . As described above, in the process of the present invention, the calculation cost can be greatly reduced without impairing the false color correction effect as compared with the method of performing the correction process on the input image as it is.

  In the process described below, the original input image to be corrected is image data having a width of WL and a height of HL, and the reduced image generated by the reduction process in step S11 of the flow shown in FIG. It is assumed that the image data has a width w and a height h. In step S11, the WL × HL image data is reduced to 1 / n to generate a w × h reduced image. Note that w is a value obtained by rounding up the decimal point of the answer obtained by dividing WL by n, and h is a value obtained by rounding up the decimal point of the answer obtained by dividing HL by n.

  Details of the reduction processing in step S11 of the flow shown in FIG. 2 will be described with reference to FIGS. 3 and 4 are flowcharts showing the procedure of the reduction process in step S11 of the flow shown in FIG. The process of each step will be described. In the present flow and the following description, the reduction amount is represented by a natural number n. For example, when n is 2, the original image data of WL × HL is reduced to ½, and a reduced image of w × h is generated. When n is 4, it means to reduce to 1/4.

The correspondence between the original image and the reduced image will be described with reference to FIG. FIG. 5 shows an original image 150 to be reduced and a reduced image 160 generated by the reduction process. The original image 150 is WL × HL, which is reduced to 1 / n to generate a reduced image 160 of w × h. When the reduced image 160 is generated based on the original image 150, it is necessary to individually set the pixel values of the reduced image 160. For example, it is necessary to determine the luminance information and the color information for the pixel 161 at the coordinates (x, y) in the reduced image 160. For this purpose, the corresponding region of the pixel 161 at the coordinate (x, y) in the reduced image 160 is selected from the original image 150, and the pixel (in the reduced image 160) (based on the luminance information and the color information of the constituent pixels of the corresponding region). The luminance L _S (x, y) and color information C _S (x, y) of x, y) are obtained. This processing procedure is shown in the flowcharts shown in FIGS.

The corresponding area of the pixel 161 at the coordinate (x, y) in the reduced image 160 is the corresponding image area 151 of the original image 150. The corresponding image area 151 has u = (n × x) to (n × x + n−1) and v = (n × y) to (n × y + n−1) as pixel ranges of the original image 150. That is, as shown in FIG.
(U, v) = (n × x, n × y),
(U, v) = (n × x + n−1, n × y)
(U, v) = (n × x + n−1, n × y + n−1)
(U, v) = (n × x, n × y + n−1)
This is an image area 151 set as a rectangular area with the four points as vertices.

The luminance L _S (x, y) and the color information C _S (x, y) of the pixel 161 at the coordinates (x, y) in the reduced image 160 are the luminance information of the pixel included in the corresponding image area 151 of the original image 150, and It is determined based on the color information.

Processing of each step in the flowcharts shown in FIGS. 3 and 4 will be described. First, in steps S101 and S102, variables x and y representing coordinates (x, y) in the reduced image are initialized. In step S101, y = 0, and in step S102, x = 0. Next, in step S103, variables L _max , C _PF , N _PF , C _NPF , and N _NPF used when obtaining the pixel at the coordinates (x, y) are all initialized to zero. Each variable has the following meaning.

L _max : Maximum luminance in the corresponding image area of the original image C _PF : Color information of purple fringe (PF) pixels in the corresponding image area of the original image N _PF : Number of pixels of purple fringe (PF) pixels in the corresponding image area of the original image C _NPF : Color information of pixels that are not purple fringe (PF) in the corresponding image area of the original image N _NPF : Number of pixels that are not purple fringe (PF) in the corresponding image area of the original image

  After initialization of variables in step S103, variables u and v representing coordinates (u, v) of the original input image are initialized in steps S104 and S105. In step S104, v = n × y, and in step S105, u = n × x. As described with reference to FIG. 5, the pixel at the coordinate (x, y) of the reduced image is a corresponding image area of the original image, that is, the u coordinate is (n × x) to (n × x + n−1), The v coordinate is calculated using pixels included in the corresponding image area corresponding to (n × y) to (n × y + n−1).

For the luminance L _S (x, y) of the target pixel (x, y) of the reduced image, the luminance L _S (x, y) of the corresponding image area of the original image is selected, and this is selected as the target pixel (x, y) of the reduced image. The luminance L _S (x, y) of

The color information C _S (x, y) of the target pixel (x, y) of the reduced image includes pixels having a hue corresponding to purple fringe (PF) among the pixels included in the corresponding image area of the original image. For example, the average value is obtained, and this average value is used as the color information C _S (x, y) of the target pixel (x, y) of the reduced image. If the pixel included in the corresponding image area of the original image does not include a pixel having a hue corresponding to purple fringe (PF), an average value of all the pixels included in the corresponding image area is calculated, and this is reduced. Is the color information C _S (x, y) of the target pixel (x, y).

In step S106, it is determined whether the luminance L (u, v) of the pixel (u, v) selected from the corresponding image area of the original image is greater than L _max .
L (u, v)> L _max
Is satisfied, in step S107, variable update processing is performed to set L _max = L (u, v), and then the process proceeds to step S108.
L (u, v)> L _max
If is not established, the process proceeds directly to S108.

  In step S108, the color C (u, v) of the coordinates (u, v) of the original image is a hue that falls within a hue range designated in advance, that is, a hue range corresponding to a false color such as purple fringe (PF). It is determined whether or not there is. The hue determination process in step S108 will be described with reference to FIG.

  FIG. 6 is a diagram showing the hue range of purple fringe (PF) in the color space. When the YCbCr space is used as an example of the color space, FIG. 6 is a two-dimensional plot of CbCr corresponding to the color information. The horizontal axis indicates the value of Cb, the vertical axis indicates the value of Cr, and the origin 180 is a color corresponding to Cb = 128 and Cr = 128.

  As a method for designating a hue range corresponding to purple fringe (PF), a method for determining whether or not the hue range is between two hues is used. The purple fringe (PF) has a purple color, and this specific hue range is set in the CbCr two-dimensional coordinate space shown in FIG. A region sandwiched between the hue line 181 and the hue line 182 shown in the drawing is set as a region indicating a hue range 183 corresponding to purple fringe (PF).

  In step S108, depending on whether or not the color C (u, v) of the selected pixel (u, v) selected from the corresponding image area of the original image is included in the hue range 183 shown in FIG. It is determined whether v) is a false color (PF) hue. If it is determined that C (u, v) is a false color (PF) hue, the process proceeds to step S109. If C (u, v) is determined not to be a false color (PF) hue, the process proceeds to step S110.

In step S109, a variable for averaging the colors of pixels determined to have a false color (PF) hue, that is,
C _PF : Color information of purple fringe (PF) pixels in the corresponding image area of the original image N _PF : Number of pixels of purple fringe (PF) pixels in the corresponding image area of the original image These variables C _PF and N _PF are updated. In particular,
C _PF = C _PF + C (u, v),
N _PF = N _PF +1
As a result, the variables _CPF and _NPF are updated.
_CPF is an integrated value of color information of purple fringe (PF) pixels extracted from the corresponding image area of the original image, and _NPF is the total number of purple fringe (PF) pixels extracted from the corresponding image area of the original image. It means a value.

On the other hand, if it is determined that C (u, v) is not a false color (PF) hue, in step S110, a variable for averaging the colors of pixels determined not to be a false color (PF) hue, that is,
C _NPF : Color information of pixels that are not purple fringe (PF) in the corresponding image area of the original image N _NPF : Number of pixels of pixels that are not purple fringe (PF) in the corresponding image area of the original image These variables C _NPF and N _NPF Update. In particular,
C _NPF = C _NPF + C (u, v),
N _NPF = N _NPF +1
As a result, the variables C _NPF and N _NPF are updated.
C _NPF is an integrated value of color information of pixels other than the purple fringe (PF) extracted from the corresponding image area of the original image, and N _NPF is not a purple fringe (PF) extracted from the corresponding image area of the original image. The value means the total number of pixels.

After the process of step S109 or step S110 ends, the process proceeds to step S111. In step S111, the u component value is increased by one in the coordinates of the corresponding image area of the original image (u, v), u in step S112 is a u <n × x + n, whether u _{<W L} is established Determine. That is, the u component value of the coordinates (u, v) does not reach the right end of the corresponding image area 151 of the original image 150 described with reference to FIG. 5 (u <n × x + n), and the original image 150 Is not reached (u <W _L ), the process returns to step S106, and the processing from step S106 onward is executed for the pixel corresponding to the updated coordinates (u, v).

U in step S112 is a u <n × x + n, if it is determined that u _{<W L} is established, in step S113, the coordinates (u, v) of the corresponding image region v component values is increased by one in step In S114, it is determined whether v is v <n × y + n and v <H _L is satisfied. That is, the v component value of the coordinates (u, v) does not reach the lower end of the corresponding image area 151 of the original image 150 described with reference to FIG. 5 (v <n × y + n), and the original image 150 Is not reached (v <H _L ), the process returns to step S105, and the processing from step S105 is executed on the pixel corresponding to the updated coordinates (u, v). If step S114 is determined to be No, processing for all the pixels included in the corresponding image area of the original image corresponding to one pixel (x, y) of the reduced image is completed.

Next, the process proceeds to step S115 shown in FIG. In steps S115 to S117, a process of setting the luminance L _S (x, y) and the color value C _S (x, y) of the pixel (x, y) of the reduced image is performed. In step S115, L _max obtained by sequentially executing the update process in step S107 is set as the luminance value L _S (x, y) of the pixel (x, y) of the reduced image. That is, the luminance L _S (x, y) of the target pixel (x, y) of the reduced image is selected from the corresponding image areas of the original image, and this is selected as the target pixel (x, y) of the reduced image. _Let y) be the luminance L _S (x, y).

Next, in step S116, it is determined whether N _PF counted in the corresponding image area of the original image, that is, whether the number of purple fringe (PF) pixels in the corresponding image area of the original image is greater than zero. .
N _PF > 0
If is established, the process proceeds to step S117, and if not, the process proceeds to step S118.

In step S117, the color value C _S (x, y) of the target pixel (x, y) of the reduced image is calculated by the following equation. That is,
C _S (x, y) = C _PF / N _PF
Calculated by the above formula. This is obtained by dividing the integrated value of the color information of purple fringe (PF) pixels in the corresponding image area of the original image by the number of purple fringe (PF) pixels. In other words, when one or more purple fringe (PF) pixels are included in the corresponding image area of the original image, the average color information of the purple fringe (PF) pixels in the corresponding image area of the original image is changed to the target pixel ( x, y) is set as a color value C _S (x, y).

In step S116,
N _PF > 0
Is not satisfied, that is, when the number of purple fringe (PF) pixels in the corresponding image area of the original image is 0, the process proceeds to step S118, and the color value C of the target pixel (x, y) of the reduced image is obtained. _S (x, y) is calculated by the following equation. That is,
C _S (x, y) = C _NPF / N _NPF
Calculated by the above formula. This is obtained by dividing the integrated value of the color information of all pixels that are not purple fringe (PF) pixels in the corresponding image area of the original image by the total number of pixels. That is, when the corresponding image area of the original image does not include any purple fringe (PF) pixels, the average color information of all the pixels in the corresponding image area of the original image is obtained from the target pixel (x, y) of the reduced image. Set as color value C _S (x, y).

With the above processing, the luminance L _S (x, y) and the color information C _S (x, y) of one pixel of the reduced image, that is, the target pixel (x, y) of the reduced image are determined. Become. Steps S119 to S122 are processing for updating the coordinate position of the target pixel (x, y) of the reduced image.

  In step S119, the x component value of the coordinate (x, y) of the target pixel is incremented by 1, and in step S120, it is determined whether x is less than x <w, that is, the width w of the image size of the reduced image. If Yes, the process returns to S103. If No, the process proceeds to S121. In step S121, the y component value of the pixel of interest (x, y) is increased by 1. In step S122, it is determined whether y is less than y <h, that is, the height h of the image size of the reduced image. If Yes, the process returns to S102, and if No, the process ends.

These processes are processes for moving the target pixel (x, y) one by one in the image data of the reduced image (width w pixels, height h pixels). If it is determined No in step S122, the reduction is performed. The processing for all the pixels of the image (width w pixels, height h pixels) is completed, and the pixel value determination processing in which all the pixels are set as the target pixel is completed. That is, the pixel information of all the constituent pixels of the reduced image, that is, the luminance L _S (x, y) and the color information C _S (x, y) are determined.

  The above processing is the image data reduction processing in step S11 of the comprehensive processing flow shown in FIG. Next, details of the image correction processing based on the averaging method in step S12 of the overall processing flow shown in FIG. 2 will be described. The image correction process based on the averaging method in step S12 is executed as a process for the reduced image data generated as a result of the above reduction process.

  Details of the image correction processing based on the averaging method will be described with reference to FIG. 7 and 8 are flowcharts showing the procedure of image processing according to the present invention. Details of the processing of each step of the flow cheat shown in FIGS. 7 and 8 will be sequentially described.

  In steps S201 and S202, initial setting of the inspection target pixel position of the correction target image data is executed. As an initial setting, the pixel position of the target pixel is set as (0, 0), that is, x = 0 and y = 0. In the processing of the present invention, as described above, the false color degree (PF degree) is calculated for each of the constituent pixels of the image data input as the correction target, and the false color degree (PF degree) corresponding to each calculated pixel is calculated. ) Based on (). As in steps S201 and S202, starting from the coordinates (0, 0), the color information is corrected with each pixel in the image as a target pixel in order.

After the target pixel is determined, a false color degree, that is, a purple fringe degree (PF degree) is calculated. In this embodiment, the following scales are used for calculating the false color degree (PF degree):
(A) proximity to whiteout (high brightness pixels),
(B) magnitude of saturation;
(C) Consider three types of measures: closeness to a specific hue corresponding to false color.

  First, “(a) proximity to whiteout (high luminance pixel)” in the scales (a) to (c) will be described. Purple fringes occur mainly around whiteout. Conversely, when the target pixel is a purple fringe, there is a whiteout around the target pixel. It can be said that the closer the target pixel is to whiteout, the higher the possibility that the pixel is purple fringe.

  By the way, the purple fringe is generated centering on the portion where the whiteout occurs, but the range in which the purple fringe occurs differs depending on the position on the whiteout image. For example, purple fringes are likely to occur in the direction from the center of the image that hits the optical center toward the image edge. This is because the characteristics of the range in which purple fringes occur vary depending on the type of optical system, the setting of the aperture / focal length, and the position of the pixel of interest.

In the present embodiment, in step S203, when the target pixel is a purple fringe, the range in which overexposure may occur, the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) is set. calculate. As a method for calculating the overexposure search range, for example, when the target pixel is overexposed, how large is the direction from the target pixel toward the center of the image and the direction opposite to 180 degrees or the direction orthogonal thereto? What is necessary is just to calculate whether the purple fringe will occur. For this purpose, the range in which purple fringes are likely to occur is examined in advance with respect to the optical system used in this embodiment, the data is stored in a lookup table or the like, and the lookup is performed based on the pixel position of the target pixel. The table is searched to extract data of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) corresponding to the pixel position. Alternatively, in the lookup table, data necessary for calculating the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) is stored, extracted, and the overexposure search range (x _{0, y 0, x 1,} y 1) may be configured to calculate a. Alternatively, without applying the look-up table, by performing arithmetic processing in accordance with a preset function, overexposure search range _{(x 0, y 0, x} 1, y 1) may be configured to calculate the .

In step S203, the search range (x ₀ , y ₀ , x, etc.) is determined based on various information such as the setting of the optical system and the target pixel position by at least one of reference to a predefined lookup table and calculation processing. ₁ , y ₁ ). The search range (x ₀ , y ₀ , x ₁ , y ₁ ) is a range that is calculated as a range that may be overexposed when the pixel of interest is purple fringe.

With reference to FIG. 9, an example of setting of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) according to the optical system will be described. Assume that a certain pixel 211 is overexposed in the image data 210 to be corrected shown in FIG. Whether or not a pixel is a whiteout pixel is determined by a process for determining whether the pixel has a luminance value equal to or higher than a predetermined threshold. If a pixel has a luminance value equal to or higher than a threshold, it is determined as a whiteout pixel. To do.

When the pixel 211 is an overexposed pixel, the range in which purple fringe may occur around the pixel 211 is calculated based on the characteristics of the lens used in the imaging unit, the F value at the time of shooting, and the focal length. be able to. Each of these data is input to a digital diagram signal processing unit (DSP) 106 (see FIG. 1) as an image processing execution unit together with the image data as attribute information corresponding to the photographed image at the time of photographing the image data. The section (DSP) 106 searches the lookup table based on the input data and the pixel position of the target pixel, extracts data necessary for calculating the overexposure range corresponding to the pixel position, The jump search range (x ₀ , y ₀ , x ₁ , y ₁ ) is calculated.

The details of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) will be described with reference to FIG. Image data 210 shown in FIG. 9 indicates a reduced image generated based on the original image in step S11 in the overall processing flow shown in FIG.

  When the pixel 211 of the image data 210 shown in FIG. 9 is a whiteout pixel, a purple fringe has a whitening width: Wa, a height: Ha, a left upper side width: Wb, and a height: Hb. Suppose that occurs. This is an area defined by the pixel area 212 and the pixel area 213 shown in FIG.

  On the other hand, assuming that the pixel 211 is a purple fringe pixel, on the other hand, since the purple fringe pixel is generated in the vicinity of the whiteout pixel, there is always a whiteout pixel in the vicinity of the purple fringe pixel 211. It should be. In the periphery of the purple fringe pixel 211, there is a possibility that overexposure occurs in the pixel 211 along the straight line connecting the pixel 211 and the pixel region 213 to the pixel 211 and the image data center pixel 250. These are regions that are set by being folded around the center, that is, the pixel region 222 and the pixel region 223 shown in FIG.

  As described above, when the pixel 211 illustrated in FIG. 9 is a purple fringe pixel, it can be estimated that overexposed pixels are generated in the pixel region 222 and the pixel region 223 illustrated in FIG. 9. Therefore, when the pixel 211 is a purple fringe pixel, an overexposed pixel can be found by performing a search process using the pixel area 230 set as a rectangular area including the pixel area 222 and the pixel area 223 shown in FIG. 9 as a search area. It should be.

In step S203 of the flow shown in FIG. 7, when the target pixel (x, y) is a purple fringe pixel, an overexposure search is performed for the purpose of detecting an overexposed pixel that is determined to exist around the pixel. A range (x ₀ , y ₀ , x ₁ , y ₁ ) is set. As understood from the above description, when the pixel 211 is the target pixel in FIG. 9, if the pixel region 230 is set as the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ), the pixel 211 Is a purple fringe pixel, it is possible to detect overexposed pixels in this search range.

In this manner, the whiteout pixel search region around the purple fringe pixel is set based on the harpling fringe generation region around the whiteout pixel determined by the characteristics of the photographing means and the photographing conditions. In step S203 of the flow shown in FIG. 7, the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) is calculated for each pixel to be processed, that is, the target pixel.

With reference to FIG. 10, the region of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) will be described. FIG. 10 shows an overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) 301 set around the pixel of interest (x, y) 300. The overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) 301 is a pixel 302 whose upper left corner is a coordinate (x ₀ , y ₀ ) and whose lower right corner is a pixel of coordinates (x ₁ , y ₁ ). This is an overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) composed of a rectangular area 303.

An overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) 301 shown in FIG. 10 is an area corresponding to the rectangular area 230 shown in FIG. 9, and a pixel area 223 shown in FIG. 9 corresponds to a pixel region 311 having a height H1 and a width W1 at the upper left of the target pixel (x, y) 300 shown. The pixel region 222 shown in FIG. 9 is the same as the target pixel (x, y) 300 shown in FIG. This corresponds to a pixel region 312 having a height H2 and a width W2 at the lower right. The pixel 321 is a pixel having coordinates (x ₀ , y ₁ ), the pixel 322 is a pixel having coordinates (x ₁ , y), the pixel 323 is a pixel having coordinates (x, y ₀ ), and the pixel 324 is a coordinate (x, y ₁ ). Pixels.

In step S203 of the flow shown in FIG. 7, for example, when the pixel (x, y) 300 shown in FIG. 10 is the target pixel, W1 is on the left of the target pixel (x, y) 300, W2 is on the right, and H1 is on the top. A search range (x ₀ , y ₀ , x ₁ , y ₁ ) 301 consisting of the H2 region is set below. With this setting, when the target pixel (x, y) 300 is a purple fringe pixel, an overexposed pixel is almost certainly detected from within the search range (x ₀ , y ₀ , x ₁ , y ₁ ) 301. Can do.

As described above, in step S203 in FIG. 7, when the target pixel (x, y) is a purple fringe pixel, the whiteout pixel is detected for the purpose of detecting a whiteout pixel that is determined to exist around the pixel. The search range (x ₀ , y ₀ , x ₁ , y ₁ ) is set. Note that, as described above, at least one of lookup table reference and calculation processing is performed to calculate the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ). The digital diagram signal processing unit (DSP) 106 (image processing unit) as an image processing execution unit together with the image data, using each data such as the characteristics of the lens used in the imaging means, the F value at the time of shooting, and the focal length as attribute information corresponding to the shot image 1), and the digital signal processing unit (DSP) 106 executes at least one of lookup table reference and calculation processing based on the input data, target pixel position information, and the like. The jump search range (x ₀ , y ₀ , x ₁ , y ₁ ) is calculated.

When the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) corresponding to the target pixel (x, y) is calculated in step S203 of FIG. 7, the calculated overexposure search range (x ₀ , In y ₀ , x ₁ , y ₁ ), an overexposed pixel is searched to calculate a minimum distance: d _min as a distance between the target pixel (x, y) and the adjacent overexposed pixel. This process is the process of steps S204 to S214 shown in the flow of FIGS.

The distance d between the pixel of interest (x, y) calculated here and the overexposure pixel: d is the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) with the attention pixel (x, y) as the center point. The distance data calculated by normalizing the area is used. The normalized distance data is compared, and the minimum distance: d _min is calculated. The reason for using the normalized distance data will be described. When calculating the distance from the target pixel to overexposure, it is necessary to consider the direction in which purple fringes occur. That is, in order to calculate the possibility that purple fringes occur in the target pixel due to the whiteout pixels around the target pixel, the position of the whiteout pixel and the distance between the target pixel and the position thereof are calculated. It is necessary to consider the relationship.

As described with reference to FIGS. 9 and 10, the overexposed pixel search area is not set evenly around the target pixel. For example, the overexposure pixel search area 301 shown in FIG. 10 is set as an area including a small pixel area 311 at the upper left of the target pixel 300 (x, y) and a large pixel area 312 at the lower right. , Y) is not set at the center of the overexposure pixel search area 301 but is set according to the upper left portion. When calculating the minimum distance: d _min between the target pixel (x, y) and the overexposure pixel, when calculating as actual distance data, the small pixel region 311 in the upper left also has overexposure, and the lower right large pixel When the region 312 has a whiteout, most of the minimum distance data is likely to be calculated as a distance from the small pixel region 311 at the upper left.

However, the overexposed pixel as the cause of the purple fringe of the target pixel 300 (x, y) may be the overexposed pixel in the lower right pixel region 312 where the actual distance from the target pixel is larger. is there. In order to prevent such misidentification, that is, to accurately calculate the distance from the overexposed pixel as the factor causing the purple fringe of the pixel of interest (x, y) as the minimum distance: _dmin , the overexposed search is performed. Normalized distance data is used to set the target pixel (x, y) at the center of the range (x ₀ , y ₀ , x ₁ , y ₁ ).

By using the normalized distance data, for example, the pixel of interest 300 (x, y) illustrated in FIG. 10 and the pixel 321 (x ₀ , x ₁ , y ₁ ) at the end point of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) Y ₁ ) distance: W 1, the target pixel 300 (x, y) and the pixel 322 (x ₁ , y): W 2, the target pixel 300 (x, y) and the pixel 323 (x, y ₀ ) The distance: H1 and the distance H2 between the target pixel 300 (x, y) and the pixel 324 (x, y ₁ ) are all processed as equal. By using this normalized distance data, the overexposed pixel having the pixel of interest 300 (x, y) and the normalized minimum distance: d _min causes a purple fringe of the pixel of interest 300 (x, y). It is possible to improve the probability that the estimation of the overexposed pixel is correct.

In the present embodiment, for example, in the setting of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) shown in FIG. 10, the absolute value of the difference between the overexposed pixel position and the x coordinate of each pixel of interest is Distance between the pixel of interest 300 (x, y) and the pixel 321 (x ₀ , y ₁ ) at the end point of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ): W1, or the pixel of interest 300 (x , Y) and the pixel 322 (x ₁ , y) distance: normalized using W2, and further, the distance between the target pixel 300 (x, y) and the pixel 323 (x, y ₀ ): H1, the target pixel 300: (x, y) and pixel 324 (x, y ₁ ) The distance: H is normalized to calculate the distance: d.

A specific distance calculation formula will be described. Assume that the position of a whiteout pixel found in the whiteout search range is (a, b). The distance from the target pixel to the overexposed position (a, b): d is calculated according to the following equation (Equation 1).

Using the above expression (Expression 1), the pixel 321 (x ₀ , y ₁ ) at the end point of the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) from 300 (x, y) from the target pixel. Distance to: W1, distance to pixel 322 (x ₁ , y): W2, distance to pixel 323 (x, y ₀ ): H1, distance to pixel 324 (x, y ₁ ): all H2 etc. It is taken as a distance. By applying the normalized distance data and selecting a whiteout pixel having the minimum distance: _dmin , the whiteout pixel that has caused the purple fringe of the pixel of interest 300 (x, y) is correctly established. Can be extracted.

The description of the flow in FIG. 7 will be continued. After calculating the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) in step S203, first, in step S204, the minimum distance: d _min is initialized, for example, d _min = ∞ or the target pixel and white The value is set to a value sufficiently larger than the maximum distance of the jump search range (x ₀ , y ₀ , x ₁ , y ₁ ). Next, in steps S205 and S206, _one of the constituent pixels in the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ) is selected, and in step S207, the selected pixel (s, t) is an overexposed pixel. It is determined whether or not. The process for determining whether or not a pixel is a whiteout pixel is executed as a process for determining, for example, a pixel having a predetermined luminance or higher as a whiteout pixel.

If the selected pixel (s, t) is not an overexposed pixel, the process of steps S211 to S214 updates the selected pixel, that is, the next within the overexposed search range (x ₀ , y ₀ , x ₁ , y ₁ ). The pixel is set as the selected pixel, and the processing from step S207 is repeated.

If it is determined that the selected pixel (s, t) is a whiteout pixel, the distance d between the target pixel (x, y) and the whiteout pixel (s, t) is calculated in step S208. The distance d is calculated as normalized distance data by a calculation applying the above-described equation (Equation 1). In step S209, a comparison is made between the calculated distance data: d and the initialized or updated minimum distance: d _min . If the calculated distance data: d has a value smaller than the minimum distance: d _min. In step S210, update processing of the minimum distance: d _min is performed with the calculated distance data: d as the minimum distance: d _min .

The above processing is executed for all pixels in the overexposure range (x ₀ , y ₀ , x ₁ , y ₁ ) set around the pixel of interest (x, y), and one minimum corresponding to the pixel of interest Distance: Determine d _min . When the overexposure pixel is not detected in the overexposure search range (x ₀ , y ₀ , x ₁ , y ₁ ), the minimum distance: d _min remains the initial value.

  Next, the process proceeds to step S215 in the flowchart shown in FIG. In step S215, a parameter for calculating the false color degree of the target pixel (x, y), that is, the PF (purple fringe) degree is determined based on the color information c (x, y) of the target pixel (x, y). calculate. This is a false color (PF) degree p as a scale for determining whether the pixel of interest (x, y) has saturation and hue close to purple fringe, that is, a false color (PF) such as purple fringe. This is a process for obtaining a parameter to be applied to the calculation of the false color (PF) degree p as a parameter indicating possibility.

  A parameter calculation process based on the color information c (x, y) of the target pixel (x, y) in step S215 will be described with reference to FIG. FIG. 11 shows an example of a two-dimensional color information space. Here, CbCr in the YCbCr space is used as color information. In FIG. 11, the horizontal axis represents Cb, the vertical axis represents Cr, and the origin 400 has color values such that Cb = 128 and Cr = 128. A purple fringe hue line 401 is a line representing a hue designated in advance as a purple fringe color. The target pixel hue 402 is obtained by plotting the hue (Cb, Cr) of the target pixel (x, y).

  In step S215 in the flow shown in FIG. 8, the saturation s of the target pixel and the proximity θ to the hue of the purple fringe are obtained as parameters. In the example shown in FIG. 11, the saturation s is a value calculated as the distance from the origin 400 to the target pixel hue 402, and the purple fringe proximity to the hue θ is the purple fringe hue line 401, the origin, and the attention. This is a value calculated as an angle formed by a line connecting the pixel hue 402. In the present embodiment, the saturation s as a parameter for calculating the false color degree of the target pixel (x, y), that is, the PF (purple fringe) degree, and the proximity θ to the hue of the purple fringe, Although the process which calculates as a parameter by the said process is applied, these are an example as a scale of saturation and the proximity to a hue, and you may apply another method. Note that θ described here is an angle formed by two hues, and is always a value of 0 or more.

Next, in step S216, parameters s and θ based on the color information c (x, y) of the target pixel (x, y) obtained in step S215 are applied, and the false color degree p of the target pixel, that is, PF ( A value p representing the degree of purple fringe is calculated. As described above, as the purple fringe degree p, in the processing of this embodiment, the following scales, that is,
(A) proximity to whiteout (high brightness pixels),
(B) magnitude of saturation;
(C) It was explained considering three kinds of scales: closeness to a specific hue corresponding to false color. The scales (a) to (c) are applied to the calculation of the final false color (PF) degree p.

Each scale of the above (a)-(c)
(A) Closeness to whiteout (high luminance pixel) = whiteout distance evaluation value P _d ,
(B) Saturation magnitude = saturation evaluation value P _s ,
(C) proximity to a specific hue corresponding to a false color = hue evaluation value P _θ ,
And

Hereinafter, details of the calculation process of each evaluation value will be described.
(A) Closeness to whiteout (high luminance pixel) = whiteout distance evaluation value P _d ,
The overexposure distance evaluation value _Pd is calculated by the following formula (Formula 2).

In the above equation, d is substituted with the data previously obtained in the processing of steps S204 to S214, that is, the minimum distance _dmin corresponding to the target pixel (x, y). d _max is distance data set in advance, and is set to a value close to the maximum value or a value equal to or greater than the maximum value in the range that d can take. However, when d _max is set as a value close to the maximum value in the range that d can take, if d> d _max is satisfied, it is assumed that d _max = d, and processing is performed with d−d _max = 0. To do. For example, according to the above formula (Formula 1), d takes a value from 0 to √2. Therefore, for example, √2 may be set as dmax.

In this way, as the pixel of interest (x, y) is a distance d from up overexposed pixel smaller, the distance evaluation value P _d approaches 1, whereas the distance from the pixel of interest (x, y) to overexposed pixels The distance evaluation value _Pd is set to a value that approaches 0 as _d increases. That is, in the range of the distance evaluation value P _d is 0-1, the pixel of interest (x, y) will be set to a value corresponding to the distance from to the overexposed pixels.

Further, (c) proximity to a specific hue corresponding to a false color = hue evaluation value P _θ , that is, a hue evaluation value p _θ as a value indicating proximity to a purple fringe hue is expressed by the following equation (Equation 3): Is calculated by

For θ in the above formula (formula 3), the closeness θ to the hue of the purple fringe of the target pixel obtained in step S215 is substituted. As described with reference to FIG. 11, the purple fringe closeness θ is a value calculated as an angle between the purple fringe hue line 401 and a line connecting the origin and the target pixel hue 402. Θ _max in the above equation (Equation 3) is a value designated in advance, and is set to a value close to the maximum value or a value equal to or greater than the maximum value in a possible range of θ. However, when θ _max is set as a value close to the maximum value in the range that θ can take, if θ> θ _max , it is assumed that θ _max = θ, and processing is performed as θ−θ _max = 0. To do. Advance what false color hue is generated by a lens used for imaging can be expected in advance. What is necessary is just to set an optimal value as (theta) _max so that all the hues of the false color which may generate | occur | produce can be covered.

In this way, the pixel of interest (x, y) closer to the hue of purple fringing as hue false color, the hue evaluation value p _theta approaches 1, whereas the hue of the pixel of interest (x, y) is false The hue evaluation value _pθ is set to a value that approaches 0 as it is farther from the hue of the purple fringe as the color. That is, in the range of hue evaluation value p _theta is 0-1, the pixel of interest (x, y) will be set to a value corresponding to the difference between the hue of the hue and purple fringing.

The size = chroma evaluation value P _s (b), saturation is calculated by the following equation (Equation 4).

For the s in the above formula (formula 4), the saturation s of the target pixel obtained in step S215 is substituted. As described with reference to FIG. 11, the saturation s is a value calculated as the distance from the origin 400 to the target pixel hue 402. In the above equation (equation 4), s _max is a value designated in advance, and is set to a value close to the maximum value or a value equal to or greater than the maximum value in the range that s can take. However, when s _max is set as a value close to the maximum value in a range that s can take, if s> s _max is satisfied, it is assumed that s _max = s and s−s _max = 0 is processed. To do.

In this way, the greater the saturation of the pixel of interest (x, y), the chroma evaluation value P _s approaches 1, whereas, as the saturation of the pixel of interest (x, y) is small, the saturation evaluation value P _s is set to a value approaching zero. That is, in the range of chroma evaluation value P _s is 0-1, the pixel of interest (x, y) will be set to a value corresponding to saturation of the.

FIG. 12 is a diagram showing the correspondence between the image area of the image data where the purple fringe occurs and the saturation. The vertical axis in FIG. 12 represents the saturation, and the horizontal axis represents the position on the image. In FIG. 12, it is assumed that there are overexposed pixels in the image range 451 of the image position, purple fringes occur over the image range 452 and the image range 453, and the original color of the subject is captured in the image range 454. . The purple fringe in the image range 452 has a high saturation and is very conspicuous, so it is desirable to correct as strongly as possible. On the other hand, since the image range 453 is a transition portion from the purple fringe to the original color of the subject, it is desirable that the correction level is continuously changed from a strong correction level to a level that is gradually weakened and not corrected. It is preferable that the saturation evaluation value P _s calculated by the above expression (Expression 4) corresponds to an optimum correction level according to such saturation. In order to achieve this setting, s _max as a fixed value applied in the above equation (equation 4) may be set as, for example, saturation near the maximum saturation of the image range 453 as shown in FIG. .

With this setting, in the image area 452 shown in FIG. 12, the saturation evaluation value P _s calculated by the above expression (expression 4) becomes 1, and in the image area 453, it is calculated by the above expression (expression 4). The saturation evaluation value P _s becomes 1 at the contact point of the image area 452, and the saturation evaluation value P _s decreases as the image area 454 approaches, and is set to a value that gradually approaches 0.

Returning to the flowchart of FIG. 8, the description of the image processing sequence will be continued. In step S216, the above three evaluation values, that is,
(A) Closeness to whiteout (high luminance pixel) = whiteout distance evaluation value P _d ,
(B) Saturation magnitude = saturation evaluation value P _s ,
(C) proximity to a specific hue corresponding to a false color = hue evaluation value P _θ ,
Based on the above, the false color (PF) degree p is calculated according to the following equation (equation 5).

The equation (Equation 5), the three evaluation values, i.e., the overexposure distance evaluation value P _d, the hue evaluation value P _theta, false color (PF) degree based on the multiplication value of the chroma evaluation value P _s p is an expression for calculating a false color (PF) degree p which is a parameter indicating the possibility of false color.

  Next, in step S217, a process for correcting the color information of the target pixel is performed. The detailed processing in step S217 will be described with reference to the flowchart in FIG. In this embodiment, the color information interpolation value obtained from the color information of the pixels around the target pixel is divided into the original color information C (x, y) according to the false color (PF) degree p. , Make corrections. As the interpolation value, a weighted average of the target pixel and its surrounding pixels is used. By using the weighted average of the color information in the interpolation value calculation range as the interpolation value, false colors such as purple fringes cannot be completely removed, but particularly high-saturation and highly conspicuous parts are made inconspicuous. As a result, processing for reducing false colors such as purple fringing can be performed.

Processing of each step in the flow shown in FIG. 13 will be described. First, in step S301, an interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) is calculated. The interpolation value calculation range is the same as the above-described whiteout search range, based on the data based on the type and setting of the optical system and the position of the target pixel, in which range the purple fringe occurs when the target pixel is whiteout A range representing this is calculated as an interpolation value calculation range. Here, correction is performed when the pixel of interest is a purple fringe. In this case, an interpolation value calculation range (x ₂ , y ₂ , x ₃ , y is assumed as a range in which the original subject color may remain. ₃ ) is used to obtain a weighted average of the color information of the pixels in the range.

A specific setting process of the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) will be described with reference to FIG. FIG. 14 is a diagram corresponding to FIG. 9 described above, and it is assumed that the target pixel is the pixel 511 in the image data 510 to be corrected. If the pixel 511 is a false color pixel such as purple fringe, the false color pixel is corrected based on the pixel values of surrounding pixels. When the pixel 511 is a false color pixel such as a purple fringe, there is a whiteout pixel in the whiteout search range 520 as described with reference to FIG. The overexposure search range 520 corresponds to the pixel area 230 in FIG. 9 and is an overexposure search area 520 set as a rectangular area including the pixel area 521 and the pixel area 522.

When the pixel 511 is a purple fringe, interpolation processing using the pixels around the pixel 511, that is, processing for determining the pixel value of the pixel 511 by applying the pixel values of the pixels around the pixel 511 is performed. At that time, it is desirable that the pixel to be applied to the interpolation process, that is, the pixel to be referenced is not a false color (purple fringe) pixel as much as possible. When the pixel 511 is purple fringe, for example, all the pixels existing between the whiteout pixel 531 detected in the whiteout search range 520 and the target pixel 511 should be purple fringe. Here, it is assumed that the overexposed pixel 531 is an overexposed pixel 531 having a minimum distance d _min from the target pixel 511 in the processing of steps S204 to S214 in the flow of FIG. In this case, the pixel 511 that is a purple fringe is a purple fringe due to the direct influence of the whiteout pixel 531, and all pixels that are closer to the whiteout pixel 531 should be a purple fringe.

  In such a case, it can be said that the original color of the subject approaches as the whiteout pixel 531 progresses from the target pixel 511 to the opposite side of the certain direction. Therefore, as the interpolation value calculation range, for example, by setting a region on the opposite side in the direction from the pixel 511 to the overexposed pixel 531, it is determined that a region including more normal pixel values can be set. . In such a case, for example, an interpolation value calculation range 541 may be set as shown in FIG.

In the example illustrated in FIG. 14, the pixel value of the target pixel 511 is determined by a value calculated based on the pixel values of the constituent pixels in the interpolation value calculation range 541. In this way, when setting the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ), pixels in a direction far from the direction of the overexposed pixel having the minimum distance d _min from the target pixel 511 are selected. By setting a region including many, more effective interpolation processing can be performed.

Returning to the flow of FIG. 13, the description of the interpolation processing sequence will be continued. In step S302, as an initial setting, a variable (vector value) C _tmp that stores an interpolation value of color information is set to a zero vector, and a variable (scalar value) W _total that stores a sum of weights is set to 0. In steps S303 and S304, initial setting processing of variables s and t representing the coordinates (s, t) of each pixel selected from the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) is performed. Initially, pixel coordinates (s, t) corresponding to one vertex of the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) are selected, and s = x ₂ , t = y ₂ , respectively. Set to.

Next, in step S305, a weight w (x, y, s, t) to be applied to the selected pixel (s, t) is calculated, and the calculated weight w (x, y, s, t) is calculated as a sum of weights. Save variable is added to the (scalar _{value) W total,} then update the weight sum stored variables (scalar _{value) W total,} further color information C (s, t) of the selected pixel (s, t) to the weight w (x , it is multiplied by y, s, t) and adds the color information interpolating values stored variable _{C tmp,} performs processing to update the color information interpolating values stored variable _{C tmp.} That is,
_Wtotal = _Wtotal + w (x, y, s, t)
C _tmp = C _tmp + w (x, y, s, t) × C (s, t)
By applying the weight w (x, y, s, t) to the processing target pixel (s, t) by these calculation formulas, the color information interpolation value storage variable (vector value) C _tmp and the weight sum storage storage variable ( updating of the scalar _{value) W total.}

In the simplest processing example, a setting in which the weight w (x, y, s, t) of all the selected pixels (s, t) is 1 can be applied. In the setting of weight w (x, y, s, t) = 1 for all pixels (s, t), all the values selected from the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) The interpolation processing applies the average value of the reference pixels to determine the pixel value of the target pixel (x, y).

  In addition, as a setting mode of the weight w (x, y, s, t), as a setting example of the weight w for optimally interpolating the purple fringe pixel, the pixel of interest (x, y) and the coordinates (s, t) ) May be used as the weight w. Further, a method that takes into account the color information of the input image is adopted, such as setting the weight to be larger as the hue of the color information C (s, t) of the pixel (s, t) is farther from the purple fringe hue. May be.

After update processing of the color information interpolation value storage variable (vector value) C _tmp and the weight sum storage storage variable (scalar value) W _total in step S305, in step S306, the x component of the coordinates (s, t) of the selected pixel Value: s is incremented by 1. In step S307, it is determined whether s exceeds s> x ₃ , that is, whether it exceeds the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ). If it is No, the process returns to S305. In step S308, the y component value: t of the selected coordinates (s, t) is increased by 1. In step S309, t is t> y ₃ , that is, the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) It is determined whether or not the value exceeds _3. If Yes, the process proceeds to S310. These processes are processes for moving the selected pixel (s, t) one by one in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ), and when step S309 is determined to be Yes. , based on all the pixels of the interpolation value calculation range _{(x 2, y 2, x} 3, y 3) processing for all the pixels are finished, interpolation value calculation range _{(x 2, y 2, x} 3, y 3) The final color information interpolation value storage variable C _tmp calculated in this way and the weight sum storage variable W _total are determined.

上記式（式６）に基づいて決定する。 Finally, in step S310, the final color information interpolation value storage variable C _tmp calculated based on all the pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ), and the weight sum Based on the storage variable W _total , the correction color information C ′ (x, y) as the correction color of the target pixel (x, y) is expressed by the following equation (Equation 6), that is,

It determines based on the said Formula (Formula 6).

In the above formula (formula 6), p is the false color (PF) degree p calculated in step S216 of FIG. 8 described above. That is,
(A) Closeness to whiteout (high luminance pixel) = whiteout distance evaluation value P _d ,
(B) Saturation magnitude = saturation evaluation value P _s ,
(C) proximity to a specific hue corresponding to a false color = hue evaluation value P _θ ,
As the multiplication value of (a) to (c), the false color (PF) degree p of the target pixel (x, y) calculated based on the above-described expression (Expression 5).

C _tmp and W _total are final color information interpolation values calculated based on all pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) obtained in the processing up to step S 309 described above. A storage variable C _tmp and a weight sum storage variable W _total , C (x, y) is color information of a pixel of interest (x, y) before interpolation processing, and (C _tmp / W _total ) is a target This corresponds to an interpolation value calculated based on all the pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) set in the vicinity region of the pixel (x, y).

The above expression corresponds to the pixel value of the original pixel before interpolation processing of the target pixel (x, y) as the value of the false color (PF) degree p of the target pixel (x, y) is higher (closer to 1). Adjustment processing in which the influence of the color information C (x, y) is small and the influence of the interpolation value calculated from the color information of the constituent pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) is high Indicates that the correction color information C ′ (x, y) as the correction pixel value of the target pixel (x, y) is set. In the above equation, [p × C _tmp / W _total ] is a value determined based on the color information of the constituent pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ).

On the other hand, as the value of the false color (PF) degree p of the target pixel (x, y) is smaller (closer to 0), the color information C () corresponding to the original pixel value before the interpolation process of the target pixel (x, y) is performed. The pixel of interest (x, y) is adjusted by an adjustment process in which the influence of the interpolation value based on the color information of the constituent pixels in the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) is small. Correction color information C ′ (x, y) of y) is set.

The above processing is the processing of step S217 in FIG. 8, and by this processing, pixel value interpolation processing for one target pixel, that is, the original pixel value of the target pixel and the interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ), and the interpolation value based on the color information of the constituent pixel is applied, and the contribution ratio between the original pixel value and the interpolation value is changed according to the false color degree p of the target pixel, and the corrected pixel value of the target pixel The process of calculating is finished.

As described above, in the image processing according to the present invention, the false color degree p of the pixel of interest as a parameter indicating the possibility of false color is set to (a) white indicating closeness to whiteout (high luminance pixel). Calculation is performed by applying the jump distance evaluation value P _d , (b) the saturation evaluation value P _s indicating the magnitude of the saturation, and (c) the hue evaluation value P _θ indicating the proximity to the specific hue corresponding to the false color. Since the corrected pixel value is calculated by applying the false color degree p according to the above equation (Equation 6), the interpolation value based on the surrounding pixels is used for a pixel having a high possibility of false color such as purple fringe. For pixels with a high contribution rate and a low possibility of false color, correction with a high contribution rate of the original pixel value is executed, and a high-quality image without failure can be acquired.

In this processing example, the processing example according to the flowchart of FIG. 13 is given as an example of a method for correcting false color pixels such as purple fringe. There is also a method of reducing the saturation using. For example, when YCbCr is used as the color space, this method is realized by a calculation process represented by the following equation (Equation 7).
Cb ′ = (1.0−p) * (Cb−128) +128
Cr ′ = (1.0−p) * (Cr−128) +128
... (Formula 7)

  In the above formula (formula 7), Cb and Cr indicate the original color of the target pixel to be corrected, and are values (Cb, Cr) in the YCbCr color space. These values have values from 0 to 128. By applying the false color (PF) degree p of the target pixel (x, y) to these values, the color information Cb ′, Cr ′ of the target pixel after correction is determined. Also in the above equation, the correction with less influence of (Cb, Cr) as the original color data of the target pixel is performed as the false color (PF) degree p of the target pixel (x, y) is higher.

Further, with respect to pixels with reduced color saturation calculated by the above equation (Equation 7), and executes the processing of steps S302~S309 shown in the flowchart of FIG. 13, the interpolation value calculation range _(x 2, _{y 2} , X ₃ , y ₃ ), a color information interpolation value storage variable C _tmp and a weight sum storage variable W _total based on all the pixels are calculated, and using these values, the process of step S310, that is, the above-described formula (formula 6) may be applied to calculate the correction color information C ′ (x, y) of the target pixel. This correction process also realizes effective false color (purple fringe) reduction.

  In addition, as a method of correcting false colors such as purple fringe, there is a method of applying a low-pass filter in which the kernel size is set based on the false color (PF) degree p calculated by the above equation (Equation 5). Applicable. For example, when the false color (PF) degree p of the target pixel (x, y) is low, only pixel values of pixels relatively close to the target pixel (x, y) are selected, and these pixel values are set. When the applied interpolation processing is executed and the false color (PF) degree p of the target pixel (x, y) is high, not only the vicinity of the target pixel (x, y) but also a wider range of peripheral pixels is referred to as the reference pixel. It is good also as a structure which selects as and performs the interpolation process which applied the pixel value of these wide range pixels.

  That is, based on the false color (PF) degree p calculated by the above formula (Formula 5), based on the interpolation value corrected by applying the low-pass filter whose filtering mode is changed, that is, the pixel value near the target pixel. The generated interpolation value may be corrected and the corrected interpolation value may be applied to calculate the corrected pixel value of the target pixel.

In addition, if the correction result is not stable if the false color (PF) degree p calculated by the above-described expression (Expression 5) is used as it is, the false color (PF) degree p or the overexposure distance evaluation value P _d , saturation By using a degree evaluation value P _s and a hue evaluation value P _θ as inputs, a lookup table in which an optimum correction mode corresponding to each value is set is prepared, and a correction process based on the lookup table is executed. For example, the purple fringing degree may be corrected so that purple fringing is reduced.

  Returning to the flow of FIG. 8, the description of the image processing sequence will be continued. In step S217, according to the process described with reference to FIG. 13, the corrected color information C ′ (x, y) of one target pixel (x, y), that is, the surrounding pixels of the target pixel by the above process. An interpolated pixel value to which the pixel value is applied is determined. When the corrected pixel value of one pixel of interest (x, y) is determined, next, in step S218, the x component value of the coordinate (x, y) of the pixel of interest is increased by 1, and in step S219, x is x> w −1, that is, whether or not the image size width w has been reached is determined. If Yes, the process proceeds to S220. If No, the process returns to S203. In step S220, the y component value of the target pixel (x, y) is increased by 1. In step S221, it is determined whether y has reached y> h−1, that is, the image size height h. If not, the process ends. If No, the process returns to S202. These processes are processes in which the target pixel (x, y) is moved one by one in the image data of the processing target image (width w pixel, height h pixel). When step S221 is determined to be Yes, The processing for all the pixels of the processing target image (width w pixel, height h pixel) is completed, and the corrected pixel value is determined after setting all the pixels as the target pixel.

  The above processing is the details of the image correction processing based on the averaging method in step S12 in the overall processing flow of FIG. The feature of this processing is that it is possible to generate an image without failure even when all processing is performed automatically without performing user assistance, that is, processing such as setting an optimum parameter according to the image. Is a point.

  As described above, the image correction processing based on the averaging method in step S12 in the overall processing flow of FIG. 2 is performed on the reduced image generated in step S11. In step S13, enlargement processing of image correction result data based on the averaging method executed in step S12 is executed.

  In step S13, processing for reflecting the correction result in step S12 on the original input image is performed. Specifically, a process of updating only the color information of the part of the original image corresponding to the part whose color is corrected on the reduced image in step S12 is performed. As a first step for that purpose, it is necessary to enlarge the reduced image to have the same resolution as the original image. Various methods for enlarging a small image are generally known. One example is enlargement processing by bilinear interpolation. In this method, when obtaining the pixel value of the image after enlargement, the coordinates when the image before enlargement is used as a reference are obtained, and linear interpolation is performed from four neighboring pixels corresponding to the four corners of the position.

Details of the enlargement process in step S13 will be described with reference to FIG. In the processing flow of FIG. 15, the image data obtained by executing the enlargement process on the corrected image generated in step S12 and the original image data as the photographed image data are blended to generate enlarged image data.
The color information of the enlarged image of the corrected image based on the averaging method is represented by C _L (x, y),
The color information of the original image is represented by C _org (x, y),
As a result of the enlargement process, the color information of the image data is changed to C _new (x, y),
As a result of the enlargement processing, color information C _new (x, y) of the image data is obtained by blending C _L (x, y) and C _org (x, y).

First, in step S401, a blend ratio image B _L (x, y) is set. The blend ratio image B _L (x, y) is an image having the same number of pixels (H _L × W _L ) as the original image, and a blend ratio of 0 to 1 is set corresponding to each pixel. The blend ratio image B _L (x, y) is generated based on the false color (PF) degree p calculated in step S12. The false color (PF) degree p is calculated for each pixel on the reduced image. First, this is enlarged by the above-described bilinear interpolation or the like, so that the number of pixels (H _L × W _L ) is the same as that of the original image. Next, by multiplying each pixel value of the enlarged false color (PF) degree image by a constant from 0 to 1, the contribution ratio of the average method correction result to the final result can be changed. That is, when the correction is made weakly, the contribution ratio of the average method correction result in the final result image can be reduced by multiplying each pixel by a constant having a smaller value. The constant from 0 to 1 is set by the user or a predetermined value is used. An image calculated based on an image obtained by expanding the false color (PF) degree in this way is defined as a blend ratio image B _L (x, y).

In step S402, the corrected image generated by the averaging method generated in step S12 is enlarged to obtain an enlarged image having color information C _L (x, y). As the enlargement method, the above-described enlargement process by bilinear interpolation is applied. By this processing, the reduced image data subjected to the correction process by the average method, that is, the low resolution image data is enlarged to the same resolution (width W _L , height H _L ) as the original input image, and the enlarged image C _L (X, y) is obtained.

Next, in step S403 and subsequent steps, color information C _new (x, y) is determined for each constituent pixel of the image data as a result of the enlargement process. First, in steps S403 and S404, y = 0 and x = 0 are set as initial settings of the processing pixel (x, y) of the enlarged image having the number of pixels (H _L × W _L ).

In step S405, the color information C _new (x, y) of the processing pixel (x, y) of the enlarged image is determined. The color information determination process is the following formula (formula 8), that is,
C _new (x, y) = B _L (x, y) × C _L (x, y) + (1−B _L (x, y)) × C _org (x, y) (Equation 8)
Ask according to.

The above equation (Equation 8) is obtained by changing the original input image and the original input image according to the blend ratio determined based on the blend ratio image B _L (x, y) in which a value in the range of 0 to 1 is set for each pixel. The color information C _new (x, y) of the processing pixel (x, y) of the enlarged image is determined by blending with the corrected image generated by the averaging method generated in step S12. The blend ratio image B _L (x, y) is an image calculated based on the correction degree of correction by the average method. By using this and blending the original image and the result of the averaging method, it is possible to update the color information of only the corrected part of the original image, that is, only the part where the false color has occurred. For this reason, since the original image is used as it is for the portion that has not been corrected in step S12, the image does not deteriorate.

The color information C _new (x, y) of one pixel of interest (x, y) is determined according to the calculation process according to the above equation (Equation 8). When the pixel value of one pixel of interest (x, y) is determined, next, in step S406, the x component value of the coordinate (x, y) of the processing pixel is increased by 1, and in step S407, x is x> W _L −1, that is, whether or not the image width W _L of the image data having the same resolution (width W _L , height H _L ) as the original input image has been reached is determined. If Yes, the process proceeds to S408. If No, the process proceeds to S405. Return. In step S408, the y component value of the processing pixel (x, y) is increased by 1. In step S409, it is determined whether y has reached y> H _L -1, that is, the image size height H _L. If Yes, the process ends. If No, the process returns to S404. These processes are processes for moving the pixel of interest (x, y) one by one in the image data of the processing target image (width W _L pixel, height H _L pixel), and step S409 is determined as Yes. Then, the processing for all the pixels of the processing target image (width W _L pixel, height H _L pixel) is completed, and the pixel values of all pixels having the same resolution (width W _L , height H _L ) as the original input image Is determined.

Note that the enlarged image generated by blending the original input image and the correction image by the averaging method generated in step S12, that is, the enlarged image generated in step S13 of the overall processing flow shown in FIG. In order to distinguish from the enlarged image to be generated, [C _avgnew (x, y)] is used.

  Next, in step S14 of the comprehensive processing flow shown in FIG. 2, image correction processing based on the hole filling method is executed. The image correction process based on the hole filling method in step S14 is executed on the reduced image generated in step S11. By executing as an image process for a reduced image, it can be executed as an efficient process, that is, a process with reduced calculation cost.

  Details of the image correction processing based on the hole filling method in step S14 will be described with reference to FIG. FIG. 16 is a flowchart showing the entire image correction processing based on the hole filling method. Details of the hole filling process will be described with reference to FIG. Note that the image correction process based on the hole filling method shown in FIG. 16 is a process executed on the reduced image generated in step S11 of the overall processing flow of FIG. The process of each step will be described.

  Step S501 is an overexposure area detection process in which an overexposed portion is detected from the input image data to be corrected. Step S502 is a false color pixel detection area setting process, in which a search area for purple pixels, which are false color generation pixels around whiteout, is set. Further, step S503 is a false color pixel detection process, in which purple pixels in the search area are detected. Step S504 is a false color pixel correction process, in which the color component of the detected purple pixel is corrected. Further, step S505 is a color blurring process (filtering process), and a filtering process for obtaining a natural result is performed.

  As described above, the image to be processed is executed on the reduced image generated in step S11 of the overall processing flow shown in FIG. Details of each processing step will be described. First, in the overexposure area detection process in step S501, an overexposure area detection process (excessive mask calculation) based on the converted image data is executed. That is, an overexposed pixel is detected from the image data, and an overexposed mask S that is a binary image is generated. With reference to FIG. 17, the details of the overexposed pixel detection process executed in step S501 will be described.

In steps S601 and S602, the initial setting of the inspection target pixel position of the reduced image that is the correction target image data is executed. As an initial setting, the inspection target pixel position of the reduced image that is the processing target image is set as x = 0 and y = 0. In step S603, for each pixel in the luminance component image _{L in} (x, y), the luminance is determined whether a predetermined threshold value or more, the process proceeds to step S604 if more than a threshold value, overexposure mask S (x, y) Is true and if it is equal to or less than the threshold value, the process proceeds to step S605, and the overexposure mask S (x, y) is set to false.

  In step S606, the pixel position: x is updated. In step S607, it is determined whether x has reached the maximum value (x = w). If not, step S603 and subsequent steps are repeatedly executed. If x reaches the maximum value (x = w) in step S607, the process proceeds to step S608 to update the pixel position: y, and whether y reaches the maximum value (y = h) in step S609. If it has not been reached, step S602 and subsequent steps are repeatedly executed. When the processing for all the pixels (x = 0 to w, y = 0 to h) is finished, the generation of the whiteout mask is finished.

  As a result, at each pixel position, a whiteout mask is generated in which a pixel region having a luminance level greater than the threshold value and a pixel region having a luminance region less than the threshold value are separated. When a luminance level of (dark) 0 to 255 (bright) is set, the threshold is set to about 250, for example, pixels having a luminance level of 250 or higher are extracted as overexposed pixels, and only these pixels are extracted. A whiteout mask is generated which makes it possible to distinguish between the two.

  Step S501 in FIG. 16 is completed by the above processing, and then the false color pixel detection area setting processing in step S502 is executed. In this process, a false color generation part search area setting (Dilate) process around the overexposed portion is executed. Here, a mask P for distinguishing the false color generation portion search region around the overexposed portion, that is, a mask P representing the purple pixel search region around the overexposed portion is generated. P is a binary mask image.

  With reference to FIG. 18, the details of the false color generation part search area setting process in step S502 will be described. First, in step S651, all the pixels of the false color area mask P are initialized to false. Next, in steps S652 and S653, initial setting of the pixel position is executed. As an initial setting, the pixel position is set as x = 0 and y = 0.

  After the pixel position is set, in step S654, the pixel value of the overexposure mask S at the set pixel position is determined. If the overexposed mask S (x, y) = false at the set pixel position (x, y), the process proceeds to step S664, and the process for the next pixel is performed. If S (x, y) = true, the process proceeds to step S655. The overexposure mask S (x, y) = false at the pixel position (x, y) means that the pixel value at the pixel position (x, y) is not an overexposure pixel. , Y), the overexposure mask S (x, y) = true means that the pixel value at the pixel position (x, y) is an overexposure pixel.

  If the overexposure mask S (x, y) = true at the pixel position (x, y), a range in which a false color may be generated around the overexposure pixel (x, y) in step S655. calculate. As described above, the purple fringe occurs around the overexposed portion, and the range in which the purple fringe may occur varies depending on the aperture and focal length of the optical system. Further, the generated range varies depending on the distance from the optical center on the image. In the present embodiment, for example, pre-set false color pixel detection area calculation information such as a look-up table (LUT) in which a false color pixel detection area corresponding to an image area is set based on the characteristics of the optical system of the imaging system is used. Apply. For example, the look-up table (LUT) is a false color pixel using, as input parameters, the distance from the optical center on the image to the overexposed pixel (x, y), the aperture at the time of image data capture, and focal length information. It has a configuration capable of outputting detection area information.

  The aperture and focal length information at the time of photographing image data is acquired from the attribute information of the image data, and the distance from the optical center on the image to the overexposed pixel (x, y) is set as the inspection object in this flow. Pixel position information is applied.

  If attribute information such as aperture and focal length information at the time of photographing image data cannot be acquired, a false color pixel detection region is based only on the distance from the optical center on the image to the overexposed pixel (x, y). In addition, a certain area around the overexposed pixel (x, y) is false color without applying information such as the distance from the optical center to the overexposed pixel (x, y). The pixel detection area may be set uniformly.

In step S655, the range of the false color pixel detection area around the overexposed pixel (x, y) is calculated as four scalar values (x ₀ , x ₁ , y ₀ , y ₁ ). This is because, as shown in FIG. 19, overexposed pixel (x, y) of the rectangular region defined by _{x 0 ~x 1, y 0 ~y} 1 around the 601, overexposed pixel (x, y) of A false color pixel detection area 602 exists in the surrounding area. In FIG. 19, a dotted line pixel 603 is a whiteout pixel 603 other than the target whiteout pixel (x, y) 601.

Next, the calculated false color pixel detection region (x ₀ , x ₁ , y ₀ , y ₁ ) is set starting from i = x ₀ , j = y ₀ (S 656, S 657), and in steps S 658 to S 662. Then, for each pixel in the false color pixel detection area (x ₀ , x ₁ , y ₀ , y ₁ ), it is determined whether the value of the overexposure mask S (i, j) is true or false. For the constituent pixels in the false color pixel detection area 602 shown in FIG. 19, it is sequentially determined whether the overexposure mask value is true or false. In FIG. 19, with respect to the target white skip pixel (x, y) 601 and the white skip pixel 603, the white skip mask S (i, j) = true.

If it is determined in step S658 that the overexposed mask S (i, j) = true, the process proceeds to step S660, and the next pixel is processed. If the whiteout mask S (i, j) = false, the process proceeds to step S659, and the value of the false color area mask P (i, j) is set to true. Thereafter, the process proceeds to step S660, and processing for the next pixel is performed. Steps S660 to S663 are a process for updating the values of i and j and a process for determining whether or not the maximum values i = x ₁ and j = y ₁ are satisfied. While sequentially updating the values of i and j, the value of the false color region mask P (i, j) set around the specific pixel (x, y) is determined.

  The false color area mask P (i, j) = true is a pixel that belongs to the false color pixel detection area and is not a whiteout pixel, and the false color area mask P (i, j) = false is a false color pixel detection area. Corresponds to a pixel that does not belong to the pixel or is a whiteout pixel.

If it is determined in step S663 that the processing has been completed for all the pixels (i, j) in the false color pixel detection region (x ₀ , x ₁ , y ₀ , y ₁ ), the process proceeds to step S664. Steps S664 to S667 are a process of updating the values of x and y and a process of determining whether or not the maximum values x = w and y = h. While sequentially updating the values of x and y, the values of the false color region mask P (i, j) are determined for all the pixels of the processing target image (x = 0 to w, y = 0 to h), and the false color is determined. A region mask P (i, j) is generated. When it is determined that all the pixels (x, y) in the image have been processed, the false color pixel detection area setting process in step S502 of FIG. 16 ends.

  The next processing is the false color pixel detection processing in step S503 in FIG. In step S502, an over-the-counter false color, that is, an area in the image where a purple fringe may exist is calculated as a false color area mask P. In the next step S503, it is actually a purple fringe. A process of detecting a possible pixel is performed. That is, pixels that are actually considered to be purple fringes are detected for pixels in the area of the false color area mask P = true.

  In step S503, the false color area mask P is updated by overwriting the false color area mask P calculated in step S502, and a false color area mask P that can specify only pixels that are finally determined to be false color is obtained. Generate. Details of the false color pixel detection processing in step S503 will be described with reference to FIG.

  In steps S701 and S702, initial setting of the pixel position of the image data is executed. As an initial setting, the pixel position is set as x = 0 and y = 0. In step S703, the value of the false color area mask P at the pixel position (x, y) is determined. If P (x, y) = false, the process proceeds to step S706. If P (x, y) = true, the process proceeds to step S704.

As described above, the false color region mask P (i, j) = true is a pixel that belongs to the false color pixel detection region and is not a whiteout pixel, and the false color region mask P (i, j) = false.
Corresponds to a pixel that does not belong to the false color pixel detection region or is a whiteout pixel.

If the pixel position (x, y) is a false color area mask P (i, j) = true, that is, a pixel that belongs to the false color pixel detection area and is not an overexposed pixel, in step S704, the position (x , Y) determines whether the color of the color component image C _in is a false color-corresponding color. For example, it is determined whether the purple fringe is purple.

A method for determining whether the color component image C _in (x, y) is a false color-corresponding color, for example, purple, differs depending on the color space applied in this process. For example, an example of a determination method when CIE L ^* a ^* b ^* is used as the color space will be described with reference to FIG. The color component is determined using the a ^* component and the b ^* component. The value of a ^* component, red (Red) The more approaches the green (Green) Invite Helle, ^{b *} yellow The more the value of the component (Yellow), with characteristics closer to blue (Blue) Invite Hereford Yes. Therefore, when (a ^* , b ^* ) exists in the fourth quadrant, it has a color close to purple. To determine whether the color is purple, the vector (a ^* , b ^* ) is determined using an angle α formed with the a ^* axis.

The range of the angle to be purple that is determined as the false color is set as a parameter as appropriate. Alternatively, a preset value is applied. Note that the saturation of the false color (purple fringe) partial pixels tends to be large. Therefore, only when the above condition is satisfied and the saturation of the color component C _in is equal to or greater than a preset threshold value may be determined to be a false color (purple fringe). In this way, if C _in (x, y) is determined to be a false color (purple fringe) pixel, the process proceeds to step S706, and if it is determined not to be a false color (purple fringe), the process proceeds to step S705.

  In step S705, the false color area mask P (x, y) is set to false, and the process proceeds to step S706. That is, in the process of the previous step S502 (FIG. 16), the false color area mask P (x, y) = true is assumed to be a false color, based on the final determination that it is not a false color. Thus, the false color area mask P (x, y) is changed (updated) to false.

  That is, in the processing described in FIG. 20, only pixels that are finally determined to be false color by color determination from pixels that may be false color in the processing of the previous step S502 (FIG. 16). The false color area mask P (x, y) is updated by sorting.

  Steps S <b> 706 to S <b> 709 are a process for updating the values of x and y and a process for determining whether or not the maximum values x = w and y = h. While sequentially updating the values of x and y, the values of the false color region mask P (i, j) are determined for all the pixels of the processing target image (x = 0 to w, y = 0 to h), and the false color is determined. A region mask P (i, j) is generated. When it is determined that all the pixels (x, y) in the image have been processed, the false color pixel detection process in step S503 in FIG. 16 ends.

  Next, the false color pixel correction process in step S504 shown in FIG. 16 will be described. In step S503, a false color area mask P for identifying pixels determined to be false colors (purple fringes) as described above is generated. In the next step S504, a process of interpolating the color of the pixel determined to be a false color (purple fringe) by the hole filling repetition process based on the pixel values of the surrounding pixels is executed.

  22A and 22B are schematic diagrams for explaining the false color pixel correction processing in step S504 shown in FIG. As shown in FIG. 22A, in the processing up to step S503, all the pixels are “whiteout pixels 651”, “false color (purple fringe) pixels 652”, “others that are neither whiteout nor purple fringe”. The pixel is classified into one of “pixels 653”.

  In the first step of the hole filling iterative process, each pixel in the image is scanned, and if the false color (purple fringe) pixel 652 (position (x, y)) is not another white or purple fringe pixel 653 When adjacent, for example, when there are other pixels 653 in the surrounding eight neighboring pixels, the average value of the color components C of the other adjacent pixels 653 is set as the color component C (x, y) of the new pixel. To do.

  In this way, when the process of setting the pixel value of the false color (purple fringe) pixel 652 by the other pixels 653 is performed once for all the pixels, the other pixels 653 are obtained as shown in FIG. The color component of the false color (purple fringe) pixel 652 adjacent to is interpolated and set as an interpolated false color (purple fringe) pixel 654.

  Further, in the next iterative filling process, the color component of the false color (purple fringe) pixel 652 adjacent to the other pixel 653 or the interpolated false color (purple fringe) pixel 654 is replaced with the other pixel 653 and the interpolated false color ( Purple fringing) Interpolate in the same manner based on pixel 654. Such repeated processing is repeated a fixed number of times. By this processing, the false color (purple fringe) pixel 652 is sequentially set as an interpolated false color (purple fringe) pixel 654 in the direction of the arrow 670 in FIG. If there is a false color (purple fringe) pixel 652 in which the color component is not interpolated even after repeating such a repetition process a fixed number of times, it is determined that color interpolation from surrounding pixels is impossible. A process is performed to uniformly reduce the saturation of the pixel color.

  As described above, the false color pixel correction processing in step S504 shown in FIG. 16 sets the pixel value of the pixel determined to be false color (purple fringe) in step S503 based on the surrounding pixels other than overexposure. This is a process, that is, a hole filling repeated process. The remaining false color (purple fringe) pixels 652 that are not interpolated by this processing are subjected to saturation reduction processing.

  Details of the false color pixel correction processing in step S504 will be described with reference to the processing flows in FIGS. In the processing flow of FIGS. 23 and 24, steps S751 to S765 shown in FIG. 23 are hole filling processing of color components of false color (purple fringe) pixels, and the processing from steps S766 to S773 shown in FIG. This is a process for reducing the saturation of pixels that could not be filled.

Description will be made sequentially from the flow of FIG. First, in step S751, the content of the false color area mask P generated in the false color pixel detection process in step S503 of FIG. 16 is copied to a binary mask P ″ having an equivalent size.
False color area mask P = true → binary mask P ″ = true (1)
False color area mask P = false → binary mask P ″ = false (0)
It is.

  In step S752, the value of variable t indicating the number of repetitions of the hole filling process is set to zero. In step S753, the value of the binary mask P ″ is copied to another equivalent size binary mask P ′. These masks P ″ and P ′ are false values that are reduced by the iterative process. This is a temporary mask image used to store the position of the color (purple fringe) pixel 652.

  Next, in steps S754 and S755, variables x and y representing the processing target pixel position coordinates are initialized, and x = 0 and y = 0. In the next step S756, the value of the binary mask P ′ (x, y) is determined. If the binary mask P ′ (x, y) = false, that is, if the pixel to be processed is not a false color pixel, the process proceeds to step S760. If the binary mask P ′ (x, y) = true, that is, if the processing target pixel is a false color pixel, the process proceeds to step S757.

  In step S757, it is determined whether or not a pixel adjacent to the pixel (x, y) has a binary mask P ′ = false and a whiteout mask S = false. That is, it is determined whether there is a pixel that is neither a false color pixel nor a whiteout pixel. If it is determined that no pixel satisfying such a condition exists in a pixel adjacent to the pixel (x, y), the process proceeds to step S760.

  If it is determined that a pixel that is neither a false color pixel nor a whiteout pixel exists as a pixel adjacent to the pixel (x, y), the process proceeds to step S758. In step S758, the binary mask P ′ = false and the whiteout mask S = false, that is, one or more pixels that are neither false-color pixels nor whiteout pixels and are adjacent to the pixel (x, y). The average value of the color component C is calculated and set to the color component C (x, y) at the coordinates (x, y). That is, it is set as the pixel value of the pixel (x, y). By this interpolation processing, the interpolated false color (purple fringe) pixel 654 shown in FIG. 22B is set.

  After this process, the process proceeds to step S759. In step S759, the value of the binary mask P ″ (x, y) is set to false. That is, the pixel changed from the false color pixel to the interpolated false color pixel is represented by the binary mask P ″ (x, y). In FIG. 5, the setting is such that it can be identified as a pixel that is not a false color pixel.

  Steps S760 to S763 are processing pixel coordinate update and maximum value determination processing. In step S760, the value of the x coordinate is incremented by one. In the next step S761, it is determined whether or not x> w−1 (w is the width of the input image). If it is determined that x> w−1, the process proceeds to step S762. If it is determined that x> w−1 is not satisfied, the process proceeds to step S756.

  In step S762, the y coordinate is incremented by 1, and in the next step S763, a comparison is made with h-1 (h is the height of the input image) which is the maximum value of the y coordinate. If y> h-1, the process proceeds to step S764, and if it is determined that y> h-1, the process proceeds to step S755.

  In step S764, t representing the number of repetition processes is incremented by one. In step S765, it is determined whether the number of repetitions t is equal to or greater than a predetermined value tmax. At the same time, it is determined whether or not the number of pixels having a true value in the mask image P ″ is 0. That is, it is determined whether or not the number of false color pixels that have not been interpolated by the hole-filling interpolation processing has become 0. If it is determined that either of the above two conditions is true, the process proceeds to step S766, and if both are determined to be false, the process returns to step S753 to perform the filling process again.

  In step S765, it is determined that either the number of repetitions t is equal to or greater than a predetermined value tmax, or the number of false color pixels that have not been interpolated by the hole-filling interpolation processing has become zero. Then, the process proceeds to step S766 in FIG.

  In steps S766 and S767, variables x and y representing the coordinates of the pixel to be processed are initialized so that x = 0 and y = 0. In the next step S768, it is determined whether or not the value at (x, y) of the binary mask P ″ is true. That is, it is determined whether or not the pixel to be processed is a false color pixel. The binary mask P ″ ( If x, y) = true, that is, if it is a false color pixel, the process proceeds to step S769. If the binary mask P ″ (x, y) = false, that is, if it is not a false color pixel, step S770. Proceed to

If the pixel to be processed is a false color pixel, a saturation reduction process is performed in step S769 to reduce the saturation of the color component C (x, y) of the pixel (x, y). For example, when the L ^* a ^* b ^* color system is used, the values of each component of a ^* and b ^* are uniformly colored by multiplying a constant of 1.0 or less and 0.0 or more. The degree can be reduced. When the saturation reduction process in step S769 ends, the process proceeds to step S770. In step S770, the value of the x coordinate is increased by one. In the next step S771, it is determined whether or not x> w-1. If it is determined that x> w−1, the process proceeds to step S772, and if it is not determined that x> w−1, the process proceeds to step S768, and the same processing is repeated for the adjacent pixel whose x coordinate is updated.

  If it is determined in step S771 that x> w−1, the maximum value of the x coordinate has been reached, the process proceeds to step S772, the y coordinate is incremented by 1, and the maximum value of the y coordinate is determined in the next step S773. Comparison with h-1 is performed. If not y> h−1, the process proceeds to step S767, and the same processing is repeated for the pixel having the updated y coordinate. If it is determined in step S773 that y> h-1, the false color pixel correction process is terminated. The false color pixel correction process in step S504 shown in FIG.

  When the false color pixel correction process in step S504 is completed, the color blurring process (filtering) in step S505 is performed. In step S504, the color of the false color (purple fringe) pixel is interpolated by repeating the hole filling process, and the process of reducing the saturation is performed on the false color (purple fringe) pixel that is not completely filled. However, in the false color (purple fringe) portion subjected to the processing in step S504, there is a possibility that a portion where the color changes relatively abruptly occurs. In the color blurring process (filtering) in step S505, a blurring filter is applied to alleviate this sudden color change.

  Details of the color blurring process (filtering) in step S505 will be described with reference to the process flow of FIG. First, in step S801, the value of a variable t indicating the number of repetitions of color blurring processing (filtering) is set to zero. Next, in step S802, the color component image C updated by the false color pixel correction process in the previous step S504 is copied to the color component image C ′ having the same size.

  Next, in steps S803 and S804, variables x and y representing the coordinates of the processing target pixel are initialized, and x = 0 and y = 0. In the next step S805, the value of the false color area mask P (x, y) is determined. If the false color area mask P (x, y) = false, that is, if it is not a false color pixel, the process proceeds to step S807. If the false color area mask P (x, y) = true, that is, if it is a false color pixel, the process proceeds to step S806.

  The false color area mask P (x, y) is a pixel determined to be a false color based on the color (pixel value) determination by the false color pixel detection process (corresponding to the process flow of FIG. 20) in the previous step S503. Is set as true, and the false color pixel region corrected by the false color pixel correction processing in the previous step S504 is also mask data maintaining the value of true. In order to apply the false color area mask P (x, y) to this color blurring process (filtering) process, in the false color pixel correction process in step S504, a copy mask P "of the false color area mask P (x, y) is used. Applied.

  The description of the color blurring process (filtering) in FIG. 25 will be continued. If it is determined in step S805 that the false color area mask P (x, y) = true, that is, a false color pixel, the process proceeds to step S806, and in step S806, a blur filter is applied to the color component. This is a process of updating the value of the pixel value C (x, y) indicating the color component of the pixel (x, y) in the color component image C ′ based on the pixel value C ′ indicating the color component of the surrounding pixels. is there. For example, an average value of nine color components C ′ including the processing target pixel (x, y) and the surrounding eight pixels is obtained, and this is used as the updated pixel value C (x of the processing target pixel (x, y). , Y) apply a moving average filter set.

  When the blur filter application process in step S806 ends, the coordinates of the processing target pixel are updated and the maximum value is confirmed in steps S807 to S810. In step S807, the value of the x coordinate is increased by one. In the next step S808, it is determined whether or not x> w-1. If it is determined that x> w−1, the process returns to step S808. If it is determined that x> w−1 is not satisfied, the process returns to step S804, and the same processing is performed on the pixel whose x coordinate has been updated. Execute blur processing.

  If it is determined in step S808 that x> w−1, the process proceeds to step S809, where the y coordinate is incremented by 1, and in the next step S810, comparison is made with h−1 which is the maximum value of the y coordinate. If it is determined that y> h−1 is not satisfied, the process proceeds to step S804, and the same process is performed on the pixel having the updated y coordinate, and the blurring process is performed as necessary.

  If it is determined in step S810 that y> h-1, the process proceeds to step S811. In step S811, a process of increasing the number of repetitions t of the color blurring process (filtering) by 1 is performed, and in the next step S812, it is determined whether or not the number of repetitions t is equal to or greater than a predetermined threshold value t′max. If t <t′max, the process returns to step S802. If t ≧ t′max, the color blurring process (filtering) ends.

  With the above processing, the image correction processing based on the hole filling method in step S14 in the overall processing flow of FIG.

  Next, in step S15 in the overall processing flow of FIG. 2, this process of performing the process of reflecting the portion corrected by the hole filling method for the reduced image in step S14 in the original input image is performed by the correction image based on the above-described averaging method. A method similar to the enlargement process for step S13, that is, the enlargement process in step S13 shown in FIG. Specifically, the method is almost the same as the method described with reference to FIG.

  However, steps S401 and S402 in FIG. 15 are replaced with the result of the hole filling method. As an overall flow, image data obtained by generating a blend ratio image and executing enlargement processing on the corrected image generated in step S14 and original image data as captured image data are blended to obtain enlarged image data. Is generated.

Processing of each step will be described with reference to FIG. First, in step S401, a blend ratio image B _L (x, y) is set. The blend ratio image B _L (x, y) is an image having the same number of pixels (H _L × W _L ) as the original image, and a blend ratio of 0 to 1 is set corresponding to each pixel. The blend ratio image B _L (x, y) is generated with reference to the false color area mask P calculated in step S14. The false color area mask P is a binary mask indicating whether or not each pixel has a false color on the reduced image. Here, an image having a value of 1.0 for a pixel with a false color area mask P of true and a value of 0.0 for a false pixel is created, and the image is enlarged. In other words, enlargement processing is performed by the above-described bilinear interpolation or the like, so that the number of pixels (H _L × W _L ) is the same as that of the original image. The pixel value of this enlarged image is 1.0 when the false color area mask P is true, 0.0 when false, and between 0.0 and 1.0 when true and false transition. It becomes an image that takes the value of. Next, by multiplying each pixel value of the enlarged image by a constant from 0 to 1, it is possible to change the contribution ratio of the hole filling method correction result to the final result. That is, when the correction is made weakly, by multiplying each pixel by a constant having a smaller value, the contribution ratio of the filling method correction result in the final result image can be lowered. As the constants from 0 to 1, a value set by the user or a predetermined value is used. An image calculated based on an image obtained by expanding the degree of false color (PF) in this way is defined as a blend ratio image B _L (x, y).

In step S402, the corrected image generated by the hole filling method generated in step S14 is enlarged to obtain an enlarged image having color information C _L (x, y). As the enlargement method, the above-described enlargement process using bilinear interpolation is applied. By this processing, the reduced image data subjected to the correction processing by the hole filling method, that is, the low resolution image data is enlarged to the same resolution (width W _L , height H _L ) as the original input image, and the enlarged image C _L (X, y) is obtained.

Next, in step S403 and subsequent steps, color information C _new (x, y) is determined for each constituent pixel of the image data as a result of the enlargement process. First, in steps S403 and S404, y = 0 and x = 0 are set as initial settings of the processing pixel (x, y) of the enlarged image having the number of pixels (H _L × W _L ).

In step S405, the color information C _new (x, y) of the processing pixel (x, y) of the enlarged image is determined. The color information determination process is the following formula (formula 9), that is,
C _new (x, y) = B _L (x, y) × C _L (x, y) + (1−B _L (x, y)) × C _org (x, y) (Equation 9)
Ask according to.

The above formula (formula 9) is the same formula as the formula (formula 8) described above, and is based on the blend ratio image B _L (x, y) in which a value in the range of 0 to 1 is set for each pixel. According to the determined blend ratio, the original input image and the correction image generated by the hole filling method generated in step S14 are blended, and the color information C _new (x, y) of the processing pixel (x, y) of the enlarged image. Is to determine. The blend ratio image B _L (x, y) is an image calculated based on the false color region mask P corrected by the hole filling method. By using this to blend the original image and the result of the hole filling method, it is possible to update the color information of only the corrected part of the original image, that is, only the part where the false color has occurred. For this reason, since the original image is used as it is for the portion that has not been corrected in step S14, the image does not deteriorate.

The color information C _new (x, y) of one target pixel (x, y) is determined according to the calculation process according to the above formula (formula 9). When the pixel value of one target pixel (x, y) is determined, in steps S406 to S409, coordinate update in the range of the processing target image (width W _L pixel, height H _L pixel) is executed, and step S405 is performed. The pixel value determination process for all pixels, that is, all pixels having the same resolution (width W _L , height H _L ) as the original input image is repeatedly executed according to the calculation formula shown, that is, the above formula (formula 9).

With this process, the enlargement process of the corrected image data based on the hole filling method performed on the reduced image is completed. The enlarged image generated by blending the original input image and the correction image generated by the hole filling method generated in step S14, that is, the enlarged image generated in step S15 of the overall processing flow shown in FIG. This is distinguished from the enlarged image [C _avgnew (x, y)] generated by the enlargement process, and is _defined as [C _holenew (x, y)].

  The processing up to step S15 in the overall processing flow shown in FIG. 2 is completed so far, and the first corrected image data obtained by enlarging the result of performing the image correction processing based on the average method for the reduced image, and the hole filling method for the reduced image Thus, the second corrected image data obtained by enlarging the result of the image correction processing based on the above is obtained. In step S16 of the overall processing flow shown in FIG. 2, a blend process of these two corrected image data is executed to generate a final corrected image.

  Each of the correction result image data by the hole filling method and the average method has a characteristic. The correction result image by the hole filling method has a large effect because the false color such as purple fringe is reduced by interpolation by the hole filling, and the false color such as purple fringe is interpolated by the color of the surrounding non-purple fringe pixel. . However, the resulting image may vary greatly depending on which pixel is determined to be purple fringe. When processing automatically, it works well for most images, but the result may be unnatural for a small portion of the image.

  On the other hand, correction of false colors such as purple fringe by the average method is a method of blending color information with low-pass filter with the color information of the original image according to the false color (PF) degree, resulting in a large breakdown in the resulting image It is difficult to occur. However, it can be said that the effect of correcting false colors is low compared to the hole filling method.

In step S15 of the overall processing flow shown in FIG. 2, the two corrected image data having the respective characteristics are blended to generate final corrected image data. Blend of the first corrected image _{C Avgnew} (x, y) obtained by the correction process using the false color (PF) degree p and, second corrected image _{C Holenew} calculated by filling interpolation and (x, y) By the processing, processing for calculating _final corrected image data C _final (x, y) is executed.

This blending process is performed by applying a predetermined blending ratio k, for example, the following formula:
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
Calculate according to The blend ratio k is determined in the range of 0-1. The value for setting the blend ratio is determined based on the characteristics of each process. For example, when a larger correction is desired, the reflection ratio of the processing result of the hole filling method is set to be large, and when it is desired to obtain a result image with no failure as much as possible, correction result data applying a false color (PF) degree p is applied. The blend ratio k is set so as to reflect more.

  The above process is a process executed as the process of step S16 which is the final step of the comprehensive process flow shown in FIG.

  Thus, in the processing of the present invention, by applying blend processing, harmonized pixel value correction that combines the advantages of hole filling interpolation processing and correction processing using a false color (PF) degree p is achieved. This makes it possible to reduce a false color such as purple fringe and obtain a corrected image that does not fail. As a result, it is possible to appropriately correct false colors such as purple fringes due to chromatic aberration occurring in an image photographed by a camera, and it is possible to generate and output high-quality image data.

  Next, a functional configuration of a digital signal processing unit (DSP) (corresponding to the DSP 106 in FIG. 1) in the image processing apparatus of the present invention that executes the above-described processing will be described with reference to FIGS. As shown in FIG. 26, the digital signal processing unit (DSP) 106 functionally includes a reduction processing unit 700, an average method processing unit 710 that performs pixel value correction based on the false color (PF) degree p, and a hole filling process. The hole filling method processing unit 720 that executes pixel value correction processing based on the method, the first enlargement processing unit 717 that executes enlargement processing of the correction result of the average method processing unit 710, and the enlargement processing of the correction result of the hole filling method processing unit 720 are executed. A second enlargement processing unit 727, and a blend processing unit 740 that executes blend processing of both correction results.

Average method processing unit 710, overexposure distance evaluation value _{P d} calculator 711, a saturation evaluation value _{P s} calculator 712, the hue evaluation value P _theta calculator 713, a false color (PF) degree p calculator 714, the interpolation value [C _tmp / W _total ] calculation unit 715 and first correction pixel value [C _avgnew (x, y)] calculation unit 716.

  On the other hand, the hole filling method processing unit 720 includes a whiteout detection unit 721, a false color pixel detection region setting unit 722, a false color (purple fringe) detection unit 723, and a pixel value correction unit 724, as shown in detail in FIG. . The pixel value correction unit 724 includes a false color (purple fringe) hole filling interpolation unit 725 and a false color (purple fringe) blurring processing unit 726.

  First, processing of each component of the averaging method processing unit 710 will be described with reference to FIG. As shown in FIG. 26, the digital signal processing unit (DSP) 106 inputs a processing target image frame from an input image frame memory 751. Here, the input image is shown as an input from the frame memory 751, but the captured image data may be directly input to the digital signal processing unit (DSP) 106 and processed without going through the frame memory.

  First, the digital signal processing unit (DSP) 106 executes a reduction process of the input image in the reduction processing unit 700. This reduction process is a process executed according to the process described with reference to the flowcharts of FIGS. 3 and 4 and reduces the luminance information and the color information by different methods. In order to prevent overexposure in the reduction processing of luminance information, in the reduction processing of color information, reduction processing is performed on the input image in consideration of not losing false color data such as purple fringe (PF). Apply.

  Next, the reduced image data is input to the average method processing unit 710 and the hole filling method processing unit 720. In the processing flow described above with reference to FIG. 2, the image correction processing based on the average method is executed in the previous step and then the image correction processing based on the hole filling method is executed. The image correction process based on this and the image correction process based on the hole filling method are completely independent processes, and can be performed in parallel.

First, the processing in the averaging method processing unit 710 will be described. The averaging method processing unit 710 selects each pixel constituting the reduced image generated by the reduction processing unit 700 as a target pixel (x, y) one by one, and will be described with reference to the flowcharts of FIGS. According to the above processing, the following evaluation value corresponding to the target pixel (x, y), that is,
(A) Closeness to whiteout (high luminance pixel) = whiteout distance evaluation value P _d ,
(B) Saturation magnitude = saturation evaluation value P _s ,
(C) proximity to a specific hue corresponding to a false color = hue evaluation value P _θ ,
Overexposure distance evaluation value _{P d} calculator 711 respectively, chroma evaluation value _{P s} calculation unit 712 calculates the hue evaluation value P _theta calculator 713. Overexposure distance evaluation value _{P d} calculation processing in overexposure distance evaluation value _{P d} calculator 711, FIG. 7 is a process to be executed by the processing of steps S204~S214 in the flow of FIG. 8, the above-mentioned formula (Formula 2 ).

Chroma evaluation value _{P s} calculator 712, a saturation evaluation value _{P s} in hue evaluation value P _theta calculator 713, the processing of the calculation step S215 of hue evaluation value P _theta, i.e., has been described with reference to FIG. 11 Calculated based on the values of s and θ, the hue evaluation value P _θ is calculated by the above-described equation (Equation 3), and the saturation evaluation value P _s is calculated by the aforementioned equation (Equation 4).

False color (PF) degree p calculating unit 714, based on the evaluation values calculated overexposed distance evaluation value _{P d} calculator 711, a saturation evaluation value _{P s} calculator 712, the hue evaluation value P _theta calculator 713 Thus, the false color (PF) degree p of the target pixel (x, y) is calculated. The false color (PF) degree p is calculated according to the above formula (Formula 5).

The processes of the interpolation value [C _tmp / W _total ] calculation unit 715 and the first correction pixel value calculation unit 716 execute the process of step S217 in the flow of FIG. 8, that is, the process according to the flow of FIG. The interpolation value [C _tmp / W _total ] calculation unit 715 sets an interpolation value calculation range (x ₂ , y ₂ , x ₃ , y ₃ ) around the target pixel (x, y), and the configuration of this range Based on the pixel color information, a weight sum storage variable W _total and a color information interpolation value storage variable C _tmp are calculated as interpolation processing parameters, and an interpolation value [C _tmp / W _total ] is calculated. This process corresponds to the processes of steps S301 to S309 in the flow of FIG.

  The first correction pixel value calculation unit 716 executes the process of step S310 in the flow of FIG. 13 to calculate color information C (x, y) as first correction pixel value information based on the average method of the target pixel. . The calculation of the color information C (x, y) is executed according to the above-described equation (Equation 6). The reduced image data for which the first correction pixel value for the pixels constituting the reduced image is determined is input to the first enlargement processing unit 717, and the enlargement process is executed.

The first enlargement processing unit 717 executes image enlargement processing according to the processing flow described above with reference to FIG. That is, the blend ratio image B _L in which a blend ratio of 0 to 1 is set on the basis of the false color area mask P configured as a binary mask indicating whether or not each pixel is a false color on the reduced image. (X, y) is generated, the generated blend ratio image B _L (x, y) is applied, and the enlarged image of the corrected image by the average method and the original input image are blended to enlarge the first corrected image. Generate an image. The first corrected enlarged image whose pixel value is corrected based on the averaging method is output to the blend processing unit 740, and the blending process with the second corrected enlarged image based on the correction data in the hole filling method processing unit 720 is executed. It will be.

  With reference to FIG. 27, the process of each component of the hole filling method processing unit 720 will be described. The hole filling method processing unit 720 receives the reduced image data generated by the reduction processing unit 700, and executes correction processing based on the hole filling method on the reduced image data.

  The overexposure detection unit 721 detects a pixel portion causing overexposure based on the reduced image data. This process is a process corresponding to step S501 of the flowchart shown in FIG. 16, and details thereof are described by referring to the flowchart of FIG. And a process of generating a whiteout mask S (x, y) for identifying a whiteout pixel is executed.

  The false color pixel detection area setting unit 722 performs a process of setting a false color pixel detection area in the peripheral part of the whiteout pixel detected by the whiteout detection unit 721. This area setting process is a process for calculating a range in which a false color may occur around a whiteout pixel (x, y) where whiteout mask S (x, y) = true. The false color pixel detection area calculation information is applied. This process corresponds to step S502 of the flowchart shown in FIG. 16, and is executed according to the process flow shown in FIG.

  The false color pixel detection area setting unit 722, for example, the false color associated with the distance from the optical center on the image to the overexposed pixel (x, y), the aperture when photographing the image data, and the focal length information. By applying a look-up table (LUT) having pixel detection area information, a false color pixel detection area is set around each overexposed pixel (x, y). If there is no aperture, focal length information, or the like when photographing image data, the false color pixel detection region is determined based only on the distance from the optical center on the image to the overexposed pixel (x, y). It is also possible to adopt a configuration in which false-color pixel detection is performed on a certain area around a whiteout pixel (x, y) without applying information such as the distance from the optical center to the whiteout pixel (x, y). It is good also as a structure set uniformly as an area | region.

As described above with reference to FIG. 19, the false color pixel detection region setting unit 722 determines the range of the false color pixel detection region around the overexposed pixel (x, y) 601 as (x ₀ , x _1). , Y ₀ , y ₁ ), and a false color area mask P is generated in which whiteout pixels are removed in this area.

  The false color (purple fringe) detection unit 723 applies the false color region mask P set by the false color pixel detection region setting unit 722, and is further determined to be a false color (purple fringe) by the color determination processing of each pixel. A process for detecting a pixel is executed. That is, the false color area mask P is updated to generate a false color area mask P that can identify only the false color pixels to be corrected. This process is a process corresponding to step S503 in the flowchart shown in FIG. 16, and is executed according to the process flow shown in FIG. As described above, the setting of the color to be recognized as a false color is arbitrary, and a configuration in which only purple having a specific color value is set as a false color, or a configuration in which multiple colors such as green and purple are set as false colors Etc. are possible.

  The false color (purple fringe) hole-filling interpolation unit 725 is a correction process for a pixel that is recognized as a false color by the false color (purple fringe) detection unit 723, and is a process corresponding to step S504 of the flowchart shown in FIG. FIG. 23 is a process executed according to the process flow shown in FIG. In this pixel value correction process, a hole filling interpolation process (FIGS. 22 and 23) is executed for a pixel that is recognized as a false color (purple fringe) based on pixel values other than the surrounding false color and overexposed pixels. Reference) and saturation reduction processing (see FIG. 24) of false color (purple fringe) pixels that could not be corrected by a predetermined number of hole-filling interpolation processing.

  The false color (purple fringe) blurring processing unit 726 executes blurring processing on the data corrected by the processing of the false color (purple fringe) hole filling interpolation unit 725. This process corresponds to step S505 in the flowchart shown in FIG. 16, and is executed according to the process flow shown in FIG.

  The false color (purple fringe) blurring processing unit 726 extracts pixels recognized as the false color by the false color (purple fringe) detection unit 723, applies a blurring filter to the color components, and, for example, processes target pixels (x, The average value of the color components of 9 pixels including y) and the surrounding 8 pixels is obtained, and moving average filter processing is performed to set this as the updated pixel value of the processing target pixel (x, y).

The image data that has been subjected to the blur processing in the false color (purple fringe) blur processing unit 726 is input to the second enlargement processing unit 727, and the enlargement process is executed. The second enlargement processing unit 727 executes image enlargement processing according to the processing flow described above with reference to FIG. That is, the blend ratio image B _L in which a blend ratio of 0 to 1 is set on the basis of the false color area mask P configured as a binary mask indicating whether or not each pixel is a false color on the reduced image. (X, y) is generated, the generated blend ratio image B _L (x, y) is applied, and the enlarged image of the corrected image by the hole filling method and the original input image are blended to enlarge the second corrected image. Generate an image. The second corrected enlarged image whose pixel value is corrected based on the hole filling method is output to the blend processing unit 740, and the blending process with the first corrected enlarged image based on the correction data in the averaging method processing unit 710 is executed. It will be.

The second corrected enlarged image in which the pixel value is corrected based on the hole filling method is output to the blend processing unit 740 shown in FIG. 26, and the first corrected enlarged image data based on the correction data in the average method processing unit 710; The blending process with the second corrected enlarged image based on the correction data in the hole filling processing unit 720 is executed. As described above, the blend process includes the first corrected image C _avgnew (x, y) acquired by the correction process using the false color (PF) degree p and the second corrected image C calculated by the hole filling interpolation process. _The final corrected image data C _final (x, y) is calculated by blending with _holenew (x, y), and a predetermined blend ratio k is applied. The formula shown, ie
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
Accordingly, a corrected image C _final (x, y) having a final corrected pixel value is generated, and this image data is stored in the output image frame memory 752. Note that, as described above, the blend ratio k is determined in the range of 0-1. The value for setting the blend ratio is determined based on the characteristics of each process. For example, when a larger correction is desired, the reflection ratio of the processing result of the hole filling method is set to be large, and when it is desired to obtain a result image with no failure as much as possible, correction result data applying a false color (PF) degree p is applied. The blend ratio k is set so as to reflect more.

  By applying the pixel value correction process of the present invention described above, harmonized pixel value correction that combines the advantages of the hole-filling interpolation process and the correction process using the false color (PF) degree p is possible, A false image such as purple fringe is reduced, and a corrected image that does not fail can be acquired. As a result, it is possible to appropriately correct false colors such as purple fringes due to chromatic aberration occurring in an image photographed by a camera, and it is possible to generate and output high-quality image data.

  According to the configuration of the present invention, it is possible to efficiently detect a false color region such as purple fringe generated in the vicinity of an overexposed pixel and to correct a partial pixel value, and to produce a high quality image without causing an influence on the entire image. Data can be generated and output.

  In particular, since the processing of the present invention is configured to perform image processing on a reduced input image, the image can be corrected with a low calculation cost. In addition, since two types of results, the average method and the hole filling method, are blended, correction with a higher degree of freedom is possible. In the above-described embodiment, the example in which the reduction process and the blend process are performed is described. However, these are independent, and only one of them may be configured. That is, it is possible to adopt a processing configuration in which two types of image corrections of the average method and the hole filling method are executed and the results are blended without performing reduction. In particular, when the calculation cost is not a problem, such a process may be executed.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and executed.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  The various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, the reduction processing of the correction target image data is executed, and the reduced image is subjected to the overexposure of the surrounding false color pixels in the first image correction processing mode. Image processing including pixel value correction is executed to generate a first corrected image, and further, the second image correction processing mode that is different from the first image correction processing mode is applied to the reduced image, and the false surroundings are overlooked. Since the second corrected image is generated by executing image processing including pixel value correction of the color pixels, and the different processed result images are enlarged and blended, the final corrected image is generated. Image correction processing can be performed on a reduced image, and the image can be corrected at a low calculation cost. In addition, a correction process that combines image corrections in different processing modes is realized, and harmonized pixel value correction that combines the respective advantages is possible.

  Further, according to the configuration of the present invention, it is possible to perform harmonized pixel value correction that combines the advantages of the hole-filling interpolation processing and the correction processing to which the false color (PF) degree p is applied. It is possible to obtain a corrected image that reduces the color and does not fail. Specifically, it is possible to appropriately correct false colors such as purple fringes due to chromatic aberration occurring in an image photographed by a camera, and it is possible to realize generation and output of high-quality image data.

  Furthermore, according to the configuration of the present invention, it is possible to generate an image without failure even when all image correction processing is performed as automatic processing. By applying the image processing of the present invention, it is possible to correct the captured image so that it looks more natural. Therefore, it is not necessary to pay attention to the lens aperture and focal length so that purple fringing does not occur during shooting, and shooting can be performed with a higher degree of freedom.

  Furthermore, according to the configuration of the present invention, the false color correction process is performed after the reduction process, and the enlargement process is performed. However, the part where the pixel value is updated in the original image is only the part where the false color is generated, and only the color information is corrected. Therefore, the portion other than the false color is not deteriorated at all. Since a special reduction process is performed on the portion where the false color is generated, the false color pixel can be reliably recognized. As described above, although the calculation cost is reduced by the reduction process, the false color correction quality is hardly deteriorated.

It is a figure which shows the structure of the image processing apparatus of this invention. It is a flowchart figure which shows the comprehensive process of the image processing of this invention. It is a figure which shows the flowchart (the 1) explaining the procedure of the reduction process performed in the image processing of this invention. It is a figure which shows the flowchart (the 2) explaining the procedure of the reduction process performed in the image processing of this invention. It is a figure explaining the detail of the reduction process performed in the image processing of this invention. It is a figure explaining the hue determination process performed in the reduction process performed in the image processing of this invention. It is the flowchart (the 1) explaining the sequence of the image correction process based on the average method to which the false color degree as an image process of this invention is applied. It is a flowchart (the 2) explaining the sequence of the image correction process based on the average method to which the false color degree as image processing of this invention is applied. It is a figure explaining the setting structure of the search area | region of a white-out pixel. It is a figure explaining the calculation process of the distance applied to the setting structure of the search area | region of a whiteout pixel, and whiteout distance evaluation value _Pd calculation. Chroma evaluation value P _s of the target _pixel, the hue evaluation value P _theta, is a diagram illustrating an example of how to determine the necessity parameters for the calculation of the. Is a diagram illustrating a suitable example of setting up saturation S _max to be applied to the calculation of the chroma evaluation value P _s. It is a flowchart explaining the calculation process procedure of the correction | amendment pixel value which applied the false color degree performed in the image processing of this invention. It is a figure explaining the example of a setting process of the interpolation value calculation range applied in the case of the calculation process of the correction pixel value to which the false color degree is applied. It is a figure which shows the flowchart explaining the procedure of an expansion process. It is a flowchart explaining the sequence of the image correction process based on the hole-filling method as an image process of this invention. It is a flowchart explaining the detailed sequence of the overexposure area | region detection process performed in the image processing based on a hole-filling method. It is a flowchart explaining the detailed sequence of the false color pixel detection area | region setting process performed in the image processing based on a hole-filling method. It is a figure explaining the process example of the false color pixel detection area | region setting process performed in the image process based on a hole-filling method. It is a flowchart explaining the detailed sequence of the false color pixel detection process performed in the image processing based on a hole-filling method. It is a figure explaining the process example of the false color pixel detection process performed in the image process based on a hole-filling method. It is a figure explaining the process example of the false color pixel correction process performed in the image process based on a hole-filling method. It is a flowchart explaining the detailed sequence of the false color pixel correction process performed in the image processing based on a hole-filling method. It is a flowchart explaining the detailed sequence of the false color pixel correction process performed in the image processing based on a hole-filling method. It is a flowchart explaining the detailed sequence of the color blurring process performed in the image processing based on a hole-filling method. It is a block diagram explaining the functional structure of the digital signal processing part which performs false color correction in the image processing apparatus of this invention. It is a block diagram explaining the functional structure of the hole-filling method process part of the digital signal process part which performs false color correction in the image processing apparatus of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Lens 102 Diaphragm 103 Solid-state image sensor 104 Correlated double sampling circuit 105 A / D converter 106 DSP block 107 Timing generator 108 D / A converter 109 Video encoder 110 Video monitor 111 Codec (CODEC)
112 memory 113 CPU
114 Input Device 115 Flash Controller 116 Flash Light Emitting Device 150 Original Image 151 Corresponding Image Area 160 Reduced Image 161 Pixel 180 Origin 181 and 182 Hue Line 183 Hue Range 210 Image Data 211 Pixel 212 and 213 Pixel Area 222 and 223 Pixel Area 230 Pixel Region 250 Image data center pixel 300 Pixel of interest 301 White search range 302,303 Pixels 311, 312 Pixel region 321-324 Pixel 400 Origin 401 Purple fringe hue line 402 Pixel of interest hue 451-454 Image range 510 Image data 511 Pixel 520 White Skip search range 521, 522 Pixel area 531 White skip pixel 601 White skip pixel 602 False color pixel detection area 603 White skip pixel 651 White skip pixel 652 False (Purple fringe) pixels 653 other pixels 654 interpolated false color (purple fringe) pixels 700 reduction processing unit 710 average method processing unit 711 overexposure distance evaluation value _{P d} calculator 712 chroma evaluation value _{P s} calculator 713 hue evaluation Value P _θ calculation unit 714 False color (PF) degree p calculation unit 715 Interpolated value [C _tmp / W _total ] calculation unit 716 First correction pixel value calculation unit 717 First enlargement processing unit 720 Filling method processing unit 721 White spot detection Unit 722 false color pixel detection region setting unit 723 false color (purple fringe) detection unit 724 pixel value correction unit 725 false color (purple fringe) filling interpolation unit 726 false color (purple fringe) blurring processing unit 727 second enlargement processing unit 740 Blend processing unit 751 Input image frame memory 752 Output image frame memory

Claims (29)

An image processing device,
A reduction processing unit that executes a reduction process of the correction target image data;
An image correction processing unit that performs image processing including pixel value correction of a false color pixel around a whiteout on the reduced image generated in the reduction processing unit;
An enlargement processing unit for executing an enlargement process of the corrected image generated in the image correction processing unit;
An image processing apparatus comprising: The reduction processing unit
The luminance information reduction process and the color information reduction process are separately performed on the original correction process target image data. In the luminance information reduction process, the overexposed pixels included in the correction process target image data are processed. The reduction processing that does not lose information is executed, and in the reduction processing of color information, the reduction processing that does not lose the false color color information included in the correction processing target image data is executed. The image processing apparatus according to 1. The reduction processing unit
In the reduction process of the luminance information for the correction process target image data, a process for executing the process of setting the brightness corresponding to the constituent pixels of the reduced image as the maximum brightness included in the corresponding image area of the correction process target image data. The image processing apparatus according to claim 2, wherein the image processing apparatus is provided. The reduction processing unit
In the color information reduction processing for the correction processing target image data, when the color information corresponding to the constituent pixels of the reduced image is included in the corresponding image area of the correction processing target image data, the corresponding image Set as an average value of false color pixels included in the area, and if the corresponding image area of the correction processing target image data does not include false color pixels, set as an average value of all pixels included in the corresponding image area The image processing apparatus according to claim 2, wherein the image processing apparatus is configured to execute processing. The image correction processing unit
A configuration for executing a correction process based on an average method of calculating a false color degree p of a constituent pixel of a reduced image generated in the reduction processing unit and executing a pixel value correction according to the calculated false color degree p;
A false color degree calculation unit for calculating a false color degree p as a parameter indicating the possibility of being a false color for a target pixel selected from the constituent pixels of the reduced image;
An interpolation value calculation unit that calculates an interpolation value corresponding to the target pixel based on the pixel value of the target pixel neighboring region;
Applying the original pixel value of the target pixel and the interpolation value, and calculating the corrected pixel value of the target pixel by changing the contribution ratio between the original pixel value and the interpolation value according to the false color degree p of the target pixel A correction pixel value calculation unit to perform,
The image processing apparatus according to claim 1, wherein the image processing apparatus includes: The false color degree calculation unit
About the pixel of interest
(A) White jump distance evaluation value P _d indicating the proximity to white jump (high luminance pixel),
(B) a saturation evaluation value P _s indicating the magnitude of saturation;
(C) Hue evaluation value P _θ indicating the proximity to the specific hue corresponding to the false color,
The image processing apparatus according to claim 5, wherein the image processing apparatus is configured to calculate a false color degree p of the target pixel by applying at least one of the following. The image correction processing unit
Correction processing based on a hole filling method that detects overexposed pixels from the constituent pixels of the reduced image generated in the reduction processing unit and corrects the pixel values of false color pixels around the overexposed pixels based on the pixel values of the surrounding pixels. Configuration to run,
A whiteout detector for detecting whiteout pixels from image data;
A false color pixel detection region setting unit that sets a false color pixel detection region around the whiteout pixels detected by the whiteout detection unit;
In the region set by the false color pixel detection region setting unit, a false color detection unit that identifies a pixel having a color corresponding to the false color as a false color pixel;
A pixel value correction unit that performs pixel value correction processing based on surrounding pixel values for the false color pixels detected by the false color detection unit;
The image processing apparatus according to claim 1, wherein the image processing apparatus includes: The pixel value correction unit
For a false color pixel, a hole-filling interpolation unit that performs hole-filling interpolation processing based on surrounding pixel values;
The image processing apparatus according to claim 7, further comprising: a color blur processing unit that performs color blur processing on the false color pixel. The pixel value correction unit
8. The configuration according to claim 7, wherein a pixel other than the false color pixel and the skipped pixel is selected from pixels existing around the false color pixel, and the filling interpolation processing based on the selected pixel is executed. Image processing device. The enlargement processing unit
The image processing apparatus according to claim 1, wherein the enlarged corrected image is generated by blending the enlarged data of the corrected image generated in the image correction processing unit and the original correction target image data. . The image correction processing unit
A first image correction processing unit that performs image processing including pixel value correction of overexposed false color pixels in a first image correction processing mode on the reduced image generated in the reduction processing unit;
The reduced image generated in the reduction processing unit is subjected to image processing including pixel value correction of a surrounding false color pixel in a second image correction processing mode different from the first image correction processing mode. A second image correction processing unit
The enlargement processing unit is configured to execute enlargement processing of the first correction image generated in the first image correction processing unit and the second correction image generated in the second image correction processing unit,
2. The image according to claim 1, further comprising a blend processing unit that generates a final corrected image by blending the enlarged first corrected image and the enlarged second corrected image enlarged in the enlargement processing unit. Processing equipment. The enlargement processing unit
Generating an enlarged first corrected image by blending the enlarged data of the first corrected image generated in the first image correction processing unit and the original correction target image data;
12. The configuration of generating an enlarged second corrected image by blending the enlarged data of the second correction image generated in the second image correction processing unit and the original correction processing target image data. An image processing apparatus according to 1. The blend processing unit
The final correction image is generated by blending the enlarged first corrected image and the enlarged second corrected image enlarged in the enlargement processing unit,
The pixel value of the constituent pixel (x, y) of the enlarged first corrected image is C _avgnew (x, y),
When the pixel value of the constituent pixel (x, y) of the enlarged second corrected image is C _holenew (x, y), the pixel value C _final (x, y) of the constituent pixel (x, y) of the final corrected image. ),
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
However, k = 0-1
The image processing apparatus according to claim 11, wherein the image processing apparatus is configured to execute a process to be calculated according to the above. The false color is purple fringe,
2. The image correction processing unit is configured to perform image processing including pixel value correction of overexposed purple fringe pixels on the reduced image generated by the reduction processing unit. 14. The image processing device according to any one of 13 to 13. An image processing method,
A reduction processing step for executing the reduction processing of the correction target image data;
An image correction processing step for performing image processing including pixel value correction of a false color pixel around the image on the reduced image generated in the reduction processing step;
An enlargement process step for executing an enlargement process of the corrected image generated in the image correction process step;
An image processing method comprising: The reduction processing step includes
Luminance information reduction processing and color information reduction processing are separately performed on original correction processing target image data. In the luminance information reduction processing, overexposed pixel information included in the correction processing target image data is lost. 16. The image processing according to claim 15, wherein a reduction process that does not cause the false color information included in the correction processing target image data is performed in the color information reduction process. Method. The reduction processing step includes
In the luminance information reduction processing for the correction processing target image data, executing processing for setting the luminance corresponding to the constituent pixels of the reduced image as the maximum luminance included in the corresponding image area of the correction processing target image data. The image processing method according to claim 16. The reduction processing step includes
In the color information reduction processing for the correction processing target image data, when the color information corresponding to the constituent pixels of the reduced image is included in the corresponding image area of the correction processing target image data, the corresponding image Set as an average value of false color pixels included in the area, and if the corresponding image area of the correction processing target image data does not include false color pixels, set as an average value of all pixels included in the corresponding image area The image processing method according to claim 16, wherein the process is executed. The image correction processing step includes
Calculating a false color degree p of the constituent pixels of the reduced image generated in the reduction process step, and executing a correction process based on an average method for executing a pixel value correction according to the calculated false color degree p;
A false color degree calculating step for calculating a false color degree p as a parameter indicating the possibility of a false color for a target pixel selected from the constituent pixels of the reduced image;
An interpolation value calculating step for calculating an interpolation value corresponding to the target pixel based on the pixel value of the target pixel neighboring area;
Applying the original pixel value of the target pixel and the interpolation value, and calculating the corrected pixel value of the target pixel by changing the contribution ratio between the original pixel value and the interpolation value according to the false color degree p of the target pixel A correction pixel value calculating step,
The image processing method according to claim 15, further comprising: The false color degree calculating step includes:
About the pixel of interest
(A) White jump distance evaluation value P _d indicating the proximity to white jump (high luminance pixel),
(B) a saturation evaluation value P _s indicating the magnitude of saturation;
(C) Hue evaluation value P _θ indicating the proximity to the specific hue corresponding to the false color,
The image processing method according to claim 19, wherein the method is a step of calculating a false color degree p of the target pixel by applying at least one of the following. The image correction processing step includes
Correction processing based on a hole filling method that detects overexposed pixels from the constituent pixels of the reduced image generated in the reduction processing step and corrects the pixel values of false color pixels around the overexposed pixels based on the pixel values of the surrounding pixels. The steps to perform,
A whiteout detection step for detecting whiteout pixels from the image data;
A false color pixel detection area setting step for setting a false color pixel detection area around a white spot pixel detected by the white spot detection step;
In the region set by the false color pixel detection region setting step, a false color detection step for specifying a pixel having a color corresponding to the false color as a false color pixel;
A pixel value correction step for executing a pixel value correction process based on surrounding pixel values for the false color pixels detected by the false color detection step;
The image processing method according to claim 15, further comprising: The pixel value correcting step includes:
For false color pixels, a filling interpolation step for performing filling interpolation processing based on surrounding pixel values;
The image processing method according to claim 21, further comprising: a color blur processing step of executing a color blur process for a false color pixel. The pixel value correcting step includes:
The method according to claim 21, further comprising: selecting other pixels excluding the false color pixel and the skipped pixel from the pixels existing around the false color pixel, and executing a filling interpolation process based on the selected pixel. Image processing method. The enlargement processing step includes
16. The image processing method according to claim 15, wherein the enlarged corrected image is generated by blending the enlarged data of the corrected image generated in the image correction processing step and the original correction target image data. . The image correction processing step includes
A first image correction processing step of executing image processing including pixel value correction of a false color pixel in a surrounding area in a first image correction processing mode on the reduced image generated in the reduction processing step;
The reduced image generated in the reduction processing step is subjected to image processing including pixel value correction of the surrounding false color pixels in a second image correction processing mode different from the first image correction processing mode. A second image correction processing step,
The enlargement processing step is a step of executing enlargement processing of the first correction image generated in the first image correction processing step and the second correction image generated in the second image correction processing step,
16. The image according to claim 15, further comprising a blending step for generating a final corrected image by blending the enlarged first corrected image and the enlarged second corrected image enlarged in the enlarging step. Processing method. The enlargement processing step includes
Generating an enlarged first corrected image by blending the enlarged data of the first corrected image generated in the first image correction processing step and the original correction target image data;
Generating an enlarged second corrected image by blending the enlarged data of the second correction image generated in the second image correction processing step and the original correction processing target image data;
The image processing method according to claim 25, further comprising: The blending step includes
A step of generating a final correction image by blending the enlarged first correction image and the enlarged second correction image enlarged in the enlargement processing step;
The pixel value of the constituent pixel (x, y) of the enlarged first corrected image is C _avgnew (x, y),
When the pixel value of the constituent pixel (x, y) of the enlarged second corrected image is C _holenew (x, y), the pixel value C _final (x, y) of the constituent pixel (x, y) of the final corrected image. ),
C _final (x, y) = k × C _avgnew (x, y) + (1−k) × C _holenew (x, y)
However, k = 0-1
26. The image processing method according to claim 25, wherein a process of calculating according to the above is executed. The false color is purple fringe,
28. The image correction processing step according to any one of claims 15 to 27, wherein the image processing including pixel value correction of an overlying purple fringe pixel is performed on the reduced image generated in the reduction processing step. An image processing method according to claim 1. A computer program for executing image processing on a computer;
A reduction processing step for executing the reduction processing of the correction target image data;
An image correction processing step for performing image processing including pixel value correction of a false color pixel around the image on the reduced image generated in the reduction processing step;
An enlargement process step for executing an enlargement process of the corrected image generated in the image correction process step;
A computer program characterized by comprising:

Images