JP6115183B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing program that perform flare correction processing on a captured image.

  When photographing with a photographing optical system having a diffractive optical element (DOE), if an extremely bright object (light source) is present within the photographing angle of view, the light source image on the photographed image is displayed. Flare with color blur (color flare) is generated around. This color flare is significantly unnatural compared to the colorless flare.

  Incidentally, Patent Document 1 discloses an electronic camera that makes the color flare inconspicuous by converting the hue of the flare occurrence region to match the hue of the background and applying a smoothing filter as flare correction processing. Yes.

Japanese Patent No. 4250506

  However, some of the light sources that are the source of color flare have unique colors. If the flare correction processing described above is applied to the color flare caused by such a light source, There is a possibility that even the original color of the light source may disappear together with the color blur.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an imaging apparatus, and an image processing program capable of executing an appropriate flare correction process regardless of the flare source color.

The image processing apparatus of the present invention includes an input unit for inputting a processing target image obtained by the photographing optical system, based on the type of the photographing optical system, before Symbol flare generation region of the flare in the image to be processed a determination unit that the light source color is determined whether or not the specific color, and a flare correcting section for performing flare correction processing on the generated region, the flare correcting section depending on the determination result by the determination unit To switch the content of the flare correction process.

  The imaging device of the present invention includes a color imaging element that captures a subject image formed by a photographing optical system, and the image processing device of the present invention that processes an image generated by the imaging device as the processing target image.

Computer-executable image processing program of the present invention includes an input procedure for entering the processing target image obtained by the photographing optical system, based on the type of the photographing optical system, generation region of the flare in the processed image image a determining procedure source color before Symbol flare it is determined whether or not the particular color, and a flare correction procedure for performing flare correction processing on the generated region, in the flare correction procedure, the by the determination procedure The content of the flare correction process is switched according to the determination result.

  According to the present invention, an image processing device, an imaging device, and an image processing program capable of executing appropriate flare correction processing regardless of the flare light source color are realized.

It is a block diagram which shows the structural example of the image processing apparatus of 1st Embodiment. It is an operation | movement flowchart of a development process. It is an operation | movement flowchart of a flare correction process. It is an operation | movement flowchart of a flare color component extraction process. It is an operation | movement flowchart of a light source color determination process. It is an operation | movement flowchart of a light source color restoration process. It is an example of an input image. It is an example of the detected high-intensity area | region. It is an example of the high-intensity area | region (saturation area | region) resulting from a flare. It is an example of a labeled saturation region. The radius r _i of the saturation region S _i, is a schematic diagram showing the relationship between the radius of the flared region A _i. It is a figure explaining the relationship between saturation area | region _Si , flare area | region _Ai, and process area | region _Ai '. It is a figure explaining a binary mask image. It is a figure explaining a gray scale mask image. It is a figure which shows the relationship between R component (with a flare color component) and G component of process area _| region _Ai '. It is a schematic diagram explaining the extraction procedure of a flare color component. It is a figure which shows the relationship between R component (there is no flare color component) and G component of process area _| region _Ai '. (A) is a pixel profile (pixel profile on the center line) of a certain processing area A _i ′ before saturation enhancement, and (B) is a pixel profile of the same processing area A _i ′ after saturation enhancement. is there. It is a figure explaining the gray scale mask image before expansion. It is a figure explaining the grayscale mask image after expansion. It is a pixel profile (pixel profile on the center line) of the grayscale mask image before and after enlargement. It is a figure explaining step S99. It is a block diagram which shows the structural example of the electronic camera of 2nd Embodiment.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described.

  FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to the present embodiment. The image processing apparatus according to this embodiment includes a computer 11 in which an image processing program is installed.

  A computer 11 illustrated in FIG. 1 includes a data reading unit 12, a storage device 13, a CPU 14, a memory 15, an input / output I / F 16, and a bus 17. The data reading unit 12, the storage device 13, the CPU 14, the memory 15, and the input / output I / F 16 are connected to each other via a bus 17. Further, an input device 18 (keyboard, pointing device, etc.) and a monitor 19 are connected to the computer 11 via an input / output I / F 16. The input / output I / F 16 receives various inputs from the input device 18 and outputs display data to the monitor 19.

  The data reading unit 12 is used when reading image data to be subjected to image processing from the outside. For example, the data reading unit 12 conforms to a reading device (such as an optical disk drive, a magnetic disk drive, or a magneto-optical disk drive) that acquires image data from a storage medium inserted into the data reading unit 12 or a known communication standard. It is composed of communication devices (USB interface, LAN module, wireless LAN module, etc.) that communicate with external devices.

  The storage device 13 is configured by a storage medium such as a hard disk or a nonvolatile semiconductor memory. The storage device 13 stores an image processing program. The storage device 13 can also store image data read from the data reading unit 12.

  The CPU 14 is a processor that executes an image processing program stored in the storage device 13 and comprehensively controls each unit of the computer 11. Here, the image processing program of the present embodiment is equipped with a function of developing processing for a RAW image taken by a camera, and when executing this developing processing, the CPU 14 functions as a control unit that controls each unit. Besides, it also functions as a flare correction processing unit 21 (the operation of the flare correction processing unit 21 will be described later).

  The memory 15 temporarily stores various calculation results in the image processing program. The memory 15 is composed of, for example, a volatile SDRAM.

  FIG. 2 is an operation flowchart of the development process. Hereafter, each step of FIG. 2 is demonstrated in order.

  Step S1: The CPU 14 reads image data of a color image (RAW image) designated by the user as an object of development processing onto the memory 15 (hereinafter referred to as “input image”). This input image is an image written in the storage medium of the data reading unit 12, for example.

  Here, there is a possibility that color flare has occurred in the input image. The image in which the color flare is generated is an image taken by the photographing optical system including the DOE, for example, as shown in FIG. 7 (however, since FIG. 7 is a monochrome image, the color flare is not colored. ). The color flare usually has a zonal pattern or a pattern close to it, but the pattern does not always form a complete zonal pattern, and the shooting conditions (imaging characteristics of the photographic optical system used to shoot the image, It depends on the wavelength spectrum of the light source existing within the angle of view. The color flare distribution (flare color component) also depends on the shooting conditions.

  However, since the flare correction process of the present embodiment can be executed even if the color flare pattern and the distribution with color are unknown, detailed imaging condition data may not be attached to the input image. Therefore, in this embodiment, it is sufficient if the type information of the photographing optical system used for photographing is attached to the input image.

  Step S2: The CPU 14 performs development processing other than flare correction processing on the input image on the memory 15. Here, for simplicity, it is assumed that development processing other than flare correction processing is only color interpolation processing and saturation enhancement processing (a type of color correction processing). As a result, the input image on the memory 15 is in a state after color interpolation and after saturation enhancement.

  Step S3: The CPU 14 determines whether or not DOE is included in the photographing optical system based on the type information of the photographing optical system attached to the input image. Since it is considered that there is no possibility of occurrence of color flare, the process proceeds to step S10, and when DOE is included in the photographing optical system, it is assumed that there is a possibility of occurrence of color flare in the input image. To step S4.

  Step S4: The CPU 14 creates a reduced size version (reduced image) of the input image. The reason why the reduced image is created is to reduce the calculation load of the subsequent steps. The size ratio of the reduced image to the input image is 1/4, 1/8, or the like. Incidentally, if the size ratio is made smaller than 1/8, a small light source may not be detected, so the size ratio is preferably 1/8 or more.

  Step S5: The CPU 14 compares the brightness of each pixel of the reduced image with a predetermined value (here, a saturation level), thereby reducing a bright area (high brightness area) where the luminance exceeds the predetermined value. (See FIG. 8). As shown in FIG. 8, there is a possibility that not only the high brightness area caused by flare but also the high brightness area unrelated to flare are mixed in the detected high brightness area.

  In this step, in order to reduce the calculation load, instead of using the luminance value (R, G, B weighted sum) of each pixel as an index of the luminance of each pixel, the sum of each color component (R , G, B) may be used. Alternatively, a color component value (G intensity) representing each pixel may be used as an index of luminance of each pixel. Further, such a method of reducing the calculation load can be employed in other steps for handling the luminance of the pixel.

  Step S6: The CPU 14 determines whether or not each of the high luminance areas detected in step S5 is caused by flare, and excludes those other than those caused by flare (see FIG. 9).

  For example, the determination as to whether or not each high-luminance region is caused by flare is performed as follows. That is, the CPU 14 compares the high luminance area with a circular pattern prepared in advance, and if the correlation between the two is greater than or equal to a certain level, the CPU 14 regards the high luminance area as being caused by flare and the correlation is less than a certain level If it is, the high brightness area is not considered to be caused by flare. As a circular pattern to be compared, an average pattern may be used as a high luminance area caused by flare.

  Alternatively, the determination as to whether or not each high brightness area is caused by flare may be performed as follows. That is, the CPU 14 sets a circular area having a predetermined radius from the center of the high luminance area, and when there is a high luminance area in the entire circular area, the high luminance area is caused by the flare. If at least a part of the circular area is out of the high luminance area, the high luminance area is not attributed to flare. Note that the radius of the circular region is set to be approximately the same as the radius of the smallest high-luminance region caused by flare.

  Therefore, the high luminance area that is not excluded in this step corresponds to a saturated area in the center of the flare. Hereinafter, the high luminance area that is not excluded in this step is simply referred to as “saturated area”.

  Step S7: The CPU 14 labels the one or more saturated regions detected in step S6 and assigns serial numbers (flare numbers i = 1, 2,...) (See FIG. 10). As a result, the total number of flares occurring in the input image (that is, the final value of the flare number i) is determined.

  Step S8: If the total number of flares is zero, the CPU 14 considers that the flare correction process is unnecessary, and proceeds to step S10. If the total number of flares is not zero, the CPU 14 should execute the flare correction process. The process proceeds to step S9.

  Step S9: The CPU 14 performs flare correction processing on the input image on the memory 15. Details of the flare correction process will be described later.

  Step S10: The CPU 14 compresses the input image on the memory 15 by a predetermined compression method such as JPEG, and then saves the input image in a storage destination designated by the user (for example, a storage medium inserted in the data reading unit 12). Then, the flow of the development process is terminated.

  FIG. 3 is an operation flowchart of the flare correction process. Hereafter, each step of FIG. 3 is demonstrated in order.

  Step S91: The flare correction processing unit 21 sets the flare number i to an initial value (1).

Step S92: When the flare correction processing unit 21 pays attention to the saturation region S _i corresponding to the flare number i on the reduced image, the flare correction processing unit 21 sets the processing region A _i ′ including the saturation region S _i on the reduced image. This processing area A _i ′ needs to include a flare generation area (flare area) A _i centered on the saturation area S _i .

Here, as shown in FIG. 11, the larger the saturation region S _i , the larger the flare region A _i . However, when the radius _{r i} of the saturation region _{S i} is equal to or less than the threshold a (FIG. 11 (a), (b) , (c)) is the radius of the flared region _{A i} is a predetermined value beta, the saturation region If the radius _{r i} of exceeds the threshold a (FIG. 11 (d), (e) ) , the radius of the flare area _{a i} is the value alpha × _{r i} in accordance with the radius _{r i.} This is because, when the radius r _i of the saturation region S _i is equal to or less than the threshold a, the brightness of the light source is less than a predetermined low-order diffracted light only with respect to not occur, the radius r _i of the saturation region S _i This is because the brightness of the light source is not less than a certain value and the high-order diffracted light is generated in addition to the low-order diffracted light.

Therefore, the flare correction processing unit 21 in this step sets the processing area A _{i according} to the following procedures (A) to (E).

(A) The flare correction processing unit 21 calculates a coordinate g _i corresponding to the luminance center of gravity of the saturation region S _i on the reduced image, and calculates an average distance from the coordinate g _i to the contour of the saturation region S _i as the saturation region S. _i is calculated as the radius _{r i} of.

(B) The flare correction processing unit 21 determines whether or not the radius r _i of the saturation region S _i is equal to or less than the threshold value a. If it is equal to or less than the threshold value a, the procedure proceeds to procedure (C), and if not, the procedure proceeds to procedure (D).

(C) As shown in FIG. 12A, the flare correction processing unit 21 defines a circular area centered on the coordinate g _i on the reduced image and having a radius β as the flare area corresponding to the flare number i. Set to A _i (where β> r _i ) and proceed to step (E).

(D) As shown in FIG. 12B, the flare correction processing unit 21 corresponds to the flare number i in the circular area centered on the coordinate g _i on the reduced image and having the radius (α × r _i ). The flare area A _i is set (where α> 1).

(E) As shown in FIG. 12A or FIG. 12B, the flare correction processing unit 21 includes the entire flare area A _i on the reduced image and an included area having a certain margin γ. , The processing area A _i ′ corresponding to the flare number i is set (where γ> 0).

In the present step, _'although the size of the set according to the size of the flare area A _i, the processing area A _i' processing area A _i is as long as it can reliably include the flare region A _i, the processing area A _i The size of 'may be a predetermined size (a sufficiently large size). Further, the shape of the processing area A _i ′ may be circular or rectangular. Further, the center of the processing area A _i ′ may be deviated from the center of the flare area A _i .

Further, in this step, but the luminance center of the saturation region S _i and a coordinate g _i, in order to reduce the computational load, the center of the saturation region S _i may coordinate g _i.

Further, in this step has been calculated radius r _i of the saturation region S _i as the size of the saturation region S _i, may be used the number of pixels saturation region S _i as the size of the saturation region S _i. In that case, a threshold value a ′ for the number of pixels is used instead of the threshold value a for the radius.

Further, since the relationship between the size of the saturation region S _{i and} the size of the flare region A _i (see FIG. 11) differs depending on the type of DOE (that is, the type of the photographing optical system), the flare correction processing unit 21 in this step is It is desirable to set the size threshold according to the type of the photographing optical system.

Step S93: flare correction processing unit 21 _'performs flare color component extraction process on the processing area A _i' processing area A _i corresponding to the flare number i to extract the flare color component occurring in. The flare color component extracted in this step is composed of two flare color components. Details of the flare color component extraction processing will be described later.

  Step S94: The flare correction processing unit 21 performs size enlargement processing on the flare color components (two flare color components) extracted in step S93. If the size ratio in step S4 is 1 / M, the size of flare color components (flare color components for two colors) is increased M times in the size enlargement process in this step. As a result, flare color components (two flare color components) having a size corresponding to the input image are acquired.

Step S95: The flare correction processing unit 21 subtracts the flare color component after the size enlargement (two flare color components) from the region corresponding to the processing region A _i ′ on the input image. However, in the subtraction, the CPU 14 obtains the coordinate g _i ′ on the input image corresponding to the coordinate g _i based on the coordinate g _i on the reduced image and the magnification M described above, and the flare color with respect to the coordinate g _i ′. The centers of the components (two flare color components) are made to coincide. As a result, the flare color component of the color flare generated in the area corresponding to the processing area A _i ′ on the input image is removed. That is, the color flare generated in the area is a colorless flare.

Step S96: The flare correction processing unit 21 performs a light source color determination process on the flare area A _i on the reduced image, and determines whether or not the light source color of the flare generated in the flare area A _i is white. To do. Details of the light source color determination processing will be described later. Here, “white” means a color that looks white when viewed from the human eye, and includes not only the color of a so-called white light source (wavelength spectrum is flat) but also a color close thereto.

  Step S97: The flare correction processing unit 21 determines whether or not the light source color determined in Step S96 is white. If the light source color is non-white, the flare correction processing unit 21 proceeds to Step S98 to execute the light source color restoration process, If YES, the process proceeds to step S911 to skip the light source color restoration process.

Step S98: The flare correction processing unit 21 acquires a restored processing area A _i ′ (= restored image imgC) by executing a light source color restoration process for the processing area A _i ′ on the reduced image. Details of the light source color restoration processing will be described later.

  Step S99: The flare correction processing unit 21 creates a size-enlarged version (= restored image IMGC) of the restored image imgC. Details of the method of creating the restored image IMGC will be described later.

Step S910: The flare correction processing unit 21 restores the color of the light source in the area by replacing the area corresponding to the processing area A _i ′ on the input image with the restored image IMGC.

  Step S911: The flare correction processing unit 21 determines whether or not the flare number i has reached the final value. If not, the process proceeds to step S912. If it has not, the flare correction processing flow ends. To do.

  Step S912: The flare correction processing unit 21 increments the flare number i, and then proceeds to step S92.

  FIG. 4 is a flowchart of the flare color component extraction process. Hereinafter, each step will be described in order.

Step S931: The flare correction processing unit 21 calculates the average value of the flare area A _i corresponding to the flare number i for each color component (for each RGB), and determines one color component having the minimum average value as the reference color. Determined by ingredients. Hereinafter, it is assumed that the G component is set as the reference color component. In this case, the flare color component is estimated for each color component other than the G component, that is, each of the R component and the B component.

Step S932: The flare correction processing unit 21 creates a binary mask image corresponding to the flare number i. As shown in FIG. 13, the binary mask image is a mask image having the same shape and size as the processing area A _i ′, and an opening is provided in the area corresponding to the flare area A _i . Further, the flare correction processing unit 21 creates a gray scale mask image as shown in FIG. 14 by spatially smoothing the binary mask image with a smoothing filter such as a Gaussian filter. This gray scale mask image corresponds to a binary mask image in which the contrast of the boundary between the opening and the non-opening is intentionally reduced. The average value per unit area of the gray scale mask image decreases from the center to the periphery of the processing area A _i ′.

Note that the size of the smoothing filter (filter diameter) used in this step is desirably set larger as the flare area A _i is larger.

Step S933: The flare correction processing unit 21 sets one color component other than the reference color component (here, the G component) as the target color component. Hereinafter, it is assumed that the R component is set as the target color component. FIG. 15 is a diagram (pixel profile) showing the relationship between the R component of a certain processing area A _i ′ and the G component of the same processing area A _i ′. As shown in FIG. 15, in the R component, a flare color component is superimposed on a region inside the flare region A _i and outside the saturation region S _i . That is, the R component in that region is significantly higher than the G component in the same region.

Flare correction processing section 21 of this step, the processing area A _i by dividing each pixel in the G component of the _'R component processing area A _{i of'} correlated color of R component based on the G component distribution image I Create a _GR . A schematic diagram of the correlated color distribution image _IGR is shown on the left side of FIG.

Here, many pixels in the central region a of the correlated color distribution image _IGR have pixel values close to 1, and many pixels in the ring-shaped region b located around the central region a are more than 1. And many pixels in the outermost peripheral region c located outside the ring-shaped region b have lower pixel values than the ring-shaped region b. Among these, the central region a corresponds to the saturation region S _i shown in FIG. 15, and the ring-shaped region b is a region inside the flare region A _i and outside the saturation region S _i shown in FIG. That is, it corresponds to a region where flare color components are superimposed.

Step S934: The flare correction processing unit 21 smoothes only the regions (the central region a and the most peripheral region c) whose values are lower than those of the surroundings in the correlated color distribution image _IGR . "acquired. in addition, the correlated color distribution image _{I GR"} correlated color distribution image _{I GR} as shown in) acquisition of it can be carried out according to the procedure shown in FIG. 16 (a) → FIG 16 (B).

That is, the flare correction processing unit 21 first spatially smoothes the entire correlated color distribution image _IGR shown on the left side of FIG. 16A with a smoothing filter such as a Gaussian filter. A correlated color distribution image I _GR ′ as shown on the right side of (A) is acquired. According to this smoothing, the value of the region having a low value compared to the surroundings is higher than that before the smoothing, and the value of the region having a high value compared to the surroundings is lower than that before the smoothing. Subsequently, the flare correction processing unit 21 compares the two correlated color distribution images I _GR and I _GR ′ before and after smoothing for each pixel, and collects the pixels with the higher value to create one composite image. Then, the synthesized image is set as a correlated color distribution image I _GR ″ (FIG. 16B). This correlated color distribution image I _GR ″ has a flare color component superimposed on the original correlated color distribution image I _GR . This corresponds to a smoothed area (dotted line frame portion) that did not exist.

Step S1835: flare correction processing section 21 _"by performing contraction processing by the minimum value filter or the like to the correlated color distribution image I _GR" correlated color distribution image I _GR values of a region of high compared to around the the area contracts, acquires the correlated color distribution image I _H as shown in FIG. 16 (C). Furthermore, the flare correction processing unit 21 spatially smoothes the correlated color distribution image I _H with a smoothing filter such as a Gaussian filter, thereby generating a correlated color distribution image I _H ′ as shown in FIG. get. This correlated color distribution image I _H ′ corresponds to an image obtained by interpolating the area where the flare color component is superimposed in the original correlated color distribution image I _GR with the surrounding area (dotted line frame portion).

Step S936: The flare correction processing unit 21 removes the flare color component by multiplying the reference color component (G component in this case) of the processing region A _i ′ by the correlated color distribution image I _H ′ for each pixel. R component I _Rnew is calculated. As a result, the flare color component related to the R component is removed from the processing area A _i ′. FIG. 17 is a pixel profile of the processing area A _i ′ after removal.

As apparent from a comparison of FIG. 17 with FIG. 15, after removal, the R intensity of the region where the flare color component is superimposed (the region inside the flare region A _i and outside the saturation region S _i ) is the same. It can be seen that it is close to the G intensity of the region. After the removal, the R / G intensity of the area where the flare color component is superimposed is determined by the R / G intensity of the area outside the flare area A _i and the R / G intensity inside the saturation area S _i . You can see that it is interpolated. Further, after the removal, it can be seen that the R intensity distribution shape of the region where the flare color component is superimposed is a shape corresponding to the G intensity distribution shape of the same region.

The flare correction processing unit 21 in this step calculates the flare color component I _FR related to the R component by subtracting the R component I _Rnew from which the flare color component has been removed from the original R component (flare color). The R component I _Rnew from which the component has been removed is converted into a flare color component I _FR .) This is because in this embodiment, it is necessary to perform some processing (mask processing, size enlargement processing, etc.) on the flare color component I _FR before calculating the final image.

Step S937: The flare correction processing unit 21 blurs the peripheral portion of the flare color component I _FR by applying the grayscale mask created in Step S932 to the flare color component I _FR acquired in Step S936. If blurring in this way the peripheral portion, _'upon removal of the flare color component I _FR from R components (subtraction) processing area A _i' processing area A _i an unnatural gradation level difference is generated in the It is because it can prevent. Further, as described above, the size of the smoothing filter used at the time of creating the grayscale mask is set to a size according to the size of the flare area A _i , so that the peripheral portion of the flare color component I _FR after applying the mask Will be more natural. Thereby, the estimation of the flare color component I _FR regarding the R component is completed.

  Step S938: The flare correction processing unit 21 determines whether or not the estimation for all the color components (here, the R component and the B component) other than the reference color component has been completed. The process proceeds to step S933 to estimate the components, and if completed, the flow of the flare color component extraction process is terminated. Here, only the estimation regarding the R component has been described, but the estimation regarding the B component is the same as the estimation regarding the R component, and thus the description thereof is omitted.

  FIG. 5 is a flowchart of the light source color determination process. Hereinafter, each step will be described in order.

Step S961: The flare correction processing unit 21 pays attention to the flare area A _i on the reduced image, and refers to the ring-shaped area (hatched portion in FIG. 11) formed by excluding the saturation area S _i from the flare area A _i . Set to region R _i .

Step S962: If the flare correction processing section 21, the average color of the reference area R _i is determined whether or not the achromatic (or achromatic equivalent), non-achromatic (or achromatic equivalent), the step S963 If it is an achromatic color (or an achromatic color equivalent), the process proceeds to step S966.

Step S963: The flare correction processing unit 21 determines whether or not the average color of the reference region R _i is a specific flare color of the photographing optical system depending on whether or not it falls within a predetermined threshold range. If there is, the process proceeds to step S964, and if it is not the intrinsic flare color, the process proceeds to step S963. Hereinafter, the specific flare color will be described in detail.

  Among photographing optical systems, there are those that easily generate red flare, those that easily generate blue flare, and those that easily generate purple flare. The color of flare that occurs most frequently is called the “specific flare color” of the photographing optical system. This inherent flare color is determined by the type of DOE included in the photographing optical system (that is, the kind of photographing optical system). This is because the response characteristic of the DOE to white light is set so that the wavelength spectrum of the flare generated on the captured image has a peak (high intensity) in a predetermined wavelength range. Alternatively, the response characteristic of the DOE to white light is set so that the wavelength spectrum of the flare generated on the captured image easily has a peak (high intensity) in a predetermined wavelength range.

  Since the inherent flare color varies depending on the type of the photographing optical system, the flare correction processing unit 21 in this step sets the threshold range of the inherent flare color according to the type of the photographing optical system.

Step S964: The flare correction processing unit 21 determines whether or not the saturation of the average color of the reference region R _i is equal to or greater than the threshold value. If the saturation is equal to or greater than the threshold value, the process proceeds to step S965. Proceeds to step S966. Note that the flare correction processing unit 21 in this step sets the saturation threshold value in this step according to the intensity of the saturation enhancement processing in step S2.

Note that the flare correction processing unit 21 in this step does not determine whether the average color saturation of the reference region R _i is equal to or greater than the threshold value, but the color-corrected pixel (color correction pixel) is the reference region R. _It may be determined whether or not it exists in _i . Hereinafter, the color correction pixel will be described.

18A shows a pixel profile (pixel profile on the center line) of a certain processing area A _i ′ before saturation enhancement, and FIG. 18B shows the same processing area A _i ′ after saturation enhancement. This is a pixel profile. The saturation enhancement applied to the processing area A _i ′ is saturation enhancement for enhancing the G component and the B component relative to the R component.

As shown in FIG. 18A, in the processing area A _i ′ before saturation enhancement, there are only pixels in which all color components are equal to or higher than the noise level T1, whereas FIG. As described above, pixels in which the R component is less than the noise level T1 are generated in the processing region A _i ′ after saturation enhancement. In addition, in at least some of the pixels, at least one of the G component and the B component reaches the saturation level T2. Thus, a pixel in which at least one color component has reached the saturation level T2 and the other at least one color component is less than the noise level T1 is a color correction pixel. This kind of color correction pixel is generated, a reference area R _i.

Step S965: The flare correction processing unit 21 determines that the light source color of the flare generated in the processing area A _i ′ is non-white, and ends the flow of the light source color determination process.

Step S966: The flare correction processing unit 21 determines that the light source color of the flare generated in the processing area A _i ′ is white, and ends the light source color determination processing flow.

  FIG. 6 is a flowchart of the light source color restoration process. Hereinafter, each step will be described in order.

  Step S981: The flare correction processing unit 21 acquires the same grayscale mask image (see FIG. 19) as that created in Step S932.

Next, the flare correction processing unit 21 multiplies the grayscale mask image by a positive predetermined value to improve the pixel value of the grayscale mask image. However, the flare correction processing unit 21 replaces the pixel value exceeding 1 by multiplication with 1, thereby keeping the pixel value of the grayscale mask image within the range of 0 to 1. As a result, the area of the opening of the gray scale mask image is enlarged (see FIG. 20). The profile (aperture ratio distribution) of the grayscale mask image before enlargement is as shown by the dotted line in FIG. 21, and the profile (aperture ratio distribution) of the grayscale mask image after enlargement is as shown by the solid line in FIG. It is. As shown in FIG. 21, the openings of the gray scale mask image after enlargement is substantially coincident with the flare region A _i. Hereinafter, the enlarged grayscale mask image is referred to as “grayscale mask image imgF”. This grayscale mask image imgF is used as a weight distribution of weighting synthesis (step S984) described later.

Step S982: The flare correction processing unit 21 prepares a reduced image imgA before flare removal. The reduced image imgA before the flare removal is a processing area A _i ′ on the reduced image.

  Step S983: The flare correction processing unit 21 prepares a reduced image imgB after flare removal by subtracting the flare color component before size enlargement from the reduced image imgA before flare removal (FIG. 22A). The flare color component before size expansion is the flare color component acquired in step S93.

  Step S984: The flare correction processing unit 21 obtains a restored image imgC in which the color of the light source is restored by weighted synthesis of the reduced image imgA before flare removal and the reduced image imgB after flare removal (see FIG. 22 (b)), the flow of the light source color restoration process is terminated.

  Here, the weight of the weighted synthesis in this step is set for each position (for each pixel) on the image. The weight distribution of the reduced image imgA is set to be the same as the pixel value distribution of the grayscale mask image imgF created in step S981. That is, the weighted synthesis in this step is performed by the following equation.

  In Expression (1), (x, y) is pixel coordinates, and r, g, and b are R, G, and B components of the pixel value.

  Therefore, in the restored image imgC acquired in this step, the central area corresponding to the flare area takes the same value as the reduced image A before flare removal, and the peripheral area corresponding to the outside of the flare area is reduced after flare removal. A value close to the image B is taken. Moreover, in the peripheral area, the reduced image B after flare removal is more strongly reflected as the distance from the central area increases.

  That is, the color of the light source is restored in the central area of the restored image imgC, and the flare coloring is removed in the peripheral area. In addition, in the peripheral area, the degree of removal increases as the distance from the central area increases.

  However, since the size of the restored image imgC is small, it is necessary to create a size-enlarged version (= restored image IMGC) of the restored image imgC in the subsequent step 99.

  Hereinafter, step S99 will be described in detail. Step S99 includes the following procedures (i) to (iv).

  (I) The flare correction processing unit 21 obtains a corrected flare color component imgD by subtracting the restored image imgC from the reduced image imgA before flare removal, as shown in FIG. The corrected flare color component imgD represents a true flare color component. That is, the corrected flare color component imgD represents the flare color component obtained by removing the color component of the light source.

  (Ii) The flare correction processing unit 21 creates a corrected flare color component IMGD after size expansion by performing size expansion processing on the corrected flare color component imgD as shown in FIG. If the size ratio in step S4 is 1 / M, the size of the corrected flare color component imgD is increased M times in the size enlargement process in this step.

(Iii) The flare correction processing unit 21 prepares an unreduced image IMGA before flare removal, as shown in FIG. The unreduced image IMGA before flare removal is the same as the region corresponding to the processing region A _i ′ on the input image.

  (Iv) The flare correction processing unit 21 subtracts the corrected flare color component IMGD after size enlargement from the non-reduced image IMGA before flare removal as shown in FIG. IMGC is acquired (the description of step S99 above).

[Effect of the first embodiment]
As described above, the image processing program according to the present embodiment performs the flare based on the input procedure (S1) for inputting the color processing target image acquired by the photographing optical system and the color of the flare occurrence area (A _i ) in the processing target image. And a determination procedure (S96, S97) for determining whether or not the light source color is white, and a flare correction procedure (S93, S94, S95, S98, S99, S910) for performing a flare correction process on the generated region. In the flare correction procedure, the content of the flare correction process is switched in accordance with the result of determination by the determination procedure.

  Therefore, according to the image processing program of the present embodiment, an appropriate flare correction process can be executed regardless of whether the light source color is white.

  In the determination procedure of the present embodiment, determination is performed based on at least one of the type of the photographing optical system and the saturation of the generated region (S963, S964). Further, in the determination procedure of the present embodiment, when the processing target image is an image after color correction, the determination is based on whether or not a color correction pixel exists in the generation area instead of based on the saturation of the generation area. Is performed (S964). Therefore, the discrimination accuracy is high.

In the determination procedure of the present embodiment, a circular area including the saturated pixel area (S _i ) on the processing target image is regarded as the generation area (A _i ) (S 92). Further, the discrimination means of the present embodiment estimates the size of the generated area based on the size of the saturated pixel area (S92). Therefore, the flare generation area is detected with high accuracy.

In the determination procedure of the present embodiment, only the region (R _i ) excluding the saturated pixel region from the generation region is used for determination (S961). Therefore, the discrimination efficiency is high.

Further, the flare correction procedure of the present embodiment includes an area setting procedure (S92) for setting a processing area (A _i ′) including an occurrence area on a processing target image, and a removal process for removing flare coloring. The removal procedure (S93, S94, S95) to be applied to and the restoration process for restoring the color unique to the flare light source to the generation area when the flare light source color is non-white, the processing area after the removal process Restoration procedure (S98, S99, S910). Therefore, when the light source color is non-white, the color of the flare light source is restored.

  In addition, the restoration process of the present embodiment is a process of weighting and combining the process area before the removal process with respect to the process area after the removal process (S981, S982, S983, S984). Therefore, the color unique to the light source can be easily restored.

  In the image processing program of this embodiment, the ratio of the weight after the removal process to the weight after the removal process is set for each position of the processing region. Therefore, the restoration can be performed finely.

  In the image processing program according to the present embodiment, the ratio of the position away from the generation area is smaller than the ratio of the position close to the generation area. Therefore, it is possible to prevent an unnatural gradation step from appearing in the processing area after restoration.

  Further, the removal procedure of the present embodiment includes a reference color setting procedure (S931) for setting a reference color component as a reference among the color components in the generation region, and a reference color component between the reference color component and the target color component in the processing region. Correlation calculation procedure for calculating correlation information indicating correlation (S933), and flare color component estimation procedure for estimating the flare color component that is a colored distribution of flare based on the correlation information of the processing region and the reference color component of the generation region ( S936). Therefore, although the removal procedure of the present embodiment reliably removes the flare coloring, there is a possibility that the color of the light source of the flare is also removed. Therefore, the effect of combining the restoration procedure of this embodiment with the removal procedure of this embodiment is high.

[Supplement to the first embodiment]
In the present embodiment, the color interpolation process and the saturation enhancement process are executed as the development process other than the flare correction process, but the saturation enhancement process may be omitted. However, if you omit the saturation enhancement processing, in step S 964, it is determined whether the color corrected image (color correction pixels), instead of determining the saturation of the height of the reference region R _i, that alternate approaches Cannot be adopted.

In this embodiment, an image before development processing (RAW image) is an input image, but an image after development processing (JPEG image or the like) may be used as an input image. In this case, if color correction processing such as saturation enhancement processing has already been performed on the input image, in step S964, instead of determining the saturation level of the reference region R _i , the color corrected pixel An alternative method of determining the presence or absence of (color correction pixels) can be employed.

Further, in step (D) in step S92 described above, but those obtained by multiplying a predetermined value α to radius r _i of the saturation region S _i regarded as the radius of the flared region A _i, the radius r _i of the saturation region S _i What added the predetermined value σ may be regarded as the radius of the flare area A _i .

In addition, the flare correction processing unit 21 of the present embodiment performs the estimation of the flare color component for each of the two color components other than the reference color component, but omits the estimation for one of the two color components. May be. Assuming that the G component is the reference color component, for example, when the difference between the average value of the G component and the average value of the B component in the flare area A _i is within a predetermined range, that is, the G component and the B component. If there is no difference from the component, estimation regarding the B component may be omitted.

  Further, although the flare correction processing unit 21 of the present embodiment automatically sets the reference color component, it may be specified by the user.

  In the development processing of this embodiment, in order to reduce the calculation load, a reduced size version of the input image (reduced image) is used instead of using the input image as it is. If priority is to be given, the input image may be used as it is.

  In the development processing of the present embodiment, the CPU 14 automatically detects the flare occurrence position in the input image, but the user may designate it.

Further, in the developing process of the present embodiment, although the flare region A _i CPU 14 detects automatically, it may be specified by the user.

In the development processing of this embodiment, the processing area A _i ′ is automatically set by the CPU 14, but may be set by the user.

Also, flare correction processing unit 21 of the present embodiment, the correlation with the reference color component of the flare region A _i between the flare color component of the flare area A _i, a reference color component in the processing area A _i and target color component However, it may be estimated by another method based on the color distribution of the flare area A _i .

[Second Embodiment]
Hereinafter, a second embodiment of the present embodiment will be described.

  FIG. 23 is a block diagram illustrating a schematic configuration of the electronic camera. As shown in FIG. 23, the electronic camera 111 includes an imaging optical system 112, a lens driving unit 113, a diaphragm 114, a diaphragm driving unit 115, a color imaging element 116, an AFE 117, an image processing engine 118, and a first. It has a memory 119, a second memory 120, a media I / F 121, a communication I / F 122, a monitor 123, and a release button 124. Among these, a lens driving unit 113, an aperture driving unit 115, and an AFE 117 are provided. The first memory 119, the second memory 120, the media I / F 121, the communication I / F 122, the monitor 123, and the release button 124 are each connected to the image processing engine 118. The photographing optical system 112 is a high-functional photographing optical system including a DOE lens, for example.

  The first memory 119 is composed of a volatile storage medium (SDRAM or the like), and temporarily stores a captured image in a pre-process or post-process of image processing by the image processing engine 118. On the other hand, the second memory 120 is configured by a nonvolatile storage medium such as a flash memory, and stores a program executed by the image processing engine 118 for a long period of time.

  The first memory 119 stores an image processing program to be executed by the image processing engine 118, and the image processing program has the same development processing function as that of the first embodiment. When executing the development processing, the image processing engine 118 functions as the flare correction processing unit 125 as in the first embodiment.

  Therefore, the image processing engine 118 of the present embodiment performs the first implementation on the RAW image acquired by the electronic camera 111 by shooting or the RAW image read via the media I / F 121 (or communication I / F 122). Development processing similar to that of the embodiment can be performed.

  Note that the development processing timing for the RAW image acquired by the electronic camera 111 may be immediately after the RAW image is captured.

  In addition, a portion indicated by a dotted line in FIG. 23 (a lens unit including the photographing optical system 112) may be replaceable with the electronic camera 111.

  In addition, in the case where development processing is performed on a RAW image shot with a replaceable lens unit, the image processing engine 118 may read the type information of the shooting optical system 112 from the lens unit.

  Further, the development processing of the present embodiment can be modified similarly to the development processing of the first embodiment.

  DESCRIPTION OF SYMBOLS 11 ... Computer, 12 ... Data reading part, 13 ... Memory | storage device, 14 ... CPU, 15 ... Memory, 16 ... Input-output I / F, 17 ... Bus

Claims (19)

An input unit for inputting a processing target image obtained by the photographing optical system,
Based on the type of the photographing optical system, and a determination unit that the light source color before Symbol flare generation region of the flare in the processing target image, it is determined whether or not the specific color,
And a flare correcting section for performing flare correction processing on the generated region,
The flare correction unit is
The flare correction processing images processing device Ru switching contents according to the result of the determination by the determination unit. An input unit for inputting a processing target image acquired by the photographing optical system;
And based on the saturation of the generation region of the flare in the process target image, and a determination unit that the light source color of the flare is determined whether or not the specific color,
A flare correction unit that performs a flare correction process on the occurrence region,
The flare correction unit is
An image processing apparatus that switches the content of the flare correction process in accordance with a result of the determination by the determination unit. The image processing apparatus according to claim 2 ,
The discrimination unit is
When the processing target image is an image after color correction, instead of based on the saturation of the generation region, the determined row planted image processing based on whether the color corrected pixel is present in the generation region apparatus. An input unit for inputting a processing target image acquired by the photographing optical system;
A discriminator for discriminating whether or not the light source color of the flare is a specific color for a region excluding a saturated pixel region from a flare generation region in the processing target image ;
A flare correction unit that performs a flare correction process on the occurrence region,
The flare correction unit is
An image processing apparatus that switches the content of the flare correction process in accordance with a result of the determination by the determination unit. An input unit for inputting a processing target image acquired by the photographing optical system;
A determination unit that determines whether the light source color of the flare in the flare generation region in the processing target image is a specific color based on the type of the photographing optical system;
A flare correction unit that performs a flare correction process on the occurrence region,
The flare correction unit is
An area setting unit for setting a processing area including the generation area on the processing target image;
A removal unit that applies a removal process to the processing region to remove the flare coloring;
When the light source color of the flare is not the particular color, a restoring portion for restoring processing for restoring the inherent color in the generation region on the source of the flare, subjected to the processing area after the removal process,
Bei obtain image processing device.   In the image processing apparatus according to any one of claims 1 to 5,
  The discrimination unit
  An image processing apparatus that regards a circular area including a saturated pixel area on the processing target image as the generation area.   The image processing apparatus according to claim 6.
  The discrimination unit
  An image processing apparatus that estimates a size of the generation region based on a size of the saturated pixel region. The image processing apparatus according to claim 5 .
The restoration process includes
The removal process the removal process prior to the treatment area weighted synthesis processing der Ru images processing device to said processing area after. The image processing apparatus according to claim 8 .
The ratio of the weight before the removal process to the weight after the removal process is:
Set Ru images processing device for each position of the processing area. The image processing apparatus according to claim 9 .
It said than the ratio of the position close to the generating region, is small bur image processing apparatus towards said ratio position away from the generating region. 5. The image processing apparatus according to any one of claims 8 to claim 10,
The removing unit is
A reference color setting unit for setting a reference color component to be a reference among the color components of the generation region;
A correlation calculating unit that calculates correlation information indicating a correlation between a reference color component and a target color component in the processing region;
Flare color component is a colored distribution of the flare, the processing region of the correlation information and the generation region including images processing device and flare color component estimator, the estimated that based on the reference color component of. A color imaging device for imaging a subject image formed by the imaging optical system;
IMAGING apparatus and an image processing apparatus according to any one of claims 1 to 11 for processing an image in which the imaging device is generated as the process target image. The imaging apparatus according to claim 12 ,
Imaging device having the imaging optical system including a diffractive optical element. The imaging apparatus according to claim 12 ,
Mountable der Ru imaging device of the photographing optical system including a diffractive optical element. In the imaging device according to claim 13 or 14 ,
The response characteristic of the diffractive optical element to white light is
Set have that imaging device to have a peak wavelength spectrum of flare generated in the image within a predetermined wavelength range. An input procedure for entering the processing target image obtained by the photographing optical system,
Based on the type of the photographing optical system, a determination procedure in which the light source color before Symbol flare generation region of the flare in the processed image image it is determined whether or not the specific color,
A flare correction procedure for performing a flare correction process on the occurrence region,
In the flare correction procedure,
The flare correction processing benzalkonium computer executable image processing program switching contents according to the result of the determination by the determination step.   An input procedure for inputting a processing target image acquired by the photographing optical system;
  A determination procedure for determining whether or not the light source color of the flare is a specific color based on the saturation of the flare occurrence region in the processing target image;
  A flare correction procedure for performing a flare correction process on the occurrence region,
  In the flare correction procedure,
  A computer-executable image processing program for switching the content of the flare correction processing in accordance with the result of the determination by the determination procedure.   An input procedure for inputting a processing target image acquired by the photographing optical system;
  A determination procedure for determining whether or not a light source color of the flare is a specific color for an area excluding a saturated pixel area from a flare occurrence area in the processing target image,
  A flare correction procedure for performing a flare correction process on the occurrence region,
  In the flare correction procedure,
  A computer-executable image processing program for switching the content of the flare correction processing in accordance with the result of the determination by the determination procedure.   An input procedure for inputting a processing target image acquired by the photographing optical system;
  A determination procedure for determining whether the light source color of the flare in the flare generation area in the processing target image is a specific color based on the type of the photographing optical system,
  A flare correction procedure for performing a flare correction process on the occurrence region,
  In the flare correction procedure,
  An area setting procedure for setting a processing area including the generation area on the processing target image;
  A removal procedure for applying a removal process to the processing region to remove the flare coloring;
  When the light source color of the flare is not the specific color, a restoration procedure for restoring the color specific to the light source of the flare to the generation area is performed on the processing area after the removal process;
  A computer-executable image processing program.