JP6650975B2 - Image processing apparatus and method - Google Patents

Description

  The present invention relates to image processing for correcting a difference in light source when capturing an image.

  A light source that converts color data under a shooting light source into color data under a viewing light source when matching the color of an image captured under a shooting light source different from the viewing light source with the color of an image captured under the viewing light source. Perform conversion processing. The color correction condition is calculated to perform the light source conversion process. However, when calculating the color correction condition of a subject having gloss such as metallic gloss, there is the following problem.

  The glossy portion has a large color change depending on the photographing conditions, and if the color correction conditions for the light source conversion processing are calculated using the photographed data of the glossy portion, a highly accurate color correction condition cannot be obtained.

  Patent Document 1 describes a method for calculating a color correction condition for color space conversion, which does not necessarily correct the difference between light sources. According to the calculation method of Patent Document 1, a color conversion matrix for matching an RGB signal value obtained by photographing a color chart with a target RGB signal value obtained from colorimetric values of the color chart is calculated by the least square method. Is done.

  Patent Document 2 describes a method for determining whether or not a document contains a metallic glossy portion. According to the determination method of Patent Literature 2, two types of images are acquired: an image obtained by scanning a document in a normal state, and an image obtained by scanning a document in a state in which reflected light from a metallic gloss portion is increased. Then, by calculating the difference between the images and comparing the difference with a threshold value, it is determined whether or not the original includes a metallic glossy portion.

  The determination method of Patent Document 2 performs determination based on the difference between two types of images acquired under different scan conditions.However, if there is an area where the color greatly changes depending on the difference in light source, it should not be originally detected. There is a possibility that a non-existent area is erroneously determined as a metallic glossy area. In other words, it is not possible to properly extract the color values required for the color correction conditions of the high-precision light source conversion processing only by the threshold value determination using the difference between the two types of images.

JP 2005-79834 A Patent No. 4024737

  An object of the present invention is to generate a color correction condition for a light source conversion process suitable for a captured image with high accuracy.

  The present invention has the following configuration as one means for achieving the above object.

The image processing apparatus according to the present invention includes a first image data obtained by shooting an object under a first lighting condition, and a second image obtained by shooting the object under a second lighting condition different from the first lighting condition. Input means for inputting data; extracting color data from each area of the first image data; extracting color data from each area of the second image data corresponding to each area of the first image data Extracting means for calculating a correlation coefficient between the color data extracted from the area of the first image data and the color data extracted from the corresponding area of the second image data, Analyzing means for setting the color data extracted from the extraction target area whose relation number exceeds a predetermined first threshold value as the representative color of the first image data and the representative color of the second image data; and Representative color of the image data of Using the representative color of the second image data, create a color correction condition for converting the first image data depending on the first lighting condition into image data depending on the second lighting condition. It has a creation means, with reference to the color correction conditions, and converting means for performing a light source conversion process on the first image data.

  According to the present invention, a color correction condition for a light source conversion process suitable for a captured image can be generated with high accuracy.

The figure explaining the outline | summary of the process at the time of applying the image processing apparatus concerning this invention to reproduction of a cultural property or a painting. The figure explaining the outline | summary of imaging | photography of a photography light source image and an observation light source image. FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment. 5 is a flowchart illustrating a light source conversion process performed by the image processing apparatus. The figure which shows an example of UI. The figure which shows an example of an extraction color addition window. 9 is a flowchart for explaining color data extraction processing. 5 is a flowchart illustrating a process of determining an extraction position. The figure explaining the update of a color extraction file. 9 is a flowchart illustrating a process of determining an extraction range. The figure explaining the color extraction file in which the extraction range was recorded. 9 is a flowchart for explaining color data extraction processing. 9 is a flowchart illustrating a correlation analysis process. 9 is a flowchart illustrating a light source conversion process. 9 is a flowchart illustrating a difference analysis process according to the second embodiment.

[Outline of light source conversion processing]
First, a description will be given of a case of cultural property duplication, which is a typical application destination of the light source conversion processing. Generally, reproduction of cultural properties is performed by photographing cultural properties with a digital camera and outputting a photographed image with a printer. Since reproduction of a cultural property is required to reproduce the same color as the real thing under the observation light source where the cultural property is displayed, it is desirable to perform photographing under the observation light source.

  However, the illuminance is often insufficient with the observation light source, and when photographing cultural assets with a digital camera under the observation light source, noise may be generated in the photographed image. Will be Therefore, light source conversion processing for converting the color of a captured image (hereinafter referred to as a photographing light source image) into a color equivalent to an image observed under the observation light source (hereinafter referred to as an observation light source image) is required. Therefore, the parameters for the light source conversion process are created so as to convert the photographing light source image into an image equivalent to the observation light source image.

  An outline of processing when the image processing apparatus according to the present invention is applied to reproduction of a cultural property or a painting will be described with reference to FIG.

  The image processing apparatus 101 inputs a photographing light source image 1001 obtained by photographing an original (hereinafter, an original) of a cultural property or a painting by emitting a flash 1011. Then, a light source conversion process 1014 is performed on the photographed light source image 1001 to output a corrected image 1003 having the same color as the observation light source image 1002 obtained by photographing the original under the observation light source 1012 for actually observing the duplicate.

  A color conversion image 1004 obtained by applying a printer color conversion 1016 to the corrected image 1003 is printed out 1017 to generate a duplicate product 1005. The duplicate 1005 is required to have a high-definition image with little noise and to reproduce the same color as the original under the observation light source 1012 where the original is placed, so the light source conversion processing 1014 is required .

  The outline of photographing of the photographing light source image 1001 and the observation light source image 1002 will be described with reference to FIG.

  As shown in FIG. 2 (A), a plurality of flashes 1011 are arranged around an original 1021, which is a subject, and a sufficient amount of light is secured for photographing, and a high-definition photographing light source image 1001 is obtained. On the other hand, as shown in FIG.2 (B), by making the positional relationship between the original 1021 and the camera 1016 the same as when the shooting light source image 1001 was shot, and shooting under the viewing light source 1012, 1002 is obtained.

  Then, color extraction 1013 is performed from the photographing light source image 1001 and the observation light source image 1002, and by using the color correction conditions obtained by the color correction coefficient calculation 1015, highly accurate light source conversion processing 1014 becomes possible. As a result, a reproduction 1005 that reproduces the same color as the original 1021 under the observation light source 1012 is obtained.

  Hereinafter, image processing according to an embodiment of the present invention will be described in detail with reference to the drawings. In the following, in order to perform the light source conversion processing 1014 with high accuracy, color correction conditions are created as described below, as will be described later in detail. A photographing light source image 1001 and an observation light source image 1002 are obtained, a representative color is extracted from the images by a process related to color extraction 1013, the correspondence between the extracted colors is analyzed, and a color correction coefficient calculation 1015 is performed based on the analysis result. Creates a color correction condition.

  An example of the configuration of the image processing apparatus 101 according to the embodiment will be described with reference to the block diagram of FIG.

  A central processing unit (CPU) 112 uses the memory 110 as a work memory to control the entire apparatus, and executes a program stored in the memory 110 to implement image processing described later. The memory 110 includes a ROM for storing an image processing program and various data described later, a nonvolatile memory such as a hard disk drive (HDD) and a solid state drive (SSD), and a RAM used as a work memory.

  The CPU 112 provides a user interface (UI) for inputting a user instruction to the UI unit 109, for example, a touch panel. Then, the input / output unit 103 such as a USB interface is controlled to input image data of two captured images having different illumination conditions according to a user instruction from the image data holding unit 102 which is a recording medium such as a memory card. . Note that one of the two captured images is a captured light source image 1001 and the other is an observation light source image 1002.

  The extraction position determination unit 104 determines an image position from which color data is extracted in the captured image. The extraction range determination unit 105 determines a range on a captured image when color data is extracted from the captured image. The color extraction unit 106 extracts color data of the determined position and range from the image data of the captured image. The correlation analysis unit 107 analyzes the correlation between the two captured images based on the color data extracted from the captured images. Note that the CPU 112 displays the analysis result of the extracted color data on the UI. Further, the color correction unit 108 performs color correction on the image data of the captured image.

  The above configuration (the extraction position determination unit 104, the extraction range determination unit 105, the color extraction unit 106, the correlation analysis unit 107, and the color correction unit 108) supplies a program that realizes its function to a computer device, and the CPU 112 Is implemented. Alternatively, the above-described configuration may be incorporated as hardware into the image processing apparatus 101, and the CPU 112 controls the operation of the configuration to realize image processing described later.

Light Source Conversion Process The light source conversion process executed by the image processing apparatus 101 will be described with reference to the flowchart of FIG.

  The CPU 112 displays a UI for inputting a user instruction indicating information necessary for image processing on the UI unit 109 (S11).

  FIG. 5 shows an example of the UI. The input unit 301 is an input unit for instructing image data of the photographing light source image 1001, and the input unit 303 is an input unit for instructing image data of the observation light source image 1002. The image display unit 302 displays the captured light source image 1001 input according to the instruction of the input unit 301, and the image display unit 304 displays the observation light source image 1002 input according to the instruction of the input unit 303. The input unit 308 is an input unit for specifying a file name when saving the image data of the corrected image 1003.

  The image display units 302 and 304 display images in a state where the image is divided into a plurality of regions. Although FIG. 5 shows an example of a UI for displaying a divided image obtained by dividing an image into 3 × 3 areas, the number of vertical and horizontal divisions is arbitrary.

  The UI includes an extraction target selection button 305, a color extraction button 306 for instructing color extraction processing, a color correction coefficient creation button 307, a light source conversion button 309 for instructing execution of color correction processing using color correction coefficients, and the like. The details of the processing when each button is pressed will be described later.

  When the file names are input to the input units 301 and 302, the CPU 112 outputs the image data of the photographing light source image 1001 and the image of the observation light source image 1002 from the image data holding unit 102 via the input / output unit 103 based on the file names. Data is acquired (S12). The acquired image data is stored in a predetermined area of the memory 110.

  As shown in FIG. 2, the photographing light source image 1001 and the observation light source image 1002 have the same positional relationship between the original 1021 and the camera 1016, and the first illumination condition for emitting the flash 1011 and the second illumination light source 1012 It is an image taken under lighting conditions. The observation light source 1012 is, for example, a light source such as a fluorescent lamp or a spotlight fixed at a predetermined position such as a ceiling or a wall. Note that the first lighting condition only needs to obtain a larger amount of light than the second lighting condition, and the first lighting condition may be obtained by further adding light emission of the flash 1011 to the second lighting condition. That is, the imaging under the first illumination condition shown in FIG. 2A may or may not include the observation light source 1012.

  Next, although details will be described later, the CPU 112 extracts color data from the area of the photographing light source image 1001 and the area of the observation light source image 1002 corresponding to each divided image (S13). Then, based on the extracted color data, the correlation between these areas is analyzed, and the representative color of the photographing light source image 1001 and the representative color of the observation light source image 1002 are set (S14).

  Next, the CPU 112 determines whether each area corresponding to the divided image is a representative color extraction target area based on the analysis result of the correlation (S15). Then, the divided image determined as the extraction target area is presented (S16). For example, presentation of a divided image determined to be an extraction target area is performed by displaying a check mark in a check box arranged near the divided image. A check box arranged near a divided image corresponding to an area determined not to be an extraction target area (hereinafter, an extraction non-target area) remains empty, and the user can select a divided image determined to be an extraction target area. And the divided image determined to be the extraction non-target area.

  Determination of whether or not the extraction target region (S15), for example, for each corresponding region of the imaging light source image 1001 and the observation light source image 1002, perform the calculation of a correlation coefficient R described later in the analysis of color data (S14), An area where the correlation coefficient R exceeds a predetermined threshold is determined as an extraction target area. In addition, an area where the correlation coefficient R is equal to or less than a predetermined threshold is an extraction non-target area.

  Next, the CPU 112 determines whether or not there is an extraction non-target area (S17). If there is no extraction non-target area, the process proceeds to step S22. If there is an extraction non-target area, the CPU 112 determines whether or not there is a characteristic color in the extraction non-target area (S18). If there is no characteristic color, the process proceeds to step S22. Further, when there is a characteristic color in the non-extraction area, the CPU 112 displays an extraction color addition window on the UI (S19).

  The determination as to whether or not there is a characteristic color (S18) uses the color data extracted in step S13 to calculate the color difference between the color data of the extraction target area and the color data of the non-extraction area of the photographing light source image 1001. This is performed based on the color difference. Equation (1) described later is used to calculate the color difference. If the color difference exceeds a predetermined threshold (for example, 5), it is determined that there is a characteristic color in the extraction non-target area, and if the color difference is equal to or less than the predetermined threshold, It is determined that there is no characteristic color in the extraction non-target area.

  FIG. 6 shows an example of the extraction color addition window. The detected characteristic color is displayed on the characteristic color display section 311 of the extraction color addition window, and the image of the non-extraction target area is displayed on the image display section 312. When the user refers to the extraction color addition window and determines that addition of the extraction color is necessary, the user operates the pointer 313 to specify the extraction color in the image displayed on the image display unit 312, and click the extraction color addition button. Press or touch 314 (hereinafter referred to as "press"). The user presses the OK button 315 when it is determined that the addition of the extracted color is unnecessary or when it is determined that the addition of the extracted color is completed.

  As described above, the user can specify the extraction color from the image of the non-extraction area displayed on the image display unit 312 with reference to the characteristic color displayed on the characteristic color display unit 311. As a method of specifying the extraction color, for example, a method of moving a rectangular frame displayed by tapping to an area corresponding to the extraction color and adjusting the size of the rectangular frame by pinching may be used. Alternatively, a method of specifying a rectangular area by moving the pointer 313 with a pointing device or a keyboard to specify one point of the area corresponding to the extracted color, and moving the pointer 313 to specify another point of the area may be used. Good. Of course, any method may be used as long as the method can efficiently specify the region of the extracted color.

  The CPU 112 determines whether the extraction color addition button 314 has been pressed or the OK button 315 has been pressed (S20). When the extraction color addition button 314 is pressed, the details will be described later, but the CPU 112 adds the color data of the area designated by the user to the extraction color (S21), and returns the processing to step S20. If the OK button 315 has been pressed, the process proceeds to step S22.

  Next, the CPU 112 uses the color data of the representative color (including the color data of the added extracted color) set in step S14 to determine a color that depends on the imaging light source 1011 using the observation light source 1012, as described in detail later. Creates a color correction coefficient for performing light source conversion to a color dependent on. Then, light source conversion processing based on the created color correction coefficient is performed on the captured light source image 1001 (S22).

● Extraction of color data (S13)
The color data extraction process (S13) will be described with reference to the flowchart of FIG.

  The CPU 112 acquires the RGB values and the position coordinates (x, y) of the pixel of interest from the photographing light source image 1001, for example, in raster order (S131), supplies the data to the extraction position determination unit 104, and extracts the data to the extraction position determination unit 104 A position determination process is executed (S132). Details of the extraction position determination processing will be described later.

  Next, the captured image acquisition unit 103 determines whether or not extraction position determination processing has been performed for all pixels of the captured light source image 1001 (S133), and if there are unexecuted pixels, returns the processing to step S131. .

  When the extraction position determination process is completed, the CPU 112 causes the extraction range determination unit 105 to perform an extraction range determination process based on the color data around the determined extraction position (S134). Details of the extraction range determination processing will be described later.

  When the extraction range determination processing is completed, the CPU 112 causes the color extraction unit 106 to execute color data extraction processing based on the determined extraction position and the determined extraction range (S135). Details of the color data extraction processing will be described later.

Determining the extraction position (S132)
The extraction position determination processing (S132) will be described with reference to the flowchart of FIG.

  When the data of the target pixel (RGB value and position coordinates (x, y)) is input (S1321), the extraction position determination unit 104 determines whether the color correction file 11 is empty (S1322). The color correction file 11 is updated based on the input data (S1329). “The color correction file 11 is empty” means that no extracted color is recorded in the color correction file 11.

  The color correction file 11 is stored in the memory 110 and describes, as data, an extraction number M indicating the number of extraction colors, an RGB value of each extraction color, position coordinates (x, y) of each extraction color, a frequency f, and the like. Have been. The frequency f is the cumulative number of colors determined to belong to the same region of the color space based on a color difference described later.

  If the extracted color is recorded in the color correction file 11, the extraction position determination unit 104 obtains the RGB value of the target extraction color from the color extraction file 11 (S1323), and determines the color difference between the color of the target pixel and the target extraction color. ΔE is calculated (S1324).

The color difference ΔE is calculated using Expression (1) by converting the RGB value of the target pixel and the RGB value of the target extraction color into color values such as CIELab values.
ΔE = √ {(L ₀ -L ₁ ) ² + (a ₀ -a ₁ ) ² + (b ₀ -b ₁ ) ² };… (1)
Here, L ₀ a ₀ b ₀ is the Lab value of the pixel of interest,
L ₁ a ₁ b ₁ is the Lab value of the target extraction color.

  The conversion to the color value uses the AdobeRGB conversion formula and the CIELab conversion formula when the image data is the AdobeRGB value, and uses the sRGB conversion formula and the CIELab conversion formula when the image data is the sRGB value. Further, the color difference may be calculated using a color appearance value obtained by a conversion formula such as CIECAM02 instead of a color value such as a CIELab value.

  Next, the extraction position determination unit 104 determines whether the color difference ΔE between the color of the pixel of interest and all the extracted colors recorded in the color extraction file 11 has been calculated (S1325), and determines the color difference ΔE. If there is an uncalculated extracted color, the process returns to step S1323.

  When the calculation of the color difference ΔE is completed for all the extracted colors recorded in the color extraction file 11, the extraction position determination unit 104 determines the magnitude relationship between the minimum color difference ΔEmin and the predetermined threshold value Eth among the calculated color differences ΔE. (S1326). If the minimum color difference is equal to or larger than the threshold (ΔEmin ≧ Eth), the data of the target pixel is added to the color extraction file 11 (S1329), and the process of determining the extraction position for the target pixel is completed. Note that, for example, Eth = 5 is used as the threshold for color difference determination, but the value of the threshold Eth is not limited to this.

  Update of the color extraction file 11 will be described with reference to FIG. FIG. 9A shows the RGB values and the position coordinates (x, y) of the pixel of interest, and FIG. 9B shows a recording example of the color extraction file 11. In the case of the minimum color difference ΔEmin ≧ Eth, as shown in FIG. 9C, in step S1329, the data of the target pixel is added to the color extraction file 11, the number of extractions M is incremented, and the frequency f = 1 is set. .

On the other hand, if the minimum color difference is less than the threshold (ΔEmin <Eth), the extraction position determination unit 104 updates the data of the extraction color (hereinafter, the minimum color difference extraction color) corresponding to the minimum color difference ΔEmin (S1327). In this update, the frequency f of the minimum color difference extraction color is incremented, and the position coordinates of the minimum color difference extraction color are updated to the position coordinates of the pixel of interest. Further, the extraction position determination unit 104 calculates a standard deviation σ of RGB values in a predetermined pixel range (hereinafter, a local range) around the target pixel of the photographing light source image 1001 according to Expression (2), and calculates the standard deviation σ The RGB value of the minimum color difference extraction color is updated according to the pixel value at which is minimized. That is, the RGB values recorded in the color extraction file 11 in step S1328 are RGB representative values corresponding to the frequency f.
Rm = Σ _x Σ _y R (x, y) / N;
Gm = Σ _x Σ _y G (x, y) / N;
Bm = Σ _x Σ _y B (x, y) / N;
σ _R = √ {Σ _i (Ri-Rm) ² / (N-1)};
σ _G = √ {Σ _i (Gi-Gm) ² / (N-1)};
σ _B = √ {Σ _i (Bi-Bm) ² / (N-1)};… (2)
Where RmGmBm is the average RGB value of the pixels in the local range,
RiGiBi is the RGB value of the pixel in the local range,
N is the number of pixels (the number of samples) in the local range.

  In Equation (2), the local range is, for example, a range of five pixels (x ± 2 for the X coordinate and y ± 2 for the Y coordinate), but the local range is not limited to this.

  FIG. 9D shows the RGB value and the position coordinates (x, y) of the target pixel, and FIG. 9E shows a recording example of the color extraction file 11. If the minimum color difference ΔEmin <Eth between the color value of the pixel of interest and the color value of each extracted color recorded in the color extraction file 11, the color difference is minimized based on the data of the pixel of interest, as shown in FIG. The data of the extracted color is updated, and the frequency f is incremented.

  Next, the extraction position determination unit 104 determines a magnitude relationship between the frequency f updated in step S1327 and a predetermined threshold Fth (S1328). If the updated frequency is equal to or less than the threshold value (f ≦ Fth), the process of determining the extraction position for the target pixel ends. If the updated frequency exceeds the threshold (f> Fth), the data of the pixel of interest is added to the color extraction file 11 (S1329), and the process of determining the extraction position for the pixel of interest ends. Note that, for example, a value corresponding to 1% of the total number of pixels of the captured light source image 1001 is used as the threshold for the frequency determination, but the value of the threshold Fth is not limited to this.

  FIG. 9 (G) shows the RGB values and the position coordinates (x, y) of the pixel of interest, and FIG. 9 (H) shows a recording example of the color extraction file 11. If the frequency f> Fth after the update of the color difference minimum extraction color is updated, as shown in FIG. 9 (I), the data of the target pixel is added to the color extraction file 11, the extraction number M is incremented, and the frequency f = 1 Is set. Note that the minimum color difference extraction color in FIG. 9 (I) is the third data, and FIG. 9 (I) shows an example where the frequency determination threshold Fth = 2000. As described above, even if the data belongs to the same area of the color space, if the frequency f exceeds the threshold value Fth, the data of the target pixel is added to the color extraction file 11.

Determining the extraction range (S134)
The extraction range determination processing (S134) will be described with reference to the flowchart of FIG.

  The extraction range determination unit 105 acquires an initial value of an extraction range used when extracting a pixel value from the captured light source image 1001 (S1341). As an initial value of the extraction range, for example, a square range of 10 × 10 pixels is set.

  Next, the extraction range determination unit 105 acquires the position coordinates (x, y) of the target extraction color from the color extraction file 11 (S1342). Then, based on the position coordinates (x, y) of the extracted color of interest and the initial value of the extraction range, pixel values of the extraction range are acquired from the captured light source image 1001, and the standard deviation Ds of those pixel values is calculated (S1343).

  It is sufficient to obtain the pixel value of the extraction range about the position coordinates (x, y) of the target extraction color, but if the position coordinates (x, y) are located near the vertices or sides of the image, the position Obtain the pixel value of the extraction range with the coordinates (x, y) on the vertex or side. Also, an example will be described in which the standard deviation is used as a statistic representing the color variation of the extraction range, but another statistic such as variance may be used.

  Next, the extraction range determination unit 105 determines whether the calculated standard deviation Ds satisfies the determination condition (Ds <Dth) based on the predetermined threshold Dth (S1344). When the standard deviation Ds does not satisfy the determination condition (Ds ≧ Dth), the extraction range determination unit 105 narrows the extraction range by one pixel (for example, 10 × 10 is changed to 9 × 9) (S1345), and the process proceeds to step S1343. And the standard deviation Ds of the pixel values in the extraction range is calculated again.

  When the standard deviation Ds satisfies the determination condition (Ds <Dth), the extraction range determining unit 105 records the extraction range satisfying the determination condition in the color extraction file 11 in association with the target extraction color (S1346). The color extraction file 11 in which the extraction range is recorded will be described with reference to FIG. As shown in FIG. 11, the extraction range (number of pixels) is recorded in the color extraction file 11 in addition to the number of extractions M, the RGB values of each extracted color, the position coordinates (x, y), and the frequency f.

  Next, the extraction range determination unit 105 determines whether or not the extraction range has been determined for all the extraction colors recorded in the color extraction file 11 (S1347). If there is an extraction color whose extraction range is undecided, the process returns to step S1341, and the processes of steps S1341 to S1346 are repeated until the extraction range is determined for all the extraction colors.

● Color data extraction processing (S135)
The color data extraction process (S135) will be described with reference to the flowchart of FIG.

  The color extraction unit 106 acquires the position coordinates (x, y) and the extraction range of the target extraction color from the color extraction file 11 (S1351). Then, based on the position coordinates (x, y) and the extraction range, RGB values within the extraction range are acquired from each of the photographing light source image 1001 and the observation light source image 1002 (S1352).

  What is necessary is just to obtain the RGB value of the extraction range whose center is approximately the position coordinates (x, y) of the target extraction color, but if the position coordinates (x, y) are located near the vertices or sides of the image, the position Get the RGB value of the extraction range with coordinates (x, y) on the vertex or side.

Next, the color extraction unit 106 calculates a representative value (representative color) of RGB values extracted from the captured light source image 1001 and a representative value (representative color) of RGB values extracted from the observation light source image 1002 (S1353). . For example, the average value of the RGB values may be calculated using Equation (3) to calculate the representative color. However, another statistic such as a center value may be used for the representative color instead of the average value. The calculated representative color is stored in the extracted color file 12 of the memory 110 in association with the position coordinates (x, y) for each of the photographing light source image 1001 and the observation light source image 1002.
Rsr = Σ _x Σ _y Rs (x, y) / n;
Gsr = Σ _x Σ _y Gs (x, y) / n;
Bsr = Σ _x Σ _y Bs (x, y) / n;
Rvr = Σ _x Σ _y Rv (x, y) / n;
Gvr = Σ _x Σ _y Gv (x, y) / n;
Bvr = Σ _x Σ _y Bv (x, y) / n;… (3)
Here, RsGsBs is an RGB value extracted from the photographing light source image 1001,
RvGvBv is an RGB value extracted from the observation light source image 1002,
RsrGsrBsr is a representative color in the captured light source image 1001,
RvrGvrBvr is a representative color in the observation light source image 1002,
n is the number of pixels (the number of samples) in the extraction range.

  Next, the color extraction unit 106 determines whether or not the calculation of representative colors (extraction of color data) has been executed for all the extracted colors recorded in the color extraction file 11 (S1354). If there is an extraction color for which extraction of color data has not been completed, the process returns to step S1351. If extraction of color data has been completed for all extraction colors, the extraction processing of color data ends.

● Correlation analysis processing (S14)
The correlation analysis processing (S14) will be described with reference to the flowchart of FIG.

  The correlation analysis unit 107 classifies the representative color data recorded in the extracted color file 12 into a data group corresponding to the divided images of the image display units 302 and 304 (S141). That is, based on the position coordinates (x, y) associated with the representative color, the data of the representative color is classified into a data group corresponding to each of the divided images. The data group is a target unit of the correlation analysis. However, the target unit of the correlation analysis is not limited to the divided image unit, but may be an image unit or a pixel unit.

Next, the correlation analysis unit 107 calculates a correlation coefficient R for one of the corresponding data groups of the captured light source image 1001 and the observation light source image 1002 by the following equation (S142).
Gsm = Σ _i Gsr _i / m;
Gvm = Σ _i Gvr _i / m;
_{R = Σ i (Gsr i -Gsm} ) (Gvr i -Gvm) / {√Σ i (Gsr i -Gsm) 2 √Σ i (Gvr i -Gvm) 2}; ... (4)
Here, Gsr is a G component of a representative color in the captured light source image 1001,
Gvr is the G component of the representative color in the observation light source image 1002,
m is the number of representative colors included in the data group.

  Equation (4) shows an example of calculating the correlation coefficient R on the assumption that the G component represents the luminance value.The luminance value is calculated from the RGB values of the representative color, and the correlation coefficient R is calculated based on the calculated luminance value. May be.

  Next, the correlation analysis unit 107 determines a magnitude relationship between the calculated correlation coefficient R and a predetermined threshold Rth (S143). If the correlation coefficient exceeds the threshold (R> Rth), the region of the divided image corresponding to the data group is the above-described extraction target region, and the data group is stored in the color correction file 13 of the memory 110 (S144). . Note that, for example, Rth = 0.95 is used as the threshold for determining the correlation coefficient R, but the value of the threshold Rth is not limited to this.

  Next, the correlation analysis unit 107 determines whether the calculation and determination of the correlation coefficient R have been performed for all data groups (S145), and if there is a data group that has not been performed, the process returns to step S142. The calculation and determination of the correlation coefficient R are repeated for all data groups.

  When the color data analysis process (S14) is completed, the color correction file 13 records color data of an extracted color for creating a color correction coefficient for each divided image of the extraction target area. Further, as described above, when there is a characteristic color in the non-extraction target area, in step S21, the color data of the area specified by the user may be added to the color correction file 13 as the color data of the extraction color.

  According to the correlation analysis process (S14), an area including a glossy portion can be classified as an extraction non-target area based on the color relationship between the divided image of the captured light source image 1001 and the divided image of the observation light source image 1002. . As a result, the color of the area including the glossy portion can be excluded from the representative colors used for calculating the color correction coefficient.

Light source conversion processing (S22)
The light source conversion processing (S22) will be described with reference to the flowchart of FIG.

The color correction unit 108 acquires the color data of the representative color (including the color data of the added extracted color) from the color correction file 13 (S221). Then, from the color data of the representative color of the photographing light source image 1001 and the color data of the representative color of the observation light source image 1002, the RGB value depending on the photographing light source 1011 is converted into the RGB value depending on the observation light source 1012 based on the following equation. A color correction coefficient is calculated (S222).
┌ ┐┌ ┐┌ ┐
│Rv _i ││m ₁₁ m ₁₂ m ₁₃ m ₁₄ ││Rs _i │
│Gv _i ││m ₂₁ m ₂₂ m ₂₃ m ₂₄ ││Gs _i │
│Bv _i ││m ₃₁ m ₃₂ m ₃₃ m ₃₄ ││Bs _i │
└ ┘└ ┘│ 1 │
└ ┘… (5)
Here, RsGsBs is the color data of the representative color of the photographing light source image 1001,
RvGvBv is the color data of the representative color of the observation light source image 1002.

The matrix coefficient m ₁₁ -m ₃₄ in equation (5) corresponds to the color correction coefficient. The optimal matrix coefficient can be calculated by applying the color data of each representative color to the equation (5) and using the least squares method or the like. Equation (5) shows an example in which a 3 × 4 matrix is used as a matrix coefficient, but a matrix of a different order may be used.

  Next, the color correction unit 108 receives image data of the shooting light source image 1001 as a conversion source image (S223), performs light source conversion processing using a color correction coefficient on the image data of the shooting light source image 1001, and Is calculated (S224). Then, the image data obtained by the light source conversion is stored in the memory 110 as image data of the corrected image 1003 (a captured image depending on the observation light source 1012) (S225).

  As described above, a plurality of representative colors are extracted from the captured image, and a color correction coefficient for converting the captured light source image 1001 dependent on the capturing light source 1011 into a captured image (corrected image 1003) dependent on the observation light source 1012 is calculated. Thus, a color correction condition suitable for light source conversion can be created.

  In addition, by analyzing the correlation (S14), the color relationship between the photographing light source image 1001 and the observation light source image 1002 is analyzed, and the color of the glossy part is previously excluded from the representative colors used for calculating the color correction coefficient. Thus, highly accurate color correction conditions can be created.

  Further, if there is a characteristic color in the non-extraction area by the processing of steps S17 to S21, the omission of the representative color is prevented by presenting the characteristic color and allowing the user to determine whether or not the characteristic color needs to be extracted. In addition, it is possible to prevent a decrease in the accuracy of the color correction conditions.

  Hereinafter, image processing according to the second embodiment of the present invention will be described. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the detailed description thereof will be omitted.

  In the first embodiment, a method of determining whether data of an extracted color includes data due to a glossy portion based on the color relationship between the divided image of the photographing light source image 1001 and the divided image of the observation light source image 1002. explained. In the second embodiment, a method of making a similar determination based on the difference and correlation between the representative color of the captured light source image 1001 and the representative color of the observation light source image 1002 will be described.

Difference Analysis Processing Difference analysis processing according to the second embodiment will be described with reference to the flowchart in FIG. The difference analysis process is performed as a part of the correlation analysis process (S14) shown in FIG.

  The correlation analysis unit 107 classifies the representative color data recorded in the extracted color file 12 into a data group corresponding to the divided images of the image display units 302 and 304 (S151). That is, the representative color is classified into a data group corresponding to each of the divided images based on the position coordinates (x, y) associated with the representative color.

  Next, the correlation analysis unit 107 acquires the reference white color of the imaging light source 1011 and the reference white color of the observation light source 1012 (S152). For example, as the reference white data, the color temperature at the time of shooting is acquired with reference to the Exif information of the shot image. Further, it is also possible to use a method of preparing a table for holding the correspondence between the pixel values of the captured image and the color temperature, and estimating the color temperature from the highlight portion of the captured image.

  Next, the correlation analysis unit 107 sets a threshold Th used for difference determination based on the acquired reference white (S153). The reason for setting the threshold value th based on the reference white color is that the difference in pixel value between captured images increases due to the difference in light source. This is to prevent color data from being extracted from the. For example, the reference value of the threshold value Th is set to 10, and the threshold value Th is increased by adding 5 to the threshold value Th every time the color temperature of the imaging light source 1011 and the color temperature of the observation light source 1012 differ by 500K. Of course, the reference value, the color temperature step, and the threshold step are not limited to these values.

Then, the correlation analysis unit 107, for one data group corresponding calculates the difference d _i between each representative color (S154) of the imaging light source image 1001 and the observation light source image 1002, the calculated difference d _i respectively The threshold Th is compared (S155). If all the differences are smaller than the threshold value (d _i <Th), the area of the divided image corresponding to the data group is the extraction target area, and the data group is stored in the color correction file 13 (S156). Further, if the threshold or more difference d _i is even one region of the divided image corresponding to the data group is extracted non-target region. Incidentally, the difference d _i, for example, may be calculated as the Euclidean distance between the representative colors in the RGB space.

Then, the correlation analysis unit 107 determines whether or not performed is calculated and the difference determines the difference d _i for all data groups (S157), the process if there is data group unexecuted returns to step S154 , it repeats the calculation and the difference determines the difference d _i for all data groups.

  According to the difference analysis processing of the second embodiment, the area including the glossy portion can be determined from the difference between the representative colors between the divided image of the captured light source image 1001 and the divided image of the observation light source image 1002. As a result, the color of the area including the glossy portion can be excluded from the representative colors used for calculating the color correction coefficient.

The CPU 112 creates the color correction file 13 by the above-described difference analysis processing, determines the extraction target area and the extraction non-target area based on the correlation analysis processing (S14, S15), and presents the extraction target area to the UI (S14, S15). S16). Then, when there is no extraction non-target area, the process proceeds to step S22. That is, in the second embodiment, the divided images large difference d _i than the threshold value Th by analysis of the difference is determined to be even one further performs analysis processing of the correlation, from the correlation coefficient R is the threshold Rth Is determined to be a non-target region to be extracted.

  In this way, it is determined whether or not the region of the captured image includes a glossy portion based on the difference between the representative colors, and the region determined to include the glossy portion is further extracted and excluded based on the correlation of the representative color. Determine the area. This makes it possible to extract color data for creating color correction conditions with high accuracy.

  The present invention is not limited to the configuration of each embodiment described above, and various modifications can be made without departing from the spirit of the present invention.

[Other Examples]
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and a computer (or CPU or MPU or the like) of the system or apparatus reads the program and reads the program. This is the process to be performed.

  104: extraction position determination unit, 105: extraction range determination unit, 106: color extraction unit 106, 107: correlation analysis unit, 108: color correction unit

Claims (11)

First image data obtained by photographing the subject under the first lighting condition, and input means for inputting second image data obtained by photographing the subject under a second lighting condition different from the first lighting condition,
Extraction means for extracting color data from each area of the first image data, and extracting color data from each area of the second image data corresponding to each area of the first image data,
Calculating a correlation coefficient between the color data extracted from the area of the first image data and the color data extracted from the corresponding area of the second image data, wherein the correlation coefficient is a predetermined value; Analysis means for setting the color data extracted from the extraction target area exceeding the first threshold as the representative color of the first image data and the representative color of the second image data ,
Using the representative color of the first image data and the representative color of the second image data, the first image data that depends on the first lighting condition is converted into image data that depends on the second lighting condition. Creating means for creating color correction conditions for conversion;
A conversion unit that performs a light source conversion process on the first image data using the color correction condition ,
Image processing apparatus having a.   The image processing apparatus according to claim 1, further comprising a user interface that presents the extraction target area and the extraction non-target area in which the correlation coefficient is equal to or less than the first threshold so as to be distinguishable. The analysis unit sets a second threshold based on the reference white under the first lighting condition and the reference white under the second lighting condition, and sets a representative color of the first image data and the second image data. The image processing apparatus according to claim 1, wherein color data extracted from an area whose difference from the representative color is smaller than the second threshold is set as color data of the representative color.   The said reference white is a color temperature, The said analysis means increases the said 2nd threshold value, so that the difference of the color temperature of the said 1st lighting condition and the color temperature of a said 2nd lighting condition is large. The described image processing device. The analyzing means calculates a correlation coefficient between the area of the first image data and the corresponding area of the second image data based on the extracted color data,
4. A user interface for presenting an extraction target area in which the correlation coefficient exceeds a predetermined first threshold and an extraction non-target area in which the correlation coefficient is equal to or less than the first threshold so as to be distinguishable. 4. The image processing device according to 4.   Means for determining whether or not there is a characteristic color in the extraction non-target area, and providing a window for adding the characteristic color to the representative color on the user interface if the characteristic color is present, The image processing device according to 2 or 5.   The image processing apparatus according to claim 1, wherein the first illumination condition has a larger light amount than the second illumination condition.   The light source according to any one of claims 1 to 7, wherein the first lighting condition includes emission of a flash in capturing the first image data, and the second lighting condition corresponds to observation light of the subject. The described image processing device. First image data obtained by shooting a subject under the first lighting condition, and inputting second image data obtained by shooting the subject under a second lighting condition different from the first lighting condition,
Extracting color data from each area of the first image data, extracting color data from each area of the second image data corresponding to each area of the first image data,
Calculating a correlation coefficient between the color data extracted from the area of the first image data and the color data extracted from the corresponding area of the second image data, wherein the correlation coefficient is a predetermined value; Setting the color data extracted from the extraction target area exceeding the first threshold as the representative color of the first image data and the representative color of the second image data ,
Using the representative color of the first image data and the representative color of the second image data, the first image data that depends on the first lighting condition is converted into image data that depends on the second lighting condition. Creating color correction conditions for conversion,
Using the color correction conditions, performing light source conversion processing on the first image data ,
Image processing methods, including.   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.   A computer-readable recording medium on which the program according to claim 10 is recorded.

Images