JP2005278213A - Production method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reproduce more faithful color, while reducing noise. <P>SOLUTION: A four-color filter 61 consists of a total of four filter as the minimum unit; an R filter through which only red light passes, a B filter through which only blue light passes, a G1 filter through which only green light in a first wavelength range passes; and a G2 filter with high relation to the G1 filter and through which only green light in a second wavelength range passes. The G1 filter and the G2 filter are arranged in diagonal positions in the minimum unit. An RGB signal is generated, based on four types of signals, which passed through the four-color filter 61 and were obtained with an image sensor. The invention is applied to an image processing device, such as a digital camera or the like. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a manufacturing method, and more particularly, to a manufacturing method of an image processing apparatus that reproduces a more faithful color and reduces noise.

  In recent years, consumer-oriented image input devices (digital cameras, color scanners, etc.) and image processing software are becoming widespread, and the number of users who edit images themselves obtained by shooting or the like has increased.

  Along with this, demands for image quality (requirements for more preferable colors, demands for less noise, etc.) have become very high. As a first condition when purchasing a digital camera or the like, the ratio of users who state that image quality is good accounts for more than half.

  In a digital camera, generally, a color filter 1 of three primary colors of RGB as shown in FIG. 1 is used. In this example, as indicated by a one-dot chain line in FIG. 1, two G filters that transmit only green (G) light, one R filter that transmits only red (R) light, and blue ( The color filter 1 is configured by a so-called Bayer arrangement (Bayer arrangement) with one B filter that transmits only light B) and a total of four B filters.

  FIG. 2 is a block diagram illustrating a configuration example of the signal processing unit 11 that performs various processes on an RGB signal acquired by a CCD (Charge Coupled Device) imaging device having the RGB color filter 1.

  The offset correction processing unit 21 removes an offset component included in the image signal supplied from the front end 13 that performs predetermined processing on the signal acquired by the CCD image sensor, and performs white balance correction processing on the obtained image signal. To the unit 22. The white balance correction processing unit 22 corrects the balance of each color based on the color temperature of the image signal supplied from the offset correction processing unit 21 and the difference in sensitivity of each filter of the color filter 1. The white balance correction processing unit 22 performs correction, and the acquired color signal is output to the gamma correction processing unit 23. The gamma correction processing unit 23 performs gamma correction on the signal supplied from the white balance correction processing unit 22, and outputs the acquired signal to the vertical direction synchronization processing unit 24. The vertical direction synchronization processing unit 24 is provided with a delay element, and the time shift in the vertical direction of the signal supplied from the gamma correction processing unit 23 is synchronized.

  The RGB signal generation processing unit 25 performs interpolation processing for interpolating the color signal supplied from the vertical direction synchronization processing unit 24 into the phase of the same space, noise removal processing for removing noise components of the signal, and limiting the signal band. A filtering process, a high frequency correction process for correcting a high frequency component of the signal band, and the like are performed, and the obtained RGB signals are output to the luminance signal generation processing unit 26 and the color difference signal generation processing unit 27.

  The luminance signal generation processing unit 26 combines the RGB signals supplied from the RGB signal generation processing unit 25 with a predetermined combining ratio to generate a luminance signal. Similarly, the color difference signal generation processing unit 27 combines the RGB signals supplied from the RGB signal generation processing unit 25 with a predetermined combination ratio to generate color difference signals (Cb, Cr). The luminance signal generated by the luminance signal generation processing unit 26 and the color difference signal generated by the color difference signal generation processing unit 27 are output to, for example, a monitor provided outside the signal processing unit 11.

  As described above, image processing is generally performed by linear conversion after performing gamma processing on the original signal.

  Incidentally, as a condition for determining a color filter, firstly, “color reproducibility” for reproducing a color faithful to how a human eye can see is given. This “color reproducibility” means “color appearance” which means that the color is close to the color that can be seen by the human eye. “Color discriminability” (metamerism matching) that means reproduction into colors is composed of two. Secondly, satisfying “physical restrictions” in creating a filter such that the spectral component has a positive sensitivity and the spectral sensitivity characteristic has one peak, and thirdly, “no noise For example, “reduction” is considered.

  Conventionally, for example, a filter evaluation coefficient such as q factor, μ factor, or FOM (Figure of Merit) is used to create and evaluate a color filter that emphasizes “color reproducibility”. These coefficients take values from 0 to 1, and show a larger value, that is, a value closer to 1, as the spectral sensitivity characteristic of the color filter approximates to the linear transformation of the spectral sensitivity characteristic (color matching function) of the human eye. In order to make the values of these coefficients close to 1, the spectral sensitivity should satisfy the router condition.

  However, when designed to satisfy the router condition, the color filter has a negative spectral component or a plurality of peak values as shown in FIG. Therefore, even if it is physically impossible or can be realized, it becomes considerably difficult.

  Therefore, when considering the above-described “physical limitations” in addition to the router condition, the spectral sensitivity characteristic is normally a characteristic that does not cause a negative spectral component as shown in FIG. The curve L1 in FIG. 3 and the curve L11 in FIG. 4 indicate the spectral sensitivity of R, the curve L2 in FIG. 3 and the curve L12 in FIG. 4 indicate the spectral sensitivity of G, and the curve L3 in FIG. 3 and the curve L13 in FIG. The spectral sensitivity of B is shown respectively.

  However, in the filter having the spectral sensitivity characteristic as shown in FIG. 4, the overlap between the R spectral sensitivity characteristic (curve L11) and the G spectral sensitivity characteristic (curve L12) becomes large, and the respective color signals. When decomposing (extracting), there is a problem that propagation noise increases. That is, it is necessary to increase the difference between the R signal and the G signal in order to decompose the color signal. However, when each signal is amplified to increase the difference, noise is also amplified together with the above-described “ The “noise reduction” is not satisfied.

  Therefore, in order to satisfy the “noise reduction”, the overlapping portion of the spectral sensitivity of G and the spectral sensitivity of R is reduced even if the “color reproducibility” is somewhat sacrificed, for example, as shown in FIG. Spectral sensitivity characteristics can also be considered.

  However, in the case of a filter having such characteristics, for example, an object that appears as a different color to the eyes is photographed as the same color in a digital camera, and so-called “color discrimination” is deteriorated. There is.

  Further description of the deterioration of the “color discrimination” is as follows. That is, FIG. 6 is a diagram showing the spectral reflectances of the object R1 and the object R2, and the spectral reflectances of the object R1 and the object R2 are different as shown in the figure. 7 shows tristimulus values (X, Y, Z values) (FIG. 7A) when the objects R1 and R2 having the spectral reflectance shown in FIG. 6 are viewed with the eyes of a standard observer, and FIG. It is a figure which shows the RGB value at the time of imaging | photography with the color filter which has the spectral sensitivity characteristic (FIG. 7B).

  In FIG. 7A, the X, Y, Z values (“0.08”, “0.06”, “0.30”) of the object R1 and the X, Y, Z values (“0.10”) of the object R2 are shown. , “0.07”, “0.33”) are different values, indicating that each object appears as a different color to the human eye. On the other hand, in FIG. 7B, the values of R, G, B of the object R1 and the values of R, G, B of the object R2 are the same values (“66.5”, “88.3”, “ 132.0 "). This means that in the digital camera (color filter) having the spectral sensitivity characteristics shown in FIG. 5, the colors of the respective objects are photographed as the same color without being distinguished.

  In addition, in the evaluation of color filters using q-factor, μ-factor, and FOM, although “noise reduction” is not considered, it is not desirable from the viewpoint of “noise reduction”, The highest evaluation (coefficient value is 1) is shown for a filter that satisfies both “color reproducibility” and “physical restriction” (a filter that satisfies the router condition and that is shown in FIG. 4).

  The present invention has been made in view of such circumstances, and is intended to reproduce more faithful colors and reduce noise.

  The manufacturing method of the present invention includes an extraction unit that extracts a component of a predetermined color from incident light, and a conversion unit that converts the light of the color component extracted by the extraction unit into a corresponding color signal. A method of manufacturing a processing apparatus, comprising: a first step of preparing a conversion unit; and a first of the first to third light of the three primary colors and a first of the first to third light of the three primary colors as the extraction unit. Each of the first to fourth extraction units for extracting the fourth light having a high correlation with the second light, and the second extraction unit and the fourth extraction unit are for viewing the luminance signal. And a second step of generating extraction means having a spectral sensitivity characteristic approximate to the sensitivity characteristic and located diagonally in the unit in front of the conversion means prepared by the processing of the first step. And

  In the second step, the spectral sensitivity characteristic of the extraction unit can be determined using a predetermined evaluation coefficient.

  The evaluation coefficient can be an evaluation coefficient that considers noise reducibility as well as color reproducibility.

  In the second step, the first to third lights of the three primary colors may be red, green, or blue light, and the fourth light may be green light.

  Generating means for generating fifth to seventh color signals corresponding to the three primary colors is generated based on the first to fourth color signals generated by converting the first to fourth light by the converting means. A third step can be further included.

  The third step is to minimize a difference between a reference value calculated based on a predetermined color patch and an output value calculated based on the spectral sensitivity characteristic of the extraction unit based on the color patch in a predetermined evaluation value. The generation means can be generated so as to generate the fifth to seventh color signals based on the conversion formula given as follows.

  According to the image processing apparatus obtained by the present invention, the photographed color can be faithfully reproduced.

  Further, according to the image processing apparatus obtained by the present invention, “color discrimination” can be improved.

  Furthermore, according to the image processing apparatus obtained by the present invention, “color reproducibility” and “noise reduction” can be improved.

  According to the image processing apparatus obtained by the present invention, the “color appearance” can be improved.

  FIG. 8 is a block diagram showing a configuration example of a digital camera to which the present invention is applied.

  The digital camera shown in FIG. 8 is provided with a color filter for identifying four types of colors (light) on the front surface (surface facing the lens 42) of an image sensor 45 made of CCD (Charge Coupled Device) or the like. .

  FIG. 9 is a diagram illustrating an example of the four-color filter 61 provided in the image sensor 45 of FIG.

  As shown by the one-dot chain line in FIG. 9, the four-color filter 61 is an R filter that transmits only red light, a B filter that transmits only blue light, and transmits only green light in the first wavelength band. And a G2 filter having a high correlation with the G1 filter and transmitting only the green light in the second wavelength band, a total of four filters are configured as a minimum unit. Further, the G1 filter and the G2 filter are disposed at positions diagonal to each other within the minimum unit.

  As will be described in detail later, the color of the image acquired by the image sensor 45 is set to four types, and the acquired color information is increased, so that only three types of colors (RGB) are acquired. It is possible to express colors accurately, and it is possible to improve the reproduction of colors that look different to the eyes with different colors and colors that look the same with the same colors ("color discrimination"). .

  As can be seen from the visibility curve shown in FIG. 10, the human eye is sensitive to luminance. Therefore, in the example of the four-color filter 61 in FIG. 9, by acquiring more accurate luminance information, it is possible to increase the luminance gradation and reproduce an image close to how the eyes are seen. A G2 color filter having a spectral sensitivity characteristic close to the visibility curve has been added so that the R, G1, and B filters corresponding to R, G, and B in FIG. Green G2 filter is added).

  Further, as a filter evaluation coefficient used when determining the four-color filter 61, for example, UMG (Unified Measure of Goodness) which is a coefficient considering both “color reproducibility” and “noise reduction performance” is used. It has been.

  In the evaluation using the UMG, the evaluation value does not increase when the filter to be evaluated simply satisfies the router condition, and the overlap of spectral sensitivity distributions of the respective filters is also taken into consideration. Therefore, noise can be further reduced as compared with the case of a color filter evaluated using q factor, μ factor, or FOM (Figure of Merit). That is, by the evaluation using the UMG, a filter is selected in which the spectral sensitivity characteristics of the respective filters have a certain degree of overlap, but almost all do not overlap, such as the R characteristic and the G characteristic in FIG. Therefore, even if each color signal is amplified for color separation, it is not necessary to increase the amplification factor so much that the noise component is suppressed from being amplified accordingly.

  The reason why noise is suppressed by the fourth filter (G2 filter) will be described. Spectral sensitivity of the primary color filter is improved in order to improve sensitivity efficiency as the CCD cell size is miniaturized to increase the number of pixels. The curve tends to be thicker and the overlap of the filters tends to increase. Adding another filter under such circumstances has the effect of suppressing the overlap of the original three primary colors and consequently preventing noise.

  FIG. 11 is a diagram illustrating characteristics of each filter evaluation coefficient. FIG. 11 shows, for each evaluation coefficient, the number of filters that can be evaluated at one time, whether or not the spectral reflectance of an object is taken into consideration, and whether or not noise reduction is taken into consideration. Yes.

  As shown in FIG. 11, the number of filters that can be evaluated at one time is only “1”, and the spectral reflectance of the object and noise reduction are not considered. In addition, although the μ factor can evaluate a plurality of filters at a time, reduction of spectral reflectance and noise of an object is not taken into consideration. Furthermore, the FOM can evaluate a plurality of filters at a time and considers the spectral reflectance of the object, but does not consider noise reduction.

  On the other hand, the UMG used when determining the four-color color filter 61 can evaluate a plurality of filters at a time, considering the spectral reflectance of the object, and considering noise reduction. Has been.

  Details of the q factor are disclosed in “HEJNeugebauer“ Quality Factor for Filters Whose Spectral Transmittances are Different from Color Mixture Curves, and Its Application to Color Photography ”JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, VOLUME 46, NUMBER 10.” The details of the μ factor are disclosed in “PLVora and HJ Trussell,“ Measure of Goodness of a set of color-scanning filters ”, JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, VOLUME 10, NUMBER 7.” Yes. Details of FOM are disclosed in “G. Sharma and HJ Trussell,“ Figures of Merit for Color Scanners, IEEE TRANSACTION ON IMAGE PROCESSING, VOLUME 6 ”, and UMG is described in“ S. Quan, N. Ohta , and N. Katoh, “Optimal Design of Camera Spectral Sensitivity Functions Based on Practical Filter Components”, CIC, 2001 ”.

  Returning to the description of FIG. 8, the microcomputer 41 controls the overall operation according to a predetermined control program. For example, the microcomputer 41 controls the exposure control by the diaphragm 43, the opening / closing control of the shutter 44, the electronic shutter control of the TG (Timing Generator) 46, the gain control at the front end 47, and the camera system LSI (Large Scale Integrated Circuit) 48. Perform mode control, parameter control, etc.

  The aperture 43 adjusts the passage (aperture) of the light collected by the lens 42 and controls the amount of light taken in by the image sensor 45. The shutter 44 controls the passage of the light collected by the lens 42 based on an instruction from the microcomputer 41.

  The image sensor 45 further includes an image sensor composed of CCD or CMOS (Complementary Metal Oxide Semiconductor), and converts the light incident through the four-color filter 61 formed on the front surface of the image sensor into an electrical signal. The four color signals (R signal, G1 signal, G2 signal, B signal) are output to the front end 47 after conversion. The color sensor 61 of FIG. 9 is provided in the image sensor 45, and the components of the wavelengths in the respective bands of R, G1, G2, and B from the light incident through the lens 42 (the details are shown in FIG. 18 will be described later with reference to FIG.

  The front end 47 performs correlated double sampling processing for removing noise components, gain control processing, digital conversion processing, and the like on the color signal supplied from the image sensor 45. Various processes are performed by the front end 47, and the obtained image data is output to the camera system LSI 48.

  As will be described later in detail, the camera system LSI 48 performs various processes on the image data supplied from the front end 47, generates, for example, a luminance signal and a color signal, and outputs them to the image monitor 50. The image corresponding to is displayed.

  The image memory 49 is configured by, for example, a DRAM (Dynamic Random Access Memory), an SDRAM (Synchronous Dynamic Random Access Memory), and the like, and is appropriately used when the camera system LSI 48 performs various processes. The external storage medium 51 configured by a semiconductor memory, a disk, and the like is configured to be detachable from the digital camera of FIG. 8, for example, and image data compressed in the JPEG (Joint Photographic Expert Group) format by the camera system LSI 48 is stored. Remembered.

  The image monitor 50 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and displays captured images, various menu screens, and the like.

  FIG. 12 is a block diagram showing a configuration example of the camera system LSI 48 of FIG. Each block constituting the camera system LSI 48 is controlled by the microcomputer 41 in FIG. 8 via a microcomputer interface (I / F) 73.

  The signal processing unit 71 performs various kinds of processing such as interpolation processing, filtering processing, matrix calculation processing, luminance signal generation processing, and color difference signal generation processing on the four types of color information supplied from the front end 47, For example, the generated image signal is output to the image monitor 50 via the monitor interface 77.

  The image detection unit 72 performs detection processing such as auto focus, auto exposure, and auto white balance based on the output of the front end 47, and outputs the result to the microcomputer 41 as appropriate.

  The memory controller 75 controls transmission / reception of data between processing blocks, or transmission / reception of data between a predetermined processing block and the image memory 49. For example, the image data supplied from the signal processing unit 71 is stored in the memory controller 75. The data is output to the image memory 49 via the interface 74 and stored.

  For example, the image compression / decompression unit 76 compresses the image data supplied from the signal processing unit 71 in the JPEG format, and outputs the obtained data to the external storage medium 51 via the microcomputer interface 73 for storage. . The image compression / decompression unit 76 also decompresses (decompresses) the compressed data read from the external storage medium 51 and outputs it to the image monitor 50 via the monitor interface 77.

  FIG. 13 is a block diagram illustrating a detailed configuration example of the signal processing unit 71 in FIG. 12. Each block constituting the signal processing unit 71 is controlled by the microcomputer 41 via the microcomputer interface 73.

  The offset correction processing unit 91 removes a noise component (offset component) included in the image signal supplied from the front end 47 and outputs the obtained image signal to the white balance correction processing unit 92. The white balance correction processing unit 92 corrects the balance of each color based on the color temperature of the image signal supplied from the offset correction processing unit 91 and the difference in sensitivity of each filter of the four-color filter 61. The white color correction processing unit 92 performs correction, and the acquired color signal is output to the vertical direction synchronization processing unit 93. The vertical direction synchronization processing unit 93 is provided with a delay element, and the time shift in the vertical direction of the signal output from the white balance correction processing unit 92 is synchronized (corrected).

  The signal generation processing unit 94 interpolates the RG1G2B minimum unit 2 × 2 pixel color signal supplied from the vertical direction synchronization processing unit 93 into the phase of the same space, and removes noise components of the signal. Noise removal processing, filtering processing for limiting the signal band, high frequency correction processing for correcting the high frequency component of the signal band, and the like are performed, and the obtained RG1G2B signal is output to the linear matrix processing unit 95.

  The linear matrix processing unit 95 performs an operation on the RG1G2B signal based on a predetermined matrix coefficient (3 × 4 matrix) according to the following equation (1) to generate an RGB signal of three colors.

  The R signal generated by the linear matrix processing unit 95 is output to the gamma correction processing unit 96-1, the G signal is output to the gamma correction processing unit 96-2, and the B signal is output to the gamma correction processing unit 96-3. The

  The gamma correction processing units 96-1 to 96-3 perform gamma correction on each of the RGB signals output from the linear matrix processing unit 95, and the acquired RGB signals are subjected to luminance (Y) signal generation processing. Output to the unit 97 and the color difference (C) signal generation processing unit 98.

  The luminance signal generation processing unit 97 combines the RGB signals supplied from the gamma correction processing units 96-1 to 96-3 with a predetermined combining ratio according to the following equation (2), for example, to generate a luminance signal. .

  Similarly, the color difference signal generation processing unit 98 combines the RGB signals supplied from the gamma correction processing units 96-1 to 96-3 with a predetermined combination ratio to generate color difference signals (Cb, Cr). The luminance signal generated by the luminance signal generation processing unit 97 and the color difference signal generated by the color difference signal generation processing unit 98 are output to the image monitor 50 via the monitor interface 77 of FIG. 12, for example.

  In the digital camera having the above-described configuration, when an instruction to capture an image is given, the microcomputer 41 controls the TG 46 and causes the image sensor 45 to capture an image. That is, light of four colors is transmitted by the four-color filter 61 formed on the front surface of the image sensor such as a CCD constituting the image sensor 45, and the transmitted light is captured by the CCD image sensor. The light captured by the CCD image sensor is converted into four color signals, which are output to the front end 47.

  The front end 47 performs correlated double sampling processing for removing noise components, gain control processing, digital conversion processing, and the like on the color signal supplied from the image sensor 45, and outputs the obtained image data. Output to the camera system LSI 48.

  In the signal processing unit 71 of the camera system LSI 48, the offset component of the color signal is removed by the offset correction processing unit 91, and the color balance of the image signal and each filter of the four-color color filter 61 are processed by the white balance correction processing unit 92. Based on the difference in sensitivity, the balance of each color is corrected.

  Further, the vertical direction synchronization processing unit 93 synchronizes (corrects) the time lag in the vertical direction of the signal corrected by the white balance correction processing unit 92, and the signal generation processing unit 94 performs the vertical direction synchronization processing unit. An interpolation process for interpolating the RG1G2B minimum unit 2 × 2 pixel color signal supplied from 93 to the phase of the same space, a noise removal process for removing noise components of the signal, a filtering process for limiting the signal band, and High frequency correction processing for correcting the high frequency component of the signal band is performed.

  Further, in the linear matrix processing unit 95, the signal (RG1G2B signal) generated by the signal generation processing unit 94 is converted based on a predetermined matrix coefficient (3 × 4 matrix) to generate an RGB signal of three colors. Is done. The R signal generated by the linear matrix processing unit 95 is output to the gamma correction processing unit 96-1, the G signal is output to the gamma correction processing unit 96-2, and the B signal is output to the gamma correction processing unit 96-3. The

  The gamma correction processing units 96-1 to 96-3 perform gamma correction on each of the RGB signals obtained by the processing of the linear matrix processing unit 95, and the acquired RGB signals are converted into luminance signal generation processing units. 97 and the color difference signal generation processing unit 98. In the luminance signal generation processing unit 97 and the color difference signal generation processing unit 98, each of the R signal, the G signal, and the B signal supplied from the gamma correction processing units 96-1 to 96-3 has a predetermined combination ratio. Are combined to generate a luminance signal and a color difference signal. The luminance signal generated by the luminance signal generation processing unit 97 and the color difference signal generated by the color difference signal generation processing unit 98 are output to the image compression / decompression unit 76 of FIG. 12, and are compressed, for example, in the JPEG format. The compressed image data obtained is output to the external storage medium 51 via the microcomputer interface 73 and stored.

  As described above, since one image data is formed based on four types of color signals, the reproducibility is closer to the appearance of human eyes.

  On the other hand, when the reproduction (display) of the image data stored in the external storage medium 51 is instructed, the microcomputer 41 reads out the image data stored in the external storage medium 51, which is read out by the camera system LSI 48. Are output to the image compression / decompression unit 76. In the image compression / decompression unit 76, the compressed image data is expanded, and an image corresponding to the obtained data is displayed on the image monitor 50 via the monitor interface 77.

  Next, processing (procedure) for creating a digital camera having the above-described configuration will be described with reference to the flowchart of FIG.

  In step S1, four-color filter determination processing for determining the spectral sensitivity characteristics of the four-color filter 61 provided in the image sensor 45 of FIG. 8 is performed. In step S2, the color matrix processing unit 95 of FIG. A linear matrix determination process for determining the matrix coefficient is performed. Refer to the flowchart of FIG. 15 for details of the four-color filter determination process executed in step S1, and refer to the flowchart of FIG. 19 for details of the linear matrix determination process executed in step S2. Each will be described later.

  After the four-color filter 61 is determined and the matrix coefficients are determined, in step S3, the signal processing unit 71 in FIG. 13 is created, and the process proceeds to step S4, where the camera system LSI 48 in FIG. 12 is created. In step S5, the entire image processing apparatus (digital camera) as shown in FIG. 8 is created. In step S6, the image quality ("color reproducibility", "color discrimination") of the digital camera created in step S5 is evaluated, and the process ends.

  Here, object colors referred to when evaluating “color reproducibility”, “color discrimination”, and the like will be described. The object color is the product of “spectral reflectance of the object”, “spectral energy distribution of standard illumination”, and “spectral sensitivity distribution (characteristic) of the sensor (color filter) that senses the object”. It is calculated by the value integrated in the range of 700 nm). That is, the object color is calculated by the following equation (3).

  For example, when a predetermined object is observed with the eyes, the “spectral sensitivity characteristic of the sensor” in Expression (3) is represented by a color matching function, and the object color of the object is represented by tristimulus values of X, Y, and Z. Is done. Specifically, the value of X is calculated by equation (4-1), the value of Y is calculated by equation (4-2), and the value of Z is calculated by equation (4-3). Note that the value of the constant k in the equations (4-1) to (4-3) is calculated by the equation (4-4).

Also, when an image of a predetermined object is captured by an image processing apparatus such as a digital camera, the “spectral sensitivity characteristic of the sensor” in the above equation (3) is represented by the spectral sensitivity characteristic of the color filter, and the object color of the object is The object color of the color value of the number of filters (for example, RGB values (three values in the case of RGB filters (three types)) is calculated. When the image processing apparatus is provided with an RGB filter that detects three types of colors, specifically, the value of R is calculated by Expression (5-1), and the value of G is calculated by Expression (5-2). And the value of B is calculated by equation (5-3). The value of the constant k _r in the formula (5-1) is calculated by the formula (5-4), the value of the constant k _g in the formula (5-2) is calculated by the formula (5-5), the formula ( the value of the constant k _b in 5-3) is calculated by the equation (5-6).

  Next, the four-color filter determining process performed in step S1 of FIG. 14 will be described with reference to the flowchart of FIG.

  There are various methods for determining the four-color filter. For example, based on an RGB filter (one of the existing G filters (FIG. 1) as a G1 filter), A process of selecting a G2 filter that transmits a highly correlated color and adding it to determine a four-color filter will be described.

  In step S21, a color target used to calculate a UMG value is selected. For example, in step S21, a color target that includes many color patches representing existing colors and includes many color patches that emphasize human memory colors (skin color, plant green, sky blue, etc.) is selected. . Examples of the color target include IT8.7, Macbeth Color Checker, GretagMacbeth DigitalCamera Color Checker, CIE, and Color Bar.

  Also, depending on the purpose, a color patch that can be a standard may be created from data such as SOCS (Standard Object Color Spectra Database) and used. The details of SOCS are disclosed in “Joji Tajima,“ Statistical Color Reproduction Evaluation by Standard Object Color Spectroscopy Database (SOCS) ”, Color Forum JAPAN 99”. Hereinafter, a case where Macbeth Color Checker is selected as a color target will be described.

In step S22, the spectral sensitivity characteristic of the G2 filter is determined. As the spectral sensitivity characteristics, those that can be created from existing materials may be used, or a virtual curve C (λ) is assumed with a cubic spline curve (cubic spline function) as shown in FIG. Peak value λ _{0 of} virtual curve C (λ), value w (value obtained by dividing the sum of w ₁ and w ₂ by 2), value Δw (value obtained by subtracting w ₂ from w ₁ and divided by 2) Those changed in the range shown in the figure may be used. The values of w and Δw are values based on the half-value width value. The method of changing λ ₀ , w, Δw is, for example, in increments of 5 nm. The virtual curve C (λ) is expressed by the following equations (6-1) to (6-5) in each range.

  In this example, only the filter G2 is added, but only the R and B filters of the filters (R, G, G, B) in FIG. 1 are used, and the remaining two G1, G2 filters are replaced with green. It can also be defined as a virtual curve of the above equations (6-1) to (6-5). Similarly, only R and G, and only G and B may be used from the filter of FIG. Furthermore, among the four color filters, three colors can be defined as virtual curves, or all four colors can be defined as virtual curves.

  In step S23, the filter to be added (G2 filter) and the existing filters (R filter, G1 filter, B filter) are combined to create a minimum unit (set) of the four-color filter. In step S24, UMG is used as a filter evaluation coefficient for the four-color filter created in step S23, and a UMG value is calculated.

  As described with reference to FIG. 11, when UMG is used, evaluation can be performed once for each of the four color filters. Further, the evaluation is performed not only in consideration of the spectral reflectance of the object but also in consideration of noise reduction. In the evaluation using UMG, the spectral sensitivity characteristic of each filter is highly evaluated for a filter having a moderate overlap. For example, the characteristic in which the R characteristic and the G characteristic overlap over a wide wavelength band. It can be suppressed that a high evaluation is shown for a filter having noise (a filter in which noise is amplified when each color signal is separated).

  FIG. 17 is a diagram illustrating an example of the UMG value calculated in the three-color filter. For example, in the filter having the characteristics as shown in FIG. 17A in which the RGB characteristics do not overlap, a UMG value of “0.7942” is calculated, and the R characteristics and the G characteristics overlap over a wide wavelength band. In the filter having the characteristic as shown in FIG. 5, a UMG value of “0.8211” is calculated. In addition, a UMG value of “0.8879” is calculated in a filter having characteristics as shown in FIG. That is, the highest evaluation is shown for a filter having characteristics as shown in FIG. The same applies to the four-color color filter. The curve L31 in FIG. 17A, the curve L41 in FIG. 17B, and the curve L51 in FIG. 17C indicate the spectral sensitivity of R, and the curve L32 in FIG. 17A, the curve L42 in FIG. 17B, and the curve L52 in FIG. 17A, a curve L43 in FIG. 17B, and a curve L53 in FIG. 17C represent the spectral sensitivity of B, respectively.

  In step S25, it is determined whether or not the UMG value calculated in step S24 is greater than or equal to a predetermined threshold value “0.95”. If it is determined that the UMG value is less than “0.95”, step S26 is performed. The created four-color filter is rejected (not used). If the four-color filter is rejected in step S26, then the process is terminated (the processes after step S2 in FIG. 14 are not executed).

  On the other hand, if it is determined in step S25 that the UMG value calculated in step S24 is “0.95” or more, in step S27, the four-color filter is used as a candidate filter used in the digital camera. Is done.

  In step S28, it is determined whether or not the four-color filter selected as a candidate filter in step S27 can be realized with existing materials and dyes. If it is difficult to obtain materials, dyes, and the like, it is determined that the material cannot be realized, and the process proceeds to step S26 where the four-color filter is rejected.

  On the other hand, if it is determined in step S28 that materials, dyes, and the like can be acquired and can be realized, the process proceeds to step S29, and the created four-color filter is determined as a filter used in the digital camera. The Thereafter, the processes after step S2 in FIG. 14 are executed.

  FIG. 18 is a diagram illustrating an example of the spectral sensitivity characteristics of the four-color filter determined in step S29.

  In FIG. 18, a curve L61 represents the spectral sensitivity of R, and a curve L62 represents the spectral sensitivity of G1. A curve L63 represents the spectral sensitivity of G2, and a curve L64 represents the spectral sensitivity of B. As shown in FIG. 18, the spectral sensitivity curve of G2 (curve L63) is highly correlated with the spectral sensitivity curve of G1 (curve L62). In addition, the spectral sensitivity of R, the spectral sensitivity of G (G1, G2), and the spectral sensitivity of B overlap each other in an appropriate range. The characteristic shown in FIG. 18 is obtained by adding the characteristic of G2 to the characteristic of the three-color filter shown in FIG.

  By using the four-color filter determined as described above, it is possible to improve “color discrimination” among “color reproducibility”.

  From the viewpoint of light utilization efficiency, it is preferable to add a filter having a high correlation with the G filter of the existing RGB filter as an additional filter (G2 filter) as described above. In this case, it is desirable that the peak value of the spectral sensitivity curve of the added filter is empirically in the range of 495 to 535 nm (near the peak value of the spectral sensitivity curve of the existing G filter).

  In addition, when adding a filter having a high correlation with the existing G filter, only one of the two G filters constituting the minimum unit (R, G, G, B) in FIG. Since a four-color filter can be created, it is not necessary to make a major change in the creation process.

  When a four-color filter is created as described above and provided in a digital camera, four types of color signals are supplied from the signal generation processing unit 94 to the signal processing device 71 in FIG. In the matrix processing unit 95, conversion processing for generating signals of three colors (R, G, B) from signals of four colors (R, G1, G2, B) is performed. Since this conversion process is a matrix process for input signal values that are linear in luminance (brightness values can be expressed by linear conversion), hereinafter, the conversion process performed in the linear matrix processing unit 95 is appropriately changed to a linear matrix. This is called processing.

  Next, the linear matrix determination process executed in step S2 of FIG. 14 will be described with reference to the flowchart of FIG.

  Note that the color target used in the processing of FIG. 19 is Macbeth Color Checker, and the four-color filter used has the spectral sensitivity characteristics shown in FIG.

  In step S41, for example, general daylight D65 (illumination light L (λ)), which is a standard light source in CIE (Commision Internationale del'Eclairange), is selected as illumination light. The illumination light may be changed to illumination light or the like in an environment where the image processing apparatus is expected to be frequently used. In addition, when there are a plurality of assumed lighting environments, it is conceivable to prepare a plurality of linear matrices. Hereinafter, a case where daylight D65 is selected as illumination light will be described.

In step S42, reference values (reference values) _Xr , _Yr , _Zr are calculated. Specifically, the reference value X _r is calculated by the equation (7-1), Y _r is calculated by the equation (7-2), Z _r is calculated by the equation (7-3).

  Further, the constant k is calculated by the equation (8).

例えば、カラーターゲットがMacbeht Color Checkerの場合、２４色分のリファレンス値が算出される。

For example, when the color target is Macbeht Color Checker, reference values for 24 colors are calculated.

Next, in step S43, the output value R _f of 4-color _{filter, G1 f, G2 f, B} f is calculated. Specifically, R _f is calculated by equation (9-1), G1 _f is calculated by equation (9-2), G2 _f is calculated by equation (9-3), and B _f is calculated by equation (9). -4).

Also, the constant k _r is calculated by the equation (10-1), the constant k _g1 is calculated by the equation (10-2), the constant k _g2 is calculated by the equation (10-3), the constant k _b is the formula ( 10-4)

例えば、カラーターゲットがMacbeht Color Checkerの場合は、２４色分の出力値Ｒ_f，Ｇ１_f，Ｇ２_f，Ｂ_fが算出される。

For example, when the color target is Macbeht Color Checker, output values R _f , G 1 _f , G 2 _f , and B _f for 24 colors are calculated.

In step S44, a matrix for performing conversion for approximating the filter output value calculated in step S43 to the reference value (XYZ _ref ) calculated in step S42 is calculated by, for example, the error least square method in the XYZ color space. .

For example, when the calculated 3 × 4 matrix is A represented by Expression (11), the matrix transformation (XYZ _exp ) is represented by the following Expression (12).

Further, the square (E ² ) of the error of the matrix conversion with respect to the reference value (expression (12)) is expressed by the following expression (13), and based on this, the matrix that minimizes the error of the matrix conversion with respect to the reference value. A is calculated.

  Further, the color space used in the error least square method may be changed to a color space other than the XYZ color space. For example, a linear matrix that enables reproduction of colors with little perceptual error by performing the same calculation after converting to Lab, Luv, Lch color space (perceptual uniform color space) equivalent to human perception Can be calculated. Since these color space values are calculated from the XYZ values by non-linear conversion, a non-linear calculation algorithm is also used in the least-squares error method.

  By the calculation as described above, for example, the matrix coefficient for the filter having the spectral sensitivity characteristic shown in FIG. 18 is calculated by Expression (14).

  In step S45, a linear matrix is determined. For example, when the final RGB image data to be created is expressed by the following equation (15), the linear matrix (LinearM) is calculated as follows.

  That is, when the illumination light is D65, the conversion equation for converting the sRGB color space to the XYZ color space is expressed by equation (16) including the ITU-R709.BT matrix, and is expressed by the inverse matrix of the ITU-R709.BT matrix. Equation (17) is calculated.

  Expression (18) is calculated from the matrix conversion expression of Expression (12) and the inverse matrix of the ITU-R709.BT matrix of Expression (15) and Expression (17). The right side of Equation (18) includes a linear matrix as a value obtained by multiplying the inverse matrix of the ITU-R709.BT matrix and the matrix A described above.

  That is, the 3 × 4 linear matrix (LinearM) is expressed by Expression (19-1). For example, the linear matrix for the four-color filter having the spectral distribution characteristics of FIG. 18 using the matrix coefficient of Expression (14) is used. Is represented by Formula (19-2).

  The linear matrix calculated as described above is given to the linear matrix processing unit 95 in FIG. As a result, matrix processing can be performed on the signals (R, G1, G2, B) whose luminance can be expressed by linear conversion, and therefore, as in the processing in the signal processing unit 11 shown in FIG. Compared with a case where matrix processing is performed on a signal obtained after processing, it is possible to reproduce more faithful colors in terms of color engineering.

  Next, the evaluation performed in step S6 of FIG. 14 will be described.

  For example, the color reproducibility of the image processing apparatus (digital camera in FIG. 8) provided with the four-color color filter having the spectral sensitivity characteristics shown in FIG. 18 and the three colors shown in FIG. When comparing the color reproducibility of image processing apparatuses provided with color filters, the following differences appear.

  For example, the color difference in the Lab color space between the output value and the reference value when a Macbeth chart is imaged by two types of image input devices (a digital camera provided with a four-color filter and a digital camera provided with a three-color filter) Are calculated by the following equation (20).

Ｌ₁−Ｌ₂は２つの試料の明度差であり、ａ₁−ａ₂、ｂ₁−ｂ₂は２つの試料の色相・彩度の成分差を表している。

L ₁ -L ₂ is the brightness difference between the two samples, and a ₁ -a ₂ and b ₁ -b ₂ represent the hue / saturation component differences between the two samples.

  FIG. 20 is a diagram illustrating a calculation result according to the equation (20). As shown in FIG. 20, in the case of a digital camera provided with a three-color filter, the color difference is “3.32”, whereas in the case of a digital camera provided with a four-color filter, “1.39”. In terms of “color appearance”, a digital camera provided with a four-color filter is superior (color difference is small).

  FIG. 21 is a diagram showing RGB values when the object R1 and the object R2 having the spectral reflectance shown in FIG. 6 are photographed by a digital camera provided with a four-color filter.

  In FIG. 21, the R value of the object R1 is “49.4”, the G value is “64.1”, the B value is “149.5”, the R value of the object R2 is “66.0”, G The value is “63.7” and the B value is “155.6”. Therefore, as described above, when the image is taken with the three-color filter, the RGB value is as shown in FIG. 7B, and the color of each object is not identified. The RGB values of the object R1 and the object R2 are different from each other, and FIG. 21 shows that the colors of the respective objects are identified as in the case of viewing with the eyes (FIG. 7A). That is, “color discrimination” is improved by providing a filter that can identify four types of colors.

  In the above, as shown in FIG. 9, the four-color filter 61 is configured by an arrangement in which B filters are provided on the left and right sides of the G1 filter and R filters are provided on the left and right sides of the G2 filter. However, it may be configured by an arrangement as shown in FIG. In the four-color filter 61 shown in FIG. 22, R filters are provided on the left and right sides of the G1 filter, and B filters are provided on the left and right sides of the G2 filter. By configuring the four-color filter 61 in this way, “color discriminability”, “color reproducibility”, and “noise reduction” can be improved as in the case shown in FIG. it can.

It is a figure which shows the example of the conventional RGB color filter. It is a block diagram which shows the structural example of the signal processing part provided in the conventional digital camera. It is a figure which shows the example of a spectral sensitivity characteristic. It is a figure which shows the other example of a spectral sensitivity characteristic. It is a figure which shows the further another example of a spectral sensitivity characteristic. It is a figure which shows the spectral reflectance of a predetermined | prescribed object. It is a figure which shows the example of the color value of a predetermined | prescribed object. It is a block diagram which shows the structural example of the digital camera to which this invention is applied. It is a figure which shows the example of the 4 color filter provided in the digital camera of FIG. It is a figure which shows the example of a visibility curve. It is a figure which shows the characteristic of an evaluation coefficient. It is a block diagram which shows the structural example of the camera system LSI of FIG. It is a block diagram which shows the structural example of the signal processing part of FIG. It is a flowchart explaining the creation processing of an image processing apparatus. It is a flowchart explaining the detail of the 4-color filter decision process of step S1 of FIG. It is a figure which shows the example of a virtual curve. It is a figure which shows the example of the UMG value for every filter. It is a figure which shows the example of the spectral sensitivity characteristic of a 4 color filter. It is a flowchart explaining the detail of the linear matrix determination process of step S2 of FIG. It is a figure which shows the example of the evaluation result of a color difference. It is a figure which shows the chromaticity of the predetermined | prescribed object by a 4 color filter. It is a figure which shows the other example of the 4 color filter provided in the digital camera of FIG.

Explanation of symbols

  45 image sensor, 61 four-color filter, 71 signal processing unit, 94 signal generation processing unit, 95 linear matrix processing unit

Claims (6)

Extraction means for extracting a component of a predetermined color from incident light;
In a method for manufacturing an image processing apparatus, comprising: conversion means for converting light of a color component extracted by the extraction means into a corresponding color signal;
A first step of preparing the conversion means;
As the extraction means, first to third light of three primary colors and fourth light having high correlation with the second light of the first to third lights of the three primary colors are extracted. The second extraction unit and the fourth extraction unit each have a spectral sensitivity characteristic that approximates a visibility characteristic of a luminance signal, and each unit includes first to fourth extraction units. And a second step of generating an extraction means located diagonally on the front surface of the conversion means prepared by the processing of the first step. The manufacturing method according to claim 1, wherein the second step determines a spectral sensitivity characteristic of the extraction unit using a predetermined evaluation coefficient. The manufacturing method according to claim 2, wherein the evaluation coefficient is an evaluation coefficient in consideration of noise reducibility as well as color reproducibility. The second step is characterized in that the first to third lights of the three primary colors are red, green, or blue light, and the fourth light is green light. The manufacturing method as described in. Generation means for generating fifth to seventh color signals corresponding to the three primary colors based on the first to fourth color signals generated by converting the first to fourth lights by the conversion means. The manufacturing method according to claim 1, further comprising a third step of generating The third step minimizes a difference in a predetermined evaluation value between a reference value calculated based on a predetermined color patch and an output value calculated based on the spectral sensitivity characteristic of the extraction unit based on the color patch. 6. The manufacturing method according to claim 5, wherein the generation means is generated so as to generate the fifth to seventh color signals based on a conversion formula given as:

Images