JP4764794B2 - Image processing apparatus, endoscope apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus that performs image processing on an imaging signal output from an imaging element.

  Endoscope apparatuses using a CCD type or CMOS type image sensor as an image sensor are already widely used in the medical field. This endoscope apparatus is roughly classified into an imaging device capable of monochrome imaging, and R (red), G (green), B (blue), and IR (red) in front of a light source that illuminates a subject via a fiber. (Outside) a filter that transmits light in the wavelength region is switched in synchronization with the field frequency of the image sensor (see, for example, Patent Document 1), the illumination light source is white light, and R, G, B There is a simultaneous imaging method in which an image is picked up using a single-plate image sensor equipped with a color filter that transmits light in the wavelength band.

  In the frame sequential imaging method, a plurality of filters having different spectral transmittances are rotated in front of a light source, and a plurality of images illuminated with light of different wavelengths are captured, and then a color image is synthesized. For this reason, for example, if an RGB transmission filter is used as a filter to be switched in front of the light source, RGB color image data in which one pixel data has three color information of RGB can be obtained and absorbed by hemoglobin in blood. Infrared image data in which only 1-pixel data is provided with only information in the infrared region can be obtained by sequentially switching easily the narrow-band, two-wavelength IR filters. According to the image based on the RGB color image data, the appearance of the part to be examined can be visually confirmed. According to the image based on the infrared image data, the capillaries on the mucous membrane surface layer of the part to be examined and Information on the fine mucosa pattern can be visually confirmed. However, in this surface sequential imaging, color shift occurs and the image is disturbed for a moving subject.

  On the other hand, the simultaneous imaging method is a method in which after obtaining RGB color image data by imaging, infrared image data is generated by performing image processing on the RGB color image data. However, there is a problem that the information accuracy of infrared image data is low.

Japanese Patent No. 2648494

  As described above, in the imaging element used in the endoscope apparatus, RGB color image data in which three pieces of RGB color information are given to one pixel data and one pixel data by the element configuration and image processing described in Patent Document 1. It is possible to obtain two pieces of image data including infrared image data having only information in the infrared region.

  Here, the color reproducibility of RGB color image data is considered. Usually, R, G, and B color filters also transmit wavelengths in the infrared region. Therefore, when RGB color image data is generated by detecting light transmitted through the R, G, and B color filters with a photoelectric conversion element. The color reproducibility is not good. Therefore, in an image pickup device using R, G, and B color filters, an infrared (IR) cut filter is provided in front of the image pickup device, and infrared light is transmitted to the light that has passed through the R, G, and B color filters. The color reproducibility is improved by not including.

  However, an IR cut filter having a steep IR cut characteristic is expensive and expensive. In addition, an IR cut filter must be provided on the front surface of the image sensor, which may be a factor that hinders downsizing of the entire system using the image sensor. Further, if an IR cut filter is to be used in the apparatus described in Patent Document 1, it is necessary to place an IR cut filter in front of the filter only when the R, G, B color filters come to the front of the image sensor. Yes, the system mechanism and control become complicated.

  The present invention has been made in view of the above circumstances, and can generate RGB color image data with good color reproducibility from an image pickup signal from the image pickup element without providing an IR cut filter in front of the image pickup element. An object is to provide a possible image processing apparatus.

(1) An image processing apparatus that generates image data from an image pickup signal output from an image pickup device, the image pickup signal of an R (red) component, the image pickup signal of a G (green) component output from the image pickup device, and RGB color image data generating means for generating RGB color image data from B (blue) component imaging signals, and infrared for generating infrared image data from IR (infrared) component imaging signals output from the imaging device High color reproduction RGB color image for generating high color reproduction RGB color image data with improved color reproducibility using the RGB color image data and the infrared image data, using the RGB color image data and the infrared image data a data generation unit, the imaging device includes a plurality of photoelectric conversion elements arranged on the same surface of the semiconductor substrate, it is formed on the same surface of the upper side of the semiconductor substrate A photoelectric conversion element on a substrate corresponding to a part of the plurality of photoelectric conversion elements, the first electrode formed above the semiconductor substrate, the photoelectric conversion layer formed on the first electrode, and The on-substrate photoelectric conversion element configured to include the second electrode formed on the photoelectric conversion layer is different from the wavelength range of light formed above the semiconductor substrate and absorbed by the photoelectric conversion layer. A color filter layer that transmits light in a wavelength region, and a signal reading unit that reads a signal corresponding to the charge generated in the photoelectric conversion element on the substrate and a signal corresponding to the charge generated in the photoelectric conversion element, and The color filter layer includes a number of color filters corresponding to each of the number of photoelectric conversion elements, and the number of color filters includes a color filter that transmits light in a wavelength region of R, and light in a wavelength region of G. There are three types of color filters: a color filter that transmits light and a color filter that transmits light in the B wavelength range, and the color filter that transmits light in at least the R wavelength range among the three color filters is red. The light in the outer region is also transmitted, the photoelectric conversion layer absorbs the light in the infrared region, generates a charge corresponding thereto, and transmits the other light, and a part of the plurality of photoelectric conversion elements. Is an image processing apparatus that is a photoelectric conversion element corresponding to a color filter that transmits light in the R wavelength range .

(2) The image processing apparatus according to (1), wherein the pixel data of the RGB color image data is R data that is R component data. G data that is G component data and B data that is B component data, and pixel data of the infrared image data is IR data that is IR component data, and the high color reproduction RGB The color image data generation means is a value obtained by multiplying R data constituting pixel data at the same position of each of the RGB color image data and the infrared image data by a coefficient r2, and a value obtained by multiplying the G data by a coefficient g2. And the value obtained by multiplying the B data by the coefficient b2 and the value obtained by multiplying the IR data by the coefficient ir2 to generate R component data constituting one pixel data of the high color reproduction RGB color image data. A value obtained by multiplying the R data by a coefficient r3, a value obtained by multiplying the G data by a coefficient g3, a value obtained by multiplying the B data by a coefficient b3, and a value obtained by multiplying the IR data by a coefficient ir3. do it G component data constituting one pixel data of the high color reproduction RGB color image data is generated, a value obtained by multiplying the R data by a coefficient r4, a value obtained by multiplying the G data by a coefficient g4, and the B data Is multiplied by a coefficient b4 and a value obtained by multiplying the IR data by a coefficient ir4 to generate B component data constituting one pixel data of the high color reproduction RGB color image data, and the coefficient r2, g2, b2, and ir2 are values obtained by multiplying the R sensitivity, which is the spectral sensitivity of the photoelectric conversion element that outputs the R component imaging signal among the photoelectric conversion elements constituting the imaging element, by the coefficient r2. A value obtained by multiplying the G sensitivity, which is the spectral sensitivity of the photoelectric conversion element that outputs the G component imaging signal among the photoelectric conversion elements that constitute the imaging element, by the coefficient g2, and among the photoelectric conversion elements that constitute the imaging element of A value obtained by multiplying the B sensitivity, which is the spectral sensitivity of the photoelectric conversion element that outputs the B component imaging signal, by the coefficient b2, and the IR component imaging signal among the photoelectric conversion elements that constitute the imaging element are output. A value obtained by integrating the IR sensitivity, which is the spectral sensitivity of the infrared detection photoelectric conversion element, multiplied by the coefficient ir2, and the ideal spectral sensitivity value of the photoelectric conversion element that outputs the R component imaging signal; Are determined to be closest to each other, and the coefficients r3, g3, b3, ir3 are values obtained by multiplying the R sensitivity by the coefficient r3, values obtained by multiplying the G sensitivity by the coefficient g3, and the B A value obtained by multiplying a value obtained by multiplying the sensitivity by the coefficient b3 and a value obtained by multiplying the IR sensitivity by the coefficient ir3, and an ideal spectral sensitivity value of the photoelectric conversion element that outputs the G component imaging signal, Was decided to be the closest The coefficients r4, g4, b4, and ir4 are values obtained by multiplying the R sensitivity by the coefficient r4, values obtained by multiplying the G sensitivity by the coefficient g4, values obtained by multiplying the B sensitivity by the coefficient b4, And the value obtained by multiplying the IR sensitivity multiplied by the coefficient ir4 and the ideal spectral sensitivity value of the photoelectric conversion element that outputs the B component imaging signal are determined to be closest to each other. An image processing apparatus.

(3) The image processing apparatus according to (1) or (2) , wherein the color filter layer is formed above the photoelectric conversion element on the substrate.

(4) The image processing apparatus according to (3) , wherein the photoelectric conversion layer includes an organic material, and is formed by an ALCVD method between the photoelectric conversion element on the substrate and the color filter layer. An image processing apparatus comprising a protective layer for protecting the photoelectric conversion element on the substrate.

(5) The image processing apparatus according to (4) , wherein the protective layer includes an inorganic material.

(6) The image processing apparatus according to (5) , wherein the protective layer has a two-layer structure of an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.

(7) The image processing device according to any one of (1) to (6) , wherein the imaging element condenses light on each of the multiple photoelectric conversion elements above the color filter layer. An image processing apparatus including a microlens for performing the above.

(8) An endoscope apparatus comprising the image processing apparatus according to any one of (1) to (7) and the imaging element.

(9) An image processing program for causing a computer to function as each unit of the image processing apparatus according to any one of (1) to (7) .

  According to the present invention, there is provided an image processing apparatus capable of generating RGB color image data with good color reproducibility from an image pickup signal from an image pickup element without providing an IR cut filter in front of the image pickup element. Can do.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a partial surface schematic diagram of an image sensor for explaining an embodiment of the present invention. FIG. 2 is a schematic cross-sectional view taken along line AA of the image sensor shown in FIG. In FIG. 1, the microlens 14 is not shown.

  A p-well layer 2 is formed on the n-type silicon substrate 1. Hereinafter, the n-type silicon substrate 1 and the p-well layer 2 are collectively referred to as a semiconductor substrate. In a row direction on the same plane above the semiconductor substrate and in a column direction perpendicular thereto, a color filter 13r that mainly transmits light in the R wavelength region, and a color filter 13g that mainly transmits light in the G wavelength region, A large number of three kinds of color filters, ie, a color filter 13b that mainly transmits light in the B wavelength region, are arranged.

  A known material can be used for the color filter 13r, but such a material transmits a part of the light in the infrared region in addition to the light in the R wavelength region. A known material can be used for the color filter 13g, but such a material transmits a part of the light in the infrared region in addition to the light in the G wavelength region. A known material can be used for the color filter 13b, but such a material transmits part of the light in the infrared region in addition to the light in the B wavelength region.

  As the arrangement of the color filters 13r, 13g, 13b, a color filter arrangement (Bayer arrangement, vertical stripe, horizontal stripe, etc.) used in a known single-plate solid-state imaging device can be adopted.

  In the p well layer 2 below the color filter 13r, an n-type impurity region (hereinafter referred to as n region) 3r is formed corresponding to the color filter 13r, and a pn junction between the n region 3r and the p well layer 2 is formed. Thus, an R photoelectric conversion element corresponding to the color filter 13r is configured.

  An n region 3g is formed in the p well layer 2 below the color filter 13g so as to correspond to the color filter 13g, and the G region corresponding to the color filter 13g is formed by a pn junction between the n region 3g and the p well layer 2. A photoelectric conversion element is configured.

  An n region 3b is formed in the p well layer 2 below the color filter 13b so as to correspond to the color filter 13b, and B corresponding to the color filter 13b is formed by a pn junction between the n region 3b and the p well layer 2. A photoelectric conversion element is configured.

  A transparent electrode 7r is formed above the n region 3r, a transparent electrode 7g is formed above the n region 3g, and a transparent electrode 7b is formed above the n region 3b. The transparent electrodes 7r, 7g, and 7b are divided corresponding to the color filters 13r, 13g, and 13b, respectively. The transparent electrodes 7r, 7g, and 7b are each made of a material that is transparent to visible light and infrared light, and for example, ITO or IZO can be used. The transparent electrodes 7r, 7g, and 7b are embedded in the insulating layer 8, respectively.

  On each of the transparent electrodes 7r, 7g, and 7b, light in the infrared region having a wavelength of 580 nm or more is mainly absorbed to generate a charge corresponding thereto, and a visible region other than the infrared region (wavelength of about 380 nm to about 580 nm). Each of the color filters 13r, 13g, and 13b that transmits the light is formed with a photoelectric conversion layer 9 that is a single sheet configuration. As a material constituting the photoelectric conversion layer 9, for example, a phthalocyanine-based organic material or a naphthalocyanine-based organic material is used.

  On the photoelectric conversion layer 9, a transparent electrode 10 having a single configuration common to each of the color filters 13 r, 13 g, and 13 b is formed. The transparent electrode 10 is made of a material transparent to visible light and infrared light, and for example, ITO or IZO can be used.

  A photoelectric conversion element corresponding to the color filter 13r is formed by the transparent electrode 7r, the transparent electrode 10 facing the transparent electrode 7r, and a part of the photoelectric conversion layer 9 sandwiched therebetween. Hereinafter, since this photoelectric conversion element is formed on a semiconductor substrate, it is referred to as an R-substrate photoelectric conversion element.

  A photoelectric conversion element corresponding to the color filter 13g is formed by the transparent electrode 7g, the transparent electrode 10 facing the transparent electrode 7g, and a part of the photoelectric conversion layer 9 sandwiched therebetween. Hereinafter, this photoelectric conversion element is referred to as a G-substrate photoelectric conversion element.

  A photoelectric conversion element corresponding to the color filter 13b is formed by the transparent electrode 7b, the transparent electrode 10 facing the transparent electrode 7b, and a part of the photoelectric conversion layer 9 sandwiched therebetween. Hereinafter, this photoelectric conversion element is referred to as a B-substrate photoelectric conversion element.

  Next to the n region 3r in the p well layer 2 is a high concentration n-type impurity region (hereinafter referred to as an n + region) 4r for accumulating charges generated in the photoelectric conversion layer 9 of the photoelectric conversion element on the R substrate. Is formed. In order to prevent light from entering the n + region 4r, it is preferable to provide a light shielding film on the n + region 4r.

  Next to the n region 3g in the p well layer 2, an n + region 4g for accumulating charges generated in the photoelectric conversion layer 9 of the photoelectric conversion element on the G substrate is formed. In order to prevent light from entering the n + region 4g, it is preferable to provide a light shielding film on the n + region 4g.

  Next to the n region 3b in the p well layer 2, an n + region 4b for accumulating charges generated in the photoelectric conversion layer 9 of the photoelectric conversion element on the B substrate is formed. In order to prevent light from entering the n + region 4b, it is preferable to provide a light shielding film on the n + region 4b.

  A contact portion 6r made of a metal such as aluminum is formed on the n + region 4r, and a transparent electrode 7r is formed on the contact portion 6r. The n + region 4r and the transparent electrode 7r are electrically connected by the contact portion 6r. ing. The contact portion 6r is embedded in the insulating layer 5 that is transparent to visible light and infrared light.

  A contact portion 6g made of a metal such as aluminum is formed on the n + region 4g, and a transparent electrode 7g is formed on the contact portion 6g. The n + region 4g and the transparent electrode 7g are electrically connected by the contact portion 6g. ing. The contact portion 6g is embedded in the insulating layer 5.

  A contact portion 6b made of a metal such as aluminum is formed on the n + region 4b, and a transparent electrode 7b is formed on the contact portion 6b. The n + region 4b and the transparent electrode 7b are electrically connected by the contact portion 6b. ing. The contact portion 6 b is embedded in the insulating layer 5.

  The regions other than the n regions 3r, 3g, 3b and n + regions 4r, 4g, 4b in the p-well layer 2 are formed according to the charges generated in the R photoelectric conversion element and accumulated in the n region 3r. A signal reading unit 5r for reading out the corresponding signal and a signal corresponding to the charge accumulated in the n + region 4r, and a signal corresponding to the charge generated in the G photoelectric conversion element and accumulated in the n region 3g and the n + region 4g. A signal reading unit 5g for reading a signal corresponding to the charge accumulated in the signal, a signal corresponding to the charge generated in the B photoelectric conversion element and accumulated in the n region 3b, and a charge accumulated in the n + region 4b. A signal reading unit 5b for reading the corresponding signals is formed. Each of the signal reading units 5r, 5g, and 5b can adopt a known configuration using a CCD or a MOS circuit. In order to prevent light from entering the signal readout units 5r, 5g, 5b, it is preferable to provide a light shielding film on the signal readout units 5r, 5g, 5b.

  FIG. 3 is a diagram illustrating a specific configuration example of the signal reading unit 5r illustrated in FIG. In FIG. 3, the same components as those in FIGS. Note that the signal readout units 5r, 5g, and 5b have the same configuration, and thus the description of the signal readout units 5g and 5b is omitted.

  The signal readout section 5r has a reset transistor 43 whose drain is connected to the n + region 4r, its source connected to the power supply Vn, and an output transistor 42 whose gate is connected to the drain of the reset transistor 43 and whose source is connected to the power supply Vcc. A row selection transistor 41 whose source is connected to the drain of the output transistor 42 and whose drain is connected to the signal output line 45; a reset transistor 46 whose drain is connected to the n region 3r and whose source is connected to the power supply Vn; The output transistor 47 whose gate is connected to the drain of the reset transistor 46, the source is connected to the power supply Vcc, and the row selection transistor 48 whose source is connected to the drain of the output transistor 47 and whose drain is connected to the signal output line 49. With.

  By applying a bias voltage between the transparent electrode 7r and the transparent electrode 10, a charge is generated according to the light incident on the photoelectric conversion layer 9, and the charge moves to the n + region 4r through the transparent electrode 7r. The charge accumulated in the n + region 4r is converted into a signal corresponding to the amount of charge by the output transistor. Then, a signal is output to the signal output line 45 by turning on the row selection transistor 41. After the signal is output, the charge in the n + region 4r is reset by the reset transistor 43.

  The charge generated in the R photoelectric conversion element and accumulated in the n region 3r is converted into a signal corresponding to the charge amount by the output transistor 47. Then, a signal is output to the signal output line 49 by turning on the row selection transistor 48. After the signal is output, the charge in the n region 3r is reset by the reset transistor 46.

  Thus, the signal reading unit 5r can be configured by a known MOS circuit including three transistors.

  Returning to FIG. 2, protective layers 11 and 12 having a two-layer structure for protecting the photoelectric conversion element on the substrate are formed on the photoelectric conversion layer 9, and color filters 13r, 13g, and 13b are formed on the protective layer 12. On each of the color filters 13r, 13g, 13b, a microlens 14 for condensing light is formed on the corresponding n regions 3r, 3g, 3b.

  The image sensor 100 is manufactured by forming the color filters 13r, 13g, 13b, the microlenses 14 and the like after forming the photoelectric conversion layer 9, but the color filters 13r, 13g, 13b and the microlenses 14 are formed by photolithography. In the case where an organic material is used as the photoelectric conversion layer 9 because the process includes a process and a baking process, if the photolithography process and the baking process are performed with the photoelectric conversion layer 9 exposed, the characteristics of the photoelectric conversion layer 9 deteriorate. Resulting in. In the image sensor 100, protective layers 11 and 12 are provided in order to prevent deterioration of the characteristics of the photoelectric conversion layer 9 due to such a manufacturing process.

The protective layer 11 is preferably an inorganic layer made of an inorganic material formed by the ALCVD method. The ALCVD method is an atomic layer CVD method, can form a dense inorganic layer, and can be an effective protective layer for the photoelectric conversion layer 9. The ALCVD method is also known as the ALE method or ALD method. The inorganic layer formed by the ALCVD method is preferably made of Al ₂ O ₃ , SiO ₂ , TiO ₂ , ZrO ₂ , MgO, HfO ₂ , Ta ₂ O ₅ , more preferably made of Al ₂ O ₃ , SiO ₂ , Most preferably, it consists of Al ₂ O ₃ .

  The protective layer 12 is formed on the protective layer 11 in order to further improve the protective performance of the photoelectric conversion layer 9, and is preferably an organic layer made of an organic polymer. Parylene is preferable as the organic polymer, and parylene C is more preferable. The protective layer 12 may be omitted, and the arrangement of the protective layer 11 and the protective layer 12 may be reversed. The protective effect of the photoelectric conversion layer 9 is particularly high in the configuration shown in FIG.

  In the imaging device 100 having the above-described configuration, infrared light in the light transmitted through the color filter 13r out of the incident light is absorbed by the photoelectric conversion layer 9, and charges corresponding to the light in the infrared light are here. appear. Similarly, light in the infrared region of the light transmitted through the color filter 13g in the incident light is absorbed by the photoelectric conversion layer 9, and charges corresponding to the light in the infrared region are generated here. Similarly, light in the infrared region of the light transmitted through the color filter 13b in the incident light is absorbed by the photoelectric conversion layer 9, and charges corresponding to the light in the infrared region are generated here.

  When a predetermined bias voltage is applied to the transparent electrode 7r and the transparent electrode 10, the charge generated in the photoelectric conversion layer 9 constituting the photoelectric conversion element on the R substrate moves to the n + region 4r via the transparent electrode 7r and the contact portion 6r. , Accumulated here. Then, a signal corresponding to the electric charge accumulated in the n + region 4r is read by the signal reading unit 5r and output to the outside of the image sensor 100.

  Similarly, when a predetermined bias voltage is applied to the transparent electrode 7g and the transparent electrode 10, charges generated in the photoelectric conversion layer 9 constituting the photoelectric conversion element on the G substrate are transferred to the n + region 4g via the transparent electrode 7g and the contact portion 6g. Go to and accumulate here. Then, a signal corresponding to the electric charge accumulated in the n + region 4g is read out by the signal reading unit 5g and output to the outside of the image sensor 100.

  Similarly, when a predetermined bias voltage is applied to the transparent electrode 7b and the transparent electrode 10, the charge generated in the photoelectric conversion layer 9 constituting the photoelectric conversion element on the B substrate is transferred to the n + region 4b via the transparent electrode 7b and the contact portion 6b. Go to and accumulate here. Then, a signal corresponding to the electric charge accumulated in the n + region 4b is read by the signal reading unit 5b and output to the outside of the image sensor 100.

  The light in the R wavelength region that has passed through the color filter 13r and has passed through the photoelectric conversion layer 9 enters the R photoelectric conversion element, and charges corresponding to the amount of incident light are accumulated in the n region 3r. Similarly, the light in the G wavelength band that has passed through the color filter 13g and passed through the photoelectric conversion layer 9 enters the G photoelectric conversion element, and charges corresponding to the amount of incident light are accumulated in the n region 3g. Similarly, light in the B wavelength region that has passed through the color filter 13b and transmitted through the photoelectric conversion layer 9 is incident on the B photoelectric conversion element, and charges corresponding to the amount of incident light are accumulated in the n region 3b. The charges accumulated in the n regions 3r, 3g, and 3b are read by the signal reading units 5r, 5g, and 5b, and are output to the outside of the image sensor 100.

  The arrangement of signals read out and output from the n regions 3r, 3g, 3b is the same as the arrangement of signals output from the single-plate color solid-state image pickup device having the color filter arrangement as shown in FIG. By performing signal processing used in the color solid-state imaging device, color image data in which data of three color components of R, G, and B is given to one pixel data can be generated. Further, infrared image data in which one pixel data has infrared color component data can be generated by signals read out and output from the n + regions 4r, 4g, and 4b.

  As described above, the image sensor 100 generates the R component signal corresponding to the charge generated in the R photoelectric conversion element, the G component signal corresponding to the charge generated in the G photoelectric conversion element, and the B photoelectric conversion element. A B component signal corresponding to the charge, an IR component signal corresponding to the charge generated in the photoelectric conversion element on the R substrate, an IR component signal corresponding to the charge generated in the photoelectric conversion element on the G substrate, and the B substrate An IR component signal corresponding to the electric charge generated in the upper photoelectric conversion element can be output to the outside. For this reason, if the image sensor 100 is used, two types of image data, that is, color image data and infrared image data, can be obtained by one imaging. Therefore, this image sensor 100 can be used as, for example, an image sensor of an endoscope apparatus that requires an appearance image of a part to be inspected of a human body and an internal image of the part.

Next, the spectral sensitivity characteristic of the image sensor 100 will be described.
First, the spectral sensitivity characteristic of each photoelectric conversion element (PD) formed in the semiconductor substrate is as shown in FIG. 4, the spectral sensitivity characteristic of the photoelectric conversion layer 9 is as shown in FIG. The transmittance is as shown in FIG. 4, and the spectral transmittance of the color filters 13r, 13g, and 13b is as shown in FIG. In FIG. 4, the vertical axis indicates the spectral sensitivity or transmittance when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 5, the vertical axis indicates the spectral transmittance when 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

  When the characteristic is determined in this way, the spectral sensitivity characteristic of the R photoelectric conversion element is the product of the spectral transmittance of the photoelectric conversion layer 9 and the spectral transmittance of the color filter 13r, and the spectral sensitivity characteristic of the G photoelectric conversion element is The spectral transmittance of the conversion layer 9 is multiplied by the spectral transmittance of the color filter 13g, and the spectral sensitivity characteristic of the B photoelectric conversion element is the product of the spectral transmittance of the photoelectric conversion layer 9 and the spectral transmittance of the color filter 13b. The characteristics are as shown in FIG. In FIG. 6, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

  The spectral sensitivity characteristic of the photoelectric conversion element on the R substrate is the product of the spectral sensitivity of the photoelectric conversion layer 9 and the spectral transmittance of the color filter 13r, and the spectral sensitivity characteristic of the photoelectric conversion element on the G substrate is the photoelectric conversion layer 9. And the spectral transmittance of the color filter 13g. The spectral sensitivity characteristic of the photoelectric conversion element on the B substrate is the product of the spectral sensitivity of the photoelectric conversion layer 9 and the spectral transmittance of the color filter 13b. The characteristics are as shown in FIG.

  Here, in order to adjust the spectral sensitivity characteristics of the photoelectric conversion elements on each substrate, if a correction filter having a spectral transmittance as shown in FIG. The sensitivity characteristic is as shown in FIG. In FIG. 7, the vertical axis indicates the spectral transmittance when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 8, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

  FIG. 9 is a diagram showing the spectral reflectances of oxygenated hemoglobin (oxy) and reduced hemoglobin (deoxy). In FIG. 9, the vertical axis indicates the spectral reflectance when 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

  As can be seen from FIG. 9, oxyhemoglobin and reduced hemoglobin have a large difference in reflectance in the wavelength range of 580 nm to 780 nm. Therefore, if a photoelectric conversion element having sensitivity in this wavelength range is used, the state of hemoglobin Change can be imaged with high contrast. As shown in FIG. 8, the image sensor 100 uses a signal obtained from the photoelectric conversion element on the R substrate because the photoelectric conversion element on the R substrate has a strong sensitivity in the wavelength range of 580 nm to 780 nm. By generating infrared image data, it is possible to obtain an RGB color appearance image of a region to be inspected and an image for knowing the change in the state of hemoglobin at that region by one imaging. .

  When generating infrared image data using only signals obtained from the photoelectric conversion element on the R substrate, the position of the signal obtained from each of the photoelectric conversion element on the G substrate and the photoelectric conversion element on the B substrate The signals obtained from the photoelectric conversion elements on the R substrate in the periphery may be interpolated to generate infrared image data having the same resolution as the color image data, or obtained from the photoelectric conversion elements on the R substrate. Infrared image data having a resolution of 1/3 of the color image data may be generated using only the received signal. Alternatively, signals obtained from each of the three photoelectric conversion elements of the R substrate photoelectric conversion elements, the G substrate photoelectric conversion elements, and the B substrate photoelectric conversion elements arranged in the row direction are added to form one signal. Based on the above, infrared image data having a resolution of 1/3 of the color image data may be generated.

  If the image sensor 100 is used, two types of image data of color image data and infrared image data can be obtained. However, such an effect is obtained by using a complementary color system in addition to the primary color system as a color filter used in the image sensor 100. Even when used, it can be obtained. Although two types of image data cannot be obtained, the resolution of the image sensor 100 is higher than that of a single-plate image sensor by adjusting the color arrangement of the color filter of the image sensor 100 and the wavelength range of light absorbed by the photoelectric conversion layer. It is also possible to obtain RGB image data. A configuration example of the image sensor 100 for obtaining these effects is shown in FIG. In FIG. 10, among the components constituting the image sensor 100, a photoelectric conversion element (PD) formed in a semiconductor substrate, a photoelectric conversion layer formed above the PD, and formed above the photoelectric conversion layer. Components other than the color filter to be used are omitted.

  The image sensor shown in FIG. 10A is the same as the image sensor 100 shown in FIGS. Then, the color filter 13g is changed to an Mg filter that transmits part of the light in the Mg (magenta) wavelength region and the light in the infrared region, and the color filter 13b is changed to the light in the wavelength region of Ye (yellow) and the light in the infrared region. The filter is changed to a Ye filter that transmits part of the filter. A known material may be used for each of the Cy filter, the Mg filter, and the Ye filter.

  According to this configuration, color image data can be generated from Cy, Mg, Ye signals obtained from the photoelectric conversion elements in the semiconductor substrate, and infrared image data can be generated from the signals obtained from the photoelectric conversion layer. Note that the arrangement of the Cy filter, Mg filter, and Ye filter may be such that a color image can be reproduced.

  The image sensor shown in FIG. 10B is obtained by changing the color filter 13r to a Cy filter and the color filter 13b to a Ye filter in the image sensor 100 shown in FIGS.

  According to this configuration, color image data can be generated from Cy, G, Ye signals obtained from the photoelectric conversion elements in the semiconductor substrate, and infrared image data can be generated from the signals obtained from the photoelectric conversion layer. The arrangement of the Cy filter, G filter, and Ye filter may be set so that a color image can be reproduced.

  The image sensor shown in FIG. 10C is the same as the image sensor 100 shown in FIGS. 1 and 2, except that the color filter 13r is changed to a Cy filter and the color filter 13g is changed to an IR filter that transmits light in the infrared region. The conversion layer 9 is changed to a G photoelectric conversion layer that absorbs light in the G wavelength range and generates a signal charge according to this, and transmits light outside the G wavelength range, and the color filter 13b is changed to a Ye filter. It is a thing. As the material constituting the G photoelectric conversion layer, for example, InGaAlP and GaPAs can be used in the case of an inorganic material, and for example, R6G / PMPS (rhodamine 6G (R6G) -doped polymethylphenylsilane) is used in the case of an organic material. Can be used.

  According to this configuration, color image data is generated from the B and R signals obtained from the photoelectric conversion element in the semiconductor substrate and the G signal obtained from the photoelectric conversion layer, and obtained from the photoelectric conversion element below the IR filter. Infrared image data can be generated from the IR signal. Note that the arrangement of the Cy filter and the Ye filter may be such that a color image can be reproduced, and the arrangement of the IR filter may be such that an infrared image can be reproduced.

  The image sensor shown in FIG. 10D is the same as the image sensor 100 shown in FIGS. 1 and 2, except that the color filters 13r and 13b are changed to Cy filters, the color filter 13g is changed to a Ye filter, and the photoelectric conversion layer 9 is changed. It is changed to the G photoelectric conversion layer.

  According to this configuration, color image data can be generated from the B and R signals obtained from the photoelectric conversion elements in the semiconductor substrate and the G signal obtained from the photoelectric conversion layer. With this configuration, since two primary color signals can be obtained for each imaging point, the resolution can be improved compared to a single-plate imaging element.

  In the above description, the case where two or three color filters are used for the image sensor 100 has been described. However, even if there are four or more color filters, the same effect can be obtained. One color filter may be used. In this case, for example, in the configuration shown in FIG. 2, a single color G color filter that transmits light in the G wavelength band may be provided instead of the color filters 13r, 13g, and 13b.

  According to this configuration, monochrome image data can be generated from the signal obtained from the photoelectric conversion element in the semiconductor substrate, and infrared image data can be generated from the signal obtained from the photoelectric conversion layer 9. . In addition, this configuration has an advantage that the spectral sensitivity characteristic of the photoelectric conversion layer can be adjusted by the spectral transmittance of the color filter provided above the photoelectric conversion layer.

  In the above description, the photoelectric conversion layer is provided above the semiconductor substrate and the color filter is provided above the semiconductor substrate. However, the same effect can be obtained even if the arrangement of the photoelectric conversion layer and the color filter is reversed. .

  In the above description, the color filters 13r, 13g, and 13b transmit infrared light, but filters having a spectral transmittance that does not transmit infrared light may be used. Is possible. However, if all the color filters do not transmit infrared light, infrared image data cannot be generated. Therefore, at least one of the one or more color filters must be in the infrared range. It is necessary to have a function of transmitting light.

  In the above description, the three types of photoelectric conversion elements on the substrate, that is, the photoelectric conversion element on the R substrate, the photoelectric conversion element on the G substrate, and the photoelectric conversion element on the B substrate are provided. It is sufficient that at least one of these is present. As shown in FIGS. 6 and 8, since the photoelectric conversion element on the R substrate has the highest sensitivity in the infrared region, infrared image data is generated using a signal output from the photoelectric conversion element on the R substrate. Most preferred. When the photoelectric conversion element on the G substrate is omitted, the transparent electrode 7g, the contact portion 6g, and the n + region 4g may be omitted in the configuration shown in FIG. In the case where the photoelectric conversion element on the B substrate is omitted, the transparent electrode 7b, the contact portion 6b, and the n + region 4b may be omitted in the configuration shown in FIG.

  In the case of the configuration as shown in FIG. 10C, if the on-substrate photoelectric conversion element provided in the image sensor is only the on-substrate photoelectric conversion element corresponding to the IR filter, this on-substrate photoelectric conversion is performed. Since almost no G component signal can be obtained from the element, the generation of color image data is hindered. For this reason, in the case of the configuration shown in FIG. 10C, it is necessary to provide at least a substrate photoelectric conversion element corresponding to the Cy filter or a substrate photoelectric conversion element corresponding to the Ye filter.

Next, a method for manufacturing the image sensor 100 will be described. The image sensor 100 can be manufactured by the following steps (A) to (C).
(A) CMOS substrate → transparent electrode formation n regions 3r, 3g, 3b and a signal readout portion are formed on a silicon substrate in the same manner as a conventional CMOS sensor.
Further, n + regions 4r, 4g, 4b and a signal reading unit are formed.
The insulating layer 5 is formed on the silicon substrate, the transparent electrodes 7r, 7g, 7b are formed thereon, and the transparent electrodes 7r, 7g, 7b and the n + regions 4r, 4g, 4b are contacted using via plugs. .
The gap between the transparent electrodes 7r, 7g, and 7b is filled with an insulating material, and the surface of the transparent electrodes 7r, 7g, and 7b including the insulating material portion is planarized using CMP.
The above process is performed by a semiconductor process.
(B) Formation of photoelectric conversion layer-The photoelectric conversion layer 9 is formed on the transparent electrodes 7r, 7g, and 7b.
-Further, a transparent electrode 10 is formed. The transparent electrode 10 is brought into contact with a pad (not shown), and a bias voltage is applied from an external power source.
The above process is performed by a vacuum deposition process. (C) Formation of a microlens and a color filter-An alumina protective layer is formed on the photoelectric conversion layer 9 by, for example, the ALCVD method, and a parylene C protective layer is further formed.
Next, a mosaic color filter is formed. The mosaic color filter is formed in the order of G resist coating → pattern exposure → development → post bake, B resist coating → pattern exposure → development → post bake, R resist coating → pattern exposure → development → post bake.
-Finally, a microlens is formed. The microlens is formed in the order of resist coating → post bake → resist coating → pattern exposure → development → melt.

(Second embodiment)
In the present embodiment, an embodiment will be described in which the imaging device 100 capable of obtaining color image data and infrared image data as described in the first embodiment is applied to an endoscope apparatus.

FIG. 11 is a diagram illustrating a schematic configuration of an endoscope apparatus for explaining the second embodiment.
The endoscope apparatus shown in FIG. 11 receives a white light source 50 for illuminating a site to be examined, an optical system 51 such as a photographing lens and a diaphragm, and light that has passed through the optical system 51. The correction filter 52 disposed between the imaging device 100 and the optical system 51 in order to correct the spectral sensitivity characteristics of the photoelectric conversion layer 9 of the imaging device 100, and the imaging device 100. An infrared image data generation unit 53 that generates infrared image data based on a signal corresponding to the electric charge generated in the photoelectric conversion layer 9, an R photoelectric conversion element, a G photoelectric conversion element, and a B photoelectric conversion element of the image sensor 100. A color image data generation unit 54 that generates color image data based on a signal corresponding to a charge generated in each of the image data, an infrared image data generated by the infrared image data generation unit 53, and a color image data generation unit 54. Living High-contrast infrared image data generation that generates high-contrast infrared image data in which the contrast of the infrared image data generated by the infrared image data generation unit 53 is improved by arithmetic processing using the color image data that has been generated Unit 55, image enhancement unit 57 that performs enhancement processing on the high-contrast infrared image data generated by high-contrast infrared image data generation unit 55, and infrared image data generated by infrared image data generation unit 53 High color reproduction color image data with improved color reproducibility of the color image data generated by the color image data generation unit 54 is generated by arithmetic processing using the color image data generated by the color image data generation unit 54 A high color reproduction color image data generation unit 56 that performs the image processing and the high color reproduction color image based on the high contrast infrared image data after the enhancement processing. And a display control unit 58 for controlling to display an image based on the image data to the display device 59.

  The imaging device 100 used in the endoscope apparatus shown in FIG. 11 includes an R component signal corresponding to light in the R wavelength region, a G component signal corresponding to light in the G wavelength region, and a B wavelength region. Any type of signal can be used as long as it can output four types of signals, ie, a B component signal corresponding to light and an IR component signal corresponding to light in the infrared region. It is not limited. For example, an image sensor having the configuration shown in FIG. 10C may be used, a color filter that transmits light in the R or Cy wavelength region, and a color filter that transmits light in the G or Mg wavelength region. A single-plate image sensor in which four color filters that transmit light in the wavelength region of B, Ye, and color filter that transmit light in the infrared region are arranged in a mosaic pattern on the same surface above the semiconductor substrate. Also good. The spectral sensitivity characteristic of the image sensor 100 is, for example, as shown in FIG.

  The color image data generation unit 54 has a signal (hereinafter referred to as R signal) corresponding to the charge generated by the R photoelectric conversion element of the image sensor 100 and a signal (hereinafter referred to as R signal) generated by the G photoelectric conversion element of the image sensor 100 ( Hereinafter, a G signal) and a signal corresponding to the electric charge generated by the B photoelectric conversion element of the image sensor 100 (hereinafter referred to as a B signal) are acquired from the image sensor 100, and these signals are used to obtain known signals. Color image data is generated by the method.

  The infrared image data generation unit 53 performs signal interpolation or the like from a signal corresponding to the charge generated by the photoelectric conversion element on the R substrate of the image sensor 100 (hereinafter referred to as IRr signal), and has the same resolution as the color image data. Infrared image data is generated.

  FIG. 12 is a diagram showing the spectral reflectance of oxyhemoglobin and reduced hemoglobin. In FIG. 12, the vertical axis indicates the spectral reflectance when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 12, when the vertical axis is the spectral sensitivity of the photoelectric conversion element when 1 is used as a reference, if hemoglobin is imaged by a photoelectric conversion element having a spectral sensitivity like the Real curve shown in FIG. Can be detected with the highest contrast.

  Therefore, the high-contrast infrared image data generation unit 55 converts the IRr signal obtained from the photoelectric conversion element on the R substrate into a signal obtained from the photoelectric conversion element having spectral sensitivity characteristics as shown by the Real curve in FIG. The contrast of the infrared image data can be improved by performing a calculation process that approaches.

Specifically, the high-contrast infrared image data generation unit 55 performs the calculation represented by the following expression (1) to generate high-contrast infrared image data.
I (x, y) = r1 * R (x, y) + g1 * G (x, y) + b1 * B (x, y) + ir1 * IR (x, y) (1)
I (x, y) indicates pixel data at the coordinates (x, y) of the high-contrast infrared image data.
R (x, y) indicates pixel data of the R component at the coordinates (x, y) of the color image data.
G (x, y) indicates pixel data of the G component at the coordinates (x, y) of the color image data.
B (x, y) indicates B component pixel data at the coordinates (x, y) of the color image data.
IR (x, y) indicates pixel data of the IR component at the coordinates (x, y) of the infrared image data.
r1, g1, b1, and ir1 are spectral sensitivity characteristics of the R photoelectric conversion element, spectral sensitivity characteristics of the G photoelectric conversion element, spectral sensitivity characteristics of the B photoelectric conversion element, and spectral sensitivity characteristics of the photoelectric conversion element on the R substrate. FIG. 13 shows coefficients determined based on the spectral sensitivity characteristic represented by the Real curve in FIG.

  The coefficients r1, g1, b1, ir1 are defined as R (λ) as the spectral sensitivity at the wavelength λ of the R photoelectric conversion element shown in FIG. 8, and the spectral sensitivity at the wavelength λ of the G photoelectric conversion element as shown in FIG. λ), the spectral sensitivity at the wavelength λ of the B photoelectric conversion element shown in FIG. 8 is B (λ), the spectral sensitivity at the wavelength λ of the photoelectric conversion element on the R substrate shown in FIG. 8 is IR (λ), When the spectral sensitivity at the wavelength λ of the photoelectric conversion element having the characteristics shown by the Real curve shown in FIG. 12 is Real (λ), Real (λ) and a value obtained by the calculation of the following equation (2) are: It is determined by the least square method so as to be closest. The determined coefficient data is stored in advance in a memory (not shown) in the endoscope apparatus.

  r1 × R (λ) + g1 × G (λ) + b1 × B (λ) + ir1 × IR (λ) (2)

  FIG. 13 is a diagram showing the spectral sensitivity characteristics obtained as a result of calculating Equation (2) using the coefficients determined by the above method. In FIG. 13, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. A curve I shown in FIG. 13 is a spectral sensitivity characteristic of a virtual photoelectric conversion element that can obtain high-contrast infrared image data obtained by the calculation of Expression (1).

  FIG. 14 is a diagram showing the detection sensitivity of oxyhemoglobin and reduced hemoglobin when light from hemoglobin is detected by the photoelectric conversion element having the spectral sensitivity characteristic shown in FIG. In FIG. 14, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. FIG. 15 is a diagram showing the spectral sensitivity characteristics of the photoelectric conversion element on the R substrate shown in FIG. In FIG. 15, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light. FIG. 16 is a diagram showing the detection sensitivity of oxygenated hemoglobin and reduced hemoglobin when light from hemoglobin is detected by the photoelectric conversion element on the R substrate having the spectral sensitivity characteristics shown in FIG. In FIG. 16, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

  Comparing FIG. 14 and FIG. 16, the area A surrounded by the waveform I (oxy) of oxyhemoglobin and the straight line of spectral sensitivity = 0 shown in FIG. 14 is compared with the waveform I (deoxy) of reduced hemoglobin and the spectrum shown in FIG. The contrast ratio of the high-contrast infrared image data represented by the value divided by the area B surrounded by the sensitivity = 0 straight line is 1.318, and the waveform Ir (oxy) of oxyhemoglobin and spectral sensitivity shown in FIG. Of the infrared image data represented by a value obtained by dividing the area C surrounded by the straight line of = 0 by the area D surrounded by the waveform Ir (deoxy) of reduced hemoglobin and the straight line of spectral sensitivity = 0 shown in FIG. The contrast ratio is 1.166, and it can be seen that the contrast of the infrared image data can be improved by performing the arithmetic processing shown in Expression (1).

  Since the image sensor used in the endoscope apparatus of the present embodiment needs to output an IR signal, an infrared cut filter that is provided in a normal digital camera cannot be disposed in front of the image sensor. In this embodiment, since the correction filter 52 for correcting the spectral sensitivity characteristics of the photoelectric conversion layer 9 is provided, each of the R photoelectric conversion element, the G photoelectric conversion element, and the B photoelectric conversion element is light in the infrared region. Has little sensitivity, but still has some sensitivity. As a result, the color reproducibility of the color image data may be reduced.

  Therefore, the high color reproduction color image data generation unit 56 performs r photoelectric conversion in which the R signal obtained from the R photoelectric conversion element has an ideal spectral sensitivity characteristic defined by the standard RGB ideal imaging characteristic shown in FIG. The G signal obtained from the G photoelectric conversion element having an ideal spectral sensitivity characteristic defined by the standard RGB ideal imaging characteristic shown in FIG. An arithmetic process is performed so as to approach the signal obtained from the conversion element, and the B signal obtained from the B photoelectric conversion element has an ideal spectral sensitivity characteristic defined by the standard RGB ideal imaging characteristic shown in FIG. By performing arithmetic processing that approaches a signal obtained from the photoelectric conversion element, it is possible to generate high-color reproduction color image data.

  Specifically, the high color reproduction color image data generation unit 56 performs the calculation represented by the following expression (3) to improve the color reproducibility of the color image data.

Ｒ_０（ｘ，ｙ）は、高色再現カラー画像データの座標（ｘ，ｙ）におけるＲ成分の画素データを示す。
Ｇ_０（ｘ，ｙ）は、高色再現カラー画像データの座標（ｘ，ｙ）におけるＧ成分の画素データを示す。
Ｂ_０（ｘ，ｙ）は、高色再現カラー画像データの座標（ｘ，ｙ）におけるＢ成分の画素データを示す。
Ｒ（ｘ，ｙ）は、カラー画像データの座標（ｘ，ｙ）におけるＲ成分の画素データを示す。
Ｇ（ｘ，ｙ）は、カラー画像データの座標（ｘ，ｙ）におけるＧ成分の画素データを示す。
Ｂ（ｘ，ｙ）は、カラー画像データの座標（ｘ，ｙ）におけるＢ成分の画素データを示す。
Ｉｒ（ｘ，ｙ）は、赤外画像データの座標（ｘ，ｙ）におけるＩＲ成分の画素データを示す。
ｒ２，ｒ３，ｒ４，ｇ２，ｇ３，ｇ４，ｂ２，ｂ３，ｂ４，ｉｒ２，ｉｒ３，ｉｒ４は、Ｒ光電変換素子の分光感度特性と、Ｇ光電変換素子の分光感度特性と、Ｂ光電変換素子の分光感度特性と、Ｒ基板上光電変換素子の分光感度特性と、図１７に示すスタンダードＲＧＢ理想撮像特性とに基づいて決められた係数を示す。

R ₀ (x, y) represents pixel data of the R component at the coordinates (x, y) of the high color reproduction color image data.
G ₀ (x, y) represents pixel data of the G component at the coordinates (x, y) of the high color reproduction color image data.
B ₀ (x, y) indicates B component pixel data at coordinates (x, y) of the high color reproduction color image data.
R (x, y) indicates pixel data of the R component at the coordinates (x, y) of the color image data.
G (x, y) indicates pixel data of the G component at the coordinates (x, y) of the color image data.
B (x, y) indicates B component pixel data at the coordinates (x, y) of the color image data.
Ir (x, y) indicates pixel data of the IR component at the coordinates (x, y) of the infrared image data.
r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3, ir4 are the spectral sensitivity characteristic of the R photoelectric conversion element, the spectral sensitivity characteristic of the G photoelectric conversion element, and the B photoelectric conversion element. 18 shows coefficients determined based on the spectral sensitivity characteristics, the spectral sensitivity characteristics of the photoelectric conversion elements on the R substrate, and the standard RGB ideal imaging characteristics shown in FIG.

  The coefficients r2, g2, b2, ir2 are defined as the spectral sensitivity at the wavelength λ of the R photoelectric conversion element shown in FIG. 8 being R (λ), and the spectral sensitivity at the wavelength λ of the G photoelectric conversion element shown in FIG. λ), the spectral sensitivity at the wavelength λ of the B photoelectric conversion element shown in FIG. 8 is B (λ), the spectral sensitivity at the wavelength λ of the photoelectric conversion element on the R substrate shown in FIG. 8 is IR (λ), When the spectral sensitivity at the wavelength λ of the r photoelectric conversion element shown in FIG. 17 is r (λ), the least square method is used so that r (λ) and the value obtained by the following equation (4) are closest. Determined by. The determined coefficient data is stored in advance in a memory (not shown) in the endoscope apparatus.

  r2 × R (λ) + g2 × G (λ) + b2 × B (λ) + ir2 × IR (λ) (4)

  The coefficients r3, g3, b3, and ir3 are g (λ) and a value obtained by the following equation (5) when the spectral sensitivity at the wavelength λ of the g photoelectric conversion element shown in FIG. Is determined by the method of least squares so that. The determined coefficient data is stored in advance in a memory (not shown) in the endoscope apparatus.

  r3 × R (λ) + g3 × G (λ) + b3 × B (λ) + ir3 × IR (λ) (5)

  The coefficients r4, g4, b4, and ir4 are b (λ) and a value obtained by the following equation (6) when the spectral sensitivity at the wavelength λ of the b photoelectric conversion element shown in FIG. Is determined by the method of least squares so that. The determined coefficient data is stored in advance in a memory (not shown) in the endoscope apparatus.

  r4 × R (λ) + g4 × G (λ) + b4 × B (λ) + ir4 × IR (λ) (6)

  FIG. 18 shows the spectrum of the R photoelectric conversion element, the G photoelectric conversion element, and the B photoelectric conversion element of the image sensor 100 obtained as a result of calculating the equations (4) to (6) using the coefficients determined by the above method. It is a figure which shows a sensitivity characteristic. In FIG. 18, the vertical axis indicates the spectral sensitivity when 1 is used as a reference, and the horizontal axis indicates the wavelength. A curve R shown in FIG. 18 shows a spectral sensitivity characteristic obtained as a result of bringing the spectral sensitivity characteristic of the R photoelectric conversion element close to the ideal spectral sensitivity characteristic, and a curve G shown in FIG. 18 shows that the G photoelectric conversion element is ideal. The spectral sensitivity characteristic obtained as a result of approaching the spectral sensitivity characteristic is shown, and the curve B shown in FIG. 18 shows the spectral sensitivity characteristic obtained as a result of bringing the B photoelectric conversion element closer to the ideal spectral sensitivity characteristic.

  As can be seen from FIG. 18, the sensitivity in the infrared region having a wavelength of 680 nm or more can be made substantially 0 or less. For this reason, it can be seen that the color reproducibility of the color image data can be improved by performing the arithmetic processing shown in Expression (3).

  The display control unit 58 controls the display device 59 to display an image based on the high-contrast infrared image data emphasized by the image enhancement unit 57, or displays an image based on the high-color reproduction color image data on the display device 59. Or a control for causing the display device 59 to display an image obtained by combining an image based on high-contrast infrared image data and an image based on high-color reproduction color image data. For high-contrast infrared image data, the signal level is expressed in pseudo color and displayed, or the signal level is converted into an oxygen absorption amount and displayed.

  As described above, according to the endoscope apparatus of the present embodiment, the color image data generated from the R signal, the G signal, and the B signal output from the image sensor 100 and the IRr signal output from the image sensor 100. High-color reproduction color image data with improved color reproducibility over color image data and high-contrast infrared with improved contrast over infrared image data through arithmetic processing using infrared image data generated from Image data can be generated. For this reason, the test | inspection precision by an endoscope apparatus can be improved compared with the past.

  In addition, by adopting the configuration described in the first embodiment as the imaging device used in the endoscope apparatus, high color reproduction color image data and high contrast infrared image data can be obtained by one imaging. Therefore, the inspection can be performed without worrying about color misregistration.

  Moreover, according to the endoscope apparatus of this embodiment, since an infrared cut filter becomes unnecessary, the part inserted into the human body can be reduced in size, and the apparatus cost can be reduced.

  In the above description, the correction filter 52 is provided in the endoscope apparatus, but this may be omitted. When the correction filter 52 is omitted, the spectral sensitivity characteristic of the image sensor 100 mounted on the endoscope apparatus is as shown in FIG. 6, and the color reproducibility of the color image data is further lowered. The processing performed by the color image data generation unit 56 is more effective.

  In the present embodiment, the endoscope apparatus is provided with both the high contrast infrared image data generation unit 55 and the high color reproduction color image data generation unit 56, but the high color reproduction color image data generation unit 56 is omitted. It doesn't matter. When the high color reproduction color image data generation unit 56 is omitted, it is preferable to provide a correction filter 52 for cutting a wavelength region of 780 nm or more.

  Further, the high color reproduction color image data generation unit 56 is not limited to an endoscope apparatus, but an image sensor that can output an R component signal, a G component signal, a B component signal, and an IR component signal. A sufficient effect can also be obtained by mounting in an imaging device such as a digital camera. In this case, since an infrared cut filter is not required for the imaging apparatus, it is possible to reduce the size and cost of the imaging apparatus.

  The infrared image data generation unit 53, the color image data generation unit 54, the high contrast infrared image data generation unit 55, the high color reproduction color image data generation unit 56, and the image enhancement unit 57 in the endoscope apparatus described above. The function of each unit can be realized by executing a program for causing the computer to function as each unit by a computer such as an arithmetic processing unit mounted on the endoscope apparatus. It can also be realized by capturing an image signal obtained from the image sensor 100 as it is in a personal computer or the like and executing the program by the computer.

  In this specification, the wavelength range of R indicates a range from about 550 nm to about 700 nm, the wavelength range of G indicates a range from about 450 nm to about 610 nm, and the wavelength range of B Indicates a range from a wavelength of about 380 nm to about 520 nm, an infrared region indicates a range from a wavelength of about 680 nm to about 3000 nm, a Cy wavelength region indicates a range from a wavelength of about 380 nm to about 610 nm, The wavelength range of Mg indicates a range from a wavelength of about 380 nm to about 500 nm and a wavelength of about 600 nm to 700 nm, and the Ye wavelength range indicates a range of a wavelength of about 470 nm to about 700 nm.

  Also, in this specification, “transmitting light in a certain wavelength range” means transmitting light in the wavelength range of about 60% or more, and “absorbing light in a certain wavelength range” The absorption of about 50% or more of light in the wavelength band.

The partial surface schematic diagram of the image pick-up element for describing embodiment of this invention Cross-sectional schematic diagram of the AA line of the image sensor shown in FIG. The figure which shows the specific structural example of the signal read-out part 5r shown in FIG. The figure which shows the characteristic of the photoelectric conversion element of the image pick-up element shown in FIG. 1, and the photoelectric conversion element on a board | substrate. The figure which shows the characteristic of the color filter of the image pick-up element shown in FIG. The figure which shows the characteristic without a correction filter of the image pick-up element shown in FIG. Diagram showing characteristics of correction filter The figure which shows the characteristic with a correction filter of the image pick-up element shown in FIG. Diagram showing the spectral reflectance of hemoglobin The figure which shows the modification of a structure of the image pick-up element shown in FIG. The figure which shows schematic structure of the endoscope apparatus for demonstrating 2nd embodiment. The figure which shows the spectral sensitivity characteristic which can detect the spectral reflectance of hemoglobin and the state change of hemoglobin with the highest contrast The figure which shows the characteristic obtained by performing the process which approaches the characteristic of the photoelectric conversion element on R board | substrate of the image pick-up element shown in FIG. 1 to the characteristic shown by the Real curve of FIG. The figure which shows the detection sensitivity characteristic of hemoglobin when it images with the image pick-up element which has the characteristic shown in FIG. The figure which shows the spectral sensitivity characteristic of the photoelectric conversion element on R board | substrate shown in FIG. The figure which shows the detection sensitivity characteristic of hemoglobin when it images with the image pick-up element which has the characteristic shown in FIG. Diagram showing standard RGB ideal imaging characteristics The figure which shows the characteristic obtained by processing each characteristic of R photoelectric conversion element of the image pick-up element shown in FIG. 1, G photoelectric conversion element, and B photoelectric conversion element close to the characteristic shown in FIG.

Explanation of symbols

100 Image sensor 1 Silicon substrate 2 P well layers 3r, 3g, 3b n regions 4r, 4g, 4b n + regions 5, 8 Transparent insulating layers 5r, 5g, 5b Signal readout portions 6r, 6g, 6b Contact portions 7r, 7g, 7b , 10 Transparent electrode 9 Photoelectric conversion layer 11, 12 Protective layer 13r, 13g, 13b Color filter 14 Micro lens

Claims (9)

An image processing device that generates image data from an imaging signal output from an imaging element,
RGB color image data generating means for generating RGB color image data from an R (red) component imaging signal, a G (green) component imaging signal, and a B (blue) component imaging signal output from the imaging device;
Infrared image data generating means for generating infrared image data from an imaging signal of an IR (infrared) component output from the imaging device;
Using the RGB color image data and the infrared image data, high color reproduction RGB color image data generating means for generating high color reproduction RGB color image data with improved color reproducibility of the RGB color image data ,
The image sensor is
A number of photoelectric conversion elements arranged on the same surface in the semiconductor substrate;
A photoelectric conversion element on a substrate corresponding to a part of the plurality of photoelectric conversion elements formed on the same surface above the semiconductor substrate, the first electrode formed above the semiconductor substrate, the first electrode A photoelectric conversion element formed on a substrate including a photoelectric conversion layer formed on one electrode and a second electrode formed on the photoelectric conversion layer;
A color filter layer that is formed above the semiconductor substrate and transmits light in a wavelength range different from the wavelength range of light absorbed by the photoelectric conversion layer;
Signal reading means for respectively reading a signal corresponding to the charge generated in the photoelectric conversion element on the substrate and a signal corresponding to the charge generated in the photoelectric conversion element;
The color filter layer is composed of a number of color filters corresponding to each of the number of photoelectric conversion elements,
The large number of color filters are classified into three color filters: a color filter that transmits light in the R wavelength region, a color filter that transmits light in the G wavelength region, and a color filter that transmits light in the B wavelength region. And
Of the three types of color filters, the color filter that transmits light in at least the R wavelength region also transmits light in the infrared region,
The photoelectric conversion layer absorbs light in the infrared region and generates a charge corresponding thereto, and transmits other light,
The part of the plurality of photoelectric conversion elements is an image processing apparatus that is a photoelectric conversion element corresponding to a color filter that transmits light in the R wavelength range . The image processing apparatus according to claim 1,
The pixel data of the RGB color image data is composed of R data that is R component data, G data that is G component data, and B data that is B component data, and the pixels of the infrared image data The data consists of IR data that is IR component data,
The high color reproduction RGB color image data generating means is configured to multiply the R data constituting the pixel data at the same position of the RGB color image data and the infrared image data by a coefficient r2, and the G data as a coefficient. An R component that constitutes one pixel data of the high-color reproduction RGB color image data by accumulating a value obtained by multiplying g2, a value obtained by multiplying B data by a coefficient b2, and a value obtained by multiplying IR data by a coefficient ir2. The R data is multiplied by a coefficient r3, the G data is multiplied by a coefficient g3, the B data is multiplied by a coefficient b3, and the IR data is multiplied by a coefficient ir3. The G component data constituting one pixel data of the high color reproduction RGB color image data is generated, and a value obtained by multiplying the R data by a coefficient r4 and a coefficient g4 are added to the G data. Multiplied value The B data multiplied by the coefficient b4 and the IR data multiplied by the coefficient ir4 are integrated to generate B component data constituting one pixel data of the high color reproduction RGB color image data. And
The coefficients r2, g2, b2, and ir2 are values obtained by multiplying the R sensitivity, which is the spectral sensitivity of the photoelectric conversion element that outputs the R component imaging signal among the photoelectric conversion elements constituting the imaging element, by the coefficient r2. A value obtained by multiplying the G sensitivity, which is the spectral sensitivity of the photoelectric conversion element that outputs the G component imaging signal among the photoelectric conversion elements that constitute the imaging element, by the coefficient g2, and the photoelectric conversion element that constitutes the imaging element Among the photoelectric conversion elements that output the B component imaging signal, a value obtained by multiplying the B sensitivity, which is the spectral sensitivity of the photoelectric conversion element, by the coefficient b2, and the IR component imaging signal of the photoelectric conversion elements constituting the imaging element. An ideal spectral sensitivity of the photoelectric conversion element that outputs the R component imaging signal and a value obtained by multiplying the IR sensitivity that is the spectral sensitivity of the infrared detection photoelectric conversion element that outputs the value ir2 multiplied by the coefficient ir2 And the value of It has been determined to be closest,
The coefficients r3, g3, b3, ir3 are a value obtained by multiplying the R sensitivity by the coefficient r3, a value obtained by multiplying the G sensitivity by the coefficient g3, a value obtained by multiplying the B sensitivity by the coefficient b3, and the IR A value obtained by multiplying a value obtained by multiplying the sensitivity by the coefficient ir3 and an ideal spectral sensitivity value of the photoelectric conversion element that outputs the G component imaging signal are determined to be closest to each other.
The coefficients r4, g4, b4, ir4 are a value obtained by multiplying the R sensitivity by the coefficient r4, a value obtained by multiplying the G sensitivity by the coefficient g4, a value obtained by multiplying the B sensitivity by the coefficient b4, and the IR An image in which a value obtained by multiplying a value obtained by multiplying the sensitivity by the coefficient ir4 and an ideal spectral sensitivity value of the photoelectric conversion element that outputs the B component imaging signal are determined to be closest to each other. Processing equipment. The image processing apparatus according to claim 1 or 2 ,
An image processing apparatus in which the color filter layer is formed above the photoelectric conversion element on the substrate. The image processing apparatus according to claim 3 ,
The photoelectric conversion layer includes an organic material,
An image processing apparatus comprising a protective layer for protecting the photoelectric conversion element on the substrate formed by ALCVD between the photoelectric conversion element on the substrate and the color filter layer. The image processing apparatus according to claim 4 ,
An image processing apparatus in which the protective layer includes an inorganic material. The image processing apparatus according to claim 5 , wherein
The image processing apparatus, wherein the protective layer has a two-layer structure of an inorganic layer made of an inorganic material and an organic layer made of an organic polymer. The image processing device according to any one of claims 1 to 6 ,
An image processing apparatus, wherein the imaging element includes a microlens for condensing light on each of the multiple photoelectric conversion elements above the color filter layer. The image processing apparatus according to any one of claims 1 to 7 ,
An endoscope apparatus comprising the imaging device. The image processing program for a computer to function as each means of the image processing apparatus of any one of claims 1-7.

Images