New! View global litigation for patent families

US8111286B2 - Image processing apparatus, endoscope, and computer readable medium - Google Patents

Image processing apparatus, endoscope, and computer readable medium Download PDF

Info

Publication number
US8111286B2
US8111286B2 US11863651 US86365107A US8111286B2 US 8111286 B2 US8111286 B2 US 8111286B2 US 11863651 US11863651 US 11863651 US 86365107 A US86365107 A US 86365107A US 8111286 B2 US8111286 B2 US 8111286B2
Authority
US
Grant status
Grant
Patent type
Prior art keywords
image
data
color
photoelectric
conversion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US11863651
Other versions
US20080079807A1 (en )
Inventor
Masafumi Inuiya
Yuichi Ohashi
Mikio Ihama
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/042Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by a proximal camera, e.g. a CCD camera
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration, e.g. from bit-mapped to bit-mapped creating a similar image
    • G06T5/007Dynamic range modification
    • G06T5/009Global, i.e. based on properties of the image as a whole
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration, e.g. from bit-mapped to bit-mapped creating a similar image
    • G06T5/50Image enhancement or restoration, e.g. from bit-mapped to bit-mapped creating a similar image by the use of more than one image, e.g. averaging, subtraction
    • HELECTRICITY
    • H01BASIC ELECTRIC ELEMENTS
    • H01LSEMICONDUCTOR DEVICES; ELECTRIC SOLID STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H01L27/00Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infra-red radiation, light, electromagnetic radiation of shorter wavelength, or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144Devices controlled by radiation
    • H01L27/146Imager structures
    • H01L27/14601Structural or functional details thereof
    • H01L27/14609Pixel-elements with integrated switching, control, storage or amplification elements
    • HELECTRICITY
    • H01BASIC ELECTRIC ELEMENTS
    • H01LSEMICONDUCTOR DEVICES; ELECTRIC SOLID STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H01L27/00Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infra-red radiation, light, electromagnetic radiation of shorter wavelength, or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144Devices controlled by radiation
    • H01L27/146Imager structures
    • H01L27/14601Structural or functional details thereof
    • H01L27/1462Coatings
    • H01L27/14621Colour filter arrangements
    • HELECTRICITY
    • H01BASIC ELECTRIC ELEMENTS
    • H01LSEMICONDUCTOR DEVICES; ELECTRIC SOLID STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H01L27/00Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infra-red radiation, light, electromagnetic radiation of shorter wavelength, or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144Devices controlled by radiation
    • H01L27/146Imager structures
    • H01L27/14643Photodiode arrays; MOS imagers
    • H01L27/14645Colour imagers
    • HELECTRICITY
    • H01BASIC ELECTRIC ELEMENTS
    • H01LSEMICONDUCTOR DEVICES; ELECTRIC SOLID STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H01L27/00Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infra-red radiation, light, electromagnetic radiation of shorter wavelength, or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144Devices controlled by radiation
    • H01L27/146Imager structures
    • H01L27/14643Photodiode arrays; MOS imagers
    • H01L27/14649Infra-red imagers
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/30Transforming light or analogous information into electric information
    • H04N5/33Transforming infra-red radiation
    • H04N5/332Multispectral imaging comprising at least a part of the infrared region
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/04Picture signal generators
    • H04N9/045Picture signal generators using solid-state devices
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10024Color image
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10048Infrared image
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N2005/2255Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscope, borescope

Abstract

An image processing apparatus includes a color image data generation section 54 that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device 100; an infrared image data generation section 53 that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device 100; and a high-contrast infrared image data generation section 55 that generates high-contrast infrared image data using the color image data and the infrared image data. Contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from the Japanese Patent Application Nos. 2006-264923 (filed on Sep. 28, 2006) and 2006-264926 (filed on Sep. 28, 2006), the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

This invention relates to an image processing apparatus for performing image processing for an image pickup signal output from an image pickup device, an endoscope and a computer readable medium storing an image processing program.

2. Description of the Related Art

An endoscope using a CCD or CMOS image sensor as an image pickup device is already often used in the medical field. The endoscopes are roughly classified into (i) a frame sequential image pickup method of using an image pickup device capable of picking up a monochrome image and switching a filter for transmitting light of wavelength ranges of R (red), G (green), B (blue), and IR (infrared) in front of a light source for illuminating a subject through a fiber in synchronization with the field frequency of the image pickup device (for example, see Japanese Patent No. 2648494) and (ii) a simultaneous image pickup method of picking up an image using a single-plate image pickup device provided with a color filter for transmitting light of wavelength ranges of R, G, and B and an illumination light source that emits white light.

The frame sequential image pickup method is a method of rotating a plurality of filters having different spectral transmissivitys in front of a light source and picking up a plurality of images illuminated by light of different wavelengths and then combining the images into a color image. Thus, for example, if RGB transmission filters are used as the filters switched in front of the light source, color image data having three color information pieces of RGB in one pixel data can be provided. Also, if IR filters of two wavelengths put into a narrow band easily absorbed by hemoglobin in blood are switched in sequence, infrared image data in which one pixel data only has information of an infrared region can be provided. An image based on the color image data enables a user to visually check an appearance of a part to be tested, and an image based on the infrared image data enables the user to visually check information, etc., of blood capillary of mucosal surface layer and a mucosal minute pattern in the part to be tested. However, in the frame sequential image pickup, a color shift occurs for a subject involving motion, resulting in image interference.

On the other hand, the simultaneous image pickup method is a method of obtaining color image data by picking up an image and then performing image processing for the color image data, thereby generating infrared image data. According to this method, a color shift does not occur for a subject involving motion, but there is a problem of low information accuracy of the infrared image data.

Thus, an image pickup device used with an endoscope can obtain color image data in which one pixel has three-color (RGB) information and infrared image data in which one pixel only has infrared information by the apparatus configuration disclosed in Japanese Patent No. 2648494 or the image processing. Particularly, the infrared image data is used to check information, etc., of blood capillary of mucosal surface layer and a mucosal minute pattern in the part to be tested. Therefore, high contrast is demanded for the infrared image data.

Also, color reproducibility of the RGB color image data will be described. Usually, R, G, and B color filters also transmit infrared-region wavelength. Thus, if light passing through the R, G, and B color filters is detected in the photoelectric conversion elements and RGB color image data is generated, the color reproducibility thereof is not good. Then, an infrared (IR) cut filter is provided in front of an image pickup device using R, G, and B color filters so that the light passing through the R, G, and B color filters does not contain infrared-region light. Thereby, the color reproducibility is improved.

However, an IR cut filter having a steep IR cut property is expensive and is at high cost. An IR cut filter must be provided in front of the image pickup device and miniaturization of whole system using the image pickup device is also hindered. If an attempt is made to use an IR cut filter in the apparatus described in Japanese Patent No. 2648494, only when R, G, and B color filters come in front of the image pickup device, the IR cut filter needs to be placed in front of the R, G or B filter, and the mechanism and control of the system become complicated.

SUMMARY OF THE INVENTION

The invention provides an image processing apparatus capable of generating infrared image data with high contrast from an image pickup signal obtained from an image pickup device.

Also, the invention may further provide an image processing apparatus capable of generating RGB color image data good in color reproducibility from the image pickup signal from the image pickup device without providing an IR cut filter in front of the image pickup device.

(1) According to an aspect of the invention, an image processing apparatus for generating image data from an image pickup signal output from an image pickup device includes a color image data generation unit, an infrared image data generation unit and a high-contrast infrared image data generation unit. The color image data generation unit generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device. The infrared image data generation unit generates infrared image data from an image pickup signal of an infrared component output from the image pickup device. The high-contrast infrared image data generation unit generates high-contrast infrared image data using the color image data and the infrared image data. Contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data.
(2) In the image processing apparatus of (1), pixel data of the color image data may include red-component data, green-component data and blue-component data. Pixel data of the infrared image data may include infrared-component data. The high-contrast infrared image data generation unit may generate the high-contrast infrared image data using the following expression:
I(x, y)=rR(x, y)+gG(x, y)+bB(x, y)+irIR(x, y)
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates (x, y), R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y), G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y), B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and r1, g1, b1 and ir1 denote coefficients. The coefficients r1, g1, b1, and ir1 are determined so that r1×R(λ)+g1×G(λ)+b1×B(λ)+ir1×IR(λ) is as close as possible to Real(λ), where R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component, G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component, B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component, IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and Real(λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast
(3) In the image processing apparatus of (2), the observation target substance may be hemoglobin.
(4) The image processing apparatus of any one of (1) to (3) may further include a high-color-reproduction color image data generation unit. The high-color-reproduction color image data generation unit generates high-color-reproduction color image data using the color image data and the infrared image data. The high-color-reproduction color image data has color reproducibility higher than the color image data.
(5) In the image processing apparatus of (4), the high-color-reproduction color image data generation unit may generate each pixel data of the high-color-reproduction color image data using the following expression:

( R O ( x , y ) G O ( x , y ) B O ( x , y ) ) = ( r 2 g 2 b 2 i r 2 r 3 g 3 b 3 i r 3 r 4 g 4 b 4 i r 4 ) ( R ( x , y ) G ( x , y ) B ( x , y ) I r ( x , y ) )
where RO(x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y), GO(x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y), BO(x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y), R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y), G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y), B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y), Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3 and ir4 denote coefficients. The coefficients r2, g2, b2, and ir2 are determined so that r2×R(λ)+g2×G(λ)+b2×B(λ)+ir2×IR(λ) is as close as possible to RO(λ), where R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component, G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component, B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component, IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and RO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component. The coefficients r3, g3, b3, and ir3 are determined so that r3×R(λ)+g3×G(λ)+b3×B(λ)+ir3×IR(λ) is as close as possible to GO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component. The coefficients r4, g4, b4, and ir4 are determined so that r4×R(λ)+g4×G(λ)+b4×B(λ)+ir4×IR(λ) is as close as possible to BO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
(6) In the image processing apparatus of any one of (1) to (5), wherein the image pickup device may include a large number of first photoelectric conversion elements, second photoelectric conversion elements, a color filter layer and a signal reading portion. The first photoelectric conversion elements are arranged on a first plane in a semiconductor substrate. The second photoelectric conversion elements are formed on a second plane, which is located above the first photoelectric conversion elements. The second photoelectric conversion elements correspond to a part of the large number of first photoelectric conversion elements. The second photoelectric conversion elements include first electrodes formed above the first photoelectric conversion elements. The photoelectric conversion layer is formed on the first electrodes, and a second electrode formed on the photoelectric conversion layer. The color filter layer is formed above the first photoelectric conversion elements. The color filter layer transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs. The signal reading portion reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements. The color filter layer include a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively. The large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range. Of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light. The photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light. The part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
(7) In the image processing apparatus of (6), the color filter layer may be formed above the second photoelectric conversion elements.
(8) In the image processing apparatus of (7), the photoelectric conversion layer may contain an organic material. The image pickup device may further include a protective layer that protects the second photoelectric conversion elements. The protective layer is formed by an ALCVD method between the first photoelectric conversion elements and the color filter layer.
(9) In the image processing apparatus of (8), the protective layer may contain an inorganic material.
(10) In the image processing apparatus of (9), the protective layer may have a two-layer structure including an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
(11) In the image processing apparatus of any one of (6) to (10), the image pickup device may further include a microlens that collects light in each of the large number of first photoelectric conversion elements.
(12) According to another aspect of the invention, an endoscope includes an image processing apparatus; and the image processing apparatus of any one of (1) to (11).
(13) According to further another aspect of the invention, a computer readable medium stores a program for causing a computer to execute a process for image processing. The image processing includes: generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device; generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and generating high-contrast infrared image data using the color image data and the infrared image data. Contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data.

According to any of the above configurations of (1) to (13), there can be provided an image processing apparatus capable of generating infrared image data in high contrast from an image pickup signal obtained from an image pickup device, an endoscope and a computer readable medium storing an image processing program.

(14) According to still further another aspect of the invention, an image processing apparatus for generating image data from an image pickup signal output from an image pickup device includes a color image data generation unit, an infrared image data generation unit and a high-color-reproduction color image data generation unit. The color image data generation unit generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device. The infrared image data generation unit generates infrared image data from an image pickup signal of an infrared component output from the image pickup device. The high-color-reproduction color image data generation unit generates high-color-reproduction color image data using the color image data and the infrared image data. The high-color-reproduction color image data has color reproducibility higher than the color image data.
(15) In the image processing apparatus of (14), the high-color-reproduction color image data generation unit may generate each pixel data of the high-color-reproduction color image data using the following expression:

( R O ( x , y ) G O ( x , y ) B O ( x , y ) ) = ( r 2 g 2 b 2 i r 2 r 3 g 3 b 3 i r 3 r 4 g 4 b 4 i r 4 ) ( R ( x , y ) G ( x , y ) B ( x , y ) I r ( x , y ) )
where RO(x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y), GO(x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y), BO(x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y), R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y), G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y), B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y), Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3 and ir4 denote coefficients. The coefficients r2, g2, b2, and ir2 are determined so that r2×R(λ)+g2×G(λ)+b2×B(λ)+ir2×IR(λ) is as close as possible to RO(λ), where R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component, G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component, B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component, IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and RO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component. The coefficients r3, g3, b3, and ir3 are determined so that r3×R(λ)+g3×G(λ)+b3×B(λ)+ir3×IR(λ) is as close as possible to GO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component, and the coefficients r4, g4, b4, and ir4 are determined so that r4×R(λ)+g4×G(λ)+b4×B(λ)+ir4×IR(λ) is as close as possible to BO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
(16) In the image processing apparatus of any one of (14) to (15), wherein the image pickup device may include a large number of first photoelectric conversion elements, second photoelectric conversion elements, a color filter layer and a signal reading portion. The first photoelectric conversion elements are arranged on a first plane in a semiconductor substrate. The second photoelectric conversion elements are formed on a second plane, which is located above the first photoelectric conversion elements. The second photoelectric conversion elements correspond to a part of the large number of first photoelectric conversion elements. The second photoelectric conversion elements include first electrodes formed above the first photoelectric conversion elements. The photoelectric conversion layer formed on the first electrodes, and a second electrode formed on the photoelectric conversion layer. The color filter layer is formed above the first photoelectric conversion elements. The color filter layer transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs. The signal reading portion reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements. The color filter layer includes a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively. The large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range. Of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light. The photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light. The part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
(17) In the image processing apparatus of (16), the color filter layer may be formed above the second photoelectric conversion elements.
(18) In the image processing apparatus of (17), the photoelectric conversion layer may contain an organic material. The image pickup device may further include a protective layer that protects the second photoelectric conversion elements. The protective layer is formed by an ALCVD method between the first photoelectric conversion elements and the color filter layer.
(19) In the image processing apparatus of (18), the protective layer may contain an inorganic material.
(20) In the image processing apparatus of (19), the protective layer may have a two-layer structure including an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
(21) In the image processing apparatus of any one of (16) to (20), the image pickup device may further include a microlens that collects light in each of the large number of first photoelectric conversion elements.
(22) According to another aspect of the invention, an endoscope includes an image processing apparatus; and the image processing apparatus of any one of (14) to (21).
(23) According to further another aspect of the invention, a computer readable medium stores a program for causing a computer to execute a process for image processing. The image processing includes: generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device; generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and generating high-color-reproduction color image data using the color image data and the infrared image data. The high-color-reproduction color image data has color reproducibility higher than the color image data.

According to any of the above configurations of (14) to (23), there can be provided an image processing apparatus capable of generating RGB color image data good in color reproducibility from the image pickup signal from the image pickup device without providing an IR cut filter in front of the image pickup device, an endoscope and a computer readable medium storing an image processing program.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a partial schematic diagram of a surface of an image pickup device to describe an embodiment of the invention;

FIG. 2 is a sectional schematic diagram of the image pickup device taken along a line A-A in FIG. 1;

FIG. 3 is a drawing to show a specific configuration example of a signal reading portion 5 r shown in FIG. 2;

FIG. 4 is a drawing to show the characteristics of first photoelectric conversion elements and second photoelectric conversion elements of the image pickup device shown in FIG. 1;

FIG. 5 is a drawing to show the characteristics of color filters of the image pickup device shown in FIG. 1;

FIG. 6 is a drawing to show the characteristics of the image pickup device shown in FIG. 1 with no correction filter;

FIG. 7 is a drawing to show the characteristics of a correction filter;

FIG. 8 is a drawing to show the characteristics of the image pickup device shown in FIG. 1 with the correction filter;

FIG. 9 is a drawing to show the spectral reflectivity of hemoglobin;

FIG. 10 is a drawing to show modified examples of the configuration of the image pickup device shown in FIG. 1;

FIG. 11 is a drawing to show the schematic configuration of an endoscope to describe a second embodiment of the invention;

FIG. 12 is a drawing to show spectral reflectivity of hemoglobin and spectral sensitivity characteristic which makes it possible to detect a state change of hemoglobin with the highest contrast;

FIG. 13 is a drawing to show characteristic obtained by performing a process of bringing the characteristic of R second photoelectric conversion element of the image pickup device shown in FIG. 1 to be close to the characteristic shown in Real curve in FIG. 12;

FIG. 14 is a drawing to show the detection sensitivity characteristic of hemoglobin when an image is picked up with the image pickup device having the characteristic shown in FIG. 13;

FIG. 15 is a drawing to show the spectral sensitivity characteristic of R second photoelectric conversion element shown in FIG. 8;

FIG. 16 is a drawing to show the detection sensitivity characteristic of hemoglobin when an image is picked up with the image pickup device having the characteristic shown in FIG. 15;

FIG. 17 is a drawing to show standard RGB ideal image pickup characteristic; and

FIG. 18 is a drawing to show characteristics obtained by performing a process of bringing the characteristics of the R, G, and B first photoelectric conversion elements of the image pickup device shown in FIG. 1 to be close to the characteristics shown in FIG. 17.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Referring now to the accompanying drawings, embodiments of the invention will be described below.

First Embodiment

FIG. 1 is a partial schematic diagram of a surface of an image pickup device to describe an embodiment of the invention. FIG. 2 is a sectional schematic diagram of the image pickup device taken along a line A-A in FIG. 1. In FIG. 1, a microlens 14 is not shown.

A p-well layer 2 is formed on an n-type silicon substrate 1. In the following description, the n-type silicon substrate 1 and the p-well layer 2 are collectively called a semiconductor substrate. Three types of color filters, namely, a large number of color filters 13 r for mainly transmitting light of R wavelength range, a large number of color filters 13 g for mainly transmitting light of G wavelength range, and a large number of color filters 13 b for mainly transmitting light of B wavelength range are arranged in a row direction and in a column direction perpendicular to the row direction on the same plane above the semiconductor substrate.

Although a known material may be used as the color filter 13 r, such a material also transmits a part of light of infrared region as well as light of R wavelength range. Although a known material may be used as the color filter 13 g, such a material also transmits a part of light of infrared region in addition to light of G wavelength range. Although a known material may be used as the color filter 13 b, such a material also transmits a part of light of infrared region in addition to light of B wavelength range.

Color filter arrangement used in a known single plate CCD (Bayer arrangement, longitudinal stripe, lateral stripe, etc.,) may be adopted for the arrangement of the color filter 13 r, 13 g, 13 b.

An n-type impurity region (n region) 3 r is formed in the p-well layer 2 below the color filter 13 r so as to correspond to the color filter 13 r, and a pn junction between the n region 3 r and the p-well layer 2 forms an R first photoelectric conversion element corresponding to the color filter 13 r.

An n region 3 g is formed in the p-well layer 2 below the color filter 13 g so as to correspond to the color filter 13 g, and a pn junction between the n region 3 g and the p-well layer 2 forms a G first photoelectric conversion element corresponding to the color filter 13 g.

An n region 3 b is formed in the p-well layer below the color filter 13 b so as to correspond to the color filter 13 b, and a pn junction between the n region 3 b and the p-well layer 2 forms a B first photoelectric conversion element corresponding to the color filter 13 b.

A transparent electrode 7 r is formed above the n region 3 r, a transparent electrode 7 g is formed above the n region 3 g, and a transparent electrode 7 b is formed above the n region 3 b. The transparent electrodes 7 r, 7 g, and 7 b are divided correspondingly to the color filters 13 r, 13 g, and 13 b, respectively. Each of the transparent electrodes 7 r, 7 g, and 7 b is formed of a material which is transparent for both of visible light and infrared light. For example, ITO, IZO, etc., may be used. Each of the transparent electrodes 7 r, 7 g, and 7 b is buried in an insulating layer 8.

Formed on the transparent electrodes 7 r, 7 g, and 7 b is a photoelectric conversion layer 9 of a one-sheet structure common to the color filters 13 r, 13 g, and 13 b. The photoelectric conversion layer 9 mainly absorbs infrared-region light of wavelength 580 nm or more, generates charges in response thereto and transmits light of visible region other than the infrared region (wavelength in a range of about 380 nm to about 580 nm). For example, a phthalocyanine-based organic material or a naphthalocyanine-based organic material is used as a material forming the photoelectric conversion layer 9.

A transparent electrode 10 having a one-sheet structure common to the color filters 13 r, 13 g, and 13 b is formed on the photoelectric conversion layer 9. The transparent electrode 10 is formed of a transparent material for both of visible light and infrared light. For example, ITO, IZO, etc., may be used.

The transparent electrode 7 r, the transparent electrode 10 facing the transparent electrode 7 r, and a part of the photoelectric conversion layer 9 sandwiched therebetween form a photoelectric conversion element corresponding to the color filter 13 r. Hereinafter, this photoelectric conversion element will be referred to as an R second photoelectric conversion element.

The transparent electrode 7 g, the transparent electrode 10 facing the transparent electrode 7 g, and a part of the photoelectric conversion layer 9 sandwiched therebetween form a photoelectric conversion element corresponding to the color filter 13 g. Hereinafter, this photoelectric conversion element may be called a G second photoelectric conversion element.

The transparent electrode 7 b, the transparent electrode 10 facing the transparent electrode 7 b, and a part of the photoelectric conversion layer 9 sandwiched therebetween form a photoelectric conversion element corresponding to the color filter 13 b. Hereinafter, this photoelectric conversion element may be called a B second photoelectric conversion element.

A high-concentration n-type impurity region (which will be hereinafter referred to as n+ region) 4 r is formed adjacent to the n region 3 r in the p-well layer 2 to store charges generated in the photoelectric conversion layer 9 of the R second photoelectric conversion element. To prevent light from entering the n+ region 4 r, preferably a shading film is provided on the n+ region 4 r.

An n+ region 4 g is formed adjacent to the n region 3 g in the p-well layer 2 to store charges generated in the photoelectric conversion layer 9 of the G second photoelectric conversion element. To prevent light from entering the n+ region 4 g, preferably a shading film is provided on the n+ region 4 g.

An n+ region 4 b is formed adjacent to the n region 3 b in the p-well layer 2 to store charges generated in the photoelectric conversion layer 9 of the B second photoelectric conversion element. To prevent light from entering the n+ region 4 b, preferably a shading film is provided on the n+ region 4 b.

A contact portion 6 r made of a metal such as aluminum is formed on the n+ region 4 r, and the transparent electrode 7 r is formed on the contact portion 6 r. As a result, the n+ region 4 r and the transparent electrode 7 r are electrically connected by the contact portion 6 r. The contact portion 6 r is buried in the insulating layer 5 transparent for both of visible light and infrared light.

A contact portion 6 g made of a metal such as aluminum is formed on the n+ region 4 g, and the transparent electrode 7 g is formed on the contact portion 6 g. As a result, the n+ region 4 g and the transparent electrode 7 g are electrically connected by the contact portion 6 g. The contact portion 6 g is buried in the insulating layer 5.

A contact portion 6 b made of a metal such as aluminum is formed on the n+ region 4 b, and the transparent electrode 7 b is formed on the contact portion 6 b. As a result, the n+ region 4 b and the transparent electrode 7 b are electrically connected by the contact portion 6 b. The contact portion 6 b is buried in the insulating layer 5.

A signal reading portion 5 r for reading signals corresponding to the charges which are generated in the R photoelectric conversion element and which are stored in the n region 3 r and signals corresponding to the charges stored in the n+ region 4 r, a signal reading portion 5 g for reading signals corresponding to the charges which are generated in the G photoelectric conversion element and which are stored in the n region 3 g and signals corresponding to the charges stored in the n+ region 4 g, and a signal reading portion 5 b for reading signals corresponding to the charges which are generated in the B photoelectric conversion element and which are stored in the n region 3 b and signals corresponding to the charges stored in the n+ region 4 b are formed in other regions than the n region 3 r, 3 g, 3 b or the n+ region 4 r, 4 g, 4 b in the p-well layer 2. A known configuration using a CCD or a MOS circuit may be adopted for each of the signal reading portions 5 r, 5 g, and 5 b. To prevent light from entering the signal reading portion 5 r, 5 g, 5 b, preferably a shading film is provided on the signal reading portion 5 r, 5 g, 5 b.

FIG. 3 is a drawing to show a specific configuration example of the signal reading portion 5 r shown in FIG. 2. Components similar to those in FIGS. 1 and 2 are denoted by the same reference numerals in FIG. 3. The configurations of the signal reading portions 5 r, 5 g, and 5 b are identical, and therefore the signal reading portions 5 g and 5 b will not be described.

The signal reading portion 5 r includes a rest transistor 43, an output transistor 42, a row selection transistor 41, a rest transistor 46, an output transistor 47 and a row selection transistor 48. The reset transistor 43 has a drain connected to the n+ region 4 r and a source connected to power supply Vn. The output transistor 42 has a gate connected to the drain of the reset transistor 43 and a source connected to a power supply Vcc. The row selection transistor 41 has a source connected to the drain of the output transistor 42 and a drain connected to a signal output line 45. The reset transistor 46 has a drain connected to the n region 3 r and a source connected to the power supply Vn. The output transistor 47 has a gate connected to the drain of the reset transistor 46 and a source connected to the power supply Vcc. The row selection transistor 48 has a source connected to the drain of the output transistor 47 and a drain connected to a signal output line 49.

When a bias voltage is applied between the transparent electrode 7 r and the transparent electrode 10, charges are generated in response to the light incident on the photoelectric conversion layer 9 and move through the transparent electrode 7 r to the n+ region 4 r. The charges stored in the n+ region 4 r are converted into signals corresponding to the charge amount by the output transistor 42. When the row selection transistor 41 is turned ON, the signals are output to the signal output line 45. After the signal is output, the charges in the n+ region 4 r are reset by the reset transistor 43.

The charges generated in the R photoelectric conversion element and stored in the n region 3 r are converted into signals corresponding to the charge amount, by the output transistor 47. When the row selection transistor 48 is turned ON, the signals are output to the signal output line 49. After the signals are output, the charges in the n region 3 r are reset by the reset transistor 46.

Thus, the signal reading portion 5 r can be configured using a known MOS circuit including three transistors.

Referring back to FIG. 2, protective layers 11 and 12 having a two-layer structure to protect the second photoelectric conversion elements are formed above the photoelectric conversion layer 9, the color filters 13 r, 13 g, and 13 b are formed on the protective layer 12, and microlenses 14 for collecting light in the corresponding n regions 3 r, 3 g, and 3 b are formed on the color filters 13 r, 13 g, and 13 b.

To manufacture the image pickup device 100, the photoelectric conversion layer 9 is formed and then the color filters 13 r, 13 g, and 13 b, the microlenses 14, etc., are formed. Since the color filters 13 r, 13 g, and 13 b and the microlenses 14 involve a photolithography step and a baking step, if an organic material is used as the photoelectric conversion layer 9, and the photolithography step and the baking step are executed with the photoelectric conversion layer 9 exposed, the characteristic of the photoelectric conversion layer 9 would be deteriorated. The image pickup device 100 is provided with the protective layers 11 and 12 to prevent characteristic deterioration of the photoelectric conversion layer 9 during the manufacturing steps.

Preferably, the protective layer 11 is an inorganic layer made of an inorganic material formed by an ALCVD method. The ALCVD method is an atomic layer CVD method and can form a tight inorganic layer; an effective protective layer for the photoelectric conversion layer 9 can be provided. The ALCVD method is also known as ALE method or ALD method. Preferably, the inorganic layer formed by the ALCVD method is made of Al2O3, SiO2, TiO2, ZrO2, MgO, HfO2, Ta2O5; more preferably, Al2O3, SiO2; most preferably, Al2O3.

The protective layer 12 is formed on the protective layer 11 for the purpose of further improving the protective performance of the photoelectric conversion layer 9. Preferably, the protective layer 12 is an organic layer made of an organic polymer. Preferably, the organic polymer is perylene; more preferably, perylene C. The protective layer 12 may be omitted. The protective layers 11 and 12 may be placed in an opposite manner. The configuration shown in FIG. 2 provides a particularly high protection effect of the photoelectric conversion layer 9.

In the image pickup device 100 described above, infrared-region light of light passing through the color filter 13 r, of incident light is absorbed by the photoelectric conversion layer 9 where charges responsive to the infrared-region light are generated. Likewise, infrared-region light of light passing through the color filter 13 g, of the incident light is absorbed by the photoelectric conversion layer 9 where charges responsive to the infrared-region light are generated. Likewise, infrared-region light of light passing through the color filter 13 b, of the incident light is absorbed by the photoelectric conversion layer 9 where charges responsive to the infrared region light are generated.

When a predetermined bias voltage is applied to the transparent electrode 7 r and the transparent electrode 10, the charges generated in the photoelectric conversion layer 9 that forms the R second photoelectric conversion element move through the transparent electrode 7 r and the contact portion 6 r to the n+ region 4 r where the charges are stored. The signals corresponding to the charges stored in the n+ region 4 r are read by the signal reading portion 5 r and are output to the outside of the image pickup device 100.

Likewise, when a predetermined bias voltage is applied to the transparent electrode 7 g and the transparent electrode 10, charges generated in the photoelectric conversion layer 9 that forms the G second photoelectric conversion element move through the transparent electrode 7 g and the contact portion 6 g to the n+ region 4 g where the charges are stored. The signals corresponding to the charges stored in the n+ region 4 g are read by the signal reading portion 5 r and are output to the outside of the image pickup device 100.

Likewise, when a predetermined bias voltage is applied to the transparent electrode 7 b and the transparent electrode 10, charges generated in the photoelectric conversion layer 9 that forms the B second photoelectric conversion element move through the transparent electrode 7 b and the contact portion 6 b to the n+ region 4 b where the charges are stored. The signals corresponding to the charges stored in the n+ region 4 b are read by the signal reading portion 5 b and are output to the outside of the image pickup device 100.

R wavelength range light passing through the color filter 13 r and passing through the photoelectric conversion layer 9 is incident on the R first photoelectric conversion element and charges responsive to the incident light amount are stored in the n region 3 r. Likewise, G wavelength range light passing through the color filter 13 g and passing through the photoelectric conversion layer 9 is incident on the G first photoelectric conversion element and charges responsive to the incident light amount are stored in the n region 3 g. Likewise, B wavelength range light passing through the color filter 13 b and passing through the photoelectric conversion layer 9 is incident on the B first photoelectric conversion element and charges responsive to the incident light amount are stored in the n region 3 b. The charges stored in the n regions 3 r, 3 g, and 3 b are read by the signal reading portions 5 r, 5 g, and 5 b and are output to the outside of the image pickup device 100.

The arrangement of the signals read and output from the n regions 3 r, 3 g, and 3 b becomes similar to the arrangement of the signals output from a single plate color CCD having a color filter arrangement as shown in FIG. 1. Thus, when signal processing used in the single plate color CCD is performed, color image data in which each pixel data has three-color-component (R, G, and B) data can be generated. Also, infrared image data in which each pixel data has an infrared color component data can be generated from the signals read and output from the n+ regions 4 r, 4 g, and 4 b.

In this manner, the image pickup device 100 can output the R component signal corresponding to the charges generated in the R first photoelectric conversion element, the G component signal corresponding to the charges generated in the G first photoelectric conversion element, the B component signal corresponding to the charges generated in the B first photoelectric conversion element, the IR component signal corresponding to the charges generated in the R second photoelectric conversion element, the IR component signal corresponding to the charges generated in the G second photoelectric conversion element, and the IR component signal corresponding to the charges generated in the B second photoelectric conversion element to the outside. Thus, the image pickup device 100 can provide two types of image data, that is, color image data and infrared image data by one image picking up process. Therefore, for example, the image pickup device 100 can be used as an image pickup device of an endoscope which is required to capture an appearance image of a part of a human being to be tested and an internal image of the part.

Next, the spectral sensitivity characteristic of the image pickup device 100 will be described.

First, it is assumed that the spectral sensitivity characteristic of each first photoelectric conversion element (PD) formed in the semiconductor substrate is as shown in FIG. 4, that the spectral sensitivity characteristic of the photoelectric conversion layer 9 is as shown in FIG. 4, that the spectral transmissivity of the photoelectric conversion layer 9 is as shown in FIG. 4, and that the spectral transmissivitys of the color filters 13 r, 13 g, and 13 b are as shown in FIG. 5. In FIG. 4, the vertical axis indicates the spectral sensitivity or the spectral transmissivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 5, the vertical axis indicates the spectral transmissivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

When the characteristic is thus determined, the spectral sensitivity characteristic of the R first photoelectric conversion element becomes a product of the spectral sensitivity of the first photoelectric conversion element (PD), the spectral transmissivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 r; the spectral sensitivity characteristic of the G photoelectric conversion element becomes the product of the spectral sensitivity of the first photoelectric conversion element (PD), the spectral transmissivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 g, and the spectral sensitivity characteristic of the B photoelectric conversion element becomes the product of the spectral sensitivity of the first photoelectric conversion element (PD), the spectral transmissivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 b, as shown in FIG. 6. In FIG. 6, the vertical axis indicates the spectral sensitivity when 1 is the reference, and the horizontal axis indicates the wavelength of light.

The spectral sensitivity characteristic of the R second photoelectric conversion element becomes the product of the spectral sensitivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 r, the spectral sensitivity characteristic of the G second photoelectric conversion element becomes the product of the spectral sensitivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 g, and the spectral sensitivity characteristic of the B second photoelectric conversion element becomes the product of the spectral sensitivity of the photoelectric conversion layer 9 and the spectral transmissivity of the color filter 13 b, as shown in FIG. 6.

Here, to adjust the spectral sensitivity characteristic of each second photoelectric conversion element, if a correction filter having a spectral transmissivity as shown in FIG. 7 is placed on the light incident face side of the image pickup device 100, the spectral sensitivity characteristic of the image pickup device 100 becomes as shown in FIG. 8. In FIG. 7, the vertical axis indicates the spectral transmissivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 8, the vertical axis indicates the spectral sensitivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

FIG. 9 is a drawing to show the spectral reflectivity of oxygenated hemoglobin (oxy) and reduced hemoglobin (deoxy). In FIG. 9, the vertical axis indicates the spectral reflection factor where 1 is used as a reference, and the horizontal axis indicates the wavelength of light.

As can be seen in FIG. 9, the reflectivity difference between oxygenated hemoglobin and reduced hemoglobin becomes large in the wavelength range of wavelengths 580 nm to 780 nm. Thus, if a photoelectric conversion element having sensitivity in this wavelength range is used, state change of hemoglobin can be put into an image with high contrast. The image pickup device 100 has the R second photoelectric conversion element having strong sensitivity in the wavelength range of wavelengths 580 nm to 780 nm as shown in FIG. 8. Therefore, when infrared image data is generated using the signals obtained from the R second photoelectric conversion element, it is made possible to provide an RGB color appearance image of the part to be tested and state change of hemoglobin of the part by a single image picking up process.

In the case where infrared image data is generated using the signals obtained only from the R second photoelectric conversion element, a signal may be interpolated, at positions of the signals obtained from the G second photoelectric conversion element and the B second photoelectric conversion element, using the signals obtained from the R second photoelectric conversion element on the periphery of the positions to generate infrared image data having the same resolution as color image data. Alternatively, only the signals obtained from the R second photoelectric conversion element may be used to generate infrared image data having one-third resolution of color image data. Further alternatively, the signals obtained from the three photoelectric conversion elements of the R second photoelectric conversion element, the G second photoelectric conversion element, and the B second photoelectric conversion element aligning in the row direction may be added to form one signal and infrared image data having one-third resolution of color image data may be generated based on this signal.

When this image pickup device 100 is used, the two types of image data, that is, color image data and infrared image data can be obtained. Such an effect can also be achieved if a complementary color system other than the primary color system is used as the color filters used with the image pickup device 100. Although the two types of image data cannot be obtained, if the color arrangement of the color filters of the image pickup device 100 and the wavelength range of light to be absorbed in the photoelectric conversion layer are adjusted, it is also made possible to provide RGB image data having higher resolution than a single plate image pickup device. FIG. 10 shows configuration examples of the image pickup device 100 to produce the effects. In FIG. 10, first photoelectric conversion elements (PD) formed in semiconductor substrate, second photoelectric conversion layer formed above the PD, and color filters formed above the photoelectric conversion layer are only shown as the components of the image pickup device 100.

An image pickup device shown in FIG. 10( a) is provided by changing the color filter 13 r to a Cy filter for transmitting light in the wavelength range of Cy (cyan) and a part of infrared region light, changing the color filter 13 g to an Mg filter for transmitting light in the wavelength range of Mg (magenta) and a part of infrared region light, and changing the color filter 13 b to a Ye filter for transmitting light in the wavelength range of Ye (yellow) and a part of infrared region light in the image pickup device 100 shown in FIGS. 1 and 2. The Cy filter, the Mg filter, and the Ye filter may be made of known materials.

With this configuration, color image data can be generated from the Cy, Mg, and Ye signals obtained from the photoelectric conversion elements in the semiconductor substrate and infrared image data can be generated from the signals obtained from the photoelectric conversion layer. The arrangement of the Cy filter, the Mg filter, and the Ye filter may be any so long as a color image can be reproduced.

An image pickup device shown in FIG. 10( b) is provided by changing the color filter 13 r to a Cy filter and changing the color filter 13 b to a Ye filter in the image pickup device 100 shown in FIGS. 1 and 2.

With this configuration, color image data can be generated from the Cy, G, and Ye signals obtained from the photoelectric conversion elements in the semiconductor substrate and infrared image data can be generated from the signals obtained from the photoelectric conversion layer. The arrangement of the Cy filter, the G filter, and the Ye filter may be any so long as a color image can be reproduced.

An image pickup device shown in FIG. 10( c) is provided by changing the color filter 13 r to a Cy filter, changing the color filter 13 g to an IR filter for transmitting infrared region light, changing the photoelectric conversion layer 9 to a G photoelectric conversion layer for absorbing light in the wavelength range of G, generating signal charges responsive thereto, and transmitting light other than light in the wavelength range of G, and changing the color filter 13 b to a Ye filter in the image pickup device 100 shown in FIGS. 1 and 2. As a material forming the G photoelectric conversion layer, InGaAlP or GaPAs may be used, for example, as an inorganic material; R6G/PMPS (rhodamine 6G (R6G)-doped polymethylphenylsilane) may be used, for example, as an organic material.

With this configuration, color image data can be generated from the B and R signals obtained from the photoelectric conversion elements in the semiconductor substrate and the G signal obtained from the photoelectric conversion layer, and infrared image data can be generated from the IR signal obtained from the photoelectric conversion layer below the IR filter. The arrangement of the Cy filter and the Ye filter may be any so long as a color image can be reproduced, and the arrangement of the Ir filter may be any so long as an infrared image can be reproduced.

An image pickup device shown in FIG. 10( d) is provided by changing the color filters 13 r and 13 b to Cy filters, changing the color filter 13 g to a Ye filter, and changing the photoelectric conversion layer 9 to a G photoelectric conversion layer in the image pickup device 100 shown in FIGS. 1 and 2.

With this configuration, color image data can be generated from the B and R signals obtained from the photoelectric conversion elements in the semiconductor substrate and the G signal obtained from the photoelectric conversion layer. In this configuration, primary color system signals of two colors can be obtained per image pickup point, so that the resolution can be improved as compared with a single plate image pickup device.

The case where the number of types of color filters used in the image pickup device 100 is two or three has been described. However, similar advantages can be achieved even if the number of types of color filters is four or more. The number of types of color filters may be one. In this case, a G color filter of a one-sheet configuration for transmitting light in the wavelength range of G may be provided in place of the color filters 13 r, 13 g, and 13 b, for example, in the configuration shown in FIG. 2.

With this configuration, monochrome image data can be generated from the signals obtained from the photoelectric conversion elements in the semiconductor substrate, and infrared image data can be generated from the signals obtained from the photoelectric conversion layer 9. When this configuration is adopted, there is also the advantage that the spectral sensitivity characteristic of the photoelectric conversion layer can be adjusted according to the spectral transmissivity of the filter provided above the photoelectric conversion layer.

In the description given above, the photoelectric conversion layer is provided above the semiconductor substrate and the color filters are provided above the photoelectric conversion layer. However, similar advantages can be achieved even if the photoelectric conversion layer and the color filters are placed in an opposite manner.

In the description given above, the color filters 13 r, 13 g, and 13 b transmit infrared region light. However, a filter having a spectral transmissivity not allowing infrared region light to pass through may also be used. However, if all color filters are those for not allowing infrared region light to pass through, it becomes impossible to generate infrared image data. Therefore, at least one of color filters of one type or more needs to be provided with a function of allowing infrared region light to pass through.

In the description given above, the three types of second photoelectric conversion elements, that is, R, G, and B second photoelectric conversion elements are provided. However, if at least one of them exists, it is sufficient to obtain infrared image data. As shown in FIGS. 6 and 8, the R second photoelectric conversion element has the most sensitivity in the infrared region and therefore most preferably, infrared image data is generated using the signal output from the R second photoelectric conversion element. In the case where the G second photoelectric conversion element is omitted, the transparent electrode 7 g, the contact portion 6 g, and the n+ region 4 g may be omitted in the configuration shown in FIG. 2. In the case where the B second photoelectric conversion element is omitted, the transparent electrode 7 b, the contact portion 6 b, and the n+ region 4 b may be omitted in the configuration shown in FIG. 2.

In the configuration shown in FIG. 10( c), if the second photoelectric conversion elements provided in the image pickup device are only the second photoelectric conversion elements corresponding to the IR filters, it is hard to obtain G component signals from the second photoelectric conversion elements and generation of color image data is hindered. Thus, in the configuration shown in FIG. 10( c), it is necessary to provide at least the second photoelectric conversion element corresponding to the Cy filter or the second photoelectric conversion element corresponding to the Ye filter.

Next, a method for manufacturing the image pickup device 100 will be described. The image pickup device 100 can be manufactured in the following processes (A) to (C):

(A) Formation of CMOS Substrate→Formation of Transparent Electrode

    • As with a conventional CMOS sensor, n regions 3 r, 3 g, and 3 b and signal reading portions are formed on a silicon substrate.
    • Further, n+ regions 4 r, 4 g, and 4 b and signal reading portions are formed.
    • Insulating layer 5 is formed on the silicon substrate, transparent electrodes 7 r, 7 g, and 7 b are formed thereon, and the transparent electrodes 7 r, 7 g, and 7 b and the n+ regions 4 r, 4 g, and 4 b are brought into contact with each other using a via plug.
    • Insulating material is filled into gaps among the transparent electrodes 7 r, 7 g, and 7 b and the surfaces of the transparent electrodes 7 r, 7 g, and 7 b are made flat containing the insulating material portion using CMP.

This process is executed as a semiconductor process.

(B) Formation of Photoelectric Conversion Layer

    • Photoelectric conversion layer 9 is formed on the transparent electrodes 7 r, 7 g, and 7 b.
    • Further, transparent electrode 10 is formed. The transparent electrode 10 is brought into contact with a pad (not shown), and a bias voltage is applied to the transparent electrode 10 from an external power supply.

This process is executed as a vacuum evaporation process.

(C) Formation of Microlenses and Color Filters

    • An alumina protective layer is formed on the photoelectric conversion layer 9 by the ALCVD method, for example, and further a perylene C protective layer is formed.
    • Next, a mosaic color filter is formed. The mosaic color filter is formed in order of G resist application→pattern exposure→developing→post bake, B resist application→pattern exposure→developing→post bake, →R resist application→pattern exposure→developing→post bake.
    • Last, microlenses are formed. The microlenses are formed in order of resist application→post bake→resist application→pattern exposure→developing, melt.
Second Embodiment

In a second embodiment of the invention, an image pickup device 100 capable of providing color image data and infrared image data as described in the first embodiment is applied to an endoscope.

FIG. 11 is a drawing to show the schematic configuration of an endoscope to describe the second embodiment.

The endoscope shown in FIG. 11 includes a white light source 50, an optical system 51, the image pickup device 100, a correction filter 52, an infrared image data generation section 53, a color image data generation section 54, a high-contrast infrared image data generation section 55, a high-color-reproduction color image data generation section 56, an image enhancement section 57 and a display control section 58. The white light source 50 illuminates a part to be tested. The optical system 51 includes an imaging lens, an aperture diaphragm and the like. The image pickup device 100 has the configuration shown in FIGS. 1 and 2 and receives light passing through the optical system 51. The correction filter 52 is disposed between the image pickup device 100 and the optical system 51 and corrects the spectral sensitivity characteristic of the photoelectric conversion layer 9 of the image pickup device 100. The infrared image data generation section 53 generates infrared image data based on signals corresponding to charges generated in the photoelectric conversion layer 9 of the image pickup device 100. The color image data generation section 54 generates color image data based on signals corresponding to charges generated in the R, G, and B first photoelectric conversion elements of the image pickup device 100. The high-contrast infrared image data generation section 55 performs a computation process using the infrared image data generated by the infrared image data generation section 53 and the color image data generated by the color image data generation section 54 to generate high-contrast infrared image data that is more enhanced in contrast than the infrared image data generated by the infrared image data generation section 53. The image enhancement section 57 performs an enhancement process for the high-contrast infrared image data generated by the high-contrast infrared image data generation section 55. The high-color-reproduction color image data generation section 56 performs a computation process using the infrared image data generated by the infrared image data generation section 53 and the color image data generated by the color image data generation section 54 to generate high-color-reproduction color image data which has higher color reproducibility than the color image data generated by the color image data generation section 54. The display control section 58 controls a display 59 to display an image based on the high-contrast infrared image data for which the enhancement process is performed and to display an image based on the high-color-reproduction color image data.

The image pickup device 100 for use in the endoscope shown in FIG. 11 may not limited to the configuration shown in FIGS. 1 and 2, but may be any so long as it can output the four types of signals of an R component signal corresponding to light in the wavelength range of R, a G component signal corresponding to light in the wavelength range of G, a B component signal corresponding to light in the wavelength range of B, and an IR component signal corresponding to infrared region light. For example, the image pickup device 100 may be an image pickup device having the configuration as shown in FIG. 10( c) or may be a single plate image pickup device in which a color filter for transmitting light in the wavelength range of R or Cy, a color filter for transmitting light in the wavelength range of G or Mg, a color filter for transmitting light in the wavelength range of B or Ye, and a color filter for transmitting infrared region light are arranged like a mosaic on the same plane above the semiconductor substrate. The spectral sensitivity characteristic of the image pickup device 100 may be one shown in FIG. 8, for example.

The color image data generation section 54 acquires, from the image pickup device 100, signals corresponding to charges generated in the R first photoelectric conversion element of the image pickup device 100 (which will be hereinafter referred to as an R signal), signals corresponding to charges generated in the G first photoelectric conversion element of the image pickup device 100 (which will be hereinafter referred to as a G signal), and signals corresponding to charges generated in the B first photoelectric conversion element of the image pickup device 100 (which will be hereinafter referred to as a B signal). The color image data generation section 54 uses these signals to generate color image data according to a known technique.

The infrared image data generation section 53 generates infrared image data having the same resolution as the color image data by performing signal interpolation, etc., from signals corresponding to charges generated in the R second photoelectric conversion element of the image pickup device 100 (which will be hereinafter referred to as an IRr signal).

FIG. 12 is a drawing to show spectral reflectivities of oxygenated hemoglobin and reduced hemoglobin. In FIG. 12, the vertical axis indicates the spectral transmissivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. In FIG. 12, the vertical axis also indicates the spectral sensitivity of a photoelectric conversion element where 1 is used as a reference. If an image of hemoglobin is picked up with the photoelectric conversion element having spectral sensitivity represented by a Real curve shown in FIG. 12, state change of hemoglobin can be detected with the highest contrast.

Then, the high-contrast infrared image data generation section 55 performs the computation process of bringing the IRr signal obtained from the R second photoelectric conversion element to be close to the signal obtained from the photoelectric conversion element having the spectral sensitivity characteristic represented by the Real curve shown in FIG. 12, to thereby improve the contrast of the infrared image data.

Specifically, the high-contrast infrared image data generation section 55 performs computation represented by the following expression (1) to generate high-contrast infrared image data:
I(x,y)=rR(x,y)+gG(x, y)+bB(x, y)+irIR(x, y)  (1)
where I(x, y) denotes pixel data of high-contrast infrared image data at coordinates (x, y),

R(x, y) denotes R component pixel data of the color image data at the coordinates (x, y),

G(x, y) denotes G component pixel data of the color image data at the coordinates (x, y),

B(x, y) denotes B component pixel data of the color image data at the coordinates (x, y),

IR(x, y) denotes IR component pixel data of the infrared image data at the coordinates (x, y), and

r1, g1, b1, and ir1 denote coefficients which are determined based on the spectral sensitivity characteristic of the R first photoelectric conversion element, the spectral sensitivity characteristic of the G first photoelectric conversion element, the spectral sensitivity characteristic of the B first photoelectric conversion element, the spectral sensitivity characteristic of the R second photoelectric conversion element, and the spectral sensitivity characteristic represented by the Real curve shown in FIG. 12.

Let the spectral sensitivity at wave length λ of the R photoelectric conversion element shown in FIG. 8 be R(λ), the spectral sensitivity at wave length λ of the G photoelectric conversion element shown in FIG. 8 be G(λ), the spectral sensitivity at wave length λ of the B photoelectric conversion element shown in FIG. 8 be B(λ), the spectral sensitivity at wave length λ of the R second photoelectric conversion element shown in FIG. 8 be IR(λ), and the spectral sensitivity at wave length λ of the photoelectric conversion element having the characteristic represented by the Real curve shown in FIG. 12 be Real(λ). In this case, the coefficients r1, g1, b1, and ir1 are determined by the least squares method so that the value obtained by performing computation of the following expression (2) becomes as close as possible to Real(λ). The determined coefficient data is previously stored in a memory (not shown) of the endoscope.
r1×R(λ)+g1×G(λ)+b1×B(λ)+ir1×IR(λ)  (2)

FIG. 13 is a drawing to show the spectral sensitivity characteristic obtained by performing the computation of the expression (2) using the coefficients determined by the above-mentioned method. In FIG. 13, the vertical axis indicates the spectral sensitivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. Curve I shown in FIG. 13 becomes the spectral sensitivity characteristic of a virtual photoelectric conversion element that can obtain high-contrast infrared image data obtained by performing the computation of the expression (1).

FIG. 14 is a drawing to show the detection sensitivities of oxygenated hemoglobin and reduced hemoglobin when light from hemoglobin is detected by the photoelectric conversion element having the spectral sensitivity characteristic shown in FIG. 13. In FIG. 14, the vertical axis indicates the spectral sensitivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. FIG. 15 is a drawing to show the spectral sensitivity characteristic of the R second photoelectric conversion element shown in FIG. 8. In FIG. 15, the vertical axis indicates the spectral sensitivity where 1 is used as a reference, and the horizontal axis indicates the wavelength of light. FIG. 16 is a drawing to show the detection sensitivities of oxygenated hemoglobin and reduced hemoglobin when light from hemoglobin is detected by the R second photoelectric conversion element having the spectral sensitivity characteristic shown in FIG. 15. In FIG. 16, the vertical axis indicates the spectral sensitivity where 1 is used as reference, and the horizontal axis indicates the wavelength of light.

Comparing FIGS. 14 and 16, the contrast ratio of high-contrast infrared image data represented by a value resulting from dividing an area A surrounded by oxygenated hemoglobin waveform I(oxy) shown in FIG. 14 and the line with spectral sensitivity=0 by an area B surrounded by reduced hemoglobin waveform I(deoxy) and the line with spectral sensitivity=0 shown in FIG. 14 is 1.318, and the contrast ratio of high-contrast infrared image data represented by a value resulting from dividing an area C surrounded by oxygenated hemoglobin waveform I(oxy) shown in FIG. 16 and the line with spectral sensitivity=0 by an area D surrounded by reduced hemoglobin waveform I(deoxy) and the line with spectral sensitivity=0 shown in FIG. 16 is 1.166. It can be seen that the contrast of the infrared image data can be improved by performing the computation process represented by the expression (1).

Since the image pickup device for use in the endoscope according to the embodiment needs to output an IR signal, an infrared cut film as installed in a usual digital camera cannot be placed in front of the image pickup device. In the embodiment, a correction filter 52 for correcting the spectral sensitivity characteristic of the photoelectric conversion layer 9 is provided. Thus, each of the R, G, and B first photoelectric conversion elements has less infrared-region light sensitivity, but has some infrared-region light sensitivity. Consequently, it is concerned that color reproducibility of color image data may be deteriorated.

Then, the high-color-reproduction color image data generation section 56 performs a computation process of bringing the R signal obtained from the R first photoelectric conversion element to be close to a signal obtained from an r photoelectric conversion element having an ideal spectral sensitivity characteristic defined by the standard RGB ideal image pickup characteristic shown in FIG. 17, performs a computation process of bringing the G signal obtained from the G photoelectric conversion element to be close to a signal obtained from a g photoelectric conversion element having an ideal spectral sensitivity characteristic defined by the standard RGB ideal image pickup characteristic shown in FIG. 17, and performs a computation process of bringing the B signal obtained from the B photoelectric conversion element to be close to a signal obtained from a b photoelectric conversion element having an ideal spectral sensitivity characteristic defined by the standard RGB ideal image pickup characteristic shown in FIG. 17, to thereby generate high-color-reproduction color image data.

Specifically, the high-color-reproduction color image data generation section 56 performs computation represented by the following expression (3) to improve the color reproducibility of the color image data:

( R O ( x , y ) G O ( x , y ) B O ( x , y ) ) = ( r 2 g 2 b 2 i r 2 r 3 g 3 b 3 i r 3 r 4 g 4 b 4 i r 4 ) ( R ( x , y ) G ( x , y ) B ( x , y ) I r ( x , y ) ) ( 3 )
where Ro(x, y) denotes an R component of pixel data of high-color-reproduction color image data at coordinates (x, y),

Go (x, y) denotes a G component of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),

Bo (x, y) denotes a B component of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),

R(x, y) denotes an R component of pixel data of the color image data at the coordinates (x, y),

G(x, y) denotes a G component of the pixel data of the color image data at the coordinates (x, y),

B(x, y) denotes a B component of the pixel data of the color image data at the coordinates (x, y),

Ir(x, y) denotes an IR component of pixel data of the infrared image data at the coordinates (x, y), and

r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3, and ir4 denote coefficients which are determined based on the spectral sensitivity characteristic of the R first photoelectric conversion element, the spectral sensitivity characteristic of the G first photoelectric conversion element, the spectral sensitivity characteristic of the B first photoelectric conversion element, the spectral sensitivity characteristic of the R second photoelectric conversion element, and the standard RGB ideal image pickup characteristic shown in FIG. 17.

Let the spectral sensitivity at wave length λ of the R first photoelectric conversion element shown in FIG. 8 be R(λ), the spectral sensitivity at wave length λ of the G first photoelectric conversion element shown in FIG. 8 be G(λ), the spectral sensitivity at wave length λ of the B first photoelectric conversion element shown in FIG. 8 be B(λ), the spectral sensitivity at wave length λ of the R second photoelectric conversion element shown in FIG. 8 be IR(λ), and the spectral sensitivity at wave length λ of the r photoelectric conversion element shown in FIG. 17 is r(λ). In this case, the coefficients r2, g2, b2, and ir2 are determined by the least squares method so that a value obtained by the following expression (4) is as close as possible to r(λ). The determined coefficient data is previously stored in the memory (not shown) of the endoscope.
r2×R(λ)+g2×G(λ)+b2×B(λ)+ir2×IR(λ)  (4)

Let the spectral sensitivity at wave length λ of the g photoelectric conversion element shown in FIG. 17 be g(λ). In this case, the coefficients r3, g3, b3, and ir3 are determined by the least squares method so that a value obtained by the following expression (5) is as close as possible to g(λ). The determined coefficient data is previously stored in the memory (not shown) of the endoscope.
r3×R(λ)+g3×G(λ)+b3×B(λ)+ir3×IR(λ)  (5)

Let the spectral sensitivity at wave length λ of the b photoelectric conversion element shown in FIG. 17 be b(λ). In this case, the coefficients r4, g4, b4, and ir4 are determined by the least squares method so that a value obtained by the following expression (6) is as close as possible to b(λ). The determined coefficient data is previously stored in the memory (not shown) of the endoscope.
r4×R(λ)+g4×G(λ)+b4×B(λ)+ir4×IR(λ)  (6)

FIG. 18 is a drawing to show the spectral sensitivity characteristics of the R, G, and B photoelectric conversion elements of the image pickup device 100 obtained as a result of performing the computations of the expressions (4) to (6) using the coefficients determined by the method described above. In FIG. 18, the vertical axis indicates the spectral sensitivity where 1 is used as a reference, and the horizontal axis indicates the wavelength. Curve R shown in FIG. 18 indicates the spectral sensitivity characteristic obtained as a result of bringing the spectral sensitivity characteristic of the R first photoelectric conversion element to be close to the ideal spectral sensitivity characteristic, curve G shown in FIG. 18 indicates the spectral sensitivity characteristic obtained as a result of bringing the spectral sensitivity characteristic of the G first photoelectric conversion element to be close to the ideal spectral sensitivity characteristic, and curve B shown in FIG. 18 indicates the spectral sensitivity characteristic obtained as a result of bringing the spectral sensitivity characteristic of the B first photoelectric conversion element to be close to the ideal spectral sensitivity characteristic.

As shown in FIG. 18, the infrared region sensitivity of wavelength 680 nm or more can be set almost to 0 or less. Thus, it can be seen that the color reproducibility of the color image data can be improved by performing the computation process represented by the expression (3).

The display control section 58 controls the display 59 to display an image based on the high-contrast infrared image data enhanced by the image enhancement section 57, controls the display 59 to display an image based on the high-color-reproduction color image data, and controls the display 59 to display an image into which the image based on the high-contrast infrared image data and the image based on the high-color-reproduction color image data are combined. For the high-contrast infrared image data, a signal level is represented in pseudo color to display an image or the signal level is converted into an oxygen absorption amount to display an image.

As described above, according to the endoscope of the embodiment, the computation process is performed using the color image data generated from the R, G, and B signals output from the image pickup device 100 and the infrared image data generated from the IRr signal output from the image pickup device 100. Thereby, high-color-reproduction color image data with the color reproducibility improved as compared with the color image data and high-contrast infrared image data with the contrast improved as compared with the infrared image data can be generated. Thus, the test accuracy of the endoscope can be improved than was previously possible.

When the image pickup device having the configuration described in the first embodiment is adopted as an image pickup device for use in the endoscope, high-color-reproduction color image data and high-contrast infrared image data can be obtained by a single image picking up process. Therefore, it is made possible to conduct test without worrying about a color shift, etc.

The endoscope of the embodiment eliminates the need for an infrared cut filter, so that it is made possible to miniaturize a part to be inserted into a human body and the endoscope cost can also be reduced.

In the description given above, the endoscope is provided with the correction filter 52, which may be omitted. If the correction filter 52 is omitted, the spectral sensitivity characteristic of the image pickup device 100 for use in the endoscope becomes as shown in FIG. 6 and the color reproducibility of the color image data is more deteriorated. Thus, the process performed by the high-color-reproduction color image data generation section 56 becomes more advantageous.

In the embodiment, the endoscope is provided with both the high-contrast infrared image data generation section 55 and the high-color-reproduction color image data generation section 56. However, the high-color-reproduction color image data generation section 56 may be omitted. If the high-color-reproduction color image data generation section 56 is omitted, preferably the correction filter 52 to cut the wavelength range of 780 nm or more is provided.

The high-color-reproduction color image data generation section 56 is installed not only in the endoscope, but also in an image pickup apparatus such as a digital camera that has an image pickup device capable of outputting an R component signal, a G component signal, a B component signal, and an IR component signal. Thereby, a sufficient effect can be produced. In this case, an infrared cut filter is not required for the image pickup apparatus, so that the image pickup apparatus can be miniaturized and the cost thereof can be reduced.

The functions of the infrared image data generation section 53, the color image data generation section 54, the high-contrast infrared image data generation section 55, the high-color-reproduction color image data generation section 56, and the image enhancement section 57 in the endoscope described above may be implemented as a computer such as a processing unit installed in the endoscope executes a program for causing the computer to function as these sections. The functions of these sections may also be implemented as an image pickup signal obtained from the image pickup device 100 is input into a personal computer, etc., as it is and the computer executes the above-mentioned program.

In the specification, the R wavelength range indicates the range of wavelengths about 550 nm to about 700 nm, the G wavelength range indicates the range of wavelengths about 450 nm to about 610 nm, the B wavelength range indicates the range of wavelengths about 380 nm to about 520 nm, the infrared region indicates the range of wavelengths about 680 nm to about 3000 nm, the Cy wavelength range indicates the range of wavelengths about 380 nm to about 610 nm, the Mg wavelength range indicates the range of wavelengths about 380 nm to about 500 nm and wavelengths about 600 nm to 700 nm, and the Ye wavelength range indicates the range of wavelengths about 470 nm to about 700 nm.

In the specification, the expression “to transmit light in one wavelength range” is used to mean transmitting about 60% or more of the light in such a wavelength range and “to absorb light in one wavelength range” is used to mean absorbing about 50% or more of the light in such a wavelength range.

Claims (21)

1. An image processing apparatus for generating image data from an image pickup signal output from an image pickup device, the apparatus comprising:
a color image data generation unit that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device;
an infrared image data generation unit that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
a high-contrast infrared image data generation unit that generates high-contrast infrared image data using the color image data and the infrared image data, wherein: contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data,
wherein:
pixel data of the color image data includes red-component data, green-component data and blue-component data,
pixel data of the infrared image data includes infrared-component data, the high-contrast infrared image data generation unit generates the high-contrast infrared image data using the following expression:

I(x,y)=r1×R(x,y)+g1×G(x,y)+b1×B(x,y)+ir1×IR(x,y)
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r1, g1, b1 and ir1 denote coefficients, and
the coefficients r1, g1, b1, and ir1 are determined so that r1×R(λ)+g1×G(λ)+b1×B(λ)+ir1×IR(λ) is as close as possible to Real(λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.
2. The apparatus according to claim 1, wherein the observation target substance is hemoglobin.
3. The apparatus according to claim 1, further comprising:
a high-color-reproduction color image data generation unit that generates high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data.
4. The apparatus according to claim 3, wherein:
the high-color-reproduction color image data generation unit generates each pixel data of the high-color-reproduction color image data using the following expression:
( R O ( x , y ) G O ( x , y ) B O ( x , y ) ) = ( r 2 g 2 b 2 i r 2 r 3 g 3 b 3 i r 3 r 4 g 4 b 4 i r 4 ) ( R ( x , y ) G ( x , y ) B ( x , y ) I r ( x , y ) )
where
RO(x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y),
GO(x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
BO(x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y),
Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and
r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3 and ir4 denote coefficients,
the coefficients r2, g2, b2, and ir2 are determined so that r2×R(λ)+g2×G(λ)+b2×B(λ)+ir2×IR(λ) is as close as possible to RO(λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
RO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component,
the coefficients r3, g3, b3, and ir3 are determined so that r3×R(λ)+g3×G(λ)+b3×B(λ)+ir3×IR(λ) is as close as possible to GO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component, and
the coefficients r4, g4, b4, and ir4 are determined so that r4×R(λ)+g4×G(λ)+b4×B(λ)+ir4×IR(λ) is as close as possible to BO(λ), where GO(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
5. The apparatus according to claim 1, wherein the image pickup device comprises:
a large number of first photoelectric conversion elements arranged on a first plane in a semiconductor substrate;
second photoelectric conversion elements formed on a second plane, which is located above the first photoelectric conversion elements, the second photoelectric conversion elements that correspond to a part of the large number of first photoelectric conversion elements, the second photoelectric conversion elements comprising:
first electrodes formed above the first photoelectric conversion elements,
a photoelectric conversion layer formed on the first electrodes, and
a second electrode formed on the photoelectric conversion layer;
a color filter layer formed above the first photoelectric conversion elements, the color filter layer that transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs; and
a signal reading portion that reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements, wherein:
the color filter layer comprises a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively,
the large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range,
of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light,
the photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light, and
the part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
6. The apparatus according to claim 5, wherein the color filter layer is formed above the second photoelectric conversion elements.
7. The apparatus according to claim 6, wherein:
the photoelectric conversion layer contains an organic material, and the image pickup device further comprises a protective layer that protects the second photoelectric conversion elements, the protective layer formed by an atomic layer chemical vapor deposition (ALCVD) method between the first photoelectric conversion elements and the color filter layer.
8. The apparatus according to claim 7, wherein the protective layer contains an inorganic material.
9. The apparatus according to claim 8, wherein the protective layer has a two-layer structure comprising an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
10. The apparatus according to claim 5, wherein the image pickup device further comprises a microlens that collects light in each of the large number of first photoelectric conversion elements.
11. An endoscope comprising:
an image processing apparatus; and
the image processing apparatus according to claim 1.
12. A non-transitory computer readable medium storing a program for causing a computer to execute a process for image processing, the image processing comprising:
generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device;
generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
generating high-contrast infrared image data using the color image data and the infrared image data, wherein: contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data,
wherein:

I(x,y)=r1×R(x,y)+g1×G(x,y)+b1×B(x,y)+ir1×IR(x,y)
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r1, g1, b1 and ir1 denote coefficients, and
the coefficients r1, g1, b1, and ir1 are determined so that r1×R(λ)+g1×G(λ)+b1×B(λ)+ir1×IR(λ) is as close as possible to Real (λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.
13. An image processing apparatus for generating image data from an image pickup signal output from an image pickup device, the apparatus comprising:
a color image data generation unit that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device;
an infrared image data generation unit that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
a high-color-reproduction color image data generation unit that generates high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data,
wherein:
the high-color-reproduction color image data generation unit generates each pixel data of the high-color-reproduction color image data using the following expression:
( R o ( x , y ) G o ( x , y ) B o ( x , y ) ) = ( r 2 g 2 b 2 ir 2 r 3 g 3 b 3 ir 3 r 4 g 4 b 4 ir 4 ) ( R ( x , y ) G ( x , y ) B ( x , y ) Ir ( x , y ) )
where Ro(x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y),
Go(x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
Bo(x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y),
Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and
r2, r3, r4, g2, g3, g4, b2, b3, b4, ir2, ir3 and ir4 denote coefficients,
the coefficients r2, g2, b2, and ir2 are determined so that r2×R(λ)+g2×G(λ)+b2×B(λ)+ir2×IR(λ)) is as close as possible to Ro(λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Ro(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component
the coefficients r3, g3, b3, and ir3 are determined so that r3×R(λ)+g3×G(λ)+b3×B(λ)+ir3×IR(λ) is as close as possible to Go(λ), where
Go(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component, and
the coefficients r4, g4, b4, and ir4 are determined so that r4×R(λ)+g4×G(λ)+b4×B(λ)+ir4×IR(λ) is as close as possible to Bo(λ), where
Go(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
14. The apparatus according to claim 13, wherein the image pickup device comprises:
a large number of first photoelectric conversion elements arranged on a first plane in a semiconductor substrate;
second photoelectric conversion elements formed on a second plane, which is located above the first photoelectric conversion elements, the second photoelectric conversion elements that correspond to a part of the large number of first photoelectric conversion elements, the photoelectric conversion elements comprising:
first electrodes formed above the first photoelectric conversion elements,
a photoelectric conversion layer formed on the first electrodes, and
a second electrode formed on the photoelectric conversion layer;
a color filter layer formed above the first photoelectric conversion elements, the color filter layer that transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs; and
a signal reading portion that reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements, wherein:
the color filter layer comprises a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively,
the large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range,
of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light,
the photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light, and
the part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
15. The apparatus according to claim 14, wherein the color filter layer is formed above the second photoelectric conversion elements.
16. The apparatus according to claim 15, wherein:
the photoelectric conversion layer contains an organic material, and
the image pickup device further comprises a protective layer that protects the second photoelectric conversion elements, the protective layer formed by an atomic layer chemical vapor deposition (ALCVD) method between the first photoelectric conversion elements and the color filter layer.
17. The apparatus according to claim 16, wherein the protective layer contains an inorganic material.
18. The apparatus according to claim 17, wherein the protective layer has a two-layer structure comprising an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
19. The apparatus according to claim 14, wherein the image pickup device further comprises a microlens that collects light in each of the large number of first photoelectric conversion elements.
20. An endoscope comprising:
an image processing apparatus; and
the image processing apparatus according to claim 13.
21. A non-transitory computer readable medium storing a program for causing a computer to execute a process for image processing, the image processing comprising:
generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device;
generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
generating high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data,
wherein:

I(x,y)=r1×R(x,y)+g1×G(x,y)+b1×B(x,y)+ir1×IR(x,y)
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r1, g1, b1 and ir1 denote coefficients, and
the coefficients r1, g1, b1, and ir1 are determined so that r1×R(λ)+g1×G(λ)+b1×B(λ)+ir1×IR(λ) is as close as possible to Real (λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.
US11863651 2006-09-28 2007-09-28 Image processing apparatus, endoscope, and computer readable medium Active 2030-12-07 US8111286B2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2006-264926 2006-09-28
JP2006264923A JP5171004B2 (en) 2006-09-28 2006-09-28 The image processing apparatus, an endoscope apparatus, and image processing program
JP2006-264923 2006-09-28
JP2006264926A JP4764794B2 (en) 2006-09-28 2006-09-28 The image processing apparatus, an endoscope apparatus, and image processing program

Publications (2)

Publication Number Publication Date
US20080079807A1 true US20080079807A1 (en) 2008-04-03
US8111286B2 true US8111286B2 (en) 2012-02-07

Family

ID=39260707

Family Applications (1)

Application Number Title Priority Date Filing Date
US11863651 Active 2030-12-07 US8111286B2 (en) 2006-09-28 2007-09-28 Image processing apparatus, endoscope, and computer readable medium

Country Status (1)

Country Link
US (1) US8111286B2 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160171718A1 (en) * 2014-12-10 2016-06-16 Fujifilm Corporation Medical image processing device and method for operating the same
US9517002B2 (en) 2014-05-09 2016-12-13 Olympus Corporation Solid state image sensor, endoscope, and endoscope system

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8169518B2 (en) * 2007-08-14 2012-05-01 Fujifilm Corporation Image pickup apparatus and signal processing method
JP5262180B2 (en) * 2008-02-26 2013-08-14 ソニー株式会社 Solid-state imaging device and a camera
JP2009207584A (en) * 2008-03-03 2009-09-17 Hoya Corp Endoscope system
JP5478043B2 (en) * 2008-09-11 2014-04-23 富士フイルム株式会社 Solid-state imaging device and the imaging apparatus
US8237834B2 (en) * 2008-09-11 2012-08-07 Fujifilm Corporation Solid-state imaging device and imaging apparatus having light-preventing partitions
US8901541B2 (en) 2009-04-07 2014-12-02 Rohm Co., Ltd. Photoelectric conversion device and image pick-up device
US9123653B2 (en) * 2009-07-23 2015-09-01 Sony Corporation Solid-state imaging device, method of manufacturing the same, and electronic apparatus
US8546737B2 (en) 2009-10-30 2013-10-01 Invisage Technologies, Inc. Systems and methods for color binning
JP4851580B2 (en) * 2009-11-20 2012-01-11 シャープ株式会社 Image reading apparatus, and an image data output processing apparatus having the same, and image reading method
JP2011239252A (en) * 2010-05-12 2011-11-24 Panasonic Corp Imaging device
KR101161979B1 (en) * 2010-08-19 2012-07-03 삼성전기주식회사 Image processing apparatus and method for night vision
US9245332B2 (en) * 2011-03-09 2016-01-26 Alcatel Lucent Method and apparatus for image production
JP5789232B2 (en) * 2011-10-12 2015-10-07 富士フイルム株式会社 The endoscope system and method of operating same
US20170163954A1 (en) * 2014-06-30 2017-06-08 Sony Corporation Image processing device, imaging device, information processing device, image processing method, and program

Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06204445A (en) 1993-01-01 1994-07-22 Canon Inc Optical sensor and image information processing device having the same
JPH06335451A (en) 1993-03-19 1994-12-06 Olympus Optical Co Ltd Picture image processor for endoscope
US5550582A (en) 1993-03-19 1996-08-27 Olympus Optical Co., Ltd. Endoscope-image processing apparatus for performing image processing of emphasis in endoscope image by pigment concentration distribution
JP2648494B2 (en) 1988-05-02 1997-08-27 オリンパス光学工業株式会社 Endoscope apparatus
WO2000007365A1 (en) 1998-07-30 2000-02-10 Intel Corporation Color imaging system with infrared correction
US20030187319A1 (en) * 2002-03-29 2003-10-02 Olympus Optical Co., Ltd. Sentinel lymph node detecting apparatus, and method thereof
JP2004080565A (en) 2002-08-21 2004-03-11 Pentax Corp Electronic endoscope capable of outputting digital video signal
JP2005006066A (en) 2003-06-12 2005-01-06 Acutelogic Corp Color filter for solid-state image pickup element and color image pickup device
US20050029456A1 (en) * 2003-07-30 2005-02-10 Helmuth Eggers Sensor array with a number of types of optical sensors
JP2005065976A (en) 2003-08-22 2005-03-17 Olympus Corp Endoscopic apparatus
US20050225656A1 (en) 2004-04-07 2005-10-13 Fuji Photo Film Co., Ltd. Imaging sensor
JP2005341470A (en) 2004-05-31 2005-12-08 Mitsubishi Electric Corp Imaging apparatus and signal processing method
US20060081887A1 (en) 2004-10-20 2006-04-20 Samsung Electronics Co., Ltd. Solid state image sensor devices having non-planar transistors
JP2006121093A (en) 2004-10-20 2006-05-11 Samsung Electronics Co Ltd Solid image sensor element having non-flat surface transistor and method of manufacturing the same
US20060119724A1 (en) 2004-12-02 2006-06-08 Fuji Photo Film Co., Ltd. Imaging device, signal processing method on solid-state imaging element, digital camera and controlling method therefor and color image data generating method
JP2006165663A (en) 2004-12-02 2006-06-22 Fuji Photo Film Co Ltd Image pickup apparatus, digital camera and color image data generating method
US20060196533A1 (en) 2005-02-28 2006-09-07 Fuji Photo Film Co., Ltd. Photoelectric conversion layer-stacked element and method for producing the same
JP2006239206A (en) 2005-03-04 2006-09-14 Fujinon Corp Endoscope apparatus
US20060211915A1 (en) 2005-03-04 2006-09-21 Fujinon Corporation Endoscope apparatus

Patent Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2648494B2 (en) 1988-05-02 1997-08-27 オリンパス光学工業株式会社 Endoscope apparatus
JPH06204445A (en) 1993-01-01 1994-07-22 Canon Inc Optical sensor and image information processing device having the same
JPH06335451A (en) 1993-03-19 1994-12-06 Olympus Optical Co Ltd Picture image processor for endoscope
US5550582A (en) 1993-03-19 1996-08-27 Olympus Optical Co., Ltd. Endoscope-image processing apparatus for performing image processing of emphasis in endoscope image by pigment concentration distribution
US5675378A (en) 1993-03-19 1997-10-07 Olympus Optical Co., Ltd. Endoscope-image processing apparatus for performing image processing of emphasis in endoscope image by pigment concentration distribution
WO2000007365A1 (en) 1998-07-30 2000-02-10 Intel Corporation Color imaging system with infrared correction
JP2002521975A (en) 1998-07-30 2002-07-16 インテル コーポレイション Infrared correction system
US20030187319A1 (en) * 2002-03-29 2003-10-02 Olympus Optical Co., Ltd. Sentinel lymph node detecting apparatus, and method thereof
JP2004080565A (en) 2002-08-21 2004-03-11 Pentax Corp Electronic endoscope capable of outputting digital video signal
JP2005006066A (en) 2003-06-12 2005-01-06 Acutelogic Corp Color filter for solid-state image pickup element and color image pickup device
US20050029456A1 (en) * 2003-07-30 2005-02-10 Helmuth Eggers Sensor array with a number of types of optical sensors
JP2005065976A (en) 2003-08-22 2005-03-17 Olympus Corp Endoscopic apparatus
JP2005302806A (en) 2004-04-07 2005-10-27 Fuji Photo Film Co Ltd Imaging sensor
US20050225656A1 (en) 2004-04-07 2005-10-13 Fuji Photo Film Co., Ltd. Imaging sensor
JP2005341470A (en) 2004-05-31 2005-12-08 Mitsubishi Electric Corp Imaging apparatus and signal processing method
US20060081887A1 (en) 2004-10-20 2006-04-20 Samsung Electronics Co., Ltd. Solid state image sensor devices having non-planar transistors
US20060084195A1 (en) 2004-10-20 2006-04-20 Samsung Electronics Co., Ltd. Methods for fabricating solid state image sensor devices having non-planar transistors
JP2006121093A (en) 2004-10-20 2006-05-11 Samsung Electronics Co Ltd Solid image sensor element having non-flat surface transistor and method of manufacturing the same
US20080153193A1 (en) 2004-10-20 2008-06-26 Jeong Ho Lyu Methods for fabricating solid state image sensor devices having non-planar transistors
JP2006165663A (en) 2004-12-02 2006-06-22 Fuji Photo Film Co Ltd Image pickup apparatus, digital camera and color image data generating method
US20060119724A1 (en) 2004-12-02 2006-06-08 Fuji Photo Film Co., Ltd. Imaging device, signal processing method on solid-state imaging element, digital camera and controlling method therefor and color image data generating method
US20060196533A1 (en) 2005-02-28 2006-09-07 Fuji Photo Film Co., Ltd. Photoelectric conversion layer-stacked element and method for producing the same
JP2006245045A (en) 2005-02-28 2006-09-14 Fuji Photo Film Co Ltd Solid imaging element of photoelectric conversion film stacked type and its manufacturing method
JP2006239206A (en) 2005-03-04 2006-09-14 Fujinon Corp Endoscope apparatus
US20060211915A1 (en) 2005-03-04 2006-09-21 Fujinon Corporation Endoscope apparatus

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Japanese Office Action issued in Application No. 2006-264926, dated Mar. 8, 2011.
Japanese Office Action, dated Aug. 23, 2011, issued in Japanese Application No. 2006-264923.

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9517002B2 (en) 2014-05-09 2016-12-13 Olympus Corporation Solid state image sensor, endoscope, and endoscope system
US20160171718A1 (en) * 2014-12-10 2016-06-16 Fujifilm Corporation Medical image processing device and method for operating the same
US9734592B2 (en) * 2014-12-10 2017-08-15 Fujifilm Corporation Medical image processing device and method for operating the same

Also Published As

Publication number Publication date Type
US20080079807A1 (en) 2008-04-03 application

Similar Documents

Publication Publication Date Title
US7242478B1 (en) Spatially corrected full-cubed hyperspectral imager
US6760475B1 (en) Colorimetric imaging system
US4438455A (en) Solid-state color imager with three layer four story structure
US5986704A (en) Solid-state image pickup device, with purposefully displaced color filters, method of manufacturing same and camera incorporating same
US4514755A (en) Solid-state color imager with two layer three story structure
US20060043261A1 (en) Solid state image pickup device and image pickup system comprising it
US20030211405A1 (en) Color filter imaging array and method of formation
US20060186322A1 (en) Color filter array and solid-state image pickup device
US4404586A (en) Solid-state color imager with stripe or mosaic filters
US20070145273A1 (en) High-sensitivity infrared color camera
US20060124833A1 (en) Method and apparatus for acquiring physical information, method for manufacturing semiconductor device including array of plurality of unit components for detecting physical quantity distribution, light-receiving device and manufacturing method therefor, and solid-state imaging device and manufacturing method therefor
US6958862B1 (en) Use of a lenslet array with a vertically stacked pixel array
US20070285540A1 (en) Image photographing device and method
US7755016B2 (en) Physical information acquisition method, physical information acquisition device, and semiconductor device
US20030193586A1 (en) Image pickup apparatus
US20110058072A1 (en) Camera sensor correction
US20090321865A1 (en) Solid-state imaging device and camera
WO2010058545A1 (en) Imaging device
US20080068475A1 (en) Image photographing apparatus, method and medium
JP2011176715A (en) Back-illuminated image sensor and imaging apparatus
JP2004228662A (en) Image pickup apparatus
US7245324B2 (en) Image pickup apparatus for photographing an object
US6590679B1 (en) Multilinear array sensor with an infrared line
JP2002135796A (en) Imaging apparatus
US20040100570A1 (en) Image sensor and digital camera

Legal Events

Date Code Title Description
AS Assignment

Owner name: FUJIFILM CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:INUIYA, MASAFUMI;OHASHI, YUICHI;IHAMA, MIKIO;REEL/FRAME:019895/0197

Effective date: 20070921

FPAY Fee payment

Year of fee payment: 4