US11368638B2 - Imaging element, imaging device, imaging method and computer-readable recording medium - Google Patents

Imaging element, imaging device, imaging method and computer-readable recording medium Download PDF

Info

Publication number
US11368638B2
US11368638B2 US17/181,022 US202117181022A US11368638B2 US 11368638 B2 US11368638 B2 US 11368638B2 US 202117181022 A US202117181022 A US 202117181022A US 11368638 B2 US11368638 B2 US 11368638B2
Authority
US
United States
Prior art keywords
color filter
same
image signal
filter
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/181,022
Other versions
US20210185248A1 (en
Inventor
Kazunori YOSHIZAKI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Assigned to OLYMPUS CORPORATION reassignment OLYMPUS CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YOSHIZAKI, KAZUNORI
Publication of US20210185248A1 publication Critical patent/US20210185248A1/en
Application granted granted Critical
Publication of US11368638B2 publication Critical patent/US11368638B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H04N5/332—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N9/0455—

Definitions

  • the present disclosure relates to an imaging element, an imaging device, an imaging method, and a computer-readable recording medium that are capable of capturing a color image and a near-infrared image.
  • an imaging element includes: a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light; and a color filter including a plurality of filters that have different spectral transmission characteristics from each other in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics from each other in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances from each other in at least one of the visible region and the near-infrared region.
  • an imaging device includes: an imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region; and a separator configured to separate an image signal in the visible region and an image signal in the near-infrared
  • an imaging method performed by an imaging device.
  • the imaging device includes an imaging element and an image processor, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region.
  • the method includes: separating, by the image processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
  • a non-transitory computer-readable recording medium with an executable program stored thereon.
  • the program is executed by an imaging device including an imaging element and an image processor, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the
  • the program causes the imaging device to execute: separating, by the image processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
  • FIG. 1 is a block diagram illustrating a functional configuration of an imaging device according to a first embodiment
  • FIG. 2 is a diagram schematically illustrating a configuration of a pixel unit according to the first embodiment
  • FIG. 3 is a diagram schematically illustrating a configuration of a color filter according to the first embodiment
  • FIG. 4 is a diagram illustrating the transmission characteristics of each filter according to the first embodiment
  • FIG. 5 is a flowchart illustrating an outline of a process executed by the imaging device according to the first embodiment
  • FIG. 6 is a diagram schematically illustrating an example of an image generated by the imaging device according to the first embodiment
  • FIG. 7 is a diagram schematically illustrating the spectral transmittance of each of trueG and IR according to the first embodiment
  • FIG. 8 is a diagram schematically illustrating the transmittance of each of a filter G 1 and a filter G 2 according to the first embodiment
  • FIG. 9 is a diagram schematically illustrating the spectral reflectance of a subject
  • FIG. 10 illustrates the spectral sensitivity of each of a G 1 pixel and a G 2 pixel according to the first embodiment
  • FIG. 11 is a diagram schematically illustrating a configuration of a color filter according to a first modification of the first embodiment
  • FIG. 12 is a diagram illustrating the transmission characteristics of the color filter according to the first modification of the first embodiment
  • FIG. 13 is a diagram schematically illustrating a color filter of a second modification of the first embodiment
  • FIG. 14 is a diagram schematically illustrating the spectral transmittance of each wavelength band according to a second embodiment
  • FIG. 15 is a block diagram illustrating a functional configuration of an imaging device according to a third embodiment.
  • FIG. 16 is a diagram illustrating the transmission characteristics of each filter and a notch filter according to the third embodiment.
  • an imaging device including an imaging element that generates image data (RAW data) by capturing a subject will be described.
  • RAW data image data
  • the embodiment does not limit the present disclosure. Further, in the description of the drawings, the same parts will be described with the same reference numerals.
  • FIG. 1 is a block diagram illustrating a functional configuration of an imaging device according to a first embodiment.
  • the imaging device 1 illustrated in FIG. 1 includes an imaging unit 2 , an image processing unit (image processor) 3 , a display unit 4 , a recording unit 5 , and a control unit 6 .
  • the imaging unit 2 generates image data by capturing a subject.
  • the imaging unit 2 includes an optical system 21 , an imaging element 22 , and an A/D converter 23 .
  • the optical system 21 forms a subject image on the light receiving face of the imaging element 22 .
  • the optical system 21 includes one or a plurality of lenses.
  • the optical system 21 has a zoom function and a focus function.
  • the optical system 21 changes the zoom magnification and the focus position by moving on an optical axis L 1 by a drive unit such as a motor (not illustrated).
  • the light receiving face of the imaging element 22 is provided perpendicular to the optical axis L 1 of the optical system 21 .
  • the imaging element 22 generates under the control of the control unit 6 an image signal (image data) by performing photoelectric conversion on the subject image formed by the optical system 21 to output this image signal to the A/D converter 23 .
  • the imaging element 22 is realized by using a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like.
  • CMOS complementary metal oxide semiconductor
  • the imaging element 22 includes a pixel unit (pixel portion) 221 and a color filter 222 laminated on the light receiving face of the pixel unit 221 . The detailed configuration of the pixel unit 221 and the color filter 222 will be described later.
  • the A/D converter 23 Under the control of the control unit 6 , the A/D converter 23 performs an A/D conversion on an analog image signal input from the imaging unit 2 to output the image signal to the image processing unit 3 .
  • the A/D converter 23 includes, for example, an A/D conversion circuit and the like.
  • the image processing unit 3 performs, under the control of the control unit 6 , various image processes, for example, a white balance adjustment process, a demosaicing process, a separation process, and the like on the image signal input from the imaging unit 2 to output the processed image signal to the display unit 4 .
  • the image processing unit 3 includes a dedicated processor such as various arithmetic circuits, such as a graphics processing unit (GPU) or an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), that execute a specific function.
  • the image processing unit 3 includes an interpolation unit 31 , a separation unit (separator) 32 , and a generation unit 33 .
  • the interpolation unit 31 performs a well-known interpolation process for interpolating the image signal (pixel value) of each pixel with respect to the image signal input from the imaging unit 2 . Specifically, the interpolation unit 31 performs a well-known interpolation process on the image signal of each filter constituting the color filter 222 to generate an image corresponding to the image signal, of each filter, that interpolates the image signal of each filter.
  • the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the pixel on which the first same-color filter is disposed and the second image signal generated by the pixel on which the second same-color filter is disposed based on the first ratio between the transmittance of the first same-color filter and the transmittance of the second same-color filter in the visible region, and the second ratio between the transmittance of the first same-color filter and the transmittance of the second same-color filter in the near-infrared region.
  • the details of the separation method by the separation unit 32 will be described later.
  • the generation unit 33 generates a visible light image and a near-infrared image (hereinafter, simply referred to as an “IR image”) based on the image signal in the visible region and the image signal in the near-infrared region separated by the separation unit 32 .
  • the generation unit 33 outputs the visible light image and the IR image to the display unit 4 .
  • the display unit 4 displays an image (at least one of the visible light image and the IR image) corresponding to the image signal input from the image processing unit 3 .
  • the display unit 4 displays various pieces of information related to the imaging device 1 .
  • the display unit 4 includes a display panel such as a liquid crystal or an organic electro luminescence (EL).
  • the recording unit 5 records a program executed by the imaging device 1 , data being processed, image data generated by the imaging unit 2 , and the like.
  • the recording unit 5 includes a program recording unit 51 that records a program executed by the imaging device 1 , and a sensitivity information recording unit 52 that records the sensitivity of each filter of the color filter 222 .
  • the recording unit 5 includes, for example, a Flash memory, a random access memory (RAM), a hard disk drive (HDD), a synchronous dynamic random access memory (SDRAM), a memory card, or the like.
  • the control unit 6 controls each unit of the imaging device 1 .
  • the control unit 6 includes a central processing unit (CPU) and the like.
  • FIG. 2 is a diagram schematically illustrating the configuration of the pixel unit 221 .
  • the pixel unit 221 illustrated in FIG. 2 is formed by arranging a plurality of pixels P that receives light transmitted through the color filter 222 in a two-dimensional grid pattern (two-dimensional matrix pattern). Each pixel P receives light which is incident from the optical system 21 , and that is transmitted through the color filter 222 to perform a photoelectric conversion to generate an image signal.
  • This image signal includes a pixel value (luminance value) of each pixel, pixel position information (pixel address), and the like.
  • the pixel disposed in the i-th row and the j-th column is expressed as a pixel P ij .
  • FIG. 3 is a diagram schematically illustrating the configuration of the color filter 222 .
  • the color filter 222 illustrated in FIG. 3 is disposed so that a filter unit composed of a plurality of filters having different spectral transmission characteristics in each of the visible region and the near-infrared region correspond to a plurality of pixels of the pixel unit 221 .
  • the color filter 222 is disposed on the light receiving face of the pixel unit 221 , and includes four types of filters that transmit the visible light in different wavelength bands in the visible region and the near-infrared rays in different wavelength bands in the near-infrared region. More specifically, the color filter 222 includes an R filter, a G 1 filter, a G 2 filter, and a B filter.
  • the color filter 222 is formed by arranging the R filters, the G 1 filters, the G 2 filters, and the B filters side by side in a predetermined array pattern. Specifically, the color filter 222 is formed by arranging an R filter column, a G 1 filter column, a B filter column, an R filter column, and a G 2 filter column in an array pattern.
  • the R filter transmits light in the red wavelength band.
  • the G 1 filter transmits light in the green wavelength band and transmits light in the near-infrared wavelength band.
  • the G 2 filter has different transmission wavelength characteristics than the G 1 filter in the same color wavelength band as the G 1 filter, transmits light in the green wavelength band, and transmits light in the near-infrared wavelength band.
  • the B filter transmits light in the blue wavelength band.
  • the G 1 filter and the G 2 filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances from each other in at least one of the visible region and the near-infrared region.
  • the G 1 filter and the G 2 filter function as a first same-color filter and a second same-color filter, respectively.
  • the pixel P ij on which each filter is provided receives light in the wavelength band transmitted through the filter.
  • the pixel on which the R filter is disposed is referred to as an R pixel
  • the pixel on which the G 1 filter is disposed is referred to as a G 1 pixel
  • the pixel on which the G 2 filter is disposed is referred to as a G 2 pixel
  • the pixel on which the B filter is disposed is referred to as a B pixel.
  • FIG. 4 is a diagram illustrating the transmission characteristics of each filter.
  • the vertical axis represents the transmittance (spectral sensitivity) and the horizontal axis represents the wavelength (nm).
  • the curve L B indicates the transmittance curve of filter B
  • the curve L G1 indicates the transmission curve of filter G 1
  • the curve L G2 indicates the transmission curve of filter G 2
  • the curve L R indicates the transmission curve of the filter R.
  • the visible region ranges from 400 nm to 700 nm
  • the near-infrared region ranges from 700 nm to 1000 nm.
  • the filter R transmits light in the red wavelength band (wavelength band 600 nm to 700 nm), which is the first visible region.
  • the filter B transmits light in the blue wavelength band (400 nm to 500 nm), which is the second visible region.
  • the filter G 1 and the filter G 2 transmit light in the green wavelength band (500 nm to 600 nm), which is the third visible band, and the near-infrared region (700 nm to 1000 nm).
  • the filter G 1 and the filter G 2 have a region having a constant spectral transmittance in each of the third visible light region and the near-infrared region. Furthermore, the filter G 1 and the filter G 2 have different transmittances from each other in each of the third visible light region and the near-infrared region, and the difference D 1 between the transmittance of the filter G 1 and the transmittance of the filter G 2 in the third visible light region, and the difference D 2 between the transmittances of the filter G 1 and the transmittance of the filter G 2 in the near-infrared region are each constant.
  • the filter G 1 and the filter G 2 have different transmittances from each other in each of the third visible light region and the near-infrared region, but the filter G 1 and the filter G 2 may have different transmittances from each other in at least one of the third visible light region and the near-infrared region.
  • FIG. 5 is a flowchart illustrating an outline of the process executed by the imaging device 1 .
  • FIG. 6 is a diagram schematically illustrating an example of an image generated by the imaging device 1 .
  • the imaging unit 2 images a subject (step S 101 ).
  • the imaging unit 2 outputs an image signal obtained by imaging the subject to the image processing unit 3 .
  • the interpolation unit 31 performs an interpolation process for interpolating the pixel value of each pixel with respect to the image signal input from the imaging unit 2 (step S 102 ). Specifically, as illustrated in FIG. 6 , the interpolation unit 31 performs the interpolation process for interpolating the image signal of each pixel with respect to the image signal input from the imaging unit 2 to generate a B image P B , an R image P R , a G 1 image P G1 , and a G 2 image P G2 .
  • the separation unit 32 executes a separation process for separating the image signal in the visible region and the image signal in the near-infrared region with respect to the G 1 image P G1 and the G 2 image P G2 (step S 103 ).
  • FIG. 7 is a diagram schematically illustrating the spectral transmittance in each of the trueG and the IR.
  • the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm).
  • the curve L TG indicates the transmittance in the trueG
  • the curve L IR indicates the transmittance in the IR.
  • FIG. 8 is a diagram schematically illustrating the transmittance of each of the filter G 1 and the filter G 2 .
  • the curve L G1 indicates the transmittance of the filter G 1
  • the curve L G2 indicates the transmittance of the filter G 2 .
  • the transmittance in each of the trueG (500 nm to 600 nm) and the IR (700 nm to 1000 nm) is set to “1.0”
  • the transmittance in each wavelength band is a, b, c and d
  • FIG. 9 is a diagram schematically illustrating the spectral reflectance of the subject.
  • FIG. 10 illustrates the spectral sensitivities of the G 1 pixel and the G 2 pixel.
  • the vertical axis represents the spectral reflectance.
  • the vertical axis represents the spectral sensitivity
  • the horizontal axis represents the wavelength (nm).
  • the curve L p indicates the spectral reflectance of the subject. Also, in FIG.
  • the area surrounded by the curve L TG1 indicates truerG 1 (a ⁇ truerG)
  • the area surrounded by the curve L TG2 indicates truerG 2 (c ⁇ truerG)
  • the area surrounded by the curve L TR2 indicates IR 2 (d ⁇ IR)
  • the area surrounded by the curve L IR1 indicates IR 1 (b ⁇ IR).
  • the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the G 1 pixel and the second image signal generated by the G 2 pixel based on the first ratio D 1 (a/c) of the transmittance of the G 1 filter to the transmittance of the G 2 filter in the visible region and the second ratio D 2 (d/b) of the transmittance of the G 1 filter to the transmittance of the G 2 filter in the near-infrared region.
  • the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the G 1 pixel and the second image signal generated by the G 2 pixel based on the first ratio D 1 (a/c) of the transmittance of the G 1 filter to the transmittance of the G 2 filter in the visible region and the second ratio D 2 (d/b) of the transmittance of the G 1 filter to the transmittance of the G 2 filter in the near-infrared region.
  • step S 103 the description after step S 103 is continued.
  • the generation unit 33 generates an IR image and a G image based on the signal value in the visible region and the signal value in the near-infrared region separated by the separation unit 32 . Specifically, as illustrated in FIG. 6 , the generation unit 33 generates the G image P G and the IR image P IR based on the signal value in the visible region and the signal value in the near-infrared region separated by the separation unit 32 (step S 104 ).
  • the generation unit 33 After that, the generation unit 33 generates a visible image (step S 105 ). Specifically, as illustrated in FIG. 6 , the generation unit 33 generates the visible image P W based on the B image P B , the R image P R , and the G image P G . After step S 105 , the imaging device 1 ends this process.
  • the color filter 222 includes the first same-color filter and the second same-color filter in which any one or more of the plurality of filters transmit light in a near-infrared region, the first same-color filter and the second same-color filter having different transmission wavelength characteristics in the same color wavelength band, and the first same-color filter and the second same-color filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region and have different spectral transmittances in at least one of the visible region and the near-infrared region, so that a color image and a near-infrared image can be acquired at the same time without installing an IR filter.
  • the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the pixel on which the first same-color filter is disposed and the second image signal generated by the pixel on which the second same-color filter is disposed based on the first ratio of the transmittance of the first same-color filter to the transmittance of the second same-color filter in the visible region, and the second ratio of the transmittance of the first same-color filter to the transmittance of the second same-color filter in the near-infrared region, so that a color image and a near-infrared image can be acquired at the same time without installing an IR filter.
  • the configuration of the color filter is different.
  • the G filters that transmit light in the green wavelength band have different spectral transmission characteristics
  • R filters that transmit light in the red wavelength band have different spectral transmission characteristics.
  • FIG. 11 describes a detailed configuration of the color filter according to the first modification of the first embodiment.
  • a color filter 222 A illustrated in FIG. 11 is disposed so that a filter unit composed of a plurality of filters having different spectral transmission characteristics in each of the visible region and the near-infrared region correspond to a plurality of pixels of the pixel unit 221 .
  • the color filter 222 A is disposed on the light receiving face of the pixel unit 221 , and includes four types of filters that transmit the visible light in different wavelength bands in the visible region and the near-infrared rays in different wavelength bands in the near-infrared region. More specifically, the color filter 222 includes an R 1 filter, an R 2 filter, a G filter, and a B filter.
  • the color filter 222 A is formed by arranging the R 1 filters, the R 2 filters, the G filters, and the B filters side by side in a predetermined array pattern. Specifically, the color filter 222 A is formed by arranging an R 1 filter column, a G filter column, a B filter column, and an R 2 filter column in an array pattern.
  • the R 1 filter transmits light in the red wavelength band and transmits light in the near-infrared wavelength band.
  • the R 2 filter has different transmission wavelength characteristics than the R 1 filter in the same color wavelength band as the R 1 filter, transmits light in the red wavelength band, and transmits light in the near-infrared wavelength band.
  • the R 1 filter and the R 2 filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region.
  • the R 1 filter and the R 2 filter function as the first same-color filter and the second same-color filter, respectively.
  • the pixel P ij on which each filter is provided receives light in the wavelength band transmitted through the filter.
  • the pixel on which the R 1 filter is disposed is referred to as an R 1 pixel
  • the pixel on which the R 2 filter is disposed is referred to as an R 2 pixel
  • the pixel on which the G filter is disposed is referred to as a G pixel
  • the pixel on which the B filter is disposed is referred to as a B pixel.
  • FIG. 12 is a diagram illustrating the transmission characteristics of the color filter according to the first modification of the first embodiment.
  • the transmission curve is simulated and standardized so that the maximum values of the filters are substantially equal.
  • the vertical axis represents the transmittance (spectral sensitivity) and the horizontal axis represents the wavelength (nm).
  • the curve L B indicates the transmittance curve of filter B
  • the curve L G1 indicates the transmission curve of filter G
  • the curve L R1 indicates the transmission curve of filter R 1
  • the curve L R2 indicates the transmission curve of the filter R 2 .
  • the visible region ranges from 400 nm to 700 nm
  • the near-infrared region ranges from 700 nm to 1000 nm.
  • the filter R 1 and the filter R 2 have a region where the difference between transmittances, D 11 , and the difference between transmittances, D 12 , are each constant in each of the first visible light region and the near-infrared region, and have different transmittances in at least one of the first visible light region and the near-infrared region.
  • the filter R 1 and the filter R 2 have a region where the difference between transmittances, D 11 , and the difference between transmittances, D 12 , are each constant in each of the first visible light region and the near-infrared region, and have different transmittances in at least one of the first visible light region and the near-infrared region.
  • the color filter 222 A configured in this way is disposed on the light receiving face of the pixel unit 221 , it is possible to simultaneously capture a visible light image and a near-infrared image as in the first embodiment described above.
  • the R filters that transmit light in the red wavelength band have different transmittances, but in addition to the R 1 filter and the R 2 filter, for example, the G 1 filter and the G 2 filter of the first embodiment described above may be used to constitute the color filter.
  • each filter is disposed for each column, but this is not limited and can be changed as appropriate.
  • FIG. 13 is a diagram schematically illustrating the color filter of the second modification of the first embodiment.
  • a color filter 222 B illustrated in FIG. 13 is configured in a Bayer array, and the G 1 filter and the G 2 filter are disposed at the locations where the G filters are disposed.
  • a conventional image processing pipeline image signal line
  • a visible light image and a near-infrared image can be acquired at the same time with a simple configuration.
  • the signal values of the R pixel and the B pixel that do not include the near-infrared are calculated.
  • the calculation method calculated by the separation unit 32 will be described.
  • the same components as those of the imaging device 1 according to the first embodiment described above are given the same reference numerals, and detailed description thereof will be omitted.
  • FIG. 14 is a diagram schematically illustrating the spectral transmittance in each wavelength band.
  • the region L TB indicates the integral value of the image signal trueB of the B filter
  • the region L TR indicates the integral value of the image signal trueR of the R filter
  • the region L TG indicates the integral value of the image signal trueG when the G 1 filter+the G 2 filter are combined
  • the L TIR indicates the integral value of the image signal IR.
  • the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm).
  • the separation unit 32 calculates the image signal trueG and the image signal IR by the method of the first embodiment described above. Then, as illustrated in FIG. 14 , since the transmittance in the near-infrared region is substantially constant, the separation unit 32 calculates the image signal of the R pixel and the image signal of the B pixel based on the image signal IR of the near-infrared light. Specifically, the following equations hold when the transmittance in the near-infrared region is almost constant.
  • the separation unit 32 calculates the image signal of each of the R pixel and the B pixel using the above equations (9) and (10).
  • the true pixel values of the R pixel and the B pixel can be output.
  • the third embodiment has a different configuration from the imaging device 1 according to the first embodiment described above. Specifically, the imaging device according to the third embodiment further includes a notch filter that cuts light in a predetermined wavelength band.
  • the configuration of the imaging device according to the third embodiment will be described. The same components as those of the imaging device 1 according to the first embodiment described above are given the same reference numerals, and detailed description thereof will be omitted.
  • FIG. 15 is a block diagram illustrating a functional configuration of the imaging device according to the third embodiment.
  • An imaging device 10 illustrated in FIG. 15 includes an imaging unit 2 C instead of the imaging unit 2 according to the first embodiment described above. Further, the imaging unit 2 C includes an imaging element 22 C instead of the imaging element 22 according to the first embodiment described above. The imaging element 22 C further includes a notch filter 223 in addition to the configuration of the imaging element 22 according to the first embodiment described above.
  • the notch filter 223 transmits light in the visible region and cuts part of light in the near-infrared region.
  • FIG. 16 is a diagram illustrating the transmission characteristics of each filter and the notch filter.
  • the curve L N indicates the transmission characteristics of the notch filter 223 .
  • the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm).
  • the notch filter 223 transmits light in the visible region and cuts part of light in the near-infrared region.
  • the notch filter 223 cuts light in a certain region in the near-infrared region, for example, in the region of 700 nm to 800 nm.
  • the spectral transmittance of each filter is made substantially constant. As a result, a highly accurate near-infrared image can be acquired.
  • the notch filter 223 cuts light in a certain region in the near-infrared region, the spectral transmittance of each filter can be made almost constant, so that highly accurate near-infrared images can be acquired.
  • Various embodiments can be formed by appropriately combining the plurality of components disclosed in the above-described first to third embodiments of the present disclosure. For example, some components may be deleted from all the components described in the above-described first to third embodiments of the present disclosure. Further, the components described in the above-described first to third embodiments of the present disclosure may be appropriately combined.
  • first to third embodiments of the present disclosure are applied to the imaging device, but they may be applied, for example, to an endoscope system or a video microscope for capturing a subject, a mobile phone having an imaging function, and a tablet terminal having an imaging function.
  • the above-mentioned “unit” can be read as “means” or “circuit”.
  • the control unit can be read as a control means or a control circuit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

An imaging element includes: a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix; and a color filter including a plurality of filters that have different spectral transmission characteristics from each other in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics from each other in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances from each other in at least one of the visible region and the near-infrared region.

Description

CROSS-REFERENCE TO RELATED APPLICATION
This application is a continuation of International Application No. PCT/JP2018/034679, filed on Sep. 19, 2018, the entire contents of which are incorporated herein by reference.
BACKGROUND 1. Technical Field
The present disclosure relates to an imaging element, an imaging device, an imaging method, and a computer-readable recording medium that are capable of capturing a color image and a near-infrared image.
2. Related Art
In the related art, in an imaging device, a technique capable of capturing a color image and a near-infrared image has been known (see, JP 2005-6066 A). With this technique, by replacing one filter with an IR filter of a Bayer array consisting of an R filter, two G filters and a B filter, a visible light image and a near-infrared image are captured.
In addition, in the color filter composed of the R filter, the G filter, the B filter and the IR filter, a technique for capturing a color image and a near-infrared image by providing an infrared absorption filter on the light receiving faces of the R filter, the G filter and the B filter is known (see, JP 2017-139286 A).
SUMMARY
In some embodiments, an imaging element includes: a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light; and a color filter including a plurality of filters that have different spectral transmission characteristics from each other in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics from each other in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances from each other in at least one of the visible region and the near-infrared region.
In some embodiments, an imaging device includes: an imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region; and a separator configured to separate an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
In some embodiments, provided is an imaging method performed by an imaging device. The imaging device includes an imaging element and an image processor, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region. The method includes: separating, by the image processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
In some embodiments, provided is a non-transitory computer-readable recording medium with an executable program stored thereon. The program is executed by an imaging device including an imaging element and an image processor, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region. The program causes the imaging device to execute: separating, by the image processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
The above and other features, advantages and technical and industrial significance of this disclosure will be better understood by reading the following detailed description of presently preferred embodiments of the disclosure, when considered in connection with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram illustrating a functional configuration of an imaging device according to a first embodiment;
FIG. 2 is a diagram schematically illustrating a configuration of a pixel unit according to the first embodiment;
FIG. 3 is a diagram schematically illustrating a configuration of a color filter according to the first embodiment;
FIG. 4 is a diagram illustrating the transmission characteristics of each filter according to the first embodiment;
FIG. 5 is a flowchart illustrating an outline of a process executed by the imaging device according to the first embodiment;
FIG. 6 is a diagram schematically illustrating an example of an image generated by the imaging device according to the first embodiment;
FIG. 7 is a diagram schematically illustrating the spectral transmittance of each of trueG and IR according to the first embodiment;
FIG. 8 is a diagram schematically illustrating the transmittance of each of a filter G1 and a filter G2 according to the first embodiment;
FIG. 9 is a diagram schematically illustrating the spectral reflectance of a subject;
FIG. 10 illustrates the spectral sensitivity of each of a G1 pixel and a G2 pixel according to the first embodiment;
FIG. 11 is a diagram schematically illustrating a configuration of a color filter according to a first modification of the first embodiment;
FIG. 12 is a diagram illustrating the transmission characteristics of the color filter according to the first modification of the first embodiment;
FIG. 13 is a diagram schematically illustrating a color filter of a second modification of the first embodiment;
FIG. 14 is a diagram schematically illustrating the spectral transmittance of each wavelength band according to a second embodiment;
FIG. 15 is a block diagram illustrating a functional configuration of an imaging device according to a third embodiment; and
FIG. 16 is a diagram illustrating the transmission characteristics of each filter and a notch filter according to the third embodiment.
 DETAILED DESCRIPTION
Hereinafter, a mode for carrying out the present disclosure (hereinafter referred to as an “embodiment”) will be described. In the embodiment, an imaging device including an imaging element that generates image data (RAW data) by capturing a subject will be described. Moreover, the embodiment does not limit the present disclosure. Further, in the description of the drawings, the same parts will be described with the same reference numerals.
First Embodiment
Configuration of Imaging Device
FIG. 1 is a block diagram illustrating a functional configuration of an imaging device according to a first embodiment. The imaging device 1 illustrated in FIG. 1 includes an imaging unit 2, an image processing unit (image processor) 3, a display unit 4, a recording unit 5, and a control unit 6.
The imaging unit 2 generates image data by capturing a subject. The imaging unit 2 includes an optical system 21, an imaging element 22, and an A/D converter 23.
The optical system 21 forms a subject image on the light receiving face of the imaging element 22. The optical system 21 includes one or a plurality of lenses. The optical system 21 has a zoom function and a focus function. The optical system 21 changes the zoom magnification and the focus position by moving on an optical axis L1 by a drive unit such as a motor (not illustrated).
The light receiving face of the imaging element 22 is provided perpendicular to the optical axis L1 of the optical system 21. The imaging element 22 generates under the control of the control unit 6 an image signal (image data) by performing photoelectric conversion on the subject image formed by the optical system 21 to output this image signal to the A/D converter 23. The imaging element 22 is realized by using a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. The imaging element 22 includes a pixel unit (pixel portion) 221 and a color filter 222 laminated on the light receiving face of the pixel unit 221. The detailed configuration of the pixel unit 221 and the color filter 222 will be described later.
Under the control of the control unit 6, the A/D converter 23 performs an A/D conversion on an analog image signal input from the imaging unit 2 to output the image signal to the image processing unit 3. The A/D converter 23 includes, for example, an A/D conversion circuit and the like.
The image processing unit 3 performs, under the control of the control unit 6, various image processes, for example, a white balance adjustment process, a demosaicing process, a separation process, and the like on the image signal input from the imaging unit 2 to output the processed image signal to the display unit 4. The image processing unit 3 includes a dedicated processor such as various arithmetic circuits, such as a graphics processing unit (GPU) or an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), that execute a specific function. The image processing unit 3 includes an interpolation unit 31, a separation unit (separator) 32, and a generation unit 33.
The interpolation unit 31 performs a well-known interpolation process for interpolating the image signal (pixel value) of each pixel with respect to the image signal input from the imaging unit 2. Specifically, the interpolation unit 31 performs a well-known interpolation process on the image signal of each filter constituting the color filter 222 to generate an image corresponding to the image signal, of each filter, that interpolates the image signal of each filter.
The separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the pixel on which the first same-color filter is disposed and the second image signal generated by the pixel on which the second same-color filter is disposed based on the first ratio between the transmittance of the first same-color filter and the transmittance of the second same-color filter in the visible region, and the second ratio between the transmittance of the first same-color filter and the transmittance of the second same-color filter in the near-infrared region. The details of the separation method by the separation unit 32 will be described later.
The generation unit 33 generates a visible light image and a near-infrared image (hereinafter, simply referred to as an “IR image”) based on the image signal in the visible region and the image signal in the near-infrared region separated by the separation unit 32. The generation unit 33 outputs the visible light image and the IR image to the display unit 4.
Under the control of the control unit 6, the display unit 4 displays an image (at least one of the visible light image and the IR image) corresponding to the image signal input from the image processing unit 3. In addition, the display unit 4 displays various pieces of information related to the imaging device 1. The display unit 4 includes a display panel such as a liquid crystal or an organic electro luminescence (EL).
The recording unit 5 records a program executed by the imaging device 1, data being processed, image data generated by the imaging unit 2, and the like. The recording unit 5 includes a program recording unit 51 that records a program executed by the imaging device 1, and a sensitivity information recording unit 52 that records the sensitivity of each filter of the color filter 222. The recording unit 5 includes, for example, a Flash memory, a random access memory (RAM), a hard disk drive (HDD), a synchronous dynamic random access memory (SDRAM), a memory card, or the like.
The control unit 6 controls each unit of the imaging device 1. The control unit 6 includes a central processing unit (CPU) and the like.
Configuration of Pixel Unit
Next, the detailed configuration of the pixel unit 221 described above will be described. FIG. 2 is a diagram schematically illustrating the configuration of the pixel unit 221.
The pixel unit 221 illustrated in FIG. 2 is formed by arranging a plurality of pixels P that receives light transmitted through the color filter 222 in a two-dimensional grid pattern (two-dimensional matrix pattern). Each pixel P receives light which is incident from the optical system 21, and that is transmitted through the color filter 222 to perform a photoelectric conversion to generate an image signal. This image signal includes a pixel value (luminance value) of each pixel, pixel position information (pixel address), and the like. In FIG. 2, the pixel disposed in the i-th row and the j-th column is expressed as a pixel Pij.
Configuration of Color Filter
Next, the detailed configuration of the color filter 222 will be described. FIG. 3 is a diagram schematically illustrating the configuration of the color filter 222.
The color filter 222 illustrated in FIG. 3 is disposed so that a filter unit composed of a plurality of filters having different spectral transmission characteristics in each of the visible region and the near-infrared region correspond to a plurality of pixels of the pixel unit 221. Specifically, the color filter 222 is disposed on the light receiving face of the pixel unit 221, and includes four types of filters that transmit the visible light in different wavelength bands in the visible region and the near-infrared rays in different wavelength bands in the near-infrared region. More specifically, the color filter 222 includes an R filter, a G1 filter, a G2 filter, and a B filter. The color filter 222 is formed by arranging the R filters, the G1 filters, the G2 filters, and the B filters side by side in a predetermined array pattern. Specifically, the color filter 222 is formed by arranging an R filter column, a G1 filter column, a B filter column, an R filter column, and a G2 filter column in an array pattern.
The R filter transmits light in the red wavelength band. The G1 filter transmits light in the green wavelength band and transmits light in the near-infrared wavelength band. The G2 filter has different transmission wavelength characteristics than the G1 filter in the same color wavelength band as the G1 filter, transmits light in the green wavelength band, and transmits light in the near-infrared wavelength band. The B filter transmits light in the blue wavelength band.
The G1 filter and the G2 filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances from each other in at least one of the visible region and the near-infrared region. In the first embodiment, the G1 filter and the G2 filter function as a first same-color filter and a second same-color filter, respectively. Further, in the following, the pixel Pij on which each filter is provided receives light in the wavelength band transmitted through the filter. Therefore, the pixel on which the R filter is disposed is referred to as an R pixel, the pixel on which the G1 filter is disposed is referred to as a G1 pixel, the pixel on which the G2 filter is disposed is referred to as a G2 pixel, and the pixel on which the B filter is disposed is referred to as a B pixel.
Spectral Transmission Characteristics of Each Filter
Next, the transmission characteristics (spectral sensitivity characteristics) of the color filter 222 described above will be described. FIG. 4 is a diagram illustrating the transmission characteristics of each filter. In FIG. 4, the vertical axis represents the transmittance (spectral sensitivity) and the horizontal axis represents the wavelength (nm). Furthermore, in FIG. 4, the curve LB indicates the transmittance curve of filter B, the curve LG1 indicates the transmission curve of filter G1, the curve LG2 indicates the transmission curve of filter G2, and the curve LR indicates the transmission curve of the filter R. Further, in the following, the visible region ranges from 400 nm to 700 nm, and the near-infrared region ranges from 700 nm to 1000 nm.
As illustrated in the curve LR illustrated in FIG. 4, the filter R transmits light in the red wavelength band (wavelength band 600 nm to 700 nm), which is the first visible region. Further, as illustrated in the curve LB, the filter B transmits light in the blue wavelength band (400 nm to 500 nm), which is the second visible region. Also, as illustrated in the curve LG1 and the curve LG2, the filter G1 and the filter G2 transmit light in the green wavelength band (500 nm to 600 nm), which is the third visible band, and the near-infrared region (700 nm to 1000 nm). Further, the filter G1 and the filter G2 have a region having a constant spectral transmittance in each of the third visible light region and the near-infrared region. Furthermore, the filter G1 and the filter G2 have different transmittances from each other in each of the third visible light region and the near-infrared region, and the difference D1 between the transmittance of the filter G1 and the transmittance of the filter G2 in the third visible light region, and the difference D2 between the transmittances of the filter G1 and the transmittance of the filter G2 in the near-infrared region are each constant. In the first embodiment, the filter G1 and the filter G2 have different transmittances from each other in each of the third visible light region and the near-infrared region, but the filter G1 and the filter G2 may have different transmittances from each other in at least one of the third visible light region and the near-infrared region.
Processing of Imaging Device
Next, the process executed by the imaging device 1 will be described. FIG. 5 is a flowchart illustrating an outline of the process executed by the imaging device 1. FIG. 6 is a diagram schematically illustrating an example of an image generated by the imaging device 1.
As illustrated in FIG. 5, first, the imaging unit 2 images a subject (step S101). In this case, the imaging unit 2 outputs an image signal obtained by imaging the subject to the image processing unit 3.
Subsequently, the interpolation unit 31 performs an interpolation process for interpolating the pixel value of each pixel with respect to the image signal input from the imaging unit 2 (step S102). Specifically, as illustrated in FIG. 6, the interpolation unit 31 performs the interpolation process for interpolating the image signal of each pixel with respect to the image signal input from the imaging unit 2 to generate a B image PB, an R image PR, a G1 image PG1, and a G2 image PG2.
After that, the separation unit 32 executes a separation process for separating the image signal in the visible region and the image signal in the near-infrared region with respect to the G1 image PG1 and the G2 image PG2 (step S103).
Details of Separation Process
Here, an outline of the separation process executed by the separation unit 32 will be described.
FIG. 7 is a diagram schematically illustrating the spectral transmittance in each of the trueG and the IR. In FIG. 7, the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm). Further, in FIG. 7, the curve LTG indicates the transmittance in the trueG, and the curve LIR indicates the transmittance in the IR. Further, FIG. 8 is a diagram schematically illustrating the transmittance of each of the filter G1 and the filter G2. In FIG. 8, the curve LG1 indicates the transmittance of the filter G1, and the curve LG2 indicates the transmittance of the filter G2.
As illustrated in FIG. 7, in the first embodiment, in a case where the transmittance in each of the trueG (500 nm to 600 nm) and the IR (700 nm to 1000 nm) is set to “1.0”, when the transmittance in each wavelength band is a, b, c and d, as illustrated in FIG. 8, the transmission ratios between the filter G1 and the filter G2 in the trueG (500 nm to 600 nm) is constant and known (a=1.0, c=0.7), and the transmission ratios between the filter G1 and the filter G2 in the IR (700 nm to 1000 nm) are constant and known (b=0.2, d=0.6).
FIG. 9 is a diagram schematically illustrating the spectral reflectance of the subject. FIG. 10 illustrates the spectral sensitivities of the G1 pixel and the G2 pixel. In FIG. 9, the vertical axis represents the spectral reflectance. In FIG. 10, the vertical axis represents the spectral sensitivity, and in FIGS. 9 and 10, the horizontal axis represents the wavelength (nm). Further, in FIG. 9, the curve Lp indicates the spectral reflectance of the subject. Also, in FIG. 10, the area surrounded by the curve LTG1 indicates truerG1 (a×truerG), the area surrounded by the curve LTG2 indicates truerG2 (c×truerG), the area surrounded by the curve LTR2 indicates IR2 (d×IR), and the area surrounded by the curve LIR1 indicates IR1 (b×IR).
As illustrated in FIGS. 9 and 10, the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the G1 pixel and the second image signal generated by the G2 pixel based on the first ratio D1 (a/c) of the transmittance of the G1 filter to the transmittance of the G2 filter in the visible region and the second ratio D2 (d/b) of the transmittance of the G1 filter to the transmittance of the G2 filter in the near-infrared region.
That is, when in the G1 image PG1, the image signal of the G1 image PG1 only in the visible region is trueG1, the image signal only in the near-infrared region is IR1, the image signal of the true G1 image PG1 only in the visible region is trueG, and the true image signal only in the near-infrared region is IR, and in the G2 image PG2, the image signal of the true G2 image PG2 only in the visible region is trueG2, the true image signal only in the near-infrared region is IR2, the image signal of the true G2 image PG2 only in the visible region is trueG, and the true image signal only in the near-infrared region is IR, the following equations (1) and (2) hold.
trueG 1+IR1=a×trueG+b×IR=G1  (1)
trueG 2+IR2=c×trueG+d×IR=G2  (2)
when solving the equations (1) and (2) above, and assuming that α=ad−bc holds, and ad−bc≠0 holds, resulting in the following:
IR=(a×G2−c×G1)/α  (3)
trueG=(d×G1−b×G2)/α  (4)
Further, when a (a=1.0), b (b=0.2), c (c=0.7) and d (d=0.6) in FIG. 8 are substituted into the above equations (1) and (2), the following equations (5) and (6) hold.
trueG ⁢ ⁢ 1 + IR ⁢ ⁢ 1 = ⁢ a × trueG + b × IR = ⁢ 1.0 × trueG + 0.2 × IR = ⁢ G ⁢ ⁢ 1 ( 5 ) trueG ⁢ ⁢ 2 + IR ⁢ ⁢ 2 = ⁢ c × trueG + d × IR = ⁢ 0.7 × trueG + 0.6 × IR = ⁢ G ⁢ ⁢ 2 ( 6 )
In this way, using equations (3) to (6), the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the G1 pixel and the second image signal generated by the G2 pixel based on the first ratio D1 (a/c) of the transmittance of the G1 filter to the transmittance of the G2 filter in the visible region and the second ratio D2 (d/b) of the transmittance of the G1 filter to the transmittance of the G2 filter in the near-infrared region.
Returning to FIG. 5, the description after step S103 is continued.
Subsequently, the generation unit 33 generates an IR image and a G image based on the signal value in the visible region and the signal value in the near-infrared region separated by the separation unit 32. Specifically, as illustrated in FIG. 6, the generation unit 33 generates the G image PG and the IR image PIR based on the signal value in the visible region and the signal value in the near-infrared region separated by the separation unit 32 (step S104).
After that, the generation unit 33 generates a visible image (step S105). Specifically, as illustrated in FIG. 6, the generation unit 33 generates the visible image PW based on the B image PB, the R image PR, and the G image PG. After step S105, the imaging device 1 ends this process.
According to the first embodiment described above, the color filter 222 includes the first same-color filter and the second same-color filter in which any one or more of the plurality of filters transmit light in a near-infrared region, the first same-color filter and the second same-color filter having different transmission wavelength characteristics in the same color wavelength band, and the first same-color filter and the second same-color filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region and have different spectral transmittances in at least one of the visible region and the near-infrared region, so that a color image and a near-infrared image can be acquired at the same time without installing an IR filter.
Further, according to the first embodiment, the separation unit 32 separates the image signal in the visible region and the image signal in the near-infrared region from each of the first image signal generated by the pixel on which the first same-color filter is disposed and the second image signal generated by the pixel on which the second same-color filter is disposed based on the first ratio of the transmittance of the first same-color filter to the transmittance of the second same-color filter in the visible region, and the second ratio of the transmittance of the first same-color filter to the transmittance of the second same-color filter in the near-infrared region, so that a color image and a near-infrared image can be acquired at the same time without installing an IR filter.
First Modification of First Embodiment
Next, a first modification of the first embodiment will be described. In the first modification of the first embodiment, the configuration of the color filter is different. Specifically, in the first embodiment described above, the G filters that transmit light in the green wavelength band have different spectral transmission characteristics, but in the first modification of the first embodiment, R filters that transmit light in the red wavelength band have different spectral transmission characteristics. The same components as those of the imaging device 1 according to the first embodiment described above are given the same reference numerals, and detailed description thereof will be omitted.
Configuration of Color Filter
FIG. 11 describes a detailed configuration of the color filter according to the first modification of the first embodiment.
A color filter 222A illustrated in FIG. 11 is disposed so that a filter unit composed of a plurality of filters having different spectral transmission characteristics in each of the visible region and the near-infrared region correspond to a plurality of pixels of the pixel unit 221. Specifically, the color filter 222A is disposed on the light receiving face of the pixel unit 221, and includes four types of filters that transmit the visible light in different wavelength bands in the visible region and the near-infrared rays in different wavelength bands in the near-infrared region. More specifically, the color filter 222 includes an R1 filter, an R2 filter, a G filter, and a B filter. The color filter 222A is formed by arranging the R1 filters, the R2 filters, the G filters, and the B filters side by side in a predetermined array pattern. Specifically, the color filter 222A is formed by arranging an R1 filter column, a G filter column, a B filter column, and an R2 filter column in an array pattern.
The R1 filter transmits light in the red wavelength band and transmits light in the near-infrared wavelength band. The R2 filter has different transmission wavelength characteristics than the R1 filter in the same color wavelength band as the R1 filter, transmits light in the red wavelength band, and transmits light in the near-infrared wavelength band.
The R1 filter and the R2 filter have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region. In the first modification of the first embodiment, the R1 filter and the R2 filter function as the first same-color filter and the second same-color filter, respectively. Further, in the following, the pixel Pij on which each filter is provided receives light in the wavelength band transmitted through the filter. Therefore, the pixel on which the R1 filter is disposed is referred to as an R1 pixel, the pixel on which the R2 filter is disposed is referred to as an R2 pixel, the pixel on which the G filter is disposed is referred to as a G pixel, and the pixel on which the B filter is disposed is referred to as a B pixel.
FIG. 12 is a diagram illustrating the transmission characteristics of the color filter according to the first modification of the first embodiment. In FIG. 12, the transmission curve is simulated and standardized so that the maximum values of the filters are substantially equal. In FIG. 12, the vertical axis represents the transmittance (spectral sensitivity) and the horizontal axis represents the wavelength (nm). Furthermore, in FIG. 12, the curve LB indicates the transmittance curve of filter B, the curve LG1 indicates the transmission curve of filter G, the curve LR1 indicates the transmission curve of filter R1, and the curve LR2 indicates the transmission curve of the filter R2. Further, in the following, the visible region ranges from 400 nm to 700 nm, and the near-infrared region ranges from 700 nm to 1000 nm.
The filter R1 and the filter R2 have a region where the difference between transmittances, D11, and the difference between transmittances, D12, are each constant in each of the first visible light region and the near-infrared region, and have different transmittances in at least one of the first visible light region and the near-infrared region. Specifically, the filter R1 and the filter R2 have a region where the difference between transmittances, D11, and the difference between transmittances, D12, are each constant in each of the first visible light region and the near-infrared region, and have different transmittances in at least one of the first visible light region and the near-infrared region.
Even when the color filter 222A configured in this way is disposed on the light receiving face of the pixel unit 221, it is possible to simultaneously capture a visible light image and a near-infrared image as in the first embodiment described above.
In addition, in the first modification of the above-described first embodiment, the R filters that transmit light in the red wavelength band have different transmittances, but in addition to the R1 filter and the R2 filter, for example, the G1 filter and the G2 filter of the first embodiment described above may be used to constitute the color filter.
Second Modification of First Embodiment
Next, a second modification of the first embodiment will be described. In the first embodiment described above, each filter is disposed for each column, but this is not limited and can be changed as appropriate.
FIG. 13 is a diagram schematically illustrating the color filter of the second modification of the first embodiment. A color filter 222B illustrated in FIG. 13 is configured in a Bayer array, and the G1 filter and the G2 filter are disposed at the locations where the G filters are disposed.
According to the second modification of the first embodiment described above, a conventional image processing pipeline (image signal line) can be used, so that a visible light image and a near-infrared image can be acquired at the same time with a simple configuration.
Second Embodiment
Next, a second embodiment will be described. In the second embodiment, the signal values of the R pixel and the B pixel that do not include the near-infrared are calculated. In the following, the calculation method calculated by the separation unit 32 will be described. The same components as those of the imaging device 1 according to the first embodiment described above are given the same reference numerals, and detailed description thereof will be omitted.
FIG. 14 is a diagram schematically illustrating the spectral transmittance in each wavelength band. In FIG. 14, the region LTB indicates the integral value of the image signal trueB of the B filter, the region LTR indicates the integral value of the image signal trueR of the R filter, the region LTG indicates the integral value of the image signal trueG when the G1 filter+the G2 filter are combined, and the LTIR indicates the integral value of the image signal IR. Furthermore, in FIG. 14, the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm).
As illustrated in FIG. 14, first, the separation unit 32 calculates the image signal trueG and the image signal IR by the method of the first embodiment described above. Then, as illustrated in FIG. 14, since the transmittance in the near-infrared region is substantially constant, the separation unit 32 calculates the image signal of the R pixel and the image signal of the B pixel based on the image signal IR of the near-infrared light. Specifically, the following equations hold when the transmittance in the near-infrared region is almost constant.
truerR+r_ir×IR=truerR+1.0×IR=R  (7)
truerB+b_ir×IR=truerB+0.3×IR=B  (8)
that is,
truerR=R−r_ir×IR=R−1.0×IR  (9)
truerB=B−b_ir×IR=B−0.3×IR  (10)
In this way, the separation unit 32 calculates the image signal of each of the R pixel and the B pixel using the above equations (9) and (10).
According to the second embodiment described above, the true pixel values of the R pixel and the B pixel can be output.
Third Embodiment
Next, a third embodiment will be described. The third embodiment has a different configuration from the imaging device 1 according to the first embodiment described above. Specifically, the imaging device according to the third embodiment further includes a notch filter that cuts light in a predetermined wavelength band. Hereinafter, the configuration of the imaging device according to the third embodiment will be described. The same components as those of the imaging device 1 according to the first embodiment described above are given the same reference numerals, and detailed description thereof will be omitted.
Configuration of Imaging Device
FIG. 15 is a block diagram illustrating a functional configuration of the imaging device according to the third embodiment. An imaging device 10 illustrated in FIG. 15 includes an imaging unit 2C instead of the imaging unit 2 according to the first embodiment described above. Further, the imaging unit 2C includes an imaging element 22C instead of the imaging element 22 according to the first embodiment described above. The imaging element 22C further includes a notch filter 223 in addition to the configuration of the imaging element 22 according to the first embodiment described above.
The notch filter 223 transmits light in the visible region and cuts part of light in the near-infrared region.
FIG. 16 is a diagram illustrating the transmission characteristics of each filter and the notch filter. In FIG. 16, the curve LN indicates the transmission characteristics of the notch filter 223. Further, in FIG. 16, the vertical axis represents the transmittance and the horizontal axis represents the wavelength (nm).
As illustrated in FIG. 16, the notch filter 223 transmits light in the visible region and cuts part of light in the near-infrared region. The notch filter 223 cuts light in a certain region in the near-infrared region, for example, in the region of 700 nm to 800 nm. As a result, the spectral transmittance of each filter is made substantially constant. As a result, a highly accurate near-infrared image can be acquired.
According to the third embodiment described above, since the notch filter 223 cuts light in a certain region in the near-infrared region, the spectral transmittance of each filter can be made almost constant, so that highly accurate near-infrared images can be acquired.
OTHER EMBODIMENTS
Various embodiments can be formed by appropriately combining the plurality of components disclosed in the above-described first to third embodiments of the present disclosure. For example, some components may be deleted from all the components described in the above-described first to third embodiments of the present disclosure. Further, the components described in the above-described first to third embodiments of the present disclosure may be appropriately combined.
Further, the first to third embodiments of the present disclosure are applied to the imaging device, but they may be applied, for example, to an endoscope system or a video microscope for capturing a subject, a mobile phone having an imaging function, and a tablet terminal having an imaging function.
Further, in the first to third embodiments of the present disclosure, the above-mentioned “unit” can be read as “means” or “circuit”. For example, the control unit can be read as a control means or a control circuit.
According to the present disclosure, there is an effect that a color image and a near-infrared image can be acquired at the same time without providing an IR filter.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the disclosure in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

Claims (5)

What is claimed is:
1. An imaging device comprising:
an imaging element including
a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and
a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region; and
a processor comprising hardware, the processor being configured to separate an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
2. The imaging device according to claim 1, wherein
the color filter is in a Bayer array, and
the first same-color filter and the second same-color filter transmit light in a green wavelength band.
3. The imaging device according to claim 2, further comprising:
a notch filter provided on a light receiving face of the color filter and cutting part of light in the near-infrared region.
4. An imaging method performed by an imaging device, the imaging device including an imaging element and a processor comprising hardware, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region, the method comprising:
separating, by the processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
5. A non-transitory computer-readable recording medium with an executable program stored thereon, the program being executed by an imaging device including an imaging element and a processor comprising hardware, the imaging element including a pixel portion in which a plurality of pixels are disposed in a two-dimensional matrix, each pixel being configured to generate an image signal by receiving light, and a color filter including a plurality of filters that have different spectral transmission characteristics in each of a visible region and a near-infrared region and that are disposed on the plurality of pixels, each filter corresponding to each pixel, any one or more of the plurality of filters being configured to transmit light in the near-infrared region, the plurality of filters including a first same-color filter and a second same-color filter which have different transmission wavelength characteristics in a same color wavelength band, have a region with a constant spectral transmittance in each of the visible region and the near-infrared region, and have different spectral transmittances in at least one of the visible region and the near-infrared region, the program causing the imaging device to execute:
separating, by the processor, an image signal in the visible region and an image signal in the near-infrared region from each of a first image signal and a second image signal based on a first ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the visible region and a second ratio between a transmittance of the first same-color filter and a transmittance of the second same-color filter in the near-infrared region, the first image signal being generated by the pixel on which the first same-color filter is disposed, the second image signal being generated by the pixel on which the second same-color filter is disposed.
US17/181,022 2018-09-19 2021-02-22 Imaging element, imaging device, imaging method and computer-readable recording medium Active US11368638B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2018/034679 WO2020059050A1 (en) 2018-09-19 2018-09-19 Imaging element, imaging device, imaging method, and program

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/034679 Continuation WO2020059050A1 (en) 2018-09-19 2018-09-19 Imaging element, imaging device, imaging method, and program

Publications (2)

Publication Number Publication Date
US20210185248A1 US20210185248A1 (en) 2021-06-17
US11368638B2 true US11368638B2 (en) 2022-06-21

Family

ID=69887001

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/181,022 Active US11368638B2 (en) 2018-09-19 2021-02-22 Imaging element, imaging device, imaging method and computer-readable recording medium

Country Status (4)

Country Link
US (1) US11368638B2 (en)
JP (1) JP7083912B2 (en)
CN (1) CN112585960B (en)
WO (1) WO2020059050A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022019025A1 (en) * 2020-07-20 2022-01-27 ソニーグループ株式会社 Information processing device, information processing system, information processing method, and information processing program
JP2023075607A (en) * 2021-11-19 2023-05-31 キヤノン株式会社 Photoelectric conversion element and photoelectric conversion device

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002142228A (en) * 2000-10-31 2002-05-17 Toyota Central Res & Dev Lab Inc Imaging device
JP2005006066A (en) 2003-06-12 2005-01-06 Acutelogic Corp Color filter for solid-state image pickup element and color image pickup device
JP2005341467A (en) 2004-05-31 2005-12-08 Mitsubishi Electric Corp Imaging apparatus and signal processing method
JP2006013567A (en) 2004-06-22 2006-01-12 Mitsubishi Electric Corp Imaging apparatus and signal processing method thereof
JP2006094112A (en) 2004-09-24 2006-04-06 Mitsubishi Electric Corp Imaging device
US20070126894A1 (en) * 2005-12-07 2007-06-07 Dirk Stroker Method for calculating color correction
US20080278592A1 (en) 2004-04-05 2008-11-13 Mitsubishi Electric Corporation Imaging Device
JP2017139286A (en) 2016-02-02 2017-08-10 ソニー株式会社 Imaging element and camera system

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5484015B2 (en) * 2009-11-26 2014-05-07 キヤノン株式会社 Imaging apparatus, imaging method, and program
US8408821B2 (en) * 2010-10-12 2013-04-02 Omnivision Technologies, Inc. Visible and infrared dual mode imaging system
JP2014075767A (en) * 2012-10-05 2014-04-24 Canon Inc Solid-state imaging device
JP6291048B2 (en) * 2014-06-24 2018-03-14 マクセル株式会社 Imaging processing apparatus and imaging processing method
CN107530033A (en) * 2015-04-30 2018-01-02 奥林巴斯株式会社 Camera device
WO2017193738A1 (en) * 2016-05-09 2017-11-16 比亚迪股份有限公司 Image sensor, imaging method, and imaging device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002142228A (en) * 2000-10-31 2002-05-17 Toyota Central Res & Dev Lab Inc Imaging device
JP2005006066A (en) 2003-06-12 2005-01-06 Acutelogic Corp Color filter for solid-state image pickup element and color image pickup device
US20080278592A1 (en) 2004-04-05 2008-11-13 Mitsubishi Electric Corporation Imaging Device
JP2005341467A (en) 2004-05-31 2005-12-08 Mitsubishi Electric Corp Imaging apparatus and signal processing method
JP2006013567A (en) 2004-06-22 2006-01-12 Mitsubishi Electric Corp Imaging apparatus and signal processing method thereof
JP2006094112A (en) 2004-09-24 2006-04-06 Mitsubishi Electric Corp Imaging device
US20070126894A1 (en) * 2005-12-07 2007-06-07 Dirk Stroker Method for calculating color correction
JP2017139286A (en) 2016-02-02 2017-08-10 ソニー株式会社 Imaging element and camera system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
International Search Report dated Dec. 11, 2018 issued in PCT/JP2018/034679.

Also Published As

Publication number Publication date
JP7083912B2 (en) 2022-06-13
JPWO2020059050A1 (en) 2021-08-30
CN112585960A (en) 2021-03-30
WO2020059050A1 (en) 2020-03-26
CN112585960B (en) 2022-02-25
US20210185248A1 (en) 2021-06-17

Similar Documents

Publication Publication Date Title
US8456565B2 (en) Imaging device
US9729805B2 (en) Imaging device and defective pixel correction method
US9172927B2 (en) Imaging device, and image processing method
US11659294B2 (en) Image sensor, imaging apparatus, electronic device, image processing system, and signal processing method
US8928785B2 (en) Imaging device and storage medium storing an imaging program
CN104412581B (en) Color image sensor and camera head
US8416325B2 (en) Imaging apparatus and color contamination correction method
US20180330529A1 (en) Image processing apparatus, image processing method, and computer readable recording medium
US20230007191A1 (en) Image sensor, imaging apparatus, electronic device, image processing system, and signal processing method
JP2015185943A (en) Micro lens with filter array and solid-state imaging device
US11368638B2 (en) Imaging element, imaging device, imaging method and computer-readable recording medium
US8723993B2 (en) Imaging device and storage medium storing an imaging program
US9727947B2 (en) Downscaling a digital raw image frame
JPWO2012153532A1 (en) Imaging device
WO2023098284A1 (en) Image sensor, camera module, electronic device, and image generation method and apparatus
US8976275B2 (en) Color imaging element
US20150042782A1 (en) Image processing apparatus, imaging apparatus, microscope system, image processing method, and computer readable recording medium
US20150077600A1 (en) Color filter array and solid-state image sensor
JP5411390B2 (en) Color image sensor
US20070269133A1 (en) Image-data noise reduction apparatus and method of controlling same
TWI590659B (en) Image processing method and camera device
US12302007B2 (en) RGBIR color filter image processing with lens shading correction
US20210203899A1 (en) Image processing device, image processing method, and imaging device

Legal Events

Date Code Title Description
AS Assignment

Owner name: OLYMPUS CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YOSHIZAKI, KAZUNORI;REEL/FRAME:055349/0592

Effective date: 20210219

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4