CN115174794B - Double-light fusion imaging chip and double-light picture fusion processing method - Google Patents
Double-light fusion imaging chip and double-light picture fusion processing method Download PDFInfo
- Publication number
- CN115174794B CN115174794B CN202210961938.4A CN202210961938A CN115174794B CN 115174794 B CN115174794 B CN 115174794B CN 202210961938 A CN202210961938 A CN 202210961938A CN 115174794 B CN115174794 B CN 115174794B
- Authority
- CN
- China
- Prior art keywords
- light
- pixel
- nano
- fusion
- route
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Solid State Image Pick-Up Elements (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
The invention provides a double-light fusion imaging chip and a double-light picture fusion processing method, and relates to the field of optical sensing chips and image processing. The chip comprises: nano light route and area array photoelectric detector; setting the geometric structure and morphology of the nano light routing surface by a nano light routing surface space optimization method; the nano optical route is integrated and adhered with the area array photoelectric detector through an optical adhesive layer, and the nano optical route is completely overlapped with the area array photoelectric detector; the size of a single pixel of the nano light route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array detector array. The light energy utilization efficiency of the double-light fusion imaging chip based on the nano light route is remarkably improved, meanwhile, the double-light fusion imaging chip comprehensively utilizes light field information of four channels including red, green, blue and near infrared, and the rear-end double-light fusion image processing algorithm and output are more compact, simple and convenient.
Description
Technical Field
The invention relates to the field of optical sensing chips and image processing, in particular to a double-light fusion imaging chip and a double-light picture fusion processing method.
Background
The existing multichannel (color) optical array sensor chip mainly relies on Bayer filters to respectively distribute light energy of different spectral bands to sub-pixels in a corresponding space, for example, an RGGB (red green blue) form is adopted, namely, each complete pixel consisting of four sub-pixels is equivalently composed of half of G,1/4 of R and 1/4 of B; or in order to improve the light sensing capability, RYYB space layout is adopted, which is more beneficial to noise control but can influence color information. Regardless of the layout employed, each subpixel includes only a portion of the spectrum, requiring the full-space color image output in conjunction with the back-end Bayer interpolation algorithm. The absorption filter realizes space correspondence through light absorption outside the passband, and the effective area of the light energy is directly changed into the corresponding area of the sub-pixels, so that the overall light transmission and utilization efficiency of the traditional color image sensor is lower, the signal intensity of each sensor pixel is limited, and the overall light energy utilization efficiency is lower.
The nano light-splitting routing technology is a new developed micro-nano optical passive device in recent years, is a colorized image sensing technology based on structural colors, has very large freedom degree in color operation based on micro-nano optical structure design, has very strong dispersion on the nano structured optical surface, can be used for generating required structural colors and realizing light splitting, has no reduction of effective utilization area of light energy, and can greatly improve the light energy utilization efficiency.
On the other hand, since visible light images generally have a higher spatial resolution and considerable detail and contrast, they are susceptible to light and environmental influences. Infrared light images can be imaged around the clock, but lack texture information and image details. The visible light image and the infrared light image have strong complementarity, and the combination of the visible light image and the infrared light image can generate a robust and information-rich image. The visible-infrared double-light fusion is widely applied in the fields of low-light imaging, infrared thermal imaging, security protection, automatic driving and the like, but complex image registration and image fusion algorithms need to be developed, and great calculation force demand pressure is brought to back-end processing, so that great challenges exist in the aspects of algorithm design calibration and special chip development.
Aiming at the problems, the invention provides a high-efficiency artificial super-surface structure (nano light route) designed by combining a micro-nano optical principle with an optimization algorithm, and the image plane light sensation information multiplexed by a plurality of color channels can be obtained under the condition of basically not losing light energy; the high-efficiency visible-infrared double-light fusion imaging chip of a common light path is realized by integrating the nano light route and a wide-spectrum (covering RGB three primary color wave bands and near infrared wave bands) area array photoelectric detector.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the problems of low light energy utilization efficiency, complex system, huge volume, large algorithm development and large calculation amount and the like commonly existing in the prior multi-channel photoelectric detection chip and the technical scheme of visible-near infrared double-light fusion, thereby providing a solution for integrating the space alignment of the RGB-IR four-channel nano light route and the wide spectrum detector.
In order to solve the above problems, the present invention provides a dual light fusion imaging chip, comprising:
nano light route and area array photoelectric detector;
setting the geometric structure and morphology of the nano light routing surface by a nano light routing surface space optimization method;
the nano optical route is integrated and adhered with the area array photoelectric detector through a solid optical adhesive layer or a liquid optical adhesive layer, and the nano optical route is completely overlapped with the area array photoelectric detector; the size of a single pixel of the nano light route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array detector array.
Preferably, the nano-optical routing surface space optimization method comprises the following steps:
setting the area array detector as a single sensing pixel element, receiving visible light and near infrared light (IR), wherein the visible light comprises: red light (R), green light (G), blue light (B);
determining surface space design parameters of the nano light route according to the set sensing pixel element area and the super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical route and the substrate material and thickness;
and establishing an optimization model according to the super-surface design parameters, and outputting an optimization scheme.
Preferably, the sensing pixel element includes:
four sub-pixel units arranged in a square matrix correspondingly receive red light (R), green light (G), blue light (B) and near infrared light (IR).
Preferably, the set wavelength range of the near infrared light (IR) is 800nm to 1000nm.
Preferably, the determining the surface space design parameter of the nano light route according to the set sensing pixel element area and the super surface semiconductor material specifically includes:
according to the four sub-pixel units of the sensing pixel element area, the super-surface corresponds to the same plane area and is used as a design area;
setting the corresponding thickness t of the super surface and the substrate material and thickness d according to the dielectric constant distribution E (x, y) of the super surface;
and calculating the distribution of the energy flow of a target light field of the central wavelengths of the red light (R), the green light (G), the blue light (B) and the near infrared light (IR), wherein the target light field is the rear emergent surface of the substrate material.
Preferably, the method for creating the optimization model of the super-surface design parameter according to the design parameter of the nano-optical path surface space, and outputting the optimization scheme specifically includes:
establishing a light energy distribution target model, as shown in a formula (1):
wherein alpha is B ，α G ，α R ，α IR As a weight factor, can be adjusted according to specific design; lambda (lambda) 1 ，λ 2 —λ 8 The minimum wavelength and the maximum wavelength of four corresponding wave bands are respectively provided, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light; p (P) z (lambda) is the longitudinal light field wave-printing vector on the single pixel face of the sensor, P zB (lambda) is the wave-printing vector of blue light wave band passing through B corresponding sub-pixel, P zG (lambda) is the wave-printing vector of the green light wave band passing through the sub-pixel corresponding to G, P zR (lambda) is the PointTing vector of the red light band through the R-corresponding sub-pixel, P zIR (lambda) is the wave-printing vector of the infrared band through the IR corresponding sub-pixel; t (T) B (lambda) is the transmittance of the blue light incident surface to the pixel surface, T G (lambda) is the transmittance of the green light incident surface to the pixel surface, T R (lambda) is the transmittance of the red light incident surface to the pixel surface, T IR (lambda) is the transmission coefficient from the infrared light incident surface to the pixel surface;
and optimizing the dielectric constant distribution E (x, y) of the super surface based on the light energy distribution target model until the light energy distribution target model reaches convergence and extremum.
Preferably, the optimizing the dielectric constant distribution e (x, y) of the super surface based on the optical energy distribution target model until the optical energy distribution target model reaches convergence and extremum specifically includes:
when optimizing free form super surface, first setting initial value of E (x, y), where E (x, y) is the interval [ E [ min ,∈ max ]The method comprises the steps of carrying out a first treatment on the surface of the Optimizing by adopting an optimization iterative algorithm (such as a moving asymptote method) and a binarization processing algorithm;
when optimizing the digital form super surface, dividing the pixels of the design area into n units, and taking the value E of each unit min Or E shaped max The method comprises the steps of carrying out a first treatment on the surface of the The genetic algorithm GA (genericgenetic algorithm) or the particle swarm optimization algorithm PSO (particle swarm optimization) is used for optimization.
The invention also provides a double-light image fusion processing method suitable for the double-light fusion imaging chip, which comprises the following steps:
using the indication values of the IR sub-pixel units in the four adjacent pixel units to perform space two-dimensional spatial interpolation to obtain corresponding IR responses at the 4 sub-pixel positions in each pixel unit, and recording as an image P IR ；
The IR sub-pixel in 4 sub-pixels in each pixel unit is replaced by the G sub-pixel in the adjacent pixel unit, which is marked as G1, and then forms RGGB type Bayer pixel arrangement together with R, G and B in the original pixel unit for subsequent generation of RGB color image, which is marked as P c ；
Will P IR And P c Regarded as double lightAnd (3) fusing the infrared light image and the visible light image required in the algorithm, processing by using the image fusion algorithm, and developing feature extraction, fusion strategy and image reconstruction to generate a double-light fusion image.
The invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the double-light image fusion processing method when being executed by a processor.
The invention has the following advantages: obtaining image plane light sensation information multiplexed by a plurality of color channels under the condition of basically not losing light energy; the high-efficiency visible-infrared double-light fusion imaging chip for realizing a common light path is provided by integrating the type of nano light route with a wide-spectrum (covering RGB three primary color wave bands and near infrared wave bands) area array photoelectric detector, and has the specific effects that:
the size of a single pixel of the first nano optical route is corresponding to that of a pixel of a wide-spectrum CCD or CMOS area array detector array, the corresponding positions also need to be completely corresponding, and the distance d between the two is generally in the micron order, and can be determined as the thickness of a substrate material for designing the nano optical route; when the nano optical router is integrated with an area array detector with wide spectrum response, an optical adhesive (OCA) or liquid optical adhesive (LOCA) process is adopted for adhesion and fixation;
the second and the fourth nanometer light routes have the advantage of high light energy utilization efficiency, so that the sensitivity and the low-light detection capability of the sensor can be greatly improved, meanwhile, the natural advantage of double-light fusion of a common light path is achieved, images after double-light fusion can be conveniently output through a similar spatial interpolation algorithm like Bayer filtering, the application range of the existing binocular vision image double-light fusion based on a visible light camera and an infrared camera can be greatly expanded, and various advantages are brought, such as miniaturization and reduction of double-light fusion calculation complexity and resource requirements;
thirdly, the nano optical route is integrated and bonded with the area array detector through a solid optical adhesive layer or a liquid optical adhesive layer, and the nano optical route is completely overlapped with the area array detector; the size of a single pixel of the nano light route is corresponding to that of a pixel of a wide-spectrum CCD or CMOS area array detector array, full-wave simulation verification is carried out, the real light field light splitting characteristic of the nano light route is obtained through calculation, and then a sample is prepared according to photoetching or EBL technology; carrying out space alignment and gluing integration on the sample and a pre-corresponding imaging detection array;
fourth, directly outputting the original image information containing R, G, B, IR channels, periodically arranging according to a space square quarter grid form to form array original data, and obtaining a final fusion image by using a back-end data processing method.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of the working principle of an RGB-IR four-channel nano-beam-splitting route according to an embodiment of the present invention;
FIG. 2 is a basic flow of density-based topology optimization design for nano-optical routing;
FIG. 3 shows an RGB-IR four-channel nano-beam-splitting routing microstructure design according to an embodiment of the present invention, wherein the white area is a medium, the black area is air, and the structure is disposed on a substrate;
FIG. 4 is a graph showing the spectral efficiency ratio obtained based on the design of FIG. 2 in accordance with an embodiment of the present invention;
FIG. 5 is a light field simulation result corresponding to a specific design of an imaging plane based on RGB-IR four-channel nano-beam-splitting routing according to an embodiment of the present invention;
fig. 6 is a schematic flow chart of a back-end dual-light fusion algorithm according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a method for acquiring IR images at all sub-pixel locations according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of an RGB Bayer bilinear spatial interpolation method according to an embodiment of the present invention;
fig. 9 is a flowchart of steps of a method for optimizing a nano-optical routing surface space according to an embodiment of the present invention.
Reference numerals:
1. nano optical routing; 2. an area array photodetector; 3. a surface structure; 4. sub-pixel unit
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Summary of the application
The absorption filter realizes space correspondence through light absorption outside the passband, and the effective area of the light energy is directly changed into the corresponding area of the sub-pixels, so that the overall light transmission and utilization efficiency of the traditional color image sensor is lower, the signal intensity of each sensor pixel is limited, and the overall light energy utilization efficiency is lower.
Exemplary Dual light fusion imaging chip
In order to solve the above problems, the present invention provides a dual light fusion imaging chip, comprising:
a nano optical route 1 and an area array photoelectric detector 2;
setting the geometric structure and morphology of the nano light routing surface, namely a surface structure 3 by a nano light routing surface space optimization method;
the nano optical path 1 is integrated and bonded with the area array photoelectric detector 2 through a solid optical glue layer or a liquid optical glue layer, and the nano optical path 1 is completely overlapped with the area array photoelectric detector 2; the size of a single pixel of the nano light route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array detector array, and further points out that the double-light fusion imaging chip directly outputs the original image information containing R, G, B, IR four channels and is arranged in a space square quarter grid (2X 2 matrix) form; the original output image information can obtain four-channel light field information of each sub-pixel point through a spatial interpolation algorithm of a rear-end electric domain, an infrared image and a visible light image with the same pixel resolution are generated, and a certain double-light fusion algorithm is combined to obtain a double-light fusion image.
Embodiment two: exemplary nanooptical path design methods
The nano light routing surface in the double light fusion imaging chip in the first embodiment is a single pixel unit surface structure according to a preset form of a preparation process, an output optimization scheme is calculated through a nano light routing surface space optimization method, and a final geometric structure and morphology are determined, wherein the nano light routing surface space optimization method comprises the following steps:
setting the area array detector as a single sensing pixel element, receiving visible light and near infrared light (IR), wherein the visible light comprises: red (R), green (G), blue (B), specifically, red (R), green (G), blue (B), and near Infrared (IR) center wavelengths are set, while four sub-pixel units 4, i.e., S, having squares arranged in a 2×2 matrix are set 1 ,S 2 ,S 3 And S 4 The single pixel element on the overall composite sensing surface is marked as S; in particular, the center wavelength of IR can be set at 800 nanometers (e.g., 850 nm), 900 nanometers (e.g., 940 nm), or 1000 nanometers (e.g., 1024 nm) according to design requirements, with corresponding design freedom; RGB is set according to the normal three primary colors;
determining surface space design parameters of the nano light route according to the set sensing pixel element area and the super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical route and the substrate material and thickness are established according to the super-surface design parameters, and an optimization scheme is output, and further points out that the nano optical route has the thickness of a micron level, the thickness is one of parameters which can be optimized by an optimization algorithm, the surface structure 3 of a corresponding single pixel unit can be in various forms, such as a digital type or a continuous medium topology distribution type, and can also be formed by super-surface unit cells with different sizes, and the number and the shape can be preset according to a preparation process; after the morphology is established, determining the final geometric structure and morphology according to the designed objective function through an optimization algorithm.
Specifically, the sensing pixel element includes: four sub-pixel units 4 arranged in a square matrix correspondingly receive red light (R), green light (G), blue light (B), and near infrared light (IR). Specifically, red (R), green (G), blue (B) and near Infrared (IR) center wavelengths are set, while four sub-pixel units S1, S2, S3, and S4 arranged in a square matrix are set, and a single sensing pixel unit is collectively synthesized, denoted as S.
Specifically, the set wavelength range of the near infrared light (IR) is 800nm-1000nm, specifically, the center wavelength of the IR can be set to 800nm, 900 nm or 1000nm according to the design requirement, the design freedom degree is provided, and RGB is set according to the response wavelength of the general color CCD or CMOS photosensor array.
Specifically, the surface space design parameters of the nano light route are determined according to the set sensing pixel element area and the super-surface semiconductor material, specifically: according to the four sub-pixel units of the sensing pixel element area, the super-surface corresponds to the same plane area and is used as a design area; setting the corresponding thickness t of the super surface and the substrate material and thickness d according to the dielectric constant distribution E (x, y) of the super surface; calculating the distribution of the energy flow of the target light fields of the central wavelengths of the concerned wavebands, namely red (R), green (G), blue (B) and near Infrared (IR), which are the rear exit surfaces of the substrate material, specifically selecting as the design area of the super surface (optical metasurface) a planar area of the same size as the single pixel S (comprising four sub-pixels) of the imaging sensor array, taking into account that it is made of a high refractive index semiconductor material (such as Si, tiO 2 Or Si (or) 3 N 4 ) The dielectric constant distribution epsilon (x, y) is formed, the corresponding thickness t of the super surface and the thickness d of the substrate material are set, and a fast electromagnetic simulation solver (for example, a strict coupled wave method RCWA) is utilized to calculate the distribution of the energy flow of a target light field (the rear emergent surface of the substrate material) of the central wavelength of the concerned wave band.
Fig. 1 shows an operation of a pixel unit of the chip. Area array photoelectric detectionThe pixel unit comprises 4 sub-pixels, and can respectively and exclusively acquire R, G, B and IR band light field intensity information, so that space area array multi-channel imaging is realized. The optical film marked here, i.e. the optical routing device to be optimally designed, is an artificial micro-nano structure, or optical supersurface, prepared with a dielectric (e.g. SiO 2 ) On the substrate, light of R, G, B and IR four wavebands is routed to each of the sub-pixels arranged in a 2×2 arrangement, respectively, to achieve that an imaging detector having four sub-pixel elements receives incident light of four different wavebands, respectively. Overall, energy efficiency is more advantageous than the form of direct filtering.
The overall optimization efficiency is first set by the following equation:
wherein alpha is B ，α G ，α R ，α IR As a weight factor, can be adjusted according to specific design; lambda (lambda) 1 ，λ 2 —λ 8 The minimum wavelength and the maximum wavelength of four corresponding wave bands are respectively provided, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light. P (P) z (lambda) is the longitudinal light field wave-printing vector on the single pixel face of the sensor, P zB (lambda) is the wave-printing vector of blue light wave band passing through B corresponding sub-pixel, P zG (lambda) is the wave-printing vector of the green light wave band passing through the sub-pixel corresponding to G, P zR (lambda) is the PointTing vector of the red light band through the R-corresponding sub-pixel, P zIR (lambda) is the wave-printing vector of the infrared band through the IR corresponding sub-pixel; t (T) B (lambda) is the transmittance of the blue light incident surface to the pixel surface, T G (lambda) is the transmittance of the green light incident surface to the pixel surface, T R (lambda) is the transmittance of the red light incident surface to the pixel surface, T IR And (lambda) is the transmission coefficient from the infrared light incident surface to the pixel surface.
Optimizing the dielectric constant distribution E (x, y) of the super surface until the light energy distribution target model reaches convergence and extremum, and specifically comprises the following steps:
optimization algorithm one:
when optimizing free form super surface, first setting initial value of E (x, y), where E (x, y) is the interval [ E [ min ,∈ max ]The method comprises the steps of carrying out a first treatment on the surface of the Optimizing by adopting an optimization iterative algorithm (such as a moving asymptote method) and a binarization processing algorithm, and further pointing out that when the epsilon (x, y) is a topological optimization problem of a continuous situation, combining the epsilon (x, y) with an optimization iterative process, such as a moving asymptote MMA (method of moving asymptotes) algorithm; as shown in fig. 2, the design parameter space e (x, y) is iterated continuously to achieve convergence of the objective function L, and based on the design objective function defined by the formula (1), the design parameter space e (x, y) is not applied with a preset value, but is enabled to have continuous distribution characteristics in a two-dimensional space, and the adopted scheme can be a topology optimization method based on density. In this example, si is used 3 N 4 The material, one unit geometry orientation obtained by topology optimization and MMA design is shown in fig. 3:
FIG. 3 (a) shows the overall morphology of the super surface structure, with a size of 2um×2um.
FIG. 3 (b) is a 3D perspective design thereof, si 3 N 4 The layers have the same height t=600 nm, the base SiO 2 Thickness d=6 um. The spectral efficiency of each central wave band (the wavelengths are set to 450nm,550nm,650nm and 850 nm) of the spectral routing device are shown in figure 4, and the average light energy utilization efficiency is higher than 35%, which is superior to that of a filter array detector under ideal conditions;
the light splitting effect of the light splitting routing device in fig. 3 is shown in fig. 5, where it can be clearly seen that most of the energy of the near infrared light, the blue light, the green light and the red light is diffracted by the light splitting routing to the IR, B, G, R detection unit sub-pixels corresponding to the upper left, the upper right, the lower right and the lower left regions respectively. In this design, weak sub-pixel light field crosstalk is also unavoidable.
And (3) optimizing algorithm II:
when optimizing the digital form super surface, dividing the pixels of the design area into n units, and taking the value of each unit∈ min Or E shaped max The problem is changed into a binary arrangement search, and the genetic algorithm GA (generic algorithm) or the particle swarm optimization algorithm PSO (particle swarm optimization) is adopted for optimization.
In the two optimization algorithms, the combined micro-nano optical electromagnetic solver does not need to be specified, and main electromagnetic wave equation solvers such as a strict coupled wave method (RCWA), a time domain finite differential method (FDTD) or a Finite Element Method (FEM) can be adopted.
After the optimization scheme is output, the size of a single pixel of the meter light route is corresponding to that of a pixel of a wide-spectrum CCD or CMOS area array detector array, the corresponding positions also need to be completely corresponding, and the distance d between the two is generally in the micron order and can be determined as the thickness of a substrate material for designing the meter light route; when the nano optical router is integrated with the area array detector with wide spectral response, an optical adhesive (OCA) or liquid optical adhesive (LOCA) process is adopted for adhesion and fixation.
Embodiment III:
after the optimization of the second embodiment, the dual-light fusion imaging chip in the first embodiment directly outputs the original image information containing R, G, B, IR four channels, arranges the original output image information in a form of a space square quarter grid (2×2 matrix), and uses a spatial interpolation algorithm of a rear end electric field to obtain four-channel light field information of each sub-pixel point, so that an infrared image and a visible light image with the same pixel resolution are generated, after high-efficiency pixel unit image sensing data are obtained, further development of data processing and dual-light image output of a rear end is required, and the overall flow is as shown in fig. 6, after the visible light image and the infrared light image with the same pixel resolution are obtained, the image fusion algorithm adopts the dual-light fusion algorithm which is mature and easy to use at present.
In general, bilinear interpolation is performed as shown in fig. 7, pixel points (x, y) to be interpolated are treated in a 2×2 square matrix having detected indication values, assuming that the spatial positions of four infrared pixel points are (x 1 ,y 1 )，(x 1 ,y 2 )，(x 2 ,y 1 )，(x 2 ,y 2 ) The pixel value of the point to be solved is P (x, y) Interpolation in the horizontal direction can obtain a temporary pixel value R in the horizontal direction 1 And R is as follows 2
Then interpolation is carried out in the vertical direction, and the pixel value to be solved can be obtained as follows:
preferably, the IR sensing data in each pixel cell can be interpolated using bilinear linear interpolation for pixel points (x, y) to be interpolated in a 4 x 4 square matrix already having a detected indication value, as calculated by the following formula,
wherein, W is the weight of the pixel to be interpolated, and the specific expression is as follows:
taking a=0.5 in general, finally obtaining the infrared light image P output according to the spatial interpolation scheme IR 。
For visible RGB images, as shown in fig. 8, similarly, in a 3 x 3 square array already having G detection indication values, the average value is directly taken for 4 diagonal G pixel values, instead of the IR image value of G1. So far, the Bayer type RGB image with RGGB arrangement form is obtained, and the visible light image P can be obtained by adopting the Bayer type image back-end processing algorithm c 。
Exemplary computer-readable storage Medium
The computer readable storage medium stores a computer program which, when executed by a processor, implements the dual light image fusion processing method described above.
The above-mentioned computer readable storage medium can implement the steps of the nano optical path design method in any of the above-mentioned embodiments, and can achieve the same technical effects, so that the method has all the advantages of any of the above-mentioned embodiments, and is not described herein.
The computer readable storage medium in the present embodiment is, for example, a Read-Only Memory (ROM), a random access Memory (Rand om AccessMemory, RAM), a magnetic disk, an optical disk, or the like.
In the description of the present invention, it should be noted that the azimuth or positional relationship indicated by the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are based on the azimuth or positional relationship shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific azimuth, be constructed and operated in a specific azimuth, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," "third," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (4)
1. A dual light fusion imaging chip, comprising:
nano light route and area array photoelectric detector;
setting the geometric structure and morphology of the nano light routing surface by a nano light routing surface space optimization method;
the nano optical route is integrated and adhered with the area array photoelectric detector through an optical adhesive layer, and the nano optical route is completely overlapped with the area array photoelectric detector; the size of a single pixel of the nano light route is correspondingly consistent with that of a pixel of a wide-spectrum CCD or CMOS area array detector array;
the nano light routing surface space optimization method comprises the following steps:
setting the area array detector as a single sensing pixel element, receiving visible light and near infrared light (IR), wherein the visible light comprises: red light (R), green light (G), blue light (B);
determining surface space design parameters of the nano light route according to the set sensing pixel element area and the super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical route and the substrate material and thickness;
establishing a super-surface design parameter optimization model according to the nano-optical routing surface space design parameters, and outputting an optimization scheme;
the sensing pixel element comprises:
four sub-pixel units arranged in a square matrix, which correspondingly receive red light (R), green light (G), blue light (B) and near infrared light (IR);
the set wavelength range of the near infrared light (IR) is 800nm-1000nm;
the surface space design parameters of the nano light route are determined according to the set sensing pixel element area and the super-surface semiconductor material, and specifically are as follows:
according to the four sub-pixel units of the sensing pixel element area, the super-surface corresponds to the same plane area and is used as a design area;
setting the corresponding thickness t of the super surface and the substrate material and thickness d according to the dielectric constant distribution E (x, y) of the super surface;
calculating target light field energy flows of red light (R), green light (G), blue light (B) and near infrared light (IR) central wavelengthsDistribution of (3)The target light field is positioned on the rear emergent surface of the substrate material;
the method comprises the steps of establishing a super-surface design parameter optimization model according to the nano-light path routing surface space design parameters, and outputting an optimization scheme, and specifically comprises the following steps:
establishing a light energy distribution target model, as shown in a formula (1):
wherein alpha is B ，α G ，α R ，α IR As a weight factor, can be adjusted according to specific design; lambda (lambda) 1 ，λ 2 —λ 8 The minimum wavelength and the maximum wavelength of four corresponding wave bands are respectively provided, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light; p (P) z (lambda) is the longitudinal light field wave-printing vector on the single pixel face of the sensor, P zB (lambda) is the wave-printing vector of blue light wave band passing through B corresponding sub-pixel, P zG (lambda) is the wave-printing vector of the green light wave band passing through the sub-pixel corresponding to G, P zR (lambda) is the PointTing vector of the red light band through the R-corresponding sub-pixel, P zIR (lambda) is the wave-printing vector of the infrared band through the IR corresponding sub-pixel; t (T) B (lambda) is the transmittance of the blue light incident surface to the pixel surface, T G (lambda) is the transmittance of the green light incident surface to the pixel surface, T R (lambda) is the transmittance of the red light incident surface to the pixel surface, T IR (lambda) from the infrared light incident surface toA transmission coefficient of the pixel surface;
and optimizing the dielectric constant distribution E (x, y) of the super surface based on the light energy distribution target model until the light energy distribution target model reaches convergence and extremum.
2. The dual-optical fusion imaging chip of claim 1, wherein the optimizing the dielectric constant distribution e (x, y) of the supersurface based on the optical energy distribution target model until the optical energy distribution target model reaches convergence and extremum comprises:
when optimizing free form super surface, first setting initial value of E (x, y), where E (x, y) is the interval [ E [ min ,∈ max ]The method comprises the steps of carrying out a first treatment on the surface of the Optimizing by adopting an optimization iterative algorithm and a binarization processing algorithm;
when optimizing the digital form super surface, dividing the pixels of the design area into n units, and taking the value E of each unit min Or E shaped max The method comprises the steps of carrying out a first treatment on the surface of the And optimizing by adopting a genetic algorithm GA or a particle swarm optimization algorithm PSO.
3. A dual-light image fusion processing method suitable for a dual-light fusion imaging chip according to any one of claims 1-2, comprising the following steps:
using the indication values of the IR sub-pixel units in the four adjacent pixel units to perform space two-dimensional spatial interpolation to obtain corresponding IR responses at the 4 sub-pixel positions in each pixel unit, and recording as an image P IR ；
The IR sub-pixel in 4 sub-pixels in each pixel unit is replaced by the G sub-pixel in the adjacent pixel unit, which is marked as G1, and then forms RGGB type Bayer pixel arrangement together with R, G and B in the original pixel unit for subsequent generation of RGB color image, which is marked as P c ；
Will P IR And P c And regarding the infrared light image and the visible light image which are required in the double-light fusion algorithm, processing by using the image fusion algorithm, and developing feature extraction, fusion strategy and image reconstruction to generate a double-light fusion image.
4. A computer-readable storage medium comprising,
the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the dual light image fusion processing method according to claim 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210961938.4A CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210961938.4A CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115174794A CN115174794A (en) | 2022-10-11 |
| CN115174794B true CN115174794B (en) | 2023-09-22 |
Family
ID=83479474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210961938.4A Active CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115174794B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965654A (en) * | 2018-02-11 | 2018-12-07 | 浙江宇视科技有限公司 | Double spectrum camera systems and image processing method based on single-sensor |
| CN112013956A (en) * | 2019-11-11 | 2020-12-01 | 中国科学院上海技术物理研究所 | Infrared color detector based on super surface |
| CN113299671A (en) * | 2021-03-11 | 2021-08-24 | 中国科学院上海技术物理研究所 | Infrared color focal plane detector of in-situ integrated super-surface phased array |
| CN113686437A (en) * | 2021-07-09 | 2021-11-23 | 湖南大学 | Time domain spectral imaging chip based on super-structure surface |
| CN114830341A (en) * | 2019-12-31 | 2022-07-29 | 华为技术有限公司 | Image sensor, light splitting and color filtering device and preparation method of image sensor |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210375960A1 (en) * | 2020-05-28 | 2021-12-02 | University Of Southern California | Design of Metasurfaces for High-Efficient IR Photodetectors |
-
2022
- 2022-08-11 CN CN202210961938.4A patent/CN115174794B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965654A (en) * | 2018-02-11 | 2018-12-07 | 浙江宇视科技有限公司 | Double spectrum camera systems and image processing method based on single-sensor |
| CN112013956A (en) * | 2019-11-11 | 2020-12-01 | 中国科学院上海技术物理研究所 | Infrared color detector based on super surface |
| CN114830341A (en) * | 2019-12-31 | 2022-07-29 | 华为技术有限公司 | Image sensor, light splitting and color filtering device and preparation method of image sensor |
| CN113299671A (en) * | 2021-03-11 | 2021-08-24 | 中国科学院上海技术物理研究所 | Infrared color focal plane detector of in-situ integrated super-surface phased array |
| CN113686437A (en) * | 2021-07-09 | 2021-11-23 | 湖南大学 | Time domain spectral imaging chip based on super-structure surface |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115174794A (en) | 2022-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107105141B (en) | Imaging sensor, image processing method, imaging device and mobile terminal | |
| CN101124462B (en) | A system for multi- and hyperspectral imaging | |
| CN101345248B (en) | Multi-optical spectrum light-sensitive device and preparation thereof | |
| CN104335246B (en) | The camera model of pattern is formed with pi optical filters group | |
| TWI535292B (en) | Capturing and processing of images using monolithic camera array with heterogeneous imagers | |
| CN106165411B (en) | Optical detection device and solid camera head | |
| CN102483323B (en) | Determination of optimal diameters for nanowires | |
| TW201925860A (en) | Light field image processing method for depth acquisition | |
| CN106454161A (en) | Method for improving image quality of image sensor | |
| CN102823230B (en) | Camera head | |
| Chen et al. | Full-color nanorouter for high-resolution imaging | |
| JP7462760B2 (en) | Image sensor, method for producing same, and electronic device | |
| TW201704784A (en) | Color filter array patterns for reduction of color aliasing | |
| WO2021136469A1 (en) | Image sensor, light splitting color filter device, and image sensor fabrication method | |
| CN104412586A (en) | Image processing device, imaging device, and image processing program | |
| CN108495115A (en) | Imaging sensor and its pixel groups and pel array, the method for obtaining image information | |
| CN115174794B (en) | Double-light fusion imaging chip and double-light picture fusion processing method | |
| WO2022020989A1 (en) | Filtering array, mobile terminal, and device | |
| CN104754321B (en) | Camera array system | |
| CN105206638B (en) | A kind of back-illuminated cmos image sensors and forming method thereof | |
| CN115950534A (en) | Focus-dividing plane type high-luminous-flux bionic multispectral polarization imaging detector | |
| CN104303302A (en) | Imaging element and imaging device | |
| WO2022016412A1 (en) | Depth information image acquisition apparatus and electronic device | |
| EP3232473B1 (en) | Masked pixel arrays | |
| CN103513384B (en) | A kind of method improving photon detection efficiency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |