CN115174794A - Double-light fusion imaging chip and double-light image fusion processing method - Google Patents
Double-light fusion imaging chip and double-light image fusion processing method Download PDFInfo
- Publication number
- CN115174794A CN115174794A CN202210961938.4A CN202210961938A CN115174794A CN 115174794 A CN115174794 A CN 115174794A CN 202210961938 A CN202210961938 A CN 202210961938A CN 115174794 A CN115174794 A CN 115174794A
- Authority
- CN
- China
- Prior art keywords
- light
- pixel
- optical
- nano
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Solid State Image Pick-Up Elements (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
The invention provides a double-light fusion imaging chip and a double-light picture fusion processing method, and relates to the field of optical sensing chips and image processing. The chip includes: nano optical routing and area array photodetectors; setting the geometric structure and the appearance of the surface of the nano optical path by a nano optical path surface space optimization method; the nano optical route is integrally bonded with the area array photoelectric detector through an optical adhesive layer or a liquid optical adhesive layer, and the nano optical route and the area array photoelectric detector are completely overlapped; the size of a single pixel of the nano optical route is correspondingly consistent with that of a pixel of a wide-spectrum CCD or CMOS area array type detector array. The light energy utilization efficiency of the double-light fusion imaging chip based on the nano light route is remarkably improved, meanwhile, the double-light fusion imaging chip comprehensively utilizes light field information of four channels including red, green, blue and near infrared, and the double-light fusion imaging chip is more compact, simple, convenient and convenient in rear-end double-light fusion image processing algorithm and output.
Description
Technical Field
The invention relates to the field of optical sensing chips and image processing, in particular to a double-light fusion imaging chip and a double-light image fusion processing method.
Background
The existing multi-channel (color) optical array sensor chip mainly depends on a Bayer (Bayer) type filter plate to respectively distribute light energy of different spectral bands to sub-pixels in a corresponding space, for example, the RGGB form is adopted, namely each complete pixel formed by four sub-pixels equivalently comprises half of G,1/4 of R and 1/4 of B; or in order to improve the light sensing capability, the RYYB space layout is adopted, which is more beneficial to noise control but can influence color information. Regardless of the layout, each sub-pixel only includes a portion of the spectrum, and a full-space color image output needs to be achieved in conjunction with a back-end Bayer interpolation algorithm. The absorption filter realizes spatial correspondence by light absorption outside a passband, and the effective area of the light energy is directly changed into the corresponding area of the sub-pixels, so that the overall light transmission and utilization efficiency of the traditional color image sensor is lower, the signal intensity of each sensor pixel is limited, and the overall light energy utilization efficiency is lower.
The nano light splitting routing technology is a micro-nano optical passive device newly developed in recent years, is a colorized image sensing technology based on structural colors, has very large freedom degree on color operation based on micro-nano optical structure design, and a nano structured optical surface generally has very strong dispersion, can be used for generating the required structural colors and realizing light splitting, does not reduce the effective utilization area of light energy, and can greatly improve the utilization efficiency of the light energy.
On the other hand, the visible light image generally has high spatial resolution and considerable detail and contrast, but is easily affected by illumination and environment. Infrared light images can be imaged around the clock, but lack texture information and image detail. The visible light image and the infrared light image have strong complementarity, and the combination of the visible light image and the infrared light image can generate a robust and rich-information image. The visible-infrared double-light fusion is widely applied in the fields of low-light-level imaging, infrared thermal imaging, security protection, automatic driving and the like, but complex image registration and image fusion algorithms need to be developed, so that greater computational demand pressure is brought to back-end processing, and great challenges exist in the aspects of algorithm design and calibration and special chip development.
Aiming at the existing problems, the invention provides a high-efficiency artificial super-surface structure (nano optical route) designed by combining a micro-nano optical principle and an optimization algorithm, and the high-efficiency artificial super-surface structure can obtain image surface light sensation information multiplexed by a plurality of color channels under the condition of basically not losing optical energy; the nano-optical route is integrated with a wide-spectrum (covering RGB three primary colors wave band and near infrared wave band) area array photoelectric detector, so that a common-path high-efficiency visible-infrared double-light fusion imaging chip is realized.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the problems of low light energy utilization efficiency of the existing multi-channel photoelectric detection chip, complex system, large volume, large algorithm development and calculation amount and the like generally existing in the technical scheme of visible-near infrared double-light fusion, so that a solution for spatial alignment integration of an RGB-IR four-channel nano light route and a wide spectrum detector is provided, light components of different wave bands of a scene are split and guided to corresponding four sub-pixel photosensitive positions, red, green, blue and infrared light scene information of spatial sampling is acquired, four-color spectrum images of all sub-pixel points are extracted through a simple back-end algorithm, and the simple and convenient visible-near infrared double-light fusion imaging chip is realized.
In order to solve the above problems, the present invention provides a dual light fusion imaging chip, comprising:
nano optical routing and area array photodetectors;
setting the geometric structure and the appearance of the surface of the nano optical path by a nano optical path surface space optimization method;
the nano optical route is integrally bonded with the area array photoelectric detector through an optical adhesive layer or a liquid optical adhesive layer, and the nano optical route and the area array photoelectric detector are completely overlapped; the size of a single pixel of the nano optical route is correspondingly consistent with that of a pixel of a wide-spectrum CCD or CMOS area array type detector array.
Preferably, the method for optimizing the nano optical routing surface space comprises the following steps:
the area array type detector is set as a single sensing pixel element, receives visible light and near infrared light (IR), and the visible light comprises: red (R), green (G), blue (B);
determining surface space design parameters of a nano optical route according to a set sensing pixel element region and a super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical path and the base material and thickness;
and establishing an optimization model according to the super-surface design parameters, and outputting an optimization scheme.
Preferably, the sensing pixel element includes:
four sub-pixel units arranged in a square matrix correspondingly receive red light (R), green light (G), blue light (B) and near infrared light (IR).
Preferably, the set wavelength range of the near infrared light (IR) is 800nm to 1000nm.
Preferably, the determining of the surface space design parameter of the nano optical route according to the set sensing pixel element region and the super-surface semiconductor material specifically includes:
according to the four sub-pixel units in the sensing pixel element area, the super surface corresponds to the same plane area and serves as a design area;
setting the corresponding thickness t of the super surface, the base material and the thickness d according to the dielectric constant distribution epsilon (x, y) of the super surface;
and calculating the distribution of the energy flow of the target light field with the central wavelength of red light (R), green light (G), blue light (B) and near infrared light (IR) along the vector vertical direction, wherein the target light field is the rear emergent surface of the substrate material.
Preferably, the establishing a super-surface design parameter optimization model according to the nano-optical routing surface space design parameter, and outputting an optimization scheme specifically includes:
establishing a light energy distribution target model, as formula (1):
wherein alpha is B ，α G ，α R ，α IR The weighting factor can be adjusted according to specific design; lambda 1 ，λ 2 -λ 8 The minimum wavelength and the maximum wavelength of the four corresponding wave bands are respectively, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light; p z (λ) is the longitudinal light field Pottingvector, P, on a single pixel plane of the sensor zB (λ) is the Boynting vector of the blue band of light through the B corresponding sub-pixel, P zG (λ) is the Boynting vector of the green band of light through the corresponding sub-pixel of G, P zR (λ) is the Boynting vector of the red band of light through the R corresponding sub-pixel, P zIR (λ) is the poynting vector of the infrared band through the IR corresponding sub-pixel; t is B (λ) is a transmission coefficient from the blue light incident surface to the pixel surface, T G (λ) is a transmission coefficient from the green light incident surface to the pixel surface, T R (λ) is a transmission coefficient from the red light incident surface to the pixel surface, T IR (λ) is a transmission coefficient from the infrared light incident surface to the pixel surface;
and optimizing the super-surface design parameter space epsilon (x, y) based on the optical energy distribution target model until the optical energy distribution target function reaches convergence and an extremum.
Preferably, the optimizing the super-surface design parameter space e (x, y) based on the optical energy distribution target model until the optical energy distribution target function reaches convergence and an extremum includes:
when the optimization of the free form super surface is realized, firstly, the initial value of the element (x, y) is set, and the value of the element (x, y) is the interval [ element [ min ，∈ max ](ii) a Optimizing by adopting an optimization iterative algorithm (such as a moving asymptote method) and a binarization processing algorithm;
or
When the digital form super surface is optimized, the pixel of the design area is converted into nEach unit is belonged to min Or e max (ii) a Optimization is carried out by using genetic algorithm GA (genetic algorithm) or particle swarm optimization PSO (particle swarm optimization).
The invention also provides a double-light image fusion processing method suitable for the double-light fusion imaging chip, which specifically comprises the following steps:
utilizing the indicating value of the IR sub-pixel unit in the adjacent four pixel units to carry out spatial two-dimensional spatial interpolation, obtaining the corresponding IR response at the position of 4 sub-pixels in each pixel unit, and recording the IR response as an image P IR ；
Replacing IR sub-pixels in 4 sub-pixels in each pixel unit with G sub-pixels in adjacent pixel units, marking as G1, and forming an RGGB type Bayer pixel arrangement together with R, G and B in the original pixel unit for subsequently generating RGB color images, marking as P c ；
Will P IR And P c And taking the infrared light image and the visible light image as required in the double-light fusion algorithm, processing by using the image fusion algorithm, developing feature extraction, fusion strategy and image reconstruction, and generating the double-light fusion image.
The invention also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the nano optical path design method.
The invention has the following advantages: the image surface light sensation information multiplexed by a plurality of color channels can be obtained under the condition of basically not losing light energy; the nano-optical route is integrated with a wide-spectrum (covering RGB three primary colors wave band and near infrared wave band) area array photoelectric detector, so that a common-path high-efficiency visible-infrared double-optical fusion imaging chip is realized, and the specific effects are as follows:
the size of a single pixel of the first nano optical route is correspondingly consistent with that of a pixel of a wide-spectrum CCD or CMOS area array type detector array, corresponding positions are required to be completely corresponding, the distance d between the two is generally in the micrometer level, and the distance d can be determined as the thickness of a substrate material for designing the nano optical route; when the nano optical route is integrated with the area array detector with the wide-spectrum response, an Optical Cement (OCA) or Liquid Optical Cement (LOCA) process is adopted for bonding and fixing;
the second and the fourth channel nanometer optical routes have the advantage of high light energy utilization efficiency, so that the sensitivity and the low-light detection capability of the sensor can be greatly improved, the natural advantage of double-light fusion of a common light path is simultaneously achieved, images after double-light fusion can be very conveniently output through a spatial interpolation algorithm similar to Bayer filtering, the application range of the double-light fusion of the existing binocular visual images based on a visible light camera and an infrared camera can be greatly expanded, and a great deal of convenience is brought, such as miniaturization and reduction of double-light fusion calculation complexity and resource requirements;
thirdly, the nano optical route is integrally bonded with the area array detector through an optical adhesive layer or a liquid optical adhesive layer, and the nano optical route and the area array detector are completely overlapped; the size of a single pixel of the nano optical route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array type detector array, full-wave simulation verification is carried out, the real light field spectral characteristic is obtained through calculation, and then a sample is prepared according to photoetching or EBL technology; carrying out space contraposition and gluing integration on the sample and a pre-corresponding imaging detection array;
and fourthly, directly outputting original image information containing R, G, B, IR four channels, periodically arranging the original image information according to a spatial square quartering lattice form to form array original data, and obtaining a final fusion image by utilizing a rear-end data processing method.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic diagram of an operating principle of an RGB-IR four-channel nano-optical splitting route according to an embodiment of the present invention;
FIG. 2 is a basic flow of a design of a density-based topology optimization for nano-optical routing;
fig. 3 is a RGB-IR four-channel nano light-splitting routing microstructure design according to an embodiment of the present invention, in which a white area is a medium and a black area is air, and the structure is disposed on a substrate;
FIG. 4 is a spectral efficiency ratio obtained based on the design of FIG. 2 according to an embodiment of the present invention;
fig. 5 is a light field simulation result corresponding to a specific design of the RGB-IR based four-channel nano-scale light splitting router on the imaging plane according to the embodiment of the present invention;
fig. 6 is a schematic flowchart of a back-end dual-light fusion algorithm according to an embodiment of the present invention;
FIG. 7 is a method for obtaining IR images at all sub-pixel locations according to an embodiment of the present invention;
fig. 8 is a RGB Bayer bilinear spatial interpolation method according to an embodiment of the present invention;
fig. 9 is a flowchart illustrating steps of a method for optimizing a surface space of a nano optical path according to an embodiment of the present invention.
Reference numerals:
1. a nano-optical route; 2. an area array photodetector; 3. a surface structure; 4. sub-pixel unit
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it is to be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Summary of the application
The absorption filter realizes spatial correspondence through light absorption outside the passband, and the effective area of the light energy is directly changed into the corresponding area of the sub-pixels, so that the overall light transmission and utilization efficiency of the traditional color image sensor is lower, the signal intensity of each sensor pixel is limited, and the overall light energy utilization efficiency is lower.
Exemplary Dual light fusion imaging chip
In order to solve the above problems, the present invention provides a dual light fusion imaging chip, comprising:
a nano optical path 1 and an area array photoelectric detector 2;
setting the geometric structure and the appearance of the surface of the nano optical path, namely a surface structure 3, by a nano optical path surface space optimization method;
the nano optical path 1 is integrally bonded with the area array photoelectric detector 2 through an optical adhesive layer or a liquid optical adhesive layer, and the nano optical path 1 is completely overlapped with the area array photoelectric detector 2; the size of a single pixel of the nano optical route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array type detector array, and it needs to be further pointed out that the double-optical fusion imaging chip directly outputs original image information containing R, G, B, IR four channels and is arranged in a spatial square quartering grid (2 multiplied by 2 matrix) form; the original output image information can obtain the light field information of four channels of each sub-pixel point through a space interpolation algorithm of a rear-end electric domain, an infrared image and a visible light image with the same pixel resolution are generated, and a double-light fusion image can be obtained by combining a certain double-light fusion algorithm.
Example two: exemplary Nanometric light Path design methods
In the nano optical routing surface in the two-optical fusion imaging chip in the first embodiment, a single pixel unit surface structure is in a preset shape according to a preparation process, an output optimization scheme is calculated by a nano optical routing surface space optimization method, and a final geometric structure and morphology are determined, wherein the nano optical routing surface space optimization method comprises the following steps:
the area array type detector is set as a single sensing pixel element, receives visible light and near infrared light (IR), and the visible light comprises: red (R), green (G), and blue (B), and specifically, red (R), green (G), and blue (B) light and near Infrared (IR) central wavelengths are set, and four square sub-pixel units 4 arranged in a 2 × 2 matrix, that is, S, are set 1 ,S 2 ,S 3 And S 4 On the general composite sensing surfaceA single pixel element, denoted as S; in particular, the center wavelength of the IR can be set at 800nm (e.g., 850 nm), 900 nm (e.g., 940 nm), or 1000nm (e.g., 1024 nm) according to design requirements, with corresponding design freedom; RGB is set according to the normal three primary color range;
determining surface space design parameters of a nano optical route according to a set sensing pixel element region and a super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical path, the substrate material and the thickness are used for establishing an optimization model according to the super-surface design parameters and outputting an optimization scheme, and further points out that the nano optical path has the micrometer-scale thickness which is also one of the parameters that can be optimized by an optimization algorithm, and the surface structure 3 corresponding to a single pixel unit can be in various forms, such as a digital form or a continuous medium topological distribution form, or can be formed by super-surface unit cells with different sizes, and the number and the appearance of the super-surface unit cells can be preset according to a preparation process; and determining the final geometric structure and morphology according to the designed objective function through an optimization algorithm after the morphology is established.
Specifically, the sensing pixel element includes: the four sub-pixel units 4 arranged in a square matrix correspondingly receive red light (R), green light (G), blue light (B) and near infrared light (IR). Specifically, red (R), green (G), blue (B) and near Infrared (IR) center wavelengths are set, and four sub-pixel units S arranged in a square matrix are set at the same time 1 ,S 2 ,S 3 And S 4 The individual sensor pixel elements are collectively combined, denoted as S.
Specifically, the set wavelength range of the near infrared light (IR) is 800nm to 1000nm, specifically, the center wavelength of the IR can be set to 800nm or more, 900 nm or 1000nm according to the design requirement, and has corresponding design freedom, and RGB is set according to the response wavelength of a general color CCD or CMOS photosensor array.
Specifically, the determining of the surface space design parameter of the nano optical route according to the set sensing pixel element region and the super-surface semiconductor material specifically includes: according to the four sub-pixel units of the sensing pixel element region, the super surface corresponds to the correspondingThe coplanar area is used as a design area; setting the corresponding thickness t of the super surface, the base material and the thickness d according to the dielectric constant distribution epsilon (x, y) of the super surface; calculating the distribution of energy flow of target light fields along the vector vertical direction of the central wavelength of the interested wave bands, namely red light (R), green light (G), blue light (B) and near infrared light (IR), wherein the target light fields are rear emergent surfaces of a substrate material, specifically, selecting a plane area with the same size as a single pixel S (comprising four sub-pixels) of an imaging sensor array as a super surface (optical measurement) design area, and considering that the plane area is made of a high-refractive-index semiconductor material (such as Si and TiO) 2 Or Si 3 N 4 ) The formed dielectric constant distribution epsilon (x, y), the corresponding super-surface thickness t and the base material and thickness d are set, and a rapid electromagnetic simulation solver (such as a strict coupled wave method RCWA) is used for calculating the distribution of the energy flow of a target optical field (rear emergent surface of the base material) of the central wavelength of the concerned waveband along the vector vertical direction.
Fig. 1 shows the operation of one pixel unit of the chip. One pixel unit of the area array photoelectric detector comprises 4 sub-pixels, and the area array photoelectric detector can respectively and exclusively acquire R, G, B and light field intensity information of an IR wave band, so that space area array multichannel imaging is realized. The marked optical film is a light splitting routing device to be optimally designed, is an artificial micro-nano structure or optical super-surface, and is prepared on a dielectric material (such as SiO) with the thickness of d 2 ) On the substrate, R, G, B and the light of the four IR wave bands are respectively routed to the sub-pixels arranged according to 2 x 2, so that an imaging detector with four sub-pixel units respectively receives incident light of four different wave bands. Overall, the energy utilization efficiency is more favorable than the form of direct filtering.
The overall optimization efficiency is first set by the following equation:
wherein alpha is B ，α G ，α R ，α IR As a weighting factor, canTo adjust for specific designs; lambda 1 ，λ 2 -λ 8 The minimum wavelength and the maximum wavelength of the four corresponding bands are respectively, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light. P is z (λ) is the longitudinal light field pointining vector, P, on a single pixel plane of the sensor zB (λ) is the Boynting vector of the blue band of light through the B corresponding sub-pixel, P zG (λ) is the Boynting vector of the green band of light through the corresponding sub-pixel of G, P zR (λ) is the Boynting vector of the red band of light through the R corresponding sub-pixel, P zIR (λ) is the poynting vector of the infrared band through the IR corresponding sub-pixel; t is B (λ) is a transmission coefficient from the blue light incident surface to the pixel surface, T G (λ) is a transmission coefficient from the green light incident surface to the pixel surface, T R (λ) is a transmission coefficient from the red light incident surface to the pixel surface, T IR (λ) is a transmission coefficient from the infrared light incident surface to the pixel surface.
Optimizing the super-surface design parameter space epsilon (x, y) until the optical energy distribution objective function reaches convergence and an extreme value, specifically comprising:
the first optimization algorithm comprises the following steps:
when the optimization of the free form super surface is realized, firstly, the initial value of the element (x, y) is set, and the value of the element (x, y) is the interval [ element [ mmin ，∈ max ](ii) a Optimizing by adopting an optimization iteration algorithm (such as a moving asymptote method) and a binarization processing algorithm, and further pointing out that when the element (x, y) is a topology optimization problem of a continuous situation, the optimization iteration algorithm is combined with an optimization iteration process, such as a moving asymptote method MMA (method of moving asymptes) algorithm; as shown in fig. 2, the design parameter space e (x, y) is iterated continuously to converge the objective function L, and based on the design objective function defined by formula (1), the design parameter space e (x, y) is not preset but has a continuous distribution characteristic in a two-dimensional space, and the adopted scheme may be a density-based topology optimization method. In this case, si is used 3 N 4 Material, one cell geometry orientation obtained by topology optimization and MMA design is shown in fig. 3:
FIG. 3 (a) shows the overall morphology of the super-surface structure, with the size of 2um.
FIG. 3 (b) is a 3D perspective view thereof, si 3 N 4 Layer with same height t =600nm, substrate SiO 2 Thickness d =6um. The light splitting routing device is verified through full-wave electromagnetic calculation simulation, the light splitting efficiency of each central wave band (the wavelength is set to be 450nm,550nm,650nm and 850 nm) is shown in figure 4, the average light energy utilization efficiency is higher than 35%, and the light splitting routing device is superior to a filter type array detector under an ideal condition;
the light splitting effect of the light splitting routing device in fig. 3 is shown in fig. 5, and it can be clearly seen that most of the energy of the near infrared, blue light, green light, and red light is diffracted to the upper left, upper right, and lower right regions by the light splitting routing respectively, and the sub-pixels of the IR, B, G, and R detection units corresponding to the lower left region are respectively diffracted to the upper left, upper right, and lower right regions. In this design, there is also inevitably weak sub-pixel light field crosstalk.
And (3) optimizing an algorithm II:
when the digital form super surface is optimized, the pixels of the design area are divided into n units, and each unit belongs to the epsilon min Or e max The problem is changed into binary permutation search, and genetic algorithm GA (genetic algorithm) or particle swarm optimization algorithm PS0 (particle swarm optimization) is adopted for optimization.
In the two optimization algorithms, the combined micro-nano optical electromagnetic solver is not required to be specified, and a main electromagnetic wave equation solver such as a strict coupled wave method (RCWA), a time domain finite differentiation method (FDTD), a Finite Element Method (FEM) and the like can be adopted.
After the optimization scheme is output, the size of a single pixel of the nano optical route is correspondingly consistent with that of a wide-spectrum CCD or CMOS area array type detector array, the corresponding position of the single pixel of the nano optical route is required to be completely corresponding, the distance d between the two is generally in the micrometer level, and the distance d can be determined as the thickness of a substrate material for designing the nano optical route; when the nano optical route is integrated with the area array detector with wide-spectrum response, the nano optical route is bonded and fixed by adopting an Optical Cement (OCA) or Liquid Optical Cement (LOCA) process.
Example three:
the two-photon fusion imaging chip in the first embodiment directly outputs original image information containing R, G, B, IR four channels after the optimization of the second embodiment, and arranges the original output image information in the form of a spatial square quartering lattice (2 × 2 matrix), and the four-channel light field information of each sub-pixel point can be obtained through a spatial interpolation algorithm of a rear-end electric domain, so as to generate an infrared image and a visible light image with the same pixel resolution, and after high-efficiency pixel unit image sensing data is obtained, further data processing and two-photon image output at the rear end need to be performed, the overall flow is shown in fig. 6, and after the visible light image and the infrared light image with the same pixel resolution are obtained, the image fusion algorithm adopts a mature and easy-to-use two-photon fusion algorithm at present.
Generally, as shown in fig. 7, bilinear interpolation is performed, and pixel points (x, y) to be interpolated are in a 2 × 2 square matrix with detection indication values, and the spatial positions of four infrared pixel points are assumed to be (x, y) respectively 1 ，y 1 )，(x 1 ，y 2 )，(x 2 ，y 1 )，(x 2 ，y 2 ) The pixel value to be calculated is P (x, y), and the interpolation is carried out in the horizontal direction to obtain a temporary pixel value R in the horizontal direction 1 And R is 2
And then carrying out interpolation in the vertical direction to obtain the pixel value to be solved as follows:
preferably, the IR sensing data in each pixel unit can be interpolated using bi-linear interpolation to treat the interpolated pixel (x, y) in a 4 x 4 square matrix already with the detection indication, as follows,
wherein, W is the weight of the pixel point to be interpolated, and the specific expression is as follows:
generally, a =0.5 is taken, and finally, the infrared light image P output according to the spatial interpolation scheme is obtained IR 。
For visible RGB images, as shown in fig. 8, again, in a 3 x 3 square array that already has G-detection values, the 4 diagonal G-pixel values are directly averaged instead of the IR image value being G1. Thus, a Bayer type RGB image in an RGGB arrangement form is obtained, and a visible light image P can be obtained by following a Bayer type image back-end processing algorithm c 。
Exemplary computer readable storage Medium
The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the nano optical path design method.
The computer-readable storage medium can implement the steps of the method for designing a nano optical path in any of the embodiments and achieve the same technical effects, so that all the advantages of any of the embodiments are achieved, and further description is omitted here.
The computer-readable storage medium in this embodiment is, for example, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications derived therefrom are intended to be within the scope of the invention.
Claims (9)
1. A bifusion imaging chip, comprising:
nano optical routing and area array photodetectors;
setting the geometric structure and the appearance of the surface of the nano optical path by a nano optical path surface space optimization method;
the nano optical route is integrally bonded with the area array photoelectric detector through an optical adhesive layer or a liquid optical adhesive layer, and the nano optical route and the area array photoelectric detector are completely overlapped; the size of a single pixel of the nano optical route is correspondingly consistent with that of a pixel of a wide-spectrum CCD or CMOS area array type detector array.
2. A dual light fusion imaging chip as claimed in claim 1 wherein the nano light routing surface space optimization method includes the steps of:
the area array type detector is set as a single sensing pixel element, receives visible light and near infrared light (IR), and the visible light comprises: red (R), green (G), blue (B);
determining surface space design parameters of a nano optical route according to a set sensing pixel element region and a super-surface semiconductor material, wherein the surface space design parameters comprise: the thickness of the nano optical path and the base material and thickness;
and establishing a super-surface design parameter optimization model according to the nano optical routing surface space design parameters, and outputting an optimization scheme.
3. A dual light fusion imaging chip as recited in claim 2 wherein the sensing pixel element comprises:
four sub-pixel units arranged in a square matrix correspondingly receive red light (R), green light (G), blue light (B) and near infrared light (IR).
4. A dual light fusion imaging chip as claimed in claim 3 wherein the set wavelength range of the near infrared light (IR) is 800nm-1000nm.
5. A dual optical fusion imaging chip as claimed in claim 3, wherein the surface space design parameters of the nano optical routing are determined according to the set sensing pixel element region and the super surface semiconductor material, specifically:
according to the four sub-pixel units in the sensing pixel element area, the super surface corresponds to the same plane area and serves as a design area;
setting the corresponding thickness t of the super surface, the base material and the thickness d according to the dielectric constant distribution epsilon (x, y) of the super surface;
and calculating the distribution of the energy flow of the target light field with the central wavelength of red light (R), green light (G), blue light (B) and near infrared light (IR) along the vector vertical direction, wherein the target light field is the rear emergent surface of the substrate material.
6. A dual-optical fusion imaging chip according to claim 5, wherein the building of the super-surface design parameter optimization model according to the nano-optical routing surface space design parameters and outputting the optimization scheme specifically comprises:
establishing a light energy distribution target model, as formula (1):
wherein alpha is B ，α G ，α R ，α IR The weighting factors can be adjusted according to specific design; lambda [ alpha ] 1 ，λ 2 —λ 8 The minimum wavelength and the maximum wavelength of the four corresponding wave bands are respectively, and the corresponding central wavelengths are respectively in blue light, green light, red light and near infrared light; p z (λ) is the longitudinal light field pointining vector, P, on a single pixel plane of the sensor zB (λ) is the Boynting vector of the blue band of light through the B corresponding sub-pixel, P zG (λ) is the Boynting vector of the green band of light through the corresponding sub-pixel of G, P zR (λ) is the Boynting vector of the red band of light through the R corresponding sub-pixel, P zIR (λ) is the poynting vector of the infrared band through the IR corresponding sub-pixel; t is B (λ) is a transmission coefficient from the blue light incident surface to the pixel surface, T G (λ) is a transmission coefficient from the green light incident surface to the pixel surface, T R (λ) is a transmission coefficient from the red light incident surface to the pixel surface, T IR (λ) is a transmission coefficient from the infrared light incident surface to the pixel surface;
and optimizing the super-surface design parameter space epsilon (x, y) based on the optical energy distribution target model until the optical energy distribution target function reaches convergence and an extremum.
7. A bifocal fusion imaging chip according to claim 6, wherein the optimization of the hyper-surface design parameter space e (x, y) based on the optical energy allocation objective model until the optical energy allocation objective function reaches convergence and extremum, specifically comprises:
when the optimization of the free form super surface is realized, firstly, the initial value of the element (x, y) is set, and the value of the element (x, y) is the interval [ element [ min ,∈ max ](ii) a Optimizing by adopting an optimization iterative algorithm and a binarization processing algorithm;
or
When the digital form super surface is optimized, the pixels of the design area are divided into n units, and the value of each unit belongs to the element E min Or e max (ii) a And optimizing by adopting a genetic algorithm GA or a particle swarm optimization algorithm PSO.
8. A dual-optical image fusion processing method suitable for the dual-optical fusion imaging chip according to any one of claims 1 to 7, specifically comprising:
utilizing the indicating value of the IR sub-pixel unit in the adjacent four pixel units to carry out spatial two-dimensional spatial interpolation, obtaining the corresponding IR response at the position of 4 sub-pixels in each pixel unit, and recording the IR response as an image P IR ；
Replacing IR sub-pixels in 4 sub-pixels in each pixel unit with G sub-pixels in adjacent pixel units, marking as G1, and forming an RGGB type Bayer pixel arrangement together with R, G and B in the original pixel unit for subsequently generating RGB color images, marking as P c ；
Will P IR And P c And taking the infrared light image and the visible light image as required in the double-light fusion algorithm, processing by using the image fusion algorithm, developing feature extraction, fusion strategy and image reconstruction, and generating the double-light fusion image.
9. A computer-readable storage medium, comprising,
the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the method of designing a nano-optic path according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210961938.4A CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210961938.4A CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115174794A true CN115174794A (en) | 2022-10-11 |
| CN115174794B CN115174794B (en) | 2023-09-22 |
Family
ID=83479474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210961938.4A Active CN115174794B (en) | 2022-08-11 | 2022-08-11 | Double-light fusion imaging chip and double-light picture fusion processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115174794B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965654A (en) * | 2018-02-11 | 2018-12-07 | 浙江宇视科技有限公司 | Double spectrum camera systems and image processing method based on single-sensor |
| CN112013956A (en) * | 2019-11-11 | 2020-12-01 | 中国科学院上海技术物理研究所 | Infrared color detector based on super surface |
| CN113299671A (en) * | 2021-03-11 | 2021-08-24 | 中国科学院上海技术物理研究所 | Infrared color focal plane detector of in-situ integrated super-surface phased array |
| CN113686437A (en) * | 2021-07-09 | 2021-11-23 | 湖南大学 | Time domain spectral imaging chip based on super-structure surface |
| US20210375960A1 (en) * | 2020-05-28 | 2021-12-02 | University Of Southern California | Design of Metasurfaces for High-Efficient IR Photodetectors |
| CN114830341A (en) * | 2019-12-31 | 2022-07-29 | 华为技术有限公司 | Image sensor, light splitting and color filtering device and preparation method of image sensor |
-
2022
- 2022-08-11 CN CN202210961938.4A patent/CN115174794B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965654A (en) * | 2018-02-11 | 2018-12-07 | 浙江宇视科技有限公司 | Double spectrum camera systems and image processing method based on single-sensor |
| CN112013956A (en) * | 2019-11-11 | 2020-12-01 | 中国科学院上海技术物理研究所 | Infrared color detector based on super surface |
| CN114830341A (en) * | 2019-12-31 | 2022-07-29 | 华为技术有限公司 | Image sensor, light splitting and color filtering device and preparation method of image sensor |
| US20210375960A1 (en) * | 2020-05-28 | 2021-12-02 | University Of Southern California | Design of Metasurfaces for High-Efficient IR Photodetectors |
| CN113299671A (en) * | 2021-03-11 | 2021-08-24 | 中国科学院上海技术物理研究所 | Infrared color focal plane detector of in-situ integrated super-surface phased array |
| CN113686437A (en) * | 2021-07-09 | 2021-11-23 | 湖南大学 | Time domain spectral imaging chip based on super-structure surface |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115174794B (en) | 2023-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108965654B (en) | Double-spectrum camera system based on single sensor and image processing method | |
| WO2021227250A1 (en) | Image sensor and electronic device | |
| CN103501416B (en) | Imaging system | |
| CN104335246B (en) | The camera model of pattern is formed with pi optical filters group | |
| TWI535292B (en) | Capturing and processing of images using monolithic camera array with heterogeneous imagers | |
| US10348990B2 (en) | Light detecting device, solid-state image capturing apparatus, and method for manufacturing the same | |
| CN104010183B (en) | Array system including at least one bayer-like video camera and the method for association | |
| CN101345248B (en) | Multi-optical spectrum light-sensitive device and preparation thereof | |
| JP5325117B2 (en) | Solid-state imaging device | |
| CN106454161A (en) | Method for improving image quality of image sensor | |
| JP7462760B2 (en) | Image sensor, method for producing same, and electronic device | |
| CN115462066A (en) | Image sensor and image sensing method | |
| WO2021136469A1 (en) | Image sensor, light splitting color filter device, and image sensor fabrication method | |
| CN104412586A (en) | Image processing device, imaging device, and image processing program | |
| CN115174794B (en) | Double-light fusion imaging chip and double-light picture fusion processing method | |
| WO2022020989A1 (en) | Filtering array, mobile terminal, and device | |
| CN105206638B (en) | A kind of back-illuminated cmos image sensors and forming method thereof | |
| CN101150731A (en) | Color filtering array for digital imaging and its imaging method | |
| CN115950534A (en) | Focus-dividing plane type high-luminous-flux bionic multispectral polarization imaging detector | |
| JP7504170B2 (en) | Image acquisition device for performing white balance, electronic device including same, and method for controlling image acquisition device | |
| CN104303302A (en) | Imaging element and imaging device | |
| WO2022016412A1 (en) | Depth information image acquisition apparatus and electronic device | |
| CN103513384B (en) | A kind of method improving photon detection efficiency | |
| Anzagira et al. | Two layer image sensor pixel concept for enhanced low light color imaging | |
| KR102667267B1 (en) | Image acquisition apparatus and electronic apparatus including the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |