WO2022194040A1

WO2022194040A1 - Spectral chip-based image sensing method and apparatus, spectral recovery method and apparatus, and electronic device

Info

Publication number: WO2022194040A1
Application number: PCT/CN2022/080327
Authority: WO
Inventors: 张鸿; 王宇; 黄志雷
Original assignee: 上海与光彩芯科技有限公司
Priority date: 2021-03-16
Filing date: 2022-03-11
Publication date: 2022-09-22

Abstract

The present application relates to a spectral chip-based image sensing method and apparatus, a spectral recovery method and apparatus, and an electronic device. The spectral chip-based image sensing method comprises: determining a first image parameter to be sensed; determining a first group of multiple physical pixels of an image sensor of a spectral chip on the basis of said first image parameter; determining a response function of the first group of multiple physical pixels and measuring a light intensity reading of each physical pixel in the first group of multiple physical pixels; and calculating said first image parameter on the basis of the light intensity readings and the response function. In this way, the response function of the physical pixels corresponding to the image parameter to be sensed is determined, such that the image parameter can be calculated on the basis of the light intensity readings of the physical pixels, thereby improving an image sensing effect.

Description

Spectral chip-based image sensing method, spectral recovery method, device and electronic device

technical field

The present application relates to the technical field of spectrum chips, and more particularly, to an image sensing method, an image sensing device, a spectrum recovery method, a spectrum recovery device and an electronic device based on a spectrum chip.

Background technique

In the field of image sensing, the demand for image signal-to-noise ratio, color effect, white balance, etc. is gradually increasing. Specifically, in terms of image signal-to-noise ratio, as the pixel becomes smaller and smaller, the amount of light entering a single pixel decreases; in addition, current image sensors generally use Bayer filter arrays to achieve color effects, while Bayer filters The array further reduces the amount of incoming light, which in turn affects the signal-to-noise ratio of the image. In some specific application scenarios, such as dark light environment, this problem is more prominent. To this end, new filter arrangement schemes, such as RYYB, RGBW, etc., have been born in recent years. Although these solutions increase the amount of light entering to a certain extent, their principles are similar to that of Bayer filter arrays, and their improvement effect is very limited.

Moreover, in the calculation process, for example, in terms of color effects, the Bayer filter uses the spectral response of the RGB three-color filter to simulate the stimulus response of the human eye, and restores the overall color effect of the photo through mosaic interpolation. There may be errors in many links in the whole process, resulting in errors in color reproduction. In terms of white balance, low-cost white balance adjustment can be performed based on the acquired RGB information. With the improvement of white balance requirements on the application side, many terminals are equipped with dedicated color temperature sensors to measure the ambient color temperature to achieve a better white balance effect, thereby increasing the cost of the overall solution and increasing the complexity of the system. degree.

In recent years, with the development of micro-nano technology, quantum technology, material science and semiconductor technology, new research directions for improving image sensing have been born. For example, the spectrum chip uses a broad-spectrum filter or filter structure to filter light, and then restores the spectrum through different data processing methods after the sensor signal is collected.

The current spectral imaging technology based on spectral chips is mainly based on a spectrometer plus a mechanical scanning structure. The precision control of the mechanical scanning and the trade-off between the scanning step size required by this solution will increase the cost and reduce the resolution of the time dimension. reduce. In addition, the spectrometer realized by the use of filters and photodetection arrays can directly realize spectral imaging through the array of spectrometers because of its natural two-dimensional photosensitive structure. Alternative advantages.

However, spectral imaging based on the arraying of optical filters and light detection array spectrometers has several problems.

First, there are certain requirements for the uniformity of the light intensity distribution. While performing spectral recovery in this scheme, the light intensity distribution in the spectral pixels needs to be as uniform as possible. However, in practical applications, it is difficult to avoid large changes in light intensity in space, which need to be avoided by corresponding methods.

Secondly, for spectral imaging applications, multiple light modulation units form a spectral pixel, which has the function of spectral measurement. Since a spectral pixel occupies multiple physical pixels (pixels), the spatial resolution of the image is reduced, and corresponding methods need to be used. Improve spatial resolution while maintaining spectral resolution.

Third, for spectral imaging applications, the light path changes at the edge of the imaged object, and the signal-to-noise ratio is low. This results in poor spectral recovery at the edge.

Fourth, for different application scenarios, the requirements for spectral resolution and spatial resolution are different, and a fixed wavelength sampling interval cannot meet the requirements.

Therefore, it is desirable to provide improved spectral chip-based image sensing schemes and spectral recovery schemes

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present application is made. Embodiments of the present application provide, on the one hand, an image sensing method based on a spectrum chip, an image sensing device, and an electronic device, which can determine the response function of the physical pixel corresponding to the image parameter to be sensed, This image parameter is calculated from the light intensity readings of the physical pixels to improve image sensing.

On the other hand, the embodiments of the present application provide a spectrum chip-based spectrum recovery method, spectrum recovery device, and electronic device, which can flexibly adjust the number and spatial distribution of physical pixel points corresponding to each spectrum pixel. Perform spectral restoration to enhance the spectral restoration effect.

According to an aspect of the present application, an image sensing method based on a spectrum chip is provided, comprising: determining a first image parameter to be sensed; determining an image of the spectrum chip based on the first image parameter to be sensed a first plurality of physical pixels of a sensor; determining a response function of the first plurality of physical pixels and measuring a light intensity reading for each of the first plurality of physical pixels; and, based on the The light intensity reading and the response function calculate the first image parameter to be sensed.

In the above image sensing method based on a spectrum chip, determining a first group of multiple physical pixels of the spectrum chip for image sensing based on the first image parameter to be sensed includes: At least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy determines the first plurality of physical pixels.

In the above-mentioned image sensing method based on a spectrum chip, the first group of multiple physical pixels is at least one of 2×2, 3×3, 5×5, and 10×10 physical pixel square arrays on the image sensor. One.

In the above-mentioned image sensing method based on a spectrum chip, the first group of multiple physical pixels are multiple physical pixels of a predetermined shape on the image sensor.

In the above image sensing method based on a spectral chip, determining the first group of multiple physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: responding to ambient light The intensity is a first intensity, determining a first number of physical pixel squares; and, in response to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel squares, the second The number is greater than the first number.

In the above-mentioned image sensing method based on a spectral chip, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: responding to the image a first area of a first brightness or a first signal-to-noise ratio in the image acquired by the sensor, determining a first number of physical pixel squares; and, in response to a second area of the image acquired by the image sensor that is less than the first brightness The brightness or the second region of the second signal-to-noise ratio smaller than the first signal-to-noise ratio determines a second number of physical pixel squares, the second number being greater than the first number.

In the above-mentioned image sensing method based on a spectral chip, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: responding to the image a first spatial resolution and/or a first spectral accuracy of the sensor, determining a first number of physical pixel squares; and, in response to a second spatial resolution of the image sensor less than the first spatial resolution and/or greater than The second spectral precision of the first spectral precision determines a second number of physical pixel squares, the second number being greater than the first number.

In the above-mentioned image sensing method based on a spectrum chip, at least one of the first plurality of physical pixels is used to restore an image.

In the above image sensing method based on a spectrum chip, the method further includes: determining a second image parameter to be sensed; determining a second group of multiple image sensors of the spectrum chip based on the second image parameter to be sensed a physical pixel; determining a response function for the second plurality of physical pixels and determining a light intensity reading for each of the second plurality of physical pixels; and, based on the light intensity reading and the response The function calculates the second image parameter to be sensed.

In the above-mentioned spectral chip-based image sensing method, determining the light intensity reading of each physical pixel in the second group of physical pixels includes: based on the second group of physical pixels and the first group of physical pixels The correspondence between the plurality of physical pixels is obtained, and the light intensity readings of the physical pixels corresponding to the first group of the plurality of physical pixels in the second group of the plurality of physical pixels are obtained.

In the above-mentioned image sensing method based on a spectral chip, the first image parameter is color data, and the second image parameter is color temperature data.

In the above-mentioned image sensing method based on a spectrum chip, the first group of multiple physical pixels is a first number of physical pixel square matrices, the second group of multiple physical pixels is a second number of physical pixel square arrays, The second number is greater than the first number.

In the above-mentioned spectral chip-based image sensing method, determining the response function of the second group of multiple physical pixels includes: determining, based on the color temperature data to be sensed, a first image sensor of the image sensor of the color temperature data to be sensed an area and a second area; and, determining response functions of a plurality of physical pixels of the first area and the second area, respectively, to obtain response functions of the second plurality of physical pixels.

In the above-mentioned image sensing method based on a spectrum chip, the spectrum chip is a spectrum chip for receiving light in a wavelength band of 350 to 1000 nanometers of a computing spectrum device.

According to another aspect of the present application, an image sensing device based on a spectrum chip is provided, comprising: a parameter determination unit for determining a first image parameter to be sensed; a pixel determination unit for The first image parameter sensed to determine a first group of multiple physical pixels of the image sensor of the spectrum chip; a data acquisition unit for determining the response function of the first group of multiple physical pixels and measuring the first group of multiple physical pixels a light intensity reading of each of the physical pixels; and a parameter calculation unit for calculating the first image parameter to be sensed based on the light intensity reading and the response function.

According to yet another aspect of the present application, a spectrum recovery method for a spectrum chip is provided, comprising: determining a predetermined pixel point on the spectrum chip based on a predetermined condition; Spectral recovery.

In the above spectrum chip-based spectrum recovery method, determining the predetermined pixel points on the spectrum chip includes: determining the predetermined pixel on the spectrum chip based on at least one of luminance distribution and spectral resolution requirements and spatial resolution requirements point.

In the above spectrum chip-based spectrum recovery method, determining the predetermined pixel point on the spectrum chip based on at least one of brightness distribution and spectral resolution requirements and spatial resolution requirements includes: acquiring an output of an image sensor of the spectrum chip. image data; performing edge detection on the image data; and determining predetermined pixel points on the spectrum chip based on the edge detection result of the image data.

In the above spectrum chip-based spectral restoration method, performing edge detection on the image data includes: equalizing the image data; denoising the equalized image data; Image data for edge detection.

In the above spectrum chip-based spectrum restoration method, performing noise reduction on the equalized image data includes: selecting a filter of a predetermined size according to the image resolution and/or the characteristics of the filter structure corresponding to the image sensor, The equalized image data is filtered.

In the above spectrum chip-based spectral restoration method, performing edge detection on the denoised image data includes: using an edge detection operator to detect an edge region in the denoised image data; The region is expanded.

In the above spectrum chip-based spectrum recovery method, determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data includes: based on the edge detection result of the image data, determining that the spectrum chip has A predetermined pixel point with a light intensity consistency higher than a predetermined threshold, the predetermined pixel point is a predetermined number of pixel points that are adjacent and connected based on the pixel center point.

In the above spectrum chip-based spectrum recovery method, determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data includes: based on the edge detection result of the image data, determining that the spectrum chip has Groups of predetermined pixel points with different numbers.

In the above spectrum chip-based spectrum restoration method, the plurality of groups of predetermined pixel points with different numbers include 2×2, 3×3, 5×5, and 10×10 physical pixel arrays.

In the above spectrum chip-based spectrum recovery method, the predetermined pixel point is a plurality of physical pixels of a predetermined shape on the image sensor.

In the above spectrum chip-based spectrum recovery method, determining the predetermined pixel points on the spectrum chip based on the luminance distribution and at least one of a spectral resolution requirement and a spatial resolution requirement includes: in response to the spectral resolution requirement being greater than the third A predetermined threshold, determining the predetermined number of pixels greater than a first predetermined number; and/or, in response to the spatial resolution requirement being greater than a second predetermined threshold, determining the predetermined number of pixels less than a second predetermined number.

In the above spectrum chip-based spectrum recovery method, using the predetermined pixel points on the spectrum chip to perform spectrum recovery includes: using the predetermined pixel points on the spectrum chip to perform spectrum recovery with a dynamically adjusted recovery step size .

In the above spectrum chip-based spectrum recovery method, using the predetermined pixel points on the spectrum chip to perform spectrum recovery includes: dynamically adjusting the wavelength sampling interval used for the spectrum recovery based on the application of the spectrum to be recovered .

In the above spectrum chip-based spectrum restoration method, determining the predetermined pixel points on the spectrum chip based on the edge detection result of the image data includes: determining the number of predetermined pixel points on the spectrum chip based on a wavelength sampling interval.

In the above-mentioned spectrum recovery method based on a spectrum chip, the spectrum chip is a spectrum chip used for a calculation spectrometer to receive light in a wavelength band of 350 to 1000 nanometers.

According to yet another aspect of the present application, there is provided a spectrum recovery device for a spectrum chip, comprising: a pixel determination unit for determining a predetermined pixel point on the spectrum chip based on a predetermined condition; and a spectrum recovery unit for using Spectral recovery is performed on the predetermined pixel points on the spectrum chip.

According to yet another aspect of the present application, there is provided an electronic device, comprising: a processor; and a memory, in which computer program instructions are stored, the computer program instructions cause the processor to run when the processor runs The processor executes the spectral chip-based image sensing method and the spectral chip-based spectral recovery method as described above.

According to yet another aspect of the present application, a computer-readable storage medium is provided, and computer program instructions are stored thereon, and when the computer program instructions are executed by a computing device, the computer program instructions are operable to execute the above The image sensing method based on the spectrum chip and the spectrum recovery method based on the spectrum chip are described.

The spectral chip-based image sensing method, image sensing device and electronic device provided by the present application can utilize the broad-spectrum filter structure of the spectrum chip to determine the physical properties corresponding to the image parameters to be sensed according to the broad-spectrum filter structure. The response function of the pixel is calculated, and the image parameter is calculated based on the light intensity reading of the physical pixel, thereby improving the signal-to-noise ratio, color and white balance of image sensing.

In addition, the spectral chip-based image sensing method, image sensing device, and electronic equipment provided by the present application can utilize the reconfigurable properties of the spectral chip to detect the signal-to-noise ratio, color, and white balance of image sensing in different situations. The effect is optimized in a targeted manner, thereby improving the image sensing effect.

In addition, the spectral chip-based spectral recovery method, spectral recovery device and electronic device provided by the present application can flexibly adjust the number and spatial distribution of physical pixel points corresponding to each spectral pixel according to the image data output by the image sensor, thereby Reduce the recovery error introduced by spatial light intensity inhomogeneity.

In addition, the spectral chip-based spectral recovery method, spectral recovery device and electronic device provided in this application can improve the spatial resolution of spectral imaging by adjusting the number and spatial distribution of physical pixel points corresponding to each spectral pixel.

In addition, the spectral chip-based spectral recovery method, spectral recovery device and electronic device provided by the present application improve the edge signal-to-noise ratio of spectral images by performing edge detection on image data, and can be advantageously applied to spectral image-based edge detection and substance detection. identify.

Description of drawings

Various other advantages and benefits of the present application will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are for the purpose of illustrating the preferred embodiments only, and are not to be considered as limitations of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort. Also, the same components are denoted by the same reference numerals throughout the drawings.

FIG. 1 illustrates an exemplary configuration diagram of a spectrum chip according to an embodiment of the present application;

FIG. 2 illustrates a flowchart of an image sensing method based on a spectrum chip according to an embodiment of the present application;

FIG. 3 illustrates a schematic diagram of determining the number of different physical pixels in a spectral chip-based image sensing method according to an embodiment of the present application;

4 illustrates a schematic diagram of the simultaneous use of a spectral chip-based image sensing method for color reproduction and color temperature measurement according to an embodiment of the present application;

5 illustrates a flowchart of the specific example of the spectroscopic chip-based image sensing method according to an embodiment of the present application;

FIG. 6 illustrates a flowchart of a spectrum recovery method based on a spectrum chip according to an embodiment of the present application;

FIG. 7 illustrates an example of pixel points selected for spectrum restoration in a spectrum chip-based spectrum restoration method according to an embodiment of the present application;

8 illustrates an example of a reconfigurable manner in a spectral chip-based spectral recovery method according to an embodiment of the present application;

9 illustrates a schematic diagram of image changes before and after image equalization in the spectral chip-based spectral restoration method according to an embodiment of the present application;

10 illustrates a schematic diagram of an image after noise reduction in a spectral chip-based spectral restoration method according to an embodiment of the present application;

11 illustrates an edge detection result obtained by using the Canny edge detection algorithm in the spectrum chip-based spectrum recovery method according to an embodiment of the present application;

12 illustrates an example of using edge information to select pixel points for spectrum restoration in a spectrum chip-based spectrum restoration method according to an embodiment of the present application;

Figures 13A to 13C illustrate examples of pixel points of different shapes selected using edge information to restore the spectrum in the example shown in Figure 12;

14 illustrates an example of spectral recovery using a step size of 3 in a spectral chip-based spectral recovery method according to an embodiment of the present application;

15 illustrates an example of spectral recovery using a step size of 1 in a spectral chip-based spectral recovery method according to an embodiment of the present application;

16 illustrates a block diagram of a spectral chip-based image sensing device according to an embodiment of the present application;

17 illustrates a block diagram of a spectral chip-based spectral recovery apparatus according to an embodiment of the present application;

18 illustrates a block diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Exemplary Spectroscopy Chip

FIG. 1 illustrates an exemplary configuration diagram of a spectrum chip according to an embodiment of the present application.

The spectroscopic chip according to the embodiment of the present application is generally a spectroscopic chip applied to a computing spectrometer device. The computational spectroscopy device may be a spectrometer or a spectral imaging device. Taking a spectrometer as an example, the most significant difference between a computational spectrometer and a traditional spectrometer is the difference in light filtering. In conventional spectrometers, the filters used for wavelength selection are bandpass filters. The higher the spectral resolution, the narrower the passband and the more filters must be used, which increases the size and complexity of the overall system. At the same time, when the spectral response curve is narrowed, the luminous flux decreases, resulting in a lower signal-to-noise ratio.

For a specific computational spectrometer, each filter generally adopts a broad-spectrum filter, which makes the raw data detected by the computational spectrometer system quite different from the original spectrum. However, by applying a computational reconstruction algorithm, the original spectrum can be recovered computationally. Since broadband filters pass more light, i.e. light loses less energy, than narrowband filters, these types of computational spectrometers can detect spectra from darker scenes. In addition, according to the compressed sensing theory, the spectral curve of the filter can be appropriately designed to recover the sparse spectrum with high probability, and the number of filters is much smaller than the desired number of spectral channels (recovering higher-dimensional vectors from lower-dimensional vectors) , which is undoubtedly very conducive to miniaturization. On the other hand, by using a larger number of filters, a regularization algorithm (a denoised lower dimensional vector is obtained from a higher dimensional vector) can be used to reduce noise, which increases the signal-to-noise ratio and makes the overall system more efficient higher robustness.

Relatively speaking, traditional spectrometers need to design filters according to the required wavelengths when designing, so that light of specific wavelengths can pass through (generally, it is designed to enhance the projection of incident light of specific wavelengths, rather than incident light of specific wavelength bands). It is impossible to project, and the resonance conditions can be controlled by changing the structural period and diameter of nanodisks, etc., and the central wavelength of incident light that can enhance the projection can be changed, so as to achieve filter characteristics). That is, the traditional spectrometer needs to focus on controlling the size and positional accuracy of the light modulation structure in the design process, and at the same time, it is necessary to find a way to improve its transmittance of specific wavelengths. For computational spectrometers, what is needed is a wide range of wavelengths (eg, 350nm to 1000nm, and even 900nm to 2500nm).

As shown in FIG. 1 , in the spectrum chip according to the embodiment of the present application, the optical system is optional, which may be an optical system such as a lens component, a uniform light component, and the like. The filter structure is a broadband filter structure in the frequency domain or the wavelength domain. The pass spectra of different wavelengths of the filter structures are not exactly the same everywhere. Filter structures can be metasurfaces, photonic crystals, nanopillars, multilayer films, dyes, quantum dots, MEMS (Micro-Electro-Mechanical Systems), FP etalon (FP etalon), cavity layer (hole layer), waveguide layer (waveguide). layer), diffractive elements and other structures or materials with filtering properties. For example, in the embodiment of the present application, the filter structure may be the light modulation layer in Chinese patent CN201921223201.2,

Continuing to refer to FIG. 1 , the image sensor (ie, the photodetector array) can be a CMOS image sensor (CIS), a CCD, an array photodetector, etc., and its material can be a silicon detector, or a detector of InGaAs or other materials. In addition, the optional data processing unit may be a processing unit such as MCU, CPU, GPU, FPGA, NPU, ASIC, etc., which can export the data generated by the image sensor to the outside for processing.

In addition, the computational spectroscopy apparatus according to the embodiment of the present application may also use a modulation unit or the like to form a filter structure, and in this case, each point of the photodetector array has different spectral responses. Further, directly adopting quantum dots, nanowires and other solutions can be realized.

In the embodiment of the present application, after the light intensity information is measured by the image sensor, it is transmitted to the data processing unit for recovery calculation. Specifically, in some cases, the compressed sensing method can be used, and by selecting an appropriate measurement base and dictionary, high-precision spectral data can be calculated only with light intensity data measured by a small number of pixels of the image sensor. The process is described in detail as follows:

The intensity signal of the incident light at different wavelengths λ is denoted as f(λ), and the transmission spectrum curve of the filter structure is denoted as T(λ). There are m groups of filter structures on the filter, and each group of transmission spectra is mutually Different, also known as "structural unit", the whole can be recorded as T _i (λ) (i=1,2,3,...,m). There are corresponding physical pixels under each group of filter structures to detect the light intensity I _i modulated by the filter structures. In a specific embodiment of the present application, a physical pixel corresponding to a group of structural units is taken as an example for description, but it is not limited to this. In other embodiments, a group of multiple physical pixels can also correspond to a group of structures unit. Therefore, unlike physical pixels, in the computational spectroscopy apparatus according to the embodiment of the present application, one or more physical pixels corresponding to a group of structural units are referred to as "spectral pixels". Further, the present invention can use at least one spectral pixel to restore the image.

The relationship between the spectral distribution of the incident light and the measurements of the photodetector array can be expressed by:

I _i =Σ(f(λ)*T _i (λ)*R(λ))

where R(λ) is the response of the detector, denoted as:

S _i (λ)=T _i (λ)*R(λ)

Then the above formula can be extended to matrix form:

Among them, I _i (i=1,2,3,...,m) is the response of the photodetector after the light to be measured passes through the broadband filter unit, corresponding to the light intensity measurement values of m photodetector units respectively, also known as m "physical pixels", which are a vector of length m. S is the light response of the system to different wavelengths, which is determined by two factors, the transmittance of the filter structure and the quantum efficiency of the photodetector response. S is a matrix, and each row vector corresponds to the response of a broadband filter unit to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, with a total of n sampling points. The number of columns of S is the same as the number of sampling points of the incident light. Here, f(λ) is the light intensity of the incident light at different wavelengths λ, that is, the incident light spectrum to be measured.

In practical applications, the response function S of the system is known, and through the light intensity reading I of the detector, the spectrum f(λ) of the input light can be obtained by inverse algorithm. Including: least squares, pseudo-inverse, equalization, least two norm, artificial neural network, etc. Therefore, to restore the relatively simple first image parameters to be sensed, such as the image color (ie, the three values of RGB), a specific matrix multiplication method can be used for restoration. For the requirement of high spectral accuracy, that is, when the value of n in the above process is large, algorithms such as artificial neural network can be used. Optionally, if this kind of method cooperates with the compressed sensing processing method, it can still obtain higher spectral accuracy in some cases when m is significantly smaller than n.

It should be noted that it can be inferred from the above description that when the first or second image parameter of image sensing is the incident light intensity f(λ), the system response S acts on the light intensity parameter in a matrix form. When the first or second image parameter is other physical quantities, the system response may act on the light intensity parameter by other functions. This application mainly takes the response matrix as an example to describe the image sensing method.

The above takes one physical pixel corresponding to a group of structural units as an example to describe how to use m groups of physical pixels (that is, pixels on the image sensor) and their corresponding m groups of structural units (the same structure on the modulation layer is defined as a structural unit) ) to recover a spectral information, also known as "spectral pixel". For example, in the embodiment of the present application, by recovering the color information of the pixels and implementing them in an array, complete image information can be obtained, thereby realizing image sensing. It should be noted that, as described above, in the embodiment of the present application, a plurality of physical pixels may also correspond to a group of structural units.

That is, in the field of spectral chips, a distinction is usually made between physical pixels, structural units, and spectral pixels. Here, a physical pixel refers to the smallest photosensitive unit on an image sensor (detector array). Structural unit refers to the fact that the spectrum chip needs to use a filter structure on the image sensor to filter light. The smallest unit of the filter structure can be called a structural unit, and one structural unit can cover one or more physical pixels. In the subsequent description of the embodiments of the present application, it is taken as an example that one structural unit corresponds to one physical pixel.

Exemplary Image Sensing Method

FIG. 2 illustrates a flowchart of an image sensing method based on a spectrum chip according to an embodiment of the present application.

As shown in FIG. 2 , the image sensing method based on a spectrum chip according to an embodiment of the present application includes the following steps:

Step S110, determining the first image parameter to be sensed. For example, in the embodiment of the present application, the first image parameter to be sensed refers to the sensed physical quantity to be measured. The first image parameter may be color data, such as RGB color information or finer spectral (color temperature) information, or even multi-spectral or hyperspectral information.

S120: Determine a first group of multiple physical pixels of the image sensor of the spectrum chip based on the first image parameter to be sensed. As described above, based on the first image parameter to be sensed, a first group of multiple physical pixels of the image sensor of the spectrum chip is determined, that is, m physical pixels as described above. Preferably, the first group of multiple physical pixels of the image sensor of the spectrum chip may also be determined by the first image parameter to be sensed and the spatial resolution or signal-to-noise requirement. Here, the process of determining the first plurality of physical pixels may be performed based on various considerations, which will be described in further detail below.

For example, for RGB color restoration, that is, to restore RGB color data, in the above process, n takes the value of 3, and m can also take the value of 3, that is, to determine a group of three physical pixels, the above matrix can be simplified as:

Therefore, by measuring the light intensity data I ₁ , I ₂ , and I ₃ , and knowing the parameters of the S matrix, the RGB value of the color pixel can be quickly obtained by means of pseudo-inverse or equalization.

Step S130, determining the response function of the first group of multiple physical pixels and measuring the light intensity reading of each physical pixel in the first group of multiple physical pixels. For example, the response function S of the first set of three physical pixels as described above and the light intensity readings I ₁ , I ₂ and I ₃ for each physical pixel therein. It is worth mentioning that the light intensity readings I ₁ , I ₂ and I ₃ obtained in the step S130 can be used to assist the system in the step S120 to automatically adjust to meet the corresponding signal-to-noise ratio or spatial resolution For example, a threshold can be set, and the judgment can be made based on the relationship between the light intensity reading and the threshold. For example, if the average of the light intensity readings is greater than the threshold, it can be understood that the light intensity is strong. For example, the light intensity reading is greater than The number of thresholds exceeds 75%, which is interpreted as a strong light intensity.

In addition, those skilled in the art can understand that in the image sensing method based on the spectrum chip according to the embodiment of the present application, the order of determining the response function and measuring the light intensity reading in step S130 can be set arbitrarily, for example, the first The light intensity readings of each of the plurality of physical pixels are grouped, and the response function of the first group of the plurality of physical pixels is determined. Alternatively, both can be performed simultaneously. That is, in this embodiment of the present application, after the first group of multiple physical pixels of the image sensor of the spectrum chip is determined based on the first image parameter to be sensed, the first group may be acquired in any order. Light intensity readings and response functions for multiple physical pixels.

Step S140, calculating the first image parameter to be sensed based on the light intensity reading and the response function. For example, the RGB color data of three physical pixels used for color reproduction as described above.

Therefore, the image sensing method based on the spectrum chip according to the embodiment of the present application can utilize the broad-spectrum filter structure of the spectrum chip, that is, the light-passing efficiency of the broad-spectrum filter structure of the spectrum chip is significantly better than that of the existing image The filter array of the sensor, such as the Bayer filter array, is comparable to the filter array structure of the existing image sensor in terms of color reproduction and spatial resolution. Therefore, the image sensing method based on the spectral chip according to the embodiment of the present application has significant advantages in image sensing in a dark light environment.

In addition, in the image sensing method based on the spectrum chip according to the embodiment of the present application, since the system of the spectrum chip has reconfigurable characteristics, the reconfigurable characteristics refer to dynamically adjusting a group of multiple physical pixels according to requirements, That is, it can be dynamically adjusted according to different environmental conditions, and can be dynamically adjusted in different areas in the same photo to obtain the best image effect.

That is, on the basis that a physical pixel corresponds to a unit structure, how many physical pixels are selected as a set of data to restore a spectral pixel, in other embodiments, it can also be implemented as how many unit structures are selected as A set of data is restored for one spectral pixel, that is, the value of m, which affects the spatial resolution of the sensor, the image signal-to-noise ratio, and the spectral accuracy. In a reconfigurable way, the value of m can be dynamically adjusted according to the environmental conditions in different situations, so as to obtain the optimal image effect.

FIG. 3 illustrates a schematic diagram of determining the number of different physical pixels in a spectral chip-based image sensing method according to an embodiment of the present application. As shown in FIG. 3 , a 10×10 square matrix of physical pixels is shown, wherein the transmission spectra of the filter structures corresponding to each physical pixel are different (this can make the correlation lower). Specifically, the above-mentioned reconfigurable characteristics are shown in that, in the actual use process, the number of physical pixels in the scale of 2×2, 3×3, 5×5, 10×10, etc. can be used as needed (that is, the value of m ) are processed as a set of data to generate a spectral pixel. It is worth mentioning that the present invention takes the one-to-one relationship between physical pixels and structural units as an example. In practice, it can also be that multiple physical pixels correspond to a group of structural units. When multiple physical pixels correspond to a group of structural units, you can It is understood that the structural unit can be reconstructed, that is, the multiple physical pixels are determined by the structural unit. For example, 2×2 can be understood as four structural units forming a group of multiple physical pixels.

Specifically, when the environment is dark, more physical pixels can be used as a set of data for processing. For example, physical pixels greater than or equal to 8×8 are used as a spectral pixel, thereby improving the image signal-to-noise ratio.

When the ambient light intensity is strong, fewer pixels can be used as a set of data for processing, such as less than or equal to 3×3 physical pixels to form a spectral pixel, preferably 2×2 physical pixels. one spectral pixel, thereby increasing the spatial resolution.

Further, in other embodiments, an area can be selected for dynamic adjustment in the same image, the value of m in the darker area in the image is set high, and the value of m in the brighter part in the image is set low, so as to achieve the maximum value of m. Excellent effect. That is, there may be spectral pixels of different sizes in an image.

Taking Figure 3 as an example, a reconfigurable method is used in a 10×10 pixel square matrix, and different numbers of physical pixels (2×2, 3×3, 5×5) are selected as a group of multiple physical pixels to restore the spectrum. pixel. Also taking RGB data as an example, when 2×2 pixels are selected, based on the light intensity data of 2×2 pixels, combined with the response function, the RGB color data of 25 color (spectrum) pixels can be obtained within the square matrix range, thus Perform color recovery. When 5×5 pixels are selected, based on the light intensity data of 5×5 pixels, combined with the response function, the RGB color data of 4 color (spectrum) pixels can be obtained within the range of the square matrix. Comparing the two, a higher spatial resolution can be obtained when 2×2 pixels are selected as a group of multiple physical pixels, and a higher signal-to-noise ratio can be obtained when 5×5 pixels are selected as a group of multiple physical pixels. . Dynamic selection can be made according to needs in different situations.

And, it is worth noting that, in the image sensing method based on the spectral chip according to the embodiment of the present application, the first image sensor is determined according to at least one of the spatial resolution of the image sensor, the image signal-to-noise ratio and the spectral accuracy. The group of multiple physical pixels can be automatically adjusted by the system itself based on default settings or actual needs, or manually adjusted by the user according to their own needs.

Therefore, in the image sensing method based on a spectrum chip according to an embodiment of the present application, determining a first group of a plurality of physical pixels of the spectrum chip for image sensing based on the first image parameter to be sensed includes: : determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor.

In addition, in the above-mentioned image sensing method based on a spectrum chip, the first group of multiple physical pixels may be 2×2, 3×3, 5×5, 10×10 physical pixel square arrays on the image sensor At least one of them may also be a rectangular square matrix or a combination of irregular non-square matrices. That is, the first plurality of physical pixels may be a plurality of physical pixels of a predetermined shape on the image sensor.

In addition, in the above-mentioned image sensing method based on a spectral chip, determining the first group of the plurality of physical pixels according to at least one of the spatial resolution, the image signal-to-noise ratio and the spectral accuracy of the image sensor includes: responding to an environment The intensity of the light is a first intensity, determining a first number of physical pixel squares; and, in response to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel squares, the The second number is greater than the first number.

In addition, in the above-mentioned image sensing method based on a spectral chip, determining the first group of the plurality of physical pixels according to at least one of the spatial resolution, the image signal-to-noise ratio and the spectral accuracy of the image sensor includes: responding to the determining the first number of physical pixel squares in the first area of the first brightness or the first signal-to-noise ratio in the image acquired by the image sensor; The second brightness or the second area of the second signal-to-noise ratio smaller than the first signal-to-noise ratio determines a second number of physical pixel squares, the second number being greater than the first number.

In the above, the image sensing method based on a spectral chip according to the embodiment of the present application is used for color restoration as an example. In addition, the image sensing method based on a spectral chip according to the embodiment of the present application can also be used to obtain other image sensing parameters. For example, the image sensing method based on the spectral chip according to the embodiment of the present application can be used to measure the ambient color temperature to perform precise white balance processing on the image.

Those skilled in the art can understand that the requirements for color reproduction and color temperature measurement are very different in the above-mentioned aspects. Specifically, color reproduction has lower requirements for spectral accuracy and higher spatial resolution, while color temperature measurement is just the opposite, with lower requirements for spatial resolution and higher spectral accuracy. Therefore, for image sensing solutions based on non-reconfigurable structures and methods, two requirements need to be met by designing different structures. However, by using the reconfigurable structure and processing mechanism of the embodiment of the present application, two functions can be implemented on the same sensor, and the effects of respective requirements can be guaranteed.

That is, in the spectral chip-based image sensing method according to the embodiment of the present application, the first set of multiple physical properties is determined according to at least one of the spatial resolution of the image sensor, the image signal-to-noise ratio and the spectral accuracy. The pixels include: determining a first number of physical pixel squares in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and, in response to a first spatial resolution of the image sensor less than the first spatial resolution With a spatial resolution and/or a second spectral precision greater than the first spectral precision, a second number of physical pixel squares is determined, the second number being greater than the first number.

FIG. 4 illustrates a schematic diagram of a spectral chip-based image sensing method for simultaneous color reproduction and color temperature measurement according to an embodiment of the present application. As shown in Figure 4, in the structure of 10 × 10 physical pixels, every 2 × 2 physical pixels are used as a data group for restoring RGB color, that is, a total of 5 × 5=25 pixels of RGB color pixels are restored. . Of course, those skilled in the art can understand that the RGB color data can also be equivalently replaced with other color data, such as XYZ, RGB-like color data, and the like. At the same time, all pixel data of 10 × 10 are used as a set of data for spectrum recovery, and relatively fine image spectrum information is obtained, and then the color temperature information of the image is calculated and white balance processing is performed. Here, those skilled in the art can understand that when the light intensity data of all pixels of 10×10 are used, the light intensity data of each pixel used in the process of color restoration can be directly used.

The process of calculating the relative color temperature from the spectrum can be briefly described as:

First, use the CIE tristimulus parameters and the image spectrum to calculate the image color value x, y:

in

is the spectral response parameter corresponding to the CIE tristimulus value. From this, the color level value can be calculated:

Then check the color temperature through the chromaticity chart, or calculate the relative color temperature CCT through the approximate formula:

n=(x-0.3320)/(y-0.1858),

CCT=-437*n^3+3601*n^2-6831*n+5517.

In this way, 2 × 2 and 10 × 10 pixel groups can be used to restore spectral pixels on the same image sensor, while meeting the needs of color imaging and color temperature sensing. Of course, those skilled in the art can understand that in the image sensing method based on a spectrum chip according to the embodiment of the present application, the image parameters to be sensed obtained at the same time are not limited to color data and color temperature data, and may also be other types of images. parameter.

Therefore, in the image sensing method based on a spectrum chip according to an embodiment of the present application, the method further includes: determining a second image parameter to be sensed; determining an image of the spectrum chip based on the second image parameter to be sensed a second plurality of physical pixels of a sensor; determining a response function for the second plurality of physical pixels and determining a light intensity reading for each of the second plurality of physical pixels; and, based on the The light intensity reading and the response function calculate the second image parameter to be sensed.

Moreover, in the above-mentioned image sensing method based on a spectrum chip, determining the light intensity reading of each physical pixel in the second group of multiple physical pixels includes: based on the second group of multiple physical pixels and the first The correspondence between a group of multiple physical pixels, and the light intensity readings of the physical pixels corresponding to the first group of multiple physical pixels in the second group of multiple physical pixels are acquired.

Furthermore, in the above-mentioned image sensing method based on a spectral chip, the first image parameter is color data, and the second image parameter is color temperature data.

Moreover, in the above-mentioned image sensing method based on a spectrum chip, the first group of multiple physical pixels is a first number of physical pixel squares, and the second group of multiple physical pixels is a second number of physical pixel squares array, the second number is greater than the first number.

Further, in the image sensing method based on the spectrum chip according to the embodiment of the present application, in the same imaging, the color temperature of the image can be measured in different regions, so as to know the color temperature of the environment more accurately, and provide more processing methods. Preconditions. In addition, due to the use of the same sensor to achieve high spatial resolution color reproduction and accurate color temperature measurement, a better image white balance effect can be obtained.

That is, in the spectral chip-based image sensing method according to the embodiment of the present application, determining the response function of the second group of multiple physical pixels includes: determining color temperature data to be sensed based on the color temperature data to be sensed the first area and the second area of the image sensor; and, determining the response functions of the plurality of physical pixels of the first area and the second area, respectively, to obtain the response of the second plurality of physical pixels function.

Therefore, the image sensing method based on the spectrum chip according to the embodiment of the present application can utilize the broad-spectrum filter structure of the spectrum chip, and determine the response function of the physical pixel corresponding to the image parameter to be sensed according to the broad-spectrum filter structure, The image parameter is calculated based on the light intensity reading of the physical pixel, thereby improving the signal-to-noise ratio, color and white balance of image sensing.

In addition, the image sensing method based on the spectrum chip according to the embodiment of the present application can utilize the reconfigurable characteristics of the spectrum chip to perform targeted optimization on the signal-to-noise ratio, color and white balance of image sensing according to different situations, thereby Improved image sensing effect.

First application example

As described above, the spectral chip-based image sensing method according to the embodiment of the present application is based on the reuse of pixel data, and can be used for color reproduction and color temperature measurement at the same time. A flowchart of this specific example of an image sensing method of a chip.

As shown in FIG. 5 , the specific example of the spectral chip-based image sensing method according to the embodiment of the present application includes: step S210 , determining color data and color temperature data to be sensed; S220 , based on the color to be sensed The data and the color temperature data respectively determine a first group of multiple physical pixels and a second group of multiple physical pixels of the image sensor of the spectrum chip; S230, determine the first response function of the first group of multiple physical pixels and the a first light intensity reading for each of the first plurality of physical pixels and determining a second response function for the second plurality of physical pixels and each of the second plurality of physical pixels a second light intensity reading of a physical pixel; and, S240, calculate the color data to be sensed based on the first light intensity reading and the first response function, and calculate the color data to be sensed based on the second light intensity reading and the first response function The second response function calculates the color temperature data to be sensed.

In the specific example of the image sensing method based on the spectrum chip, determining the first light intensity reading of each physical pixel in the first group of multiple physical pixels includes: based on the first group of multiple physical pixels and the The corresponding relationship of the second group of multiple physical pixels is obtained by acquiring the light intensity readings of the physical pixels corresponding to the first group of multiple physical pixels in the second group of multiple physical pixels. That is, as mentioned above, because the light intensity data of all 10×10 physical pixels needs to be measured when sensing the color temperature data, the 2×10 light intensity data to be used can be directly selected from the light intensity data of the 10×10 physical pixels. 2 physical pixels of light intensity data.

In the specific example of the image sensing method based on the spectrum chip, the first group of multiple physical pixels is a first number of physical pixel square arrays, such as the above-mentioned 2×2 physical pixel square matrix, the The second group of multiple physical pixels is a second number of physical pixel square matrices, such as the 10×10 physical pixel square matrix as described above. Of course, those skilled in the art can understand that the first group of multiple physical pixels may also be a 3×3 or 5×5 square matrix of physical pixels.

Also, as described above, the first group of physical pixels may also be rectangular or irregular-shaped physical pixels on the image sensor.

In addition, in the specific example of the image sensing method based on the spectrum chip, the first group of multiple physical pixels may also be determined according to at least one of the spatial resolution of the image sensor, the image signal-to-noise ratio and the spectral accuracy and/or the second plurality of physical pixels.

That is, determining the first plurality of physical pixels and/or the second plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor comprises: in response to The intensity of the ambient light is a first intensity, and a first number of physical pixel square matrices are determined; and, in response to the ambient light intensity being a second intensity less than the first intensity, a second number of physical pixel square matrices are determined, so The second number is greater than the first number.

Alternatively, determining the first plurality of physical pixels and/or the second plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor comprises: in response to the determining the first number of physical pixel squares in the first area of the first brightness or the first signal-to-noise ratio in the image acquired by the image sensor; The second brightness or the second area of the second signal-to-noise ratio smaller than the first signal-to-noise ratio determines a second number of physical pixel squares, the second number being greater than the first number.

Alternatively, determining the first plurality of physical pixels and/or the second plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor comprises: in response to the a first spatial resolution and/or a first spectral accuracy of the image sensor, determining a first number of physical pixel arrays; and, in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or or a second spectral precision greater than the first spectral precision, a second number of physical pixel squares is determined, the second number being greater than the first number.

Moreover, in the specific example of the image sensing method based on the spectral chip, determining the second response function of the second group of multiple physical pixels includes: determining the color temperature data to be sensed based on the color temperature data to be sensed. a first region and a second region of the image sensor; and determining response functions of a plurality of physical pixels of the first region and the second region, respectively, to obtain response functions of the second plurality of physical pixels .

In this way, this specific example can improve the color and white balance effect of image sensing by calculating color data and color temperature data at the same time. In addition, because the reconfigurable characteristics of the spectrum chip can be used, the effects of color and white balance of image sensing can be optimized according to different situations, which can improve the effect of image sensing.

Exemplary spectral recovery method

FIG. 6 illustrates a flowchart of a spectrum recovery method based on a spectrum chip according to an embodiment of the present application.

As shown in FIG. 6 , the spectrum recovery method based on a spectrum chip according to an embodiment of the present application includes the following steps:

Step S310, determining a predetermined pixel point on the spectrum chip based on a predetermined condition. As described above, in the spectrum chip, the predetermined pixel points are used to restore the spectrum, so they should correspond to the above-mentioned structural units, but in the case of one-to-one correspondence between physical pixels and structural units, it is also possible Corresponds to physical pixels as described above.

In addition, as mentioned above, the spectral pixel refers to that the data detected under several structural units can be used to recover a set of spectral data, and these structural units used to recover the spectral pixel, or the physical properties covered (corresponding) thereof. Dynamic adjustment of pixels (preset pixels) is called reconfigurable.

In this embodiment of the present application, reconfigurable refers to dynamically adjusting the preset pixels of each point according to specific requirements or predetermined conditions. In the embodiment of the present application, the above-mentioned specific and requirements or predetermined conditions may include: requirements for spectral resolution, requirements for brightness uniformity derived for improving the accuracy of spectral restoration, and requirements for total brightness thresholds of spectral pixels.

That is, in the spectral recovery method based on a spectral chip according to an embodiment of the present application, determining the predetermined pixel points on the spectral chip includes: determining based on at least one of brightness distribution and spectral resolution requirements and spatial resolution requirements. predetermined pixel points on the spectrum chip.

Specifically, there may be a correlation between spectral resolution requirements and spatial resolution requirements, that is, the higher the spectral resolution, the lower the corresponding spatial resolution. In this embodiment of the present application, the spectral resolution requirement refers to the spectral accuracy that is to be obtained based on the requirement, such as 10 nm accuracy, or 1 nm accuracy. The number of predetermined pixel points included in each spectral pixel can be adjusted according to this precision requirement. In addition, as mentioned above, this requirement and the spatial resolution requirement will affect each other, so it can also be extended to the spatial resolution requirement, that is, reducing the spectral resolution is equivalent to increasing the spatial resolution, and vice versa.

That is, in the spectral chip-based spectral recovery method according to the embodiment of the present application, determining the predetermined pixel point on the spectral chip based on at least one of the luminance distribution and spectral resolution requirements and spatial resolution requirements includes: responding to: The spectral resolution requirement is greater than a first predetermined threshold, determining that the predetermined number of pixels is greater than a first predetermined number; and/or, in response to the spatial resolution requirement being greater than a second predetermined threshold, determining that the number of pixels is less than a second predetermined number. the predetermined pixel point.

The requirement of brightness uniformity means that for each spectral pixel, when the brightness uniformity of the included pixels is relatively good, the spectral recovery effect is better; in order to achieve this requirement, edge detection can be performed according to the image information first, and then select the A physical pixel group with better luminance uniformity is used to restore a spectral pixel, which will be described in further detail below with reference to specific embodiments 1 and 2.

The total brightness threshold requirement of spectral pixels means that when the brightness of the scene is low, the spectral pixels can be restored by increasing the number of predetermined pixel points to improve the spectral accuracy of the corresponding spectral pixels.

Step S320, using the predetermined pixel points on the spectrum chip to perform spectrum recovery.

That is, after a certain number of predetermined pixel points are selected, next, spectral recovery is performed using a spectral recovery algorithm, such as a compressed sensing algorithm. Here, the compressed sensing algorithm relies on a dictionary used to project features. A dictionary learning algorithm can be used to train a suitable dictionary for spectral recovery using known spectra of materials commonly found in nature. Using compressed sensing algorithm, by selecting appropriate measurement bases and dictionaries, high-precision spectral data can be calculated with only a small amount of light intensity data measured by image sensors.

Hereinafter, specific examples of the spectrum recovery method based on the spectrum chip according to the embodiments of the present application will be described in detail.

Example 1

As described above, the predetermined pixel point can be reconstructed based on the luminance distribution, that is, the light intensity distribution. In addition, when the reconfiguration based on the brightness distribution is performed, the brightness of each pixel in the image is first determined, and then the predetermined pixel point is determined based on the brightness of the pixel.

As described above, in this embodiment of the present application, one physical pixel may correspond to a group of structural units, but it may also be that a group of multiple physical pixels corresponds to a group of structural units. Therefore, unlike physical pixels, in the spectral chip-based spectral recovery method according to the embodiment of the present application, a group of structural units corresponding to one or more groups is recovered to recover a set of spectral information, and the units are called "" Spectral Pixels". Further, the solution of the embodiment of the present application can use at least one spectral pixel to restore the image. Here, the predetermined pixel points as described above refer to physical pixels of the image sensor corresponding to a group of structural units.

In addition, in the embodiment of the present application, on the basis that one physical pixel corresponds to a group of structural units, how many physical pixels may be selected as a group of data to restore a spectral pixel. In other embodiments, also It can be implemented as how many structural units are selected as a set of data to restore one spectral pixel, that is, the value of the number of structural units affects the spatial resolution, image signal-to-noise ratio and spectral accuracy of the sensor. In the embodiment of the present application, in a reconfigurable manner, the value of the number of structural units can be dynamically adjusted according to environmental conditions in different situations, so as to obtain an optimal image effect.

For example, in the actual use process, the number of physical pixels in the scale of 2 × 2, 3 × 3, 5 × 5, 10 × 10 can be used as a set of data for processing as required to generate a spectral pixel. It is worth mentioning that the embodiments of the present application take the one-to-one relationship between physical pixels and structural units as an example. In practice, multiple physical pixels may correspond to a group of structural units, and when multiple physical pixels correspond to a group of structural units , can be understood as the reconfiguration of the structural unit, that is, the multiple physical pixels are determined by the structural unit, for example, 2×2 can be understood as a group of the multiple physical pixels constituting a spectral pixel with four structural units.

FIG. 7 illustrates an example of pixel points selected for recovering a spectrum in a spectrum chip-based spectrum restoration method according to an embodiment of the present application. As shown in FIG. 7 , it shows that a 5×5 square matrix with a total of 25 pixels is used for spectral recovery. And, using the reconfigurable spectral restoration algorithm, a specific number can be selected, for example, 10×10 total 100 pixels or 15×15 total 225 pixels can be selected for spectral restoration. Dynamically adjust the size of the pixel matrix used to restore the spectrum according to the local light intensity change of the image, that is, when the light intensity is small, the size of the pixel matrix can be increased to obtain higher spectral restoration accuracy, Conversely also optionally reduce its size. And using the compressed sensing algorithm, on the premise that the spatial light intensity consistency of the pixel points is satisfied, by dynamically selecting the number of pixels used to restore the spectrum, and selecting an appropriate dictionary for compressed sensing restoration, it is also possible to dynamically adjust the spectral frequency. purpose of resolution. For example, FIG. 8 illustrates an example of a reconfigurable manner in a spectral chip-based spectral recovery method according to an embodiment of the present application. As shown in Figure 8, the outermost area can be understood as using 10×10 pixels as a spectral pixel for restoration, the middle area can be understood as 5×5 pixels for restoration, and the innermost area is 3×3 Pixel points are restored, that is, in the embodiment of the present application, the pixel points corresponding to the spectral pixels are dynamically selected according to different application requirements.

Therefore, in the spectrum recovery method based on the spectrum chip according to the embodiment of the present application, determining the predetermined pixel points on the spectrum chip based on the luminance distribution includes: dynamically adjusting the light intensity changes of the local image acquired by the image sensor based on the brightness distribution. The size of the pixel square matrix formed by the predetermined pixel points.

Example 2

In Example 2, the brightness distribution of the pixel points on the spectrum chip can also be determined more precisely by performing edge detection on the image data output by the image sensor. That is, in order to determine the predetermined pixel points on the spectrum chip based on the brightness distribution, the image data output by the image sensor of the spectrum chip is first acquired, and then edge detection is performed on the image data, and the edge detection based on the image data is performed. As a result, predetermined pixel points on the spectrum chip are determined.

Therefore, in the spectral chip-based spectral restoration method according to the embodiment of the present application, determining the predetermined pixel point on the spectral chip based on at least one of luminance distribution and spectral resolution requirements and spatial resolution requirements includes: acquiring the image data output by the image sensor of the spectrum chip; edge detection is performed on the image data; and predetermined pixel points on the spectrum chip are determined based on the edge detection result of the image data.

Hereinafter, each of the above steps will be described in detail.

First, the image data output by the image sensor of the spectrum chip is acquired. That is, as described above, the image sensor of the spectrum chip obtains the differential response to the modulated spectrum after receiving the modulated spectrum, and uses the signal processing circuit layer to process the differential response into image data.

After that, edge detection is performed on the image data. Here, optionally, before performing edge detection, the image data may be preprocessed first.

Specifically, in the spectral chip-based spectral restoration method according to the embodiment of the present application, the preprocessing of the image data includes image equalization and image noise reduction. Since a filter structure is added to the photodetector array, the image data output by the photodetector array will contain noise caused by the filter structure, and it will be darker on the whole. The image is pre-equalized and denoised.

The equalization algorithm can be global histogram equalization or local adaptive histogram equalization. Because the filter structure makes the image darker as a whole, global histogram equalization can generally be used to equalize the image. FIG. 9 illustrates a schematic diagram of image changes before and after image equalization in the spectral chip-based spectral restoration method according to an embodiment of the present application. As shown in Figure 9, the overall brightness of the image after equalization processing increases, the contrast increases, and the details are highlighted.

And, after image equalization, a specific filtering algorithm is used to denoise the image. For example, Gaussian filtering is performed on the equalized image. According to the image resolution and the characteristics of the light filtering structure on the sensor, a filter kernel of an appropriate size can be selected for filtering, such as 5×5, 9×9, 11×11, etc. Taking into account the characteristics of the filtering structure itself, such as periodicity, median filtering can also be performed on the image. Likewise, the size of the filter kernel can be selected according to the actual situation. FIG. 10 illustrates a schematic diagram of an image after noise reduction in the spectral chip-based spectral restoration method according to an embodiment of the present application. As shown in FIG. 10 , it is the result obtained by performing Gaussian filtering with a 9×9 filter kernel and performing median filtering with a 5×5 filter kernel. Here, the purpose of filter noise reduction is to blur the noise brought by the filter structure to the image and reduce its influence on edge detection.

Therefore, in the spectral chip-based spectral restoration method according to the embodiment of the present application, performing edge detection on the image data includes: performing equalization on the image data; performing noise reduction on the equalized image data; and, Edge detection is performed on the denoised image data.

In addition, in the above-mentioned spectral chip-based spectral restoration method, performing noise reduction on the equalized image data includes: selecting a predetermined filter according to the image resolution and/or the characteristics of the filter structure corresponding to the image sensor. A filter kernel of size filters the equalized image data.

After the image is preprocessed, edge detection is performed. The edge detection operator can be Sobel, Laplacian, Scharr, Canny, etc., and is used to detect the edge area in the image. Optionally, after edge detection, an appropriate expansion operation may be performed on the edge to expand the area occupied by the edge. FIG. 11 illustrates an edge detection result obtained by using the Canny edge detection algorithm in the spectrum chip-based spectrum recovery method according to an embodiment of the present application.

Therefore, in the spectral chip-based spectral restoration method according to the embodiment of the present application, performing edge detection on the denoised image data includes: using an edge detection operator to detect edge regions in the denoised image data ; and, performing an expansion operation on the edge region.

As mentioned above, when performing spectral recovery, since the spectral recovery algorithm requires the light intensity of the selected pixels (measured values) to have a certain consistency, the accuracy of spectral recovery is affected by the number of selected pixels and the consistency of light intensity. Influence, too many pixels will reduce the uniformity of light intensity, too few will cause the amount of data required for recovery to be too undetermined. The detected edges and their neighborhoods are areas with large image gradients, that is, areas with poor light intensity consistency, which should be avoided when selecting points for spectral recovery. When selecting points, for example, the following steps can be taken: first, select the center point for restoring the spectral region, and then, based on the center point, one of "connected", "nearest neighbor", "specific number" or Combine the points to calculate the spectrum. “Connected” means that the selected points form a connected area in the image, and its connectivity can be four-neighborhood connectivity or eight-neighborhood connectivity, and the selected points need to avoid edge areas. "Nearest" means that the selected point has the smallest average distance from the center point, and the distance can be Euclidean distance. "Specific number" refers to selecting a specified number of points for spectral recovery, such as 25, 100, etc. The number of selected points will affect the accuracy and frequency resolution of spectral recovery. In order to achieve higher computational efficiency, when selecting points, the greedy algorithm and the breadth-first search algorithm can be used to quickly select the required number of points. The ultimate purpose of point selection is to ensure the consistency of the light intensity of the pixels used to restore the spectrum.

Therefore, in the spectrum recovery method based on the spectrum chip according to the embodiment of the present application, determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data includes: based on the edge detection result of the image data, determining The predetermined pixel points on the spectrum chip having a light intensity consistency higher than a predetermined threshold value are a predetermined number of adjacent and connected pixel points based on the pixel center point.

In addition, in the spectrum recovery method based on the spectrum chip according to the embodiment of the present application, determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data includes: based on the edge detection result of the image data, determining There are multiple groups of predetermined pixel points with different numbers on the spectrum chip.

In the spectrum recovery method of the spectrum chip, the plurality of groups of predetermined pixel points with different numbers include 3×3, 5×5, and 10×10 physical pixel arrays.

In addition, in Example 2, by preprocessing the image data of the spectrum chip, the edge of the photodetector-level sensitivity in the image data of the spectrum chip is determined, and the edge portion is dynamically adjusted for the recovery filter set. The edge detection operator is used to detect the edge of the image, avoid areas with large pixel gradients in the image, and ensure the consistency of the pixel light intensity used to restore the spectrum. When selecting pixels for spectral restoration, avoid the edge areas in the image, and select the nearest, connected, and specific number of pixels for spectral restoration. Here, the significance of the spectral restoration reconfigurable algorithm is that the position of the pixel points used to restore the spectrum can be dynamically selected. Spectrum is not distorted. FIG. 12 illustrates an example of using edge information to select pixel points for spectrum restoration in the spectrum chip-based spectrum restoration method according to an embodiment of the present application. FIGS. 13A to 13C illustrate examples of pixel points of different shapes selected using edge information for restoring the spectrum in the example shown in FIG. 12 .

Therefore, in the spectrum recovery method based on a spectrum chip according to an embodiment of the present application, the predetermined pixel point is a plurality of physical pixels of a predetermined shape on the image sensor.

Of course, those skilled in the art can understand that the relationship between the above example 1 and example 2 is juxtaposed, and they can also be used in combination.

Example three

In Example 3, in the spectrum recovery method based on the spectrum chip according to the embodiment of the present application, the purpose of the spectrum recovery algorithm is to obtain a spectrum image. The reconfigurability of the spectral recovery algorithm also lies in the fact that the step size when recovering the spectrum can be dynamically adjusted. For example, if a 3×3 pixel square matrix is used for spectral restoration, and the step size during restoration is also 3, the 9×9 pixel data can finally be restored to obtain a spectral image with a spatial resolution of 3×3. FIG. 14 illustrates an example of spectral recovery using a step size of 3 in the spectral chip-based spectral recovery method according to an embodiment of the present application.

In order to improve the spatial resolution of the spectral image, the step size when restoring the spectrum can be dynamically adjusted, for example, adjusted to 1. At this time, if a 3×3 pixel square matrix is used for spectral restoration, the step size is 1 for offset, and then 3×3 pixels are used for spectral restoration. The 9×9 pixel data can be restored to obtain a spatial resolution of 7×7. spectral image. Using the feature that the step size of the spectral restoration algorithm can be reconstructed, the spatial resolution of the restored spectral image can be dynamically adjusted. By improving the local spatial resolution of the image, the effect of local amplification of the spectral image can also be achieved. FIG. 15 illustrates an example of spectral recovery using a step size of 1 in the spectral chip-based spectral recovery method according to an embodiment of the present application.

Therefore, in the spectrum recovery method based on a spectrum chip according to an embodiment of the present application, performing spectrum recovery using the predetermined pixel points on the spectrum chip includes: using the predetermined pixel points on the spectrum chip to dynamically adjust recovery step size for spectral recovery.

Example four

In the fourth embodiment, for the spectrum reconstruction algorithm, the reconfiguration is also reflected in the restored spectrum, and the wavelength sampling interval can be dynamically adjusted for different application scenarios. For scenarios with low spectral resolution requirements, the wavelength sampling interval can be larger, such as 5 nm as the sampling interval, and there are a total of 61 wavelength sampling points from 450 to 750 nm, and spectral reconstruction can be performed with fewer units, such as 10 units. For scenarios with high spectral resolution requirements, the wavelength sampling interval can be small, for example, 0.5 nm is the sampling interval, and there are 301 wavelength sampling points from 450 nm to 750 nm, and more units can be used for spectral reconstruction, such as 50. Therefore, the wavelength sampling interval can be dynamically adjusted according to the application scenario to achieve a balance between the spatial resolution and the spectral resolution at the same time. Specifically, it can be understood that according to the application scenario, the number in the vector Y is adjusted, and if the spectral resolution requirement is higher, the number in the Y is also larger.

Therefore, in the spectrum recovery method based on the spectrum chip according to the embodiment of the present application, performing spectrum recovery using the predetermined pixel points on the spectrum chip includes: dynamically adjusting the spectrum for the spectrum to be recovered based on the application of the spectrum to be recovered. The wavelength sampling interval for the spectral recovery described above.

In addition, in the spectrum recovery method of the spectrum chip, determining the predetermined pixel points on the spectrum chip based on the edge detection result of the image data includes: determining the number of predetermined pixel points on the spectrum chip based on the wavelength sampling interval.

That is, similar to the above, the determination of the number of predetermined pixels may be affected by two factors. One is the uniformity of brightness. Generally speaking, when the brightness uniformity of predetermined pixels is relatively good, the spectral recovery It is more accurate, otherwise the error is relatively large, which needs to be satisfied first. The edge detection mentioned above is also for this purpose. It is also related to the brightness that needs to reach the threshold. When the brightness is generally low, you can increase the predetermined pixel. The number of points to achieve the threshold; the second is the spectral resolution, which is determined by demand. Generally speaking, the more predetermined pixel points are, the higher the spectral resolution and the lower the spatial resolution, so it can be adjusted by adjusting the number. Adjust spectral resolution (as well as spatial resolution).

To sum up, the spectrum recovery method based on the spectrum chip according to the embodiment of the present application can perform effective identification and dynamic adjustment according to the information of the image imaged by the image sensor of the spectrum chip by not fixing the structure selected for the recovery spectrum.

In this way, the spectral chip-based spectral restoration method according to the embodiment of the present application can flexibly adjust the number and spatial distribution of physical pixels corresponding to each spectral pixel according to the image data output by the image sensor, thereby reducing spatial light intensity unevenness Recovery error introduced by sex.

In addition, the spectral chip-based spectral recovery method according to the embodiment of the present application can improve the spatial resolution of spectral imaging by adjusting the number and spatial distribution of physical pixels corresponding to each spectral pixel, and improve the effect to filter-based The theoretical limit of chip and photodetection array schemes.

Furthermore, the spectral chip-based spectral restoration method according to the embodiment of the present application improves the edge signal-to-noise ratio of the spectral image by performing edge detection on the image data, and can be advantageously applied to edge detection and substance identification based on the spectral image.

Example five

As mentioned above, the reconfiguration of predetermined pixel points can be performed with spectral resolution or spatial resolution requirements.

For example, in actual use, the number of physical pixels of 2×2, 3×3, 5×5, 10×10 and other scales can be used as a set of data for processing according to the spectral resolution or spatial resolution to generate a spectrum pixel. It is worth mentioning that the embodiments of the present application take the one-to-one relationship between physical pixels and structural units as an example. In practice, multiple physical pixels may correspond to a group of structural units, and when multiple physical pixels correspond to a group of structural units , can be understood as the reconfiguration of the structural unit, that is, the multiple physical pixels are determined by the structural unit, for example, 2×2 can be understood as a group of the multiple physical pixels constituting a spectral pixel with four structural units.

FIG. 8 illustrates an example of pixel points selected for spectrum restoration in the spectrum chip-based spectrum restoration method according to an embodiment of the present application. As shown in FIG. 8 , it shows that a 5×5 square matrix with a total of 25 pixels is used for spectral recovery. In addition, using the reconfigurable spectral recovery algorithm, a specific number can be selected. For example, according to the spectral resolution requirements, 10×10 total 100 pixels can be selected for multispectral imaging, or 15×15 total 225 pixels can be used for multispectral imaging. Perform hyperspectral imaging.

That is, in the embodiment of the present application, the reconfiguration of the spectral pixels may be performed only based on the spectral resolution or spatial resolution requirements irrespective of the brightness.

Exemplary device

16 illustrates a block diagram of an image sensing device based on a spectrum chip according to an embodiment of the present application, according to an embodiment of the present application.

As shown in FIG. 16 , an image sensing device 400 based on a spectrum chip according to an embodiment of the present application includes: a parameter determination unit 410 for determining a first image parameter to be sensed; a pixel determination unit 420 for The first image parameter to be sensed determines a first group of multiple physical pixels of the image sensor of the spectrum chip; the data acquisition unit 430 is configured to determine the first response function of the first group of multiple physical pixels and measure all the physical pixels. the light intensity reading of each physical pixel in the first group of multiple physical pixels; and, a parameter calculation unit 440, configured to calculate the first image parameter to be sensed based on the light intensity reading and the response function .

In an example, in the above-mentioned spectral chip-based image sensing device 400, the matrix determining unit 420 is configured to: determine the image sensor according to at least one of the spatial resolution, the image signal-to-noise ratio and the spectral accuracy of the image sensor. The first group of multiple physical pixels is described.

In one example, in the above-mentioned image sensing device 400 based on a spectrum chip, the first plurality of physical pixels are 2×2, 3×3, 5×5, 10×10 physical pixels on the image sensor At least one of the pixel squares.

In one example, in the above-mentioned spectral chip-based image sensing device 400, the first group of multiple physical pixels are multiple physical pixels of a predetermined shape on the image sensor.

In one example, in the above-mentioned spectral chip-based image sensing device 400, the matrix determining unit 420 determines the first The grouping of the plurality of physical pixels includes: determining a first number of physical pixel square arrays in response to the intensity of the ambient light being a first intensity; and determining a first number of physical pixels in response to the intensity of the ambient light being a second intensity less than the first intensity A square matrix of physical pixels of two numbers, the second number is greater than the first number.

In one example, in the above-mentioned spectral chip-based image sensing device 400, the matrix determining unit 420 determines the first The grouping of the plurality of physical pixels includes: determining a first number of physical pixel squares in response to a first region of a first brightness or a first signal-to-noise ratio in an image acquired by the image sensor; and, in response to the image sensor In the acquired image, in a second region with a second brightness smaller than the first brightness or a second signal-to-noise ratio smaller than the first signal-to-noise ratio, determine a second number of physical pixel square matrices, the second number being greater than the first number .

In one example, in the above-mentioned spectral chip-based image sensing device 400, the matrix determining unit 420 determines the first The grouping of the plurality of physical pixels includes: determining a first number of physical pixel squares in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and, in response to the image sensor being smaller than the first spatial resolution A second spatial resolution of the resolution and/or a second spectral accuracy greater than the first spectral accuracy determines a second number of physical pixel squares, the second number being greater than the first number.

In one example, in the above-mentioned image sensing device 400 based on a spectral chip, at least one first group of multiple physical pixels is used to restore an image.

In an example, in the above-mentioned spectral chip-based image sensing device 400, the parameter determining unit 410 is further configured to determine the second image parameter to be sensed; the pixel determining unit 420 is further configured to determine the second image parameter to be sensed based on the The sensed second image parameter determines a second group of multiple physical pixels of the image sensor of the spectrum chip; the data acquisition unit 430 is further configured to determine a second response function of the second group of multiple physical pixels and determine The light intensity reading of each physical pixel in the second group of multiple physical pixels; and, the parameter calculation unit 440 is further configured to calculate the to-be-sensed first based on the light intensity reading and the response function. Two image parameters.

In one example, in the above-mentioned spectral chip-based image sensing device 400, the light intensity measurement unit 430 is configured to: based on the correspondence between the second group of physical pixels and the first group of physical pixels relationship, and obtain the light intensity readings of the physical pixels corresponding to the first group of the plurality of physical pixels in the second group of the plurality of physical pixels.

In one example, in the above-mentioned spectral chip-based image sensing device 400, the first image parameter is color data, and the second image parameter is color temperature data.

In one example, in the above-mentioned image sensing device 400 based on a spectral chip, the first group of multiple physical pixels is a square matrix of physical pixels of a first number, and the second group of multiple physical pixels is a second number of physical pixels The physical pixel square matrix of , the second number is greater than the first number.

In an example, in the above-mentioned image sensing device 400 based on a spectrum chip, the matrix determining unit 420 is configured to: determine the first image sensor of the image sensor of the color temperature data to be sensed based on the color temperature data to be sensed an area and a second area; and, determining response functions of a plurality of physical pixels of the first area and the second area, respectively, to obtain response functions of the second plurality of physical pixels.

In one example, in the above-mentioned spectral chip-based image sensing device 400 , the spectral chip is a spectral chip used for computing a spectral device that receives light in a wavelength band of 350 to 1000 nanometers.

Here, those skilled in the art can understand that the specific functions and operations of each unit and module in the above-mentioned spectral chip-based image sensing device 400 have been described in the spectral chip-based image sensing method described above with reference to FIGS. 1 to 5 . It is introduced in detail, and thus, its repeated description will be omitted.

FIG. 17 illustrates a block diagram of a spectral chip-based spectral recovery apparatus according to an embodiment of the present application.

As shown in FIG. 17 , the spectrum recovery device 500 based on a spectrum chip according to an embodiment of the present application includes: a pixel determination unit 510, configured to determine a predetermined pixel point on the spectrum chip based on a predetermined condition; and, a spectrum recovery unit 520, for spectrum recovery using the predetermined pixel points on the spectrum chip.

In an example, in the above-mentioned spectral chip-based spectral restoration device 500, the pixel determination unit 510 is configured to: determine the pixel on the spectral chip based on at least one of luminance distribution and spectral resolution requirements and spatial resolution requirements. predetermined pixels.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 determines predetermined pixel points on the spectral chip based on luminance distribution and at least one of spectral resolution requirements and spatial resolution requirements The method includes: acquiring image data output by an image sensor of the spectrum chip; performing edge detection on the image data; and determining predetermined pixel points on the spectrum chip based on the edge detection result of the image data.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 performs edge detection on the image data includes: equalizing the image data; denoising; and, performing edge detection on the denoised image data.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 performs noise reduction on the equalized image data including: according to the image resolution and/or the filter corresponding to the image sensor According to the characteristics of the optical structure, a filter kernel of a predetermined size is selected to filter the equalized image data.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 performs edge detection on the denoised image data including: detecting the denoised image by using an edge detection operator an edge region in the data; and, performing a dilation operation on the edge region.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 determining a predetermined pixel point on the spectral chip based on the edge detection result of the image data includes: based on the image data Based on the edge detection result, a predetermined pixel point on the spectrum chip with light intensity consistency higher than a predetermined threshold is determined, and the predetermined pixel point is a predetermined number of adjacent and connected pixel points based on the pixel center point.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 determining a predetermined pixel point on the spectral chip based on the edge detection result of the image data includes: based on the image data Based on the edge detection result, multiple groups of predetermined pixel points with different numbers on the spectrum chip are determined.

In one example, in the above-mentioned spectral chip-based spectral restoration device 500, the plurality of groups of predetermined pixel points with different numbers include 2×2, 3×3, 5×5, and 10×10 physical pixel arrays.

In one example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the predetermined pixel points are a plurality of physical pixels of a predetermined shape on the image sensor.

In an example, in the above-mentioned spectral chip-based spectral restoration apparatus 500, the pixel determination unit 510 determines predetermined pixel points on the spectral chip based on luminance distribution and at least one of spectral resolution requirements and spatial resolution requirements comprising: in response to the spectral resolution requirement being greater than a first predetermined threshold, determining that the predetermined number of pixel points is greater than a first predetermined number; and/or, in response to the spatial resolution requirement being greater than a second predetermined threshold, determining that the number is smaller than the first predetermined number of pixels. Two predetermined number of said predetermined pixel points.

In an example, in the above spectrum chip-based spectrum restoration apparatus 500, the spectrum restoration unit 520 is configured to perform spectrum restoration with a dynamically adjusted restoration step size using the predetermined pixel points on the spectrum chip.

In an example, in the above-mentioned spectrum chip-based spectrum restoration apparatus 500, the spectrum restoration unit 520 is configured to: dynamically adjust the wavelength sampling interval used for the spectrum restoration based on the application of the spectrum to be restored.

In an example, in the above-mentioned spectrum chip-based spectrum recovery apparatus 500, the pixel determination unit 510 is configured to: determine the number of predetermined pixel points on the spectrum chip based on a wavelength sampling interval.

In an example, in the above-mentioned spectrum chip-based spectrum recovery apparatus 500, the spectrum chip is a spectrum chip used for calculating a spectrometer that receives light in a wavelength band of 350 to 1000 nanometers

Here, those skilled in the art can understand that the specific functions and operations of each unit and module in the above-mentioned spectral chip-based spectral recovery apparatus 500 have been described in detail in the spectral chip-based spectral recovery method described above with reference to FIGS. 6 to 15 . , and therefore, a repeated description thereof will be omitted.

As described above, the spectral chip-based image sensing apparatus 400 and the spectral recovery apparatus 500 according to the embodiments of the present application may be implemented in various terminal devices, such as spectrometers, or set in a server in the cloud. In one example, the spectral chip-based image sensing apparatus 400 and the spectral recovery apparatus 500 according to the embodiments of the present application may be integrated into the terminal device as a software module and/or a hardware module. For example, the spectrum chip-based image sensing device 400 and the spectrum recovery device 500 may be a software module in the operating system of the terminal device, or may be an application program developed for the terminal device; The image sensing device 400 and the spectrum recovery device 500 of the spectrum chip can also be one of many hardware modules of the terminal device.

Alternatively, in another example, the spectrum chip-based image sensing apparatus 400 and the spectrum restoration apparatus 500 and the terminal device may also be separate devices, and the spectrum chip-based image sensing apparatus 400 and the spectrum restoration apparatus 500 may be connected to the terminal device through a wired and/or wireless network, and transmit interaction information according to an agreed data format.

Exemplary Electronics

Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to FIG. 18 .

As shown in FIG. 18 , the electronic device 10 includes one or more processors 11 and a memory 12 .

Processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the spectral chip-based image sensing of the various embodiments of the present application described above Methods and spectral recovery methods and/or other desired functions. Various contents such as light intensity data, corresponding parameters and the like may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may also include an input device 13 and an output device 14 interconnected by a bus system and/or other form of connection mechanism (not shown).

For example, the input device 13 may be, for example, a keyboard, a mouse, or the like.

The output device 14 can output various information, such as image sensing parameters and spectral restoration results, to the outside. The output devices 14 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 10 related to the present application are shown in FIG. 18 , and components such as buses, input/output interfaces and the like are omitted. Besides, the electronic device 10 may also include any other suitable components according to the specific application.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatuses described above, embodiments of the present application may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary methods" described above in this specification The steps in the spectral chip-based image sensing method and the spectral recovery method according to various embodiments of the present application described in the section.

The computer program product can write program codes for performing the operations of the embodiments of the present application in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present application may also be computer-readable storage media having computer program instructions stored thereon, the computer program instructions, when executed by a processor, cause the processor to perform the above-mentioned "Example Method" section of this specification Steps in a spectral chip-based image sensing method and a spectral recovery method according to various embodiments of the present application described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present application are only examples rather than limitations, and these advantages, advantages, effects, etc., are not considered to be Required for each embodiment of this application. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, rather than limiting, and the above-mentioned details do not limit the application to be implemented by using the above-mentioned specific details.

The block diagrams of devices, apparatus, apparatuses, and systems referred to in this application are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

It should also be pointed out that in the apparatus, equipment and method of the present application, each component or each step can be decomposed and/or recombined. These disaggregations and/or recombinations should be considered as equivalents of the present application.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Therefore, this application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

An image sensing method based on a spectrum chip, comprising:

determining the first image parameter to be sensed;

determining a first plurality of physical pixels of the image sensor of the spectrum chip based on the first image parameter to be sensed;

determining a response function of the first plurality of physical pixels and measuring light intensity readings for each of the first plurality of physical pixels; and

The first image parameter to be sensed is calculated based on the light intensity reading and the response function.
The image sensing method based on a spectrum chip according to claim 1, wherein a first group of a plurality of physical pixels of the spectrum chip used for image sensing is determined based on the first image parameter to be sensed include:

The first plurality of physical pixels is determined based on at least one of a spatial resolution of the image sensor, an image signal-to-noise ratio, and spectral accuracy.
The image sensing method based on a spectrum chip according to claim 2, wherein the first group of multiple physical pixels is 2×2, 3×3, 5×5, 10× on the image sensor At least one of the 10 physical pixel squares.
The image sensing method based on a spectrum chip according to claim 2, wherein the first group of multiple physical pixels are multiple physical pixels of a predetermined shape on the image sensor.
The image sensing method based on a spectrum chip according to claim 2, wherein the first set of multiple physical properties is determined according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor. Pixels include:

determining a first number of physical pixel squares in response to the intensity of the ambient light being the first intensity; and

In response to the intensity of the ambient light being a second intensity less than the first intensity, a second number of physical pixel squares is determined, the second number being greater than the first number.
The image sensing method based on a spectrum chip according to claim 2, wherein the first set of multiple physical properties is determined according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor. Pixels include:

determining a first number of physical pixel squares in response to a first brightness or a first region of a first signal-to-noise ratio in an image acquired by the image sensor; and

In response to a second area of a second brightness smaller than the first brightness or a second signal-to-noise ratio less than the first signal-to-noise ratio in the image acquired by the image sensor, determining a second number of physical pixel square matrices, the first The second number is greater than the first number.
The image sensing method based on a spectrum chip according to claim 2, wherein the first set of multiple physical properties is determined according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor. Pixels include:

determining a first number of physical pixel squares in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and

In response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy greater than the first spectral accuracy, determining a second number of physical pixel squares, the second number being greater than the first a number.
The image sensing method based on a spectrum chip according to claim 1, wherein at least one of the first group of multiple physical pixels is used to restore an image.
The image sensing method based on a spectrum chip according to claim 1, further comprising:

determining the second image parameter to be sensed;

determining a second plurality of physical pixels of the image sensor of the spectrum chip based on the second image parameter to be sensed;

determining a response function for the second plurality of physical pixels and determining a light intensity reading for each physical pixel in the second plurality of physical pixels; and

The second image parameter to be sensed is calculated based on the light intensity reading and the response function.
The spectral chip-based image sensing method according to claim 9, wherein determining the light intensity reading of each physical pixel in the second group of multiple physical pixels comprises:

Based on the correspondence between the plurality of physical pixels in the second group and the plurality of physical pixels in the first group, acquiring the physical pixels of the plurality of physical pixels in the second group corresponding to the physical pixels in the first group Light intensity reading.
The image sensing method based on a spectrum chip according to claim 10, wherein the first image parameter is color data, and the second image parameter is color temperature data.
The image sensing method based on a spectrum chip according to claim 11, wherein the first group of physical pixels is a square matrix of physical pixels of a first number, and the second group of physical pixels is a first number of physical pixels. A square matrix of physical pixels of two numbers, the second number is greater than the first number.
The image sensing method based on a spectrum chip according to claim 11, wherein determining the response function of the second group of multiple physical pixels comprises:

determining, based on the color temperature data to be sensed, a first area and a second area of the image sensor for which color temperature data to be sensed; and

The response functions of the plurality of physical pixels of the first region and the second region are determined respectively to obtain the response functions of the second plurality of physical pixels.
The image sensing method based on a spectrum chip according to any one of claims 1 to 13, wherein the spectrum chip is a spectrum chip for receiving light in a wavelength band of 350 to 1000 nanometers of a computing spectrum device.
An image sensing device based on a spectrum chip, characterized in that it includes:

a parameter determination unit, configured to determine the first image parameter to be sensed;

a pixel determination unit, configured to determine a first group of multiple physical pixels of the image sensor of the spectrum chip based on the first image parameter to be sensed;

a data acquisition unit for determining a response function of the first plurality of physical pixels and measuring light intensity readings for each of the first plurality of physical pixels; and

A parameter calculation unit, configured to calculate the first image parameter to be sensed based on the light intensity reading and the response function.
A spectral recovery method based on a spectral chip, comprising:

determining predetermined pixel points on the spectrum chip based on predetermined conditions; and

Spectral recovery is performed using the predetermined pixel points on the spectrum chip.
The spectrum recovery method based on a spectrum chip according to claim 16, wherein determining the predetermined pixel points on the spectrum chip comprises:

The predetermined pixel points on the spectral chip are determined based on the luminance distribution and at least one of spectral resolution requirements and spatial resolution requirements.
The spectrum recovery method based on a spectrum chip according to claim 17, wherein determining the predetermined pixel points on the spectrum chip based on at least one of luminance distribution and spectral resolution requirements and spatial resolution requirements comprises:

acquiring image data output by an image sensor of the spectrum chip;

performing edge detection on the image data; and

A predetermined pixel point on the spectrum chip is determined based on the edge detection result of the image data.
The spectrum recovery method based on a spectrum chip according to claim 18, wherein performing edge detection on the image data comprises:

equalizing the image data;

denoising the equalized image data; and

Edge detection is performed on the denoised image data.
The spectrum recovery method based on a spectrum chip according to claim 19, wherein performing noise reduction on the equalized image data comprises:

According to the image resolution and/or the characteristics of the filter structure corresponding to the image sensor, a filter kernel of a predetermined size is selected to filter the equalized image data.
The spectrum recovery method based on a spectrum chip according to claim 19, wherein performing edge detection on the denoised image data comprises:

detecting edge regions in the denoised image data using an edge detection operator; and

A dilation operation is performed on the edge region.
The spectrum recovery method based on a spectrum chip according to claim 18, wherein determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data comprises:

Based on the edge detection result of the image data, a predetermined pixel point on the spectrum chip having a light intensity consistency higher than a predetermined threshold is determined, and the predetermined pixel point is based on a predetermined number of adjacent and connected pixel center points. pixel.
The spectrum recovery method based on a spectrum chip according to claim 22, wherein determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data comprises:

Based on the edge detection result of the image data, a plurality of groups of predetermined pixel points with different numbers are determined on the spectrum chip.
The spectrum recovery method based on a spectrum chip according to claim 17 or 23, wherein the plurality of groups of predetermined pixel points with different numbers include 2×2, 3×3, 5×5, 10×10 physical pixel array.
The spectrum recovery method based on a spectrum chip according to claim 22, wherein the predetermined pixel point is a plurality of physical pixels of a predetermined shape on the image sensor.
The spectrum recovery method based on a spectrum chip according to claim 17, wherein determining the predetermined pixel points on the spectrum chip based on at least one of luminance distribution and spectral resolution requirements and spatial resolution requirements comprises:

in response to the spectral resolution requirement being greater than a first predetermined threshold, determining that the predetermined number of pixels is greater than a first predetermined number; and/or

In response to the spatial resolution requirement being greater than a second predetermined threshold, determining that the predetermined number of pixels is less than a second predetermined number.
The spectrum recovery method based on a spectrum chip according to claim 16, wherein the performing spectrum recovery using the predetermined pixel points on the spectrum chip comprises:

Spectral recovery is performed with a dynamically adjusted recovery step size using the predetermined pixel points on the spectroscopic chip.
The spectrum recovery method based on a spectrum chip according to claim 16, wherein the performing spectrum recovery using the predetermined pixel points on the spectrum chip comprises:

Based on the application of the spectrum to be recovered, the wavelength sampling interval used for the spectrum recovery is dynamically adjusted.
The spectrum recovery method based on a spectrum chip according to claim 28, wherein determining the predetermined pixel point on the spectrum chip based on the edge detection result of the image data comprises:

The number of predetermined pixel points on the spectrum chip is determined based on the wavelength sampling interval.
The spectrum recovery method based on a spectrum chip according to any one of claims 16 to 29, wherein the spectrum chip is a spectrum chip used for calculating a spectrometer that receives light in a wavelength band of 350 to 1000 nanometers.
A spectrum recovery device based on a spectrum chip, characterized in that it comprises:

a pixel determination unit, configured to determine a predetermined pixel point on the spectrum chip based on a predetermined condition; and

A spectral recovery unit, configured to perform spectral recovery using the predetermined pixel points on the spectral chip.
An electronic device, comprising:

processor; and

A memory in which computer program instructions are stored, the computer program instructions causing the processor to perform the spectroscopic chip-based image transmission according to any one of claims 1-14 when the processor is run. The sensor method or the spectrum recovery method based on a spectrum chip according to any one of claims 16-30.
A computer-readable storage medium, characterized in that the computer-readable storage medium has computer program instructions stored thereon, and when the computer program instructions are executed by a computing device, it is operable to execute any one of claims 1-14. The image sensing method based on a spectrum chip or the spectrum recovery method based on a spectrum chip according to any one of claims 16-30.