CN115086581B - Image sensing method and device based on spectrum chip - Google Patents
Image sensing method and device based on spectrum chip Download PDFInfo
- Publication number
- CN115086581B CN115086581B CN202110281439.6A CN202110281439A CN115086581B CN 115086581 B CN115086581 B CN 115086581B CN 202110281439 A CN202110281439 A CN 202110281439A CN 115086581 B CN115086581 B CN 115086581B
- Authority
- CN
- China
- Prior art keywords
- image
- physical
- determining
- physical pixels
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001228 spectrum Methods 0.000 title claims abstract description 139
- 238000000034 method Methods 0.000 title claims abstract description 95
- 230000003595 spectral effect Effects 0.000 claims abstract description 78
- 238000005316 response function Methods 0.000 claims abstract description 51
- 230000004044 response Effects 0.000 claims description 38
- 239000011159 matrix material Substances 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 25
- 238000012545 processing Methods 0.000 description 16
- 238000001914 filtration Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 230000003287 optical effect Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 9
- 230000009467 reduction Effects 0.000 description 9
- 230000008901 benefit Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000003491 array Methods 0.000 description 6
- 238000011084 recovery Methods 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 238000009529 body temperature measurement Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000007613 environmental effect Effects 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000000411 transmission spectrum Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004611 spectroscopical analysis Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- BJQHLKABXJIVAM-UHFFFAOYSA-N bis(2-ethylhexyl) phthalate Chemical compound CCCCC(CC)COC(=O)C1=CC=CC=C1C(=O)OCC(CC)CCCC BJQHLKABXJIVAM-UHFFFAOYSA-N 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000002107 nanodisc Substances 0.000 description 1
- 239000002061 nanopillar Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000004038 photonic crystal Substances 0.000 description 1
- 239000002096 quantum dot Substances 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Abstract
The application relates to an image sensing method and device based on a spectrum chip and electronic equipment. The image sensing method based on the spectrum chip comprises the following steps: determining a first image parameter to be sensed; determining a first plurality of physical pixels of an image sensor of the spectral chip based on the first image parameter to be sensed; determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and calculating the first image parameter to be sensed based on the light intensity reading and the response function. In this way, the response function of the physical pixel corresponding to the image parameter to be sensed can be determined, so that the image parameter can be calculated based on the light intensity reading of the physical pixel, and the image sensing effect is improved.
Description
Technical Field
The present disclosure relates to the field of spectrum sensing technologies, and in particular, to an image sensing method, an image sensing device, and an electronic device based on a spectrum chip.
Background
In the field of image sensing, demands on image signal-to-noise ratio, color effect, white balance and the like are gradually increasing at present. Specifically, in terms of the image signal-to-noise ratio, as the pixel points become smaller, the amount of light entering a single pixel decreases; in addition, the current image sensor generally adopts a bayer filter array to realize a color effect, and the bayer filter array further greatly reduces the light incoming quantity, thereby affecting the signal-to-noise ratio of the image. This problem is particularly pronounced in certain applications, such as in dim light environments. For this reason, new filter arrangement schemes such as RYYB, RGBW, etc. have been developed in recent years. Although these schemes improve the light intake to some extent, the principle is similar to that of the bayer filter array, and the improvement effect is very limited.
In addition, in the calculation process, for example, in terms of color effect, the Bayer filter adopts the frequency spectrum response of the RGB three-color filter to simulate the stimulus response of human eyes, and the overall color effect of the photo is restored by a mosaic interpolation mode. Errors may exist in multiple links in the whole process, thereby causing errors in color reproduction. In terms of white balance, the adjustment of white balance, which is low in cost, can be performed based on the acquired RGB information. With the improvement of the white balance requirement of the application end, a plurality of terminals are provided with special color temperature sensors to measure the ambient color temperature so as to achieve a good white balance effect, thereby improving the cost of the whole scheme and increasing the complexity of the system.
In recent years, with the development of micro-nano technology, quantum technology, material science and semiconductor technology, new research directions for improving image sensing are being created. For example, the spectrum chip adopts a wide spectrum filter or a filtering structure to filter light, and then the spectrum is restored through different data processing modes after the sensor signal is acquired.
It is therefore desirable to provide an improved spectrum chip based image sensing scheme.
Disclosure of Invention
The present application has been made in order to solve the above technical problems. The embodiment of the application provides an image sensing method, an image sensing device and electronic equipment based on a spectrum chip, which can calculate an image parameter based on the light intensity reading of a physical pixel by determining the response function of the physical pixel corresponding to the image parameter to be sensed, so that the image sensing effect is improved.
According to an aspect of the present application, there is provided an image sensing method based on a spectrum chip, including: determining a first image parameter to be sensed; determining a first plurality of physical pixels of an image sensor of the spectral chip based on the first image parameter to be sensed; determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and calculating the first image parameter to be sensed based on the light intensity reading and the response function.
In the above-described spectrum chip-based image sensing method, determining a first plurality of physical pixels of the spectrum chip for image sensing based on the first image parameter to be sensed includes: the first plurality of physical pixels is determined based on at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor.
In the above-mentioned spectrum chip-based image sensing method, the first plurality of physical pixels is at least one of a 2×2, 3×3, 5×5, 10×10 physical pixel matrix on the image sensor.
In the above-described spectrum chip-based image sensing method, the first plurality of physical pixels are a plurality of physical pixels of a predetermined shape on the image sensor.
In the above-mentioned image sensing method based on a spectrum chip, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to the intensity of the ambient light being a first intensity; and responsive to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel matrices, the second number being greater than the first number.
In the above-mentioned image sensing method based on a spectrum chip, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first region of a first luminance or a first signal-to-noise ratio in an image acquired by the image sensor; and determining a second number of physical pixel arrays in response to a second area of a second signal-to-noise ratio in the image acquired by the image sensor that is less than the first brightness or less than the first signal-to-noise ratio, the second number being greater than the first number.
In the above-mentioned image sensing method based on a spectrum chip, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and determining a second number of physical pixel matrices in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy that is greater than the first spectral accuracy, the second number being greater than the first number.
In the above image sensing method based on a spectrum chip, at least one of the first plurality of physical pixels is used for restoring an image.
In the above image sensing method based on a spectrum chip, further comprising: determining a second image parameter to be sensed; determining a second plurality of physical pixels of an image sensor of the spectral chip based on the second image parameter to be sensed; determining a response function of the second plurality of physical pixels and determining a light intensity reading for each physical pixel in the second plurality of physical pixels; and calculating the second image parameter to be sensed based on the light intensity reading and the response function.
In the above-described spectral chip-based image sensing method, determining the light intensity reading for each physical pixel in the second plurality of physical pixels comprises: and acquiring light intensity readings of physical pixels corresponding to the first plurality of physical pixels in the second plurality of physical pixels based on the corresponding relation between the second plurality of physical pixels and the first plurality of physical pixels.
In the above image sensing method based on a spectrum chip, the first image parameter is color data, and the second image parameter is color temperature data.
In the above image sensing method based on a spectrum chip, the first plurality of physical pixels is a first number of physical pixel square arrays, the second plurality of physical pixels is a second number of physical pixel square arrays, and the second number is greater than the first number.
In the above-described spectrum chip-based image sensing method, determining the response function of the second plurality of physical pixels includes: determining a first region and a second region of the image sensor of color temperature data to be sensed based on the color temperature data to be sensed; and determining response functions of a plurality of physical pixels of the first region and the second region, respectively, to obtain response functions of the second plurality of physical pixels.
In the above-described image sensing method based on a spectrum chip, the spectrum chip is a spectrum chip for calculating light of a wavelength band of 350 to 1000 nm received by a spectrum device.
According to another aspect of the present application, there is provided an image sensing device based on a spectrum chip, including: a parameter determining unit for determining a first image parameter to be sensed; a pixel determination unit for determining a first plurality of physical pixels of an image sensor of the spectrum chip based on the first image parameter to be sensed; a data acquisition unit for determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and a parameter calculation unit for calculating the first image parameter to be sensed based on the light intensity reading and the response function.
According to still another aspect of the present application, there is provided an electronic apparatus including: a processor; and a memory having stored therein computer program instructions that, when executed by the processor, cause the processor to perform the spectrum chip based image sensing method as described above.
According to a further aspect of the present application, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a computing device, are operable to perform a spectroscopic chip based image sensing method as described above.
According to the spectrum chip-based image sensing method, the spectrum chip-based image sensing device and the spectrum chip-based electronic equipment, the broad spectrum filtering structure can be utilized, the response function of the physical pixel corresponding to the image parameter to be sensed is determined according to the broad spectrum filtering structure, and the image parameter is calculated based on the light intensity reading of the physical pixel, so that the effects of signal-to-noise ratio, color, white balance and the like of image sensing are improved.
In addition, the image sensing method, the image sensing device and the electronic equipment based on the spectrum chip can utilize the reconfigurable characteristic of the spectrum chip to pertinently optimize the effects of signal to noise ratio, color, white balance and the like of image sensing according to different conditions, so that the image sensing effect is improved.
Drawings
Various other advantages and benefits of the present application will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. It is apparent that the drawings described below are only some embodiments of the present application and that other drawings may be obtained from these drawings by those of ordinary skill in the art without inventive effort. Also, like reference numerals are used to designate like parts throughout the figures.
FIG. 1 illustrates a flowchart of a spectrum chip-based image sensing method according to an embodiment of the present application;
FIG. 2 illustrates an exemplary configuration diagram of a computing spectroscopy apparatus according to embodiments of the present application;
FIG. 3 illustrates a schematic diagram of determining different numbers of physical pixels in a spectral chip-based image sensing method according to an embodiment of the present application;
FIG. 4 illustrates a schematic diagram of a spectral chip-based image sensing method for simultaneous color reduction and color temperature measurement according to an embodiment of the present application;
FIG. 5 illustrates a flowchart of this particular example of a spectral chip-based image sensing method according to an embodiment of the present application;
FIG. 6 illustrates a block diagram of a spectral chip-based image sensing device according to an embodiment of the present application;
fig. 7 illustrates a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application and not all of the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
Exemplary method
Fig. 1 illustrates a flowchart of a spectrum chip-based image sensing method according to an embodiment of the present application.
As shown in fig. 1, the spectrum chip-based image sensing method according to the embodiment of the application includes the following steps:
in step S110, a first image parameter to be sensed is determined.
Here, the spectrum chip according to the embodiment of the present application is generally a spectrum chip applied to a calculation spectrum device. The calculating spectrometer device can be a spectrometer or a spectrum imaging device, taking a spectrometer as an example, and the most remarkable difference between the calculating spectrometer and the traditional spectrometer is the difference of filtering. In a conventional spectrometer, the filter used for wavelength selection is a bandpass filter. The higher the spectral resolution, the narrower and more filters of the passband must be used, which increases the volume and complexity of the overall system. Meanwhile, when the spectral response curve is narrowed, the luminous flux is decreased, resulting in a decrease in signal-to-noise ratio.
For a specific calculation spectrometer, each filter generally adopts a wide-spectrum filter, which makes the raw data detected by the calculation spectrometer system and the raw spectrum greatly different. However, by applying a computational reconstruction algorithm, the original spectrum can be restored by computation. Since broadband filters pass more light than narrowband filters, i.e. less energy is lost to the light, such computational spectrometers can detect spectra from darker scenes. Furthermore, according to the compressed sensing theory, the spectral curve of the filter can be appropriately designed to recover the sparse spectrum with high probability, and the number of filters is much smaller than the desired number of spectral channels (recovering higher-dimensional vectors from lower-dimensional vectors), which is certainly very advantageous for miniaturization. On the other hand, by using a greater number of filters, the regularization algorithm (obtaining a lower-dimensional vector after noise reduction from a higher-dimensional vector) can be used to reduce noise, which increases the signal-to-noise ratio and makes the overall system more robust.
In contrast, when designing a conventional spectrometer, a filter needs to be designed according to a required wavelength, so that light with a specific wavelength can be transmitted (generally, the design is designed to enhance the projection of incident light with the specific wavelength, but not the incident light with the specific wavelength band cannot be projected, the resonance condition can be controlled by changing the period and the diameter of structures such as a nano-disc, and the central wavelength of the incident light which can be enhanced by changing the structure, thereby realizing the filtering characteristic). That is, the conventional spectrometer needs to control the size and position accuracy of the light modulation structure with emphasis in the design process, and needs to improve the transmittance of a specific wavelength. While for a computational spectrometer it is desirable to be able to receive light in a wide range of wavelength bands (e.g., 350nm to 1000 nm).
Fig. 2 illustrates an exemplary configuration diagram of a computing spectroscopy apparatus according to an embodiment of the present application. As shown in fig. 2, in the optical spectrum calculating device according to the embodiment of the present application, the optical system is optional, which may be an optical system such as a lens assembly, a dodging assembly, or the like. The filtering structure is a broadband filtering structure in the frequency domain or the wavelength domain. The passband spectra of different wavelengths of the filter structure are not identical. The filter structure may be a structure or a material having a filter property such as a super surface, a photonic crystal, a nano-pillar, a multilayer film, a dye, a quantum dot, a MEMS (micro electro mechanical system), an FP etalon, a cavity layer, a waveguiding layer, a diffraction element, or the like. In the present embodiment, for example, the light filtering structure may be a light modulating layer in chinese patent CN201921223201.2,
With continued reference to fig. 2, the image sensor (i.e., photodetector array) may be a CMOS Image Sensor (CIS), CCD, array photodetector, or the like. In addition, the optional data processing unit may be a processing unit such as MCU, CPU, GPU, FPGA, NPU, ASIC, which may export data generated by the image sensor to the outside for processing.
In this embodiment of the present application, the first image parameter to be sensed refers to a sensed physical quantity to be measured. The first image parameter may be color data such as RGB color information or finer spectral (color temperature) information, or even multispectral or hyperspectral information, etc.
In the embodiment of the application, after the image sensor detects the light intensity information, the light intensity information is transmitted into the data processing unit for recovery calculation. Specifically, in some cases, by selecting an appropriate measurement basis and dictionary, high-precision spectral data can be calculated using only a small number of light intensity data measured by pixels of an image sensor, using a compressed sensing method. The process is specifically described as follows:
the intensity signal of incident light under different wavelength lambda is marked as f (lambda), the transmission spectrum curve of the optical filter structure is marked as T (lambda), the optical filter is provided with m groups of optical filter structures, each group of transmission spectrums are different from each other, and the whole optical filter can be marked as T i (λ) (i=1, 2,3, …, m). The lower part of each group of filtering structures is provided with corresponding physical pixels for detecting the light intensity I modulated by the filtering structures i . In the specific embodiment of the present application, the description is given taking the case that one physical pixel corresponds to one group of structural units as an example, but the present invention is not limited thereto, and in other embodiments, a plurality of physical pixels may be formed as a group corresponding to one group of structural units. Thus, unlike physical pixels, in a computing spectroscopic apparatus according to an embodiment of the present application, one or more physical pixels corresponding to a set of structural units are referred to as "spectroscopic pixels". Further, the present invention may use at least one spectral pixel to restore an image.
The relationship between the spectral distribution of the incident light and the measured values of the photodetector array can be expressed by:
I i ＝Σ(f(λ)*T i (λ)*R(λ))
where R (λ) is the response of the detector, denoted as:
S i (λ)＝T i (λ)*R(λ)
the above equation can be extended to a matrix form:
wherein I is i (i=1, 2,3, …, m) is the transmission of the light to be measuredThe response of the photodetector after the broadband filter unit corresponds to the light intensity measurements of m photodetector units, also called m "physical pixels", which are vectors of length m. S is the optical response of the system for different wavelengths, and is determined by two factors, namely the transmissivity of the filter structure and the quantum efficiency of the response of the photodetector. S is a matrix, each row vector corresponds to the response of a wideband filter element to incident light of a different wavelength, where the incident light is sampled discretely and uniformly, for a total of n sample points. The number of columns of S is the same as the number of samples of the incident light. Here, f (λ) is the intensity of the incident light at different wavelengths λ, i.e. the spectrum of the incident light to be measured.
In practical application, the response function S of the system is known, and the spectrum f (λ) of the input light can be obtained by using the algorithm to back-push through the light intensity reading I of the detector, and the process can adopt different data processing modes according to specific situations, including: least squares, pseudo-inverses, equalizations, least squares, artificial neural networks, etc. Thus, the need to recover a relatively simple first image parameter to be sensed, such as image color (i.e. RGB three values), can be recovered simply using a specific matrix multiplication. And when the requirement on the spectrum precision is high, namely the value of n in the process is high, an algorithm such as an artificial neural network can be adopted. Alternatively, such methods, if combined with compressed sensing processing, may still achieve higher spectral accuracy in some cases where m is significantly smaller than n.
It should be noted that from the above description, it can be inferred that when the first or second image parameter of the image sensor is the incident light intensity f (λ), the system response S acts on the light intensity parameter in a matrix form. When the first or second image parameter is another physical quantity, the system response may be acted upon by another function with the light intensity parameter. The application mainly takes a response matrix as an example to describe the image sensing method.
Taking one physical pixel corresponding to one group of structural units as an example, how to recover one spectrum information, which is also called as a "spectrum pixel", by using m groups of physical pixels (i.e., pixel points on an image sensor) and m groups of corresponding structural units (the same structure on a modulation layer is defined as a structural unit) are described above. For example, in the embodiment of the present application, by recovering the color information of the pixels and performing the array implementation, the complete image information can be obtained, so as to implement image sensing. It should be noted that, as described above, in the embodiment of the present application, a plurality of physical pixels may also correspond to a group of structural units.
S120, determining a first plurality of physical pixels of an image sensor of the spectrum chip based on the first image parameter to be sensed. As described above, a first plurality of physical pixels of the image sensor of the spectral chip, i.e. m physical pixels as described above, is determined based on the first image parameter to be sensed. Preferably, the first plurality of physical pixels of the image sensor of the spectrum chip can also be determined by the first image parameter to be sensed and the spatial resolution or signal-to-noise requirement. Here, the determination of the first plurality of physical pixels may be based on a variety of considerations, which will be described in further detail below.
For example, for RGB color reproduction, i.e. to recover RGB color data, n may be 3 in the above process, m may also be 3, i.e. a group of three physical pixels is determined, and the above matrix may be simplified as:
thus, by measuring the light intensity data I 1 ，I 2 ，I 3 Under the condition that each parameter of the S matrix is known, RGB values of the color pixel points can be obtained rapidly through pseudo-inverse or equalization and the like.
Step S130, determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels. For example, the response function S of the first set of three physical pixels and the light intensity reading I of each physical pixel therein as described above 1 ，I 2 And I 3 . It is noted that the light is obtained in the step S130Strong reading I 1 ，I 2 And I 3 The system in step S120 may be used to assist in automatic adjustment of the system to meet the optimization requirement of the corresponding signal-to-noise ratio or spatial resolution, for example, a threshold may be set, and the relationship between the light intensity readings and the threshold may be used to determine that the light intensity is stronger, for example, when the average number of the light intensity readings is greater than the threshold, for example, when the number of the light intensity readings is greater than 75% of the threshold, the light intensity is understood to be stronger.
Moreover, it will be understood by those skilled in the art that in the spectrum chip-based image sensing method according to the embodiment of the present application, the order of determining the response function and measuring the light intensity readings in step S130 may be arbitrarily set, for example, the light intensity readings of each physical pixel in the first plurality of physical pixels may also be measured, and then the response function of the first plurality of physical pixels may be determined. In addition, both may be performed simultaneously. That is, in the embodiment of the present application, after determining the first plurality of physical pixels of the image sensor of the spectrum chip based on the first image parameter to be sensed, the light intensity readings and the response functions of the first plurality of physical pixels may be acquired in any order.
Step S140, calculating the first image parameter to be sensed based on the light intensity reading and the response function. For example, RGB color data of three physical pixels for color reproduction as described above.
Therefore, the image sensing method based on the spectrum chip according to the embodiment of the application can utilize the broad spectrum filtering structure of the spectrum chip, that is, the broad spectrum filtering structure of the spectrum chip has significantly better light transmission efficiency than the filter array of the existing image sensor, such as bayer filter array, but has color reduction and spatial resolution equivalent to the filter array structure of the existing image sensor. Therefore, the spectrum chip-based image sensing method according to the embodiment of the application has significant advantages in image sensing in a dark light environment.
In addition, in the image sensing method based on the spectrum chip according to the embodiment of the application, since the system of the spectrum chip has the reconfigurable characteristic, the reconfigurable characteristic refers to that a group of a plurality of physical pixels are dynamically adjusted according to requirements, namely, the dynamic adjustment can be performed according to different environmental conditions, and the dynamic adjustment can be performed in different areas in the same photo, so as to obtain the optimal image effect.
That is, based on a physical pixel point corresponding to a unit structure, how many physical pixel points are selected as a set of data to perform recovery of a spectrum pixel, and in other embodiments, how many unit structures are selected as a set of data to perform recovery of a spectrum pixel, that is, the value of m affects the spatial resolution, the image signal-to-noise ratio, and the spectrum precision of the sensor. The reconfigurable mode can dynamically adjust the value of m according to the environmental conditions under different conditions so as to obtain the optimal image effect.
Fig. 3 illustrates a schematic diagram of determining different numbers of physical pixels in a spectral chip-based image sensing method according to an embodiment of the present application. As shown in fig. 3, a 10×10 matrix of physical pixels is shown, where the transmission spectrum of the filter structure corresponding to each physical pixel is different (this may make the correlation low). Specifically, the reconfigurable characteristic as described above is that, in actual use, the number of physical pixels (i.e., m values) of 2×2,3×3,5×5, 10×10 etc. scales can be used as a set of data to be processed as needed, to generate one spectral pixel. It should be noted that, in the present invention, a one-to-one relationship between a physical pixel and a structural unit is taken as an example, in practice, a plurality of physical pixels may also correspond to a group of structural units, and when a plurality of physical pixels corresponds to a group of structural units, it may be understood that the physical pixels are reconfigurable by the structural units, that is, the physical pixels are determined by the structural units, for example, 2×2 may be understood that 4 structural units form a group of a plurality of physical pixels.
Specifically, when the environment is darker, more physical pixels can be used as a group of data for processing, for example, physical pixel points with the size of 8×8 or more are used as a spectrum pixel, so that the signal-to-noise ratio of the image is improved.
And when the environmental light intensity is strong, fewer pixel points can be used as a group of data for processing, for example, 3×3 physical pixel points or less form a spectrum pixel, preferably 2×2 physical pixel points form a spectrum pixel, so that the spatial resolution is improved.
Further, in other embodiments, the dynamic adjustment may be performed by selecting an area in the same image, setting the m value of the darker area in the image high, and setting the m value of the brighter portion in the image low, so as to achieve the optimal effect. That is, there may be spectral pixels of different specifications in one image.
Taking fig. 3 as an example, in a 10×10 pixel matrix, a reconfigurable mode is adopted, and different numbers of physical pixels (2×2,3×3,5×5) are respectively selected as a group of a plurality of physical pixels to recover spectrum pixels. Also taking RGB data as an example, when 2×2 pixels are selected, RGB color data of 25 color (spectrum) pixels can be obtained in the square matrix range based on the light intensity data of the 2×2 pixels in combination with a response function, thereby performing color recovery. When 5×5 pixels are selected, RGB color data of 4 color (spectrum) pixels can be obtained in the square matrix range based on the light intensity data of the 5×5 pixels in combination with the response function. Compared with the two, when 2×2 pixels are selected as a group of a plurality of physical pixels, higher spatial resolution can be obtained, and when 5×5 pixels are selected as a group of a plurality of physical pixels, higher signal-to-noise ratio effect can be obtained. The dynamic selection can be made according to the requirements in different situations.
Furthermore, it should be noted that, in the image sensing method based on the spectrum chip according to the embodiment of the present application, the first plurality of physical pixels may be determined according to at least one of the spatial resolution, the image signal-to-noise ratio and the spectrum precision of the image sensor, and may be automatically adjusted by the system itself based on a default setting or an actual requirement, or may be manually adjusted by a user according to the requirement thereof.
Thus, in a spectroscopic chip based image sensing method according to an embodiment of the present application, determining a first plurality of physical pixels of the spectroscopic chip for image sensing based on the first image parameter to be sensed comprises: the first plurality of physical pixels is determined based on at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor.
In the above-mentioned image sensing method based on a spectrum chip, the first plurality of physical pixels may be at least one of 2×2, 3×3, 5×5, and 10×10 physical pixel matrixes on the image sensor, or may be a rectangular matrix or an irregular non-square matrix combination. That is, the first plurality of physical pixels may be a plurality of physical pixels of a predetermined shape on the image sensor.
Further, in the above-described spectrum chip-based image sensing method, determining the first plurality of physical pixels according to at least one of a spatial resolution, an image signal-to-noise ratio, and a spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to the intensity of the ambient light being a first intensity; and responsive to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel matrices, the second number being greater than the first number.
In addition, in the above-mentioned image sensing method based on a spectrum chip, determining the first plurality of physical pixels according to at least one of a spatial resolution, an image signal-to-noise ratio and a spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first region of a first luminance or a first signal-to-noise ratio in an image acquired by the image sensor; and determining a second number of physical pixel arrays in response to a second area of a second signal-to-noise ratio in the image acquired by the image sensor that is less than the first brightness or less than the first signal-to-noise ratio, the second number being greater than the first number.
The above takes the example that the image sensing method based on the spectrum chip according to the embodiment of the application is used for color recovery, in addition, the image sensing method based on the spectrum chip according to the embodiment of the application can also be used for obtaining other image sensing parameters, for example, the image sensing method based on the spectrum chip according to the embodiment of the application can be used for measuring the ambient color temperature to perform accurate white balance processing on an image.
Those skilled in the art will appreciate that the need for color reduction and color temperature measurement varies greatly in several ways. Specifically, color reduction has lower requirements on spectral accuracy, higher requirements on spatial resolution, and the color temperature measurement is the contrary, and has lower requirements on spatial resolution and higher requirements on spectral accuracy. Thus, for image sensing schemes based on non-reconfigurable structures and methods, two needs need to be satisfied by designing different structures. By utilizing the reconfigurable structure and the processing mechanism of the embodiment of the application, two functions can be realized on the same sensor, and the respective required effects are ensured.
That is, in the spectrum chip-based image sensing method according to the embodiment of the present application, determining the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and determining a second number of physical pixel matrices in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy that is greater than the first spectral accuracy, the second number being greater than the first number.
Fig. 4 illustrates a schematic diagram of a spectral chip-based image sensing method for simultaneous color reduction and color temperature measurement according to an embodiment of the present application. As shown in fig. 4, on a 10×10 physical pixel structure, every 2×2 physical pixels are employed as one RGB color restoration data set, that is, RGB color pixels of 5×5=25 pixel points in total are restored. Of course, those skilled in the art will appreciate that the RGB color data may be equivalently replaced with other color data, such as XYZ, RGB-like color data, and the like. And simultaneously, performing spectrum recovery by taking all pixel data of 10 multiplied by 10 as a group of data to obtain finer image spectrum information, further calculating color temperature information of an image and performing white balance processing. Here, it is understood by those skilled in the art that when using the light intensity data of all pixels of 10×10, the light intensity data of each pixel used in performing color reduction may be directly used.
The process of calculating the relative color temperature from the spectrum can be briefly described as:
firstly, calculating an image color position value x and y by utilizing CIE three-stimulus parameters and an image frequency spectrum:
wherein the method comprises the steps ofThe spectrum response parameter corresponding to the CIE tristimulus value. From this, the color level value can be calculated:
And then the color temperature is checked through a chromaticity diagram, or the relative color temperature CCT is calculated through an approximate formula:
n＝(x-0.3320)/(y-0.1858)，
CCT＝-437*n^3+3601*n^2-6831*n+5517。
in this way, the spectral pixels can be restored by adopting the pixel groups of 2×2 and 10×10 on the same image sensor, and the requirements of color imaging and color temperature sensing are met. Of course, it will be understood by those skilled in the art that in the spectrum chip-based image sensing method according to the embodiment of the present application, the image parameters to be sensed obtained simultaneously are not limited to color data and color temperature data, but may be other types of image parameters.
Accordingly, in the spectrum chip-based image sensing method according to the embodiment of the present application, further comprising: determining a second image parameter to be sensed; determining a second plurality of physical pixels of an image sensor of the spectral chip based on the second image parameter to be sensed; determining a response function of the second plurality of physical pixels and determining a light intensity reading for each physical pixel in the second plurality of physical pixels; and calculating the second image parameter to be sensed based on the light intensity reading and the response function.
And, in the above-described spectral chip-based image sensing method, determining the light intensity reading for each physical pixel in the second plurality of physical pixels comprises: and acquiring light intensity readings of physical pixels corresponding to the first plurality of physical pixels in the second plurality of physical pixels based on the corresponding relation between the second plurality of physical pixels and the first plurality of physical pixels.
In the above image sensing method based on a spectrum chip, the first image parameter is color data, and the second image parameter is color temperature data.
In the above image sensing method based on a spectrum chip, the first plurality of physical pixels is a first number of physical pixel matrixes, the second plurality of physical pixels is a second number of physical pixel matrixes, and the second number is greater than the first number.
Further, in the spectrum chip-based image sensing method according to the embodiment of the application, in the same imaging, the color temperature of the image can be measured in different areas, so that the color temperature condition of the environment can be known more accurately, and preconditions are provided for more processing modes. In addition, the same sensor is used for realizing the accurate measurement of color reduction and color temperature with high spatial resolution, so that a better image white balance effect can be obtained.
That is, in the spectrum chip-based image sensing method according to the embodiment of the present application, determining the response function of the second plurality of physical pixels includes: determining a first region and a second region of the image sensor of color temperature data to be sensed based on the color temperature data to be sensed; and determining response functions of a plurality of physical pixels of the first region and the second region, respectively, to obtain response functions of the second plurality of physical pixels.
Therefore, the image sensing method based on the spectrum chip can utilize the broad spectrum filtering structure of the spectrum chip, determine the response function of the physical pixel corresponding to the image parameter to be sensed according to the broad spectrum filtering structure, and calculate the image parameter based on the light intensity reading of the physical pixel, so that the effects of signal-to-noise ratio, color, white balance and the like of image sensing are improved.
In addition, the image sensing method based on the spectrum chip can utilize the reconfigurable characteristic of the spectrum chip to pertinently optimize the effects of signal to noise ratio, color, white balance and the like of image sensing according to different conditions, so that the image sensing effect is improved.
Application example
As described above, the spectrum chip-based image sensing method according to the embodiment of the present application is based on the repeated use of pixel data, and can be used for color reduction and color temperature measurement at the same time, and fig. 5 below illustrates a flowchart of this specific example of the spectrum chip-based image sensing method according to the embodiment of the present application.
As shown in fig. 5, this specific example of the spectrum chip-based image sensing method according to the embodiment of the present application includes: step S210, determining color data and color temperature data to be sensed; s220, respectively determining a first plurality of physical pixels and a second plurality of physical pixels of an image sensor of the spectrum chip based on the color data and the color temperature data to be sensed; s230, determining a first response function of the first plurality of physical pixels and a first light intensity reading of each physical pixel in the first plurality of physical pixels, and determining a second response function of the second plurality of physical pixels and a second light intensity reading of each physical pixel in the second plurality of physical pixels; and S240, calculating the color data to be sensed based on the first light intensity reading and the first response function, and calculating the color temperature data to be sensed based on the second light intensity reading and the second response function.
In a specific example of the above spectral chip-based image sensing method, determining a first light intensity reading for each physical pixel in the first plurality of physical pixels comprises: and acquiring light intensity readings of physical pixels corresponding to the first plurality of physical pixels in the second plurality of physical pixels based on the corresponding relation between the first plurality of physical pixels and the second plurality of physical pixels. That is, as described above, since it is necessary to measure the light intensity data of all 10×10 physical pixels when sensing color temperature data, the light intensity data of 2×2 physical pixels to be used can be directly selected from the light intensity data of 10×10 physical pixels.
In a specific example of the above-described spectrum chip-based image sensing method, the first plurality of physical pixels is a first number of physical pixel squares, such as a 2×2 physical pixel square as described above, and the second plurality of physical pixels is a second number of physical pixel squares, such as a 10×10 physical pixel square as described above. Of course, those skilled in the art will appreciate that the first plurality of physical pixels may be a 3×3, 5×5 matrix of physical pixels.
Also, as described above, the first plurality of physical pixels may be rectangular or irregularly shaped physical pixels on the image sensor.
In addition, in the specific example of the spectrum chip-based image sensing method described above, the first plurality of physical pixels and/or the second plurality of physical pixels may also be determined according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor.
That is, determining the first plurality of physical pixels and/or the second plurality of physical pixels based on at least one of an image signal-to-noise ratio and a spectral accuracy of the image sensor comprises: determining a first number of physical pixel matrices in response to the intensity of the ambient light being a first intensity; and responsive to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel matrices, the second number being greater than the first number.
Alternatively, determining the first plurality of physical pixels and/or the second plurality of physical pixels based on at least one of an image signal-to-noise ratio and a spectral accuracy of the image sensor comprises: determining a first number of physical pixel matrices in response to a first region of a first luminance or a first signal-to-noise ratio in an image acquired by the image sensor; and determining a second number of physical pixel arrays in response to a second area of a second signal-to-noise ratio in the image acquired by the image sensor that is less than the first brightness or less than the first signal-to-noise ratio, the second number being greater than the first number.
Alternatively, determining the first plurality of physical pixels and/or the second plurality of physical pixels based on at least one of an image signal-to-noise ratio and a spectral accuracy of the image sensor comprises: determining a first number of physical pixel matrices in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and determining a second number of physical pixel matrices in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy that is greater than the first spectral accuracy, the second number being greater than the first number.
Also, in the specific example of the spectrum chip-based image sensing method described above, determining the second response function of the second plurality of physical pixels includes: determining a first region and a second region of the image sensor of color temperature data to be sensed based on the color temperature data to be sensed; and determining response functions of a plurality of physical pixels of the first region and the second region, respectively, to obtain response functions of the second plurality of physical pixels.
Thus, this specific example can enhance the color and white balance effect of image sensing by calculating color data and color temperature data at the same time. In addition, the reconfigurable characteristic of the spectrum chip can be utilized to pertinently optimize the effects of color, white balance and the like of the image sensing according to different conditions, so that the image sensing effect can be improved.
Exemplary apparatus
Fig. 6 illustrates a block diagram of a spectral chip-based image sensing device according to an embodiment of the present application, according to an embodiment of the present application.
As shown in fig. 6, the spectrum chip-based image sensing apparatus 300 according to the embodiment of the present application includes: a parameter determining unit 310, configured to determine a first image parameter to be sensed; a pixel determination unit 320, configured to determine a first plurality of physical pixels of the image sensor of the spectrum chip based on the first image parameter to be sensed; a data acquisition unit 330 for determining a first response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and a parameter calculation unit 340 for calculating the first image parameter to be sensed based on the light intensity reading and the response function.
In one example, in the above-mentioned spectrum chip-based image sensing apparatus 300, the matrix determining unit 320 is configured to: the first plurality of physical pixels is determined based on at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor.
In one example, in the above-described spectrum chip-based image sensing device 300, the first plurality of physical pixels is at least one of a 2×2, 3×3, 5×5, 10×10 matrix of physical pixels on the image sensor.
In one example, in the above-described spectrum chip-based image sensing device 300, the first plurality of physical pixels are a plurality of physical pixels of a predetermined shape on the image sensor.
In one example, in the above-described spectrum chip-based image sensing apparatus 300, the matrix determining unit 320 determines the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to the intensity of the ambient light being a first intensity; and responsive to the intensity of the ambient light being a second intensity less than the first intensity, determining a second number of physical pixel matrices, the second number being greater than the first number.
In one example, in the above-described spectrum chip-based image sensing apparatus 300, the matrix determining unit 320 determines the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first region of a first luminance or a first signal-to-noise ratio in an image acquired by the image sensor; and determining a second number of physical pixel arrays in response to a second area of a second signal-to-noise ratio in the image acquired by the image sensor that is less than the first brightness or less than the first signal-to-noise ratio, the second number being greater than the first number.
In one example, in the above-described spectrum chip-based image sensing apparatus 300, the matrix determining unit 320 determines the first plurality of physical pixels according to at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor includes: determining a first number of physical pixel matrices in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and determining a second number of physical pixel matrices in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy that is greater than the first spectral accuracy, the second number being greater than the first number.
In one example, in the above-described spectrum chip-based image sensing device 300, at least a first plurality of physical pixels are used to restore an image.
In one example, in the above-mentioned spectrum chip based image sensing apparatus 300, the parameter determining unit 310 is further configured to determine a second image parameter to be sensed; the pixel determination unit 320 is further configured to determine a second plurality of physical pixels of the image sensor of the spectrum chip based on the second image parameter to be sensed; the data acquisition unit 330 is further configured to determine a second response function for the second plurality of physical pixels and determine a light intensity reading for each physical pixel in the second plurality of physical pixels; and, the parameter calculation unit 340 is further configured to calculate the second image parameter to be sensed based on the light intensity reading and the response function.
In one example, in the above-mentioned spectrum chip-based image sensing apparatus 300, the light intensity measuring unit 330 is configured to: and acquiring light intensity readings of physical pixels corresponding to the first plurality of physical pixels in the second plurality of physical pixels based on the corresponding relation between the second plurality of physical pixels and the first plurality of physical pixels.
In one example, in the spectrum chip-based image sensing apparatus 300 described above, the first image parameter is color data, and the second image parameter is color temperature data.
In one example, in the above-described spectrum chip-based image sensing device 300, the first plurality of physical pixels is a first number of physical pixel matrices, and the second plurality of physical pixels is a second number of physical pixel matrices, the second number being greater than the first number.
In one example, in the above-mentioned spectrum chip-based image sensing apparatus 300, the matrix determining unit 320 is configured to: determining a first region and a second region of the image sensor of color temperature data to be sensed based on the color temperature data to be sensed; and determining response functions of a plurality of physical pixels of the first region and the second region, respectively, to obtain response functions of the second plurality of physical pixels.
In one example, in the above-described spectrum chip-based image sensing apparatus 300, the spectrum chip is a spectrum chip for calculating light of a band of 350 to 1000 nm range of the spectrum apparatus.
Here, it will be understood by those skilled in the art that the specific functions and operations of the respective units and modules in the above-described spectrum chip-based image sensing apparatus 300 have been described in detail in the spectrum chip-based image sensing method described above with reference to fig. 1 to 5, and thus, repetitive descriptions thereof will be omitted.
As described above, the spectrum chip-based image sensing apparatus 300 according to the embodiment of the present application may be implemented in various terminal devices, such as a spectrometer, or a server provided in the cloud. In one example, the spectrum chip based image sensing apparatus 300 according to the embodiments of the present application may be integrated into the terminal device as one software module and/or hardware module. For example, the spectrum chip based image sensing apparatus 300 may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the spectrum chip based image sensing apparatus 300 may also be one of a plurality of hardware modules of the terminal device.
Alternatively, in another example, the spectrum chip-based image sensing apparatus 300 and the terminal device may be separate devices, and the spectrum chip-based image sensing apparatus 300 may be connected to the terminal device through a wired and/or wireless network and transmit the interactive information in a agreed data format.
Exemplary electronic device
Next, an electronic device according to an embodiment of the present application is described with reference to fig. 7.
Fig. 7 illustrates a block diagram of an electronic device according to an embodiment of the present application.
As shown in fig. 7, the electronic device 10 includes one or more processors 11 and a memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by the processor 11 to implement the spectral chip-based image sensing methods and/or other desired functions of the various embodiments of the present application described above. Various contents such as light intensity data, corresponding parameters, etc. may also be stored in the computer-readable storage medium.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other forms of connection mechanisms (not shown).
The input means 13 may be, for example, a keyboard, a mouse, etc.
The output device 14 may output various information such as image sensing parameters and the like to the outside. The output device 14 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
Of course, only some of the components of the electronic device 10 that are relevant to the present application are shown in fig. 7 for simplicity, components such as buses, input/output interfaces, etc. are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer readable storage Medium
In addition to the methods and apparatus described above, embodiments of the present application may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in a spectroscopic chip based image sensing method according to various embodiments of the present application described in the "exemplary methods" section of the present specification.
The computer program product may write program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps in a spectroscopic chip based image sensing method according to various embodiments of the present application described in the above "exemplary method" section of the present specification.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present application have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not intended to be limited to the details disclosed herein as such.
The block diagrams of the devices, apparatuses, devices, systems referred to in this application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent to the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.
Claims (14)
1. An image sensing method based on a spectrum chip, comprising:
determining a first image parameter to be sensed, wherein the first image parameter is color data or spectrum information;
Determining a first plurality of physical pixels of an image sensor of the spectral chip based on the first image parameter to be sensed;
determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and
calculating the first image parameter to be sensed based on the light intensity reading and the response function;
wherein determining a first plurality of physical pixels of the spectral chip for image sensing based on the first image parameter to be sensed comprises:
the first plurality of physical pixels is determined based on at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor.
2. The spectral chip-based image sensing method of claim 1, wherein the first plurality of physical pixels is at least one of a 2 x 2, 3 x 3, 5 x 5, 10 x 10 matrix of physical pixels on the image sensor.
3. The spectral chip-based image sensing method of claim 1, wherein the first plurality of physical pixels are a plurality of physical pixels of a predetermined shape on the image sensor.
4. The spectral chip-based image sensing method of claim 1, wherein determining the first plurality of physical pixels based on at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor comprises:
determining a first number of physical pixel matrices in response to the intensity of the ambient light being a first intensity; and
in response to the intensity of the ambient light being a second intensity less than the first intensity, a second number of physical pixel matrices is determined, the second number being greater than the first number.
5. The spectral chip-based image sensing method of claim 1, wherein determining the first plurality of physical pixels based on at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor comprises:
determining a first number of physical pixel matrices in response to a first region of a first luminance or a first signal-to-noise ratio in an image acquired by the image sensor; and
and determining a second number of physical pixel matrixes in response to a second area of a second brightness smaller than the first brightness or a second signal-to-noise ratio smaller than the first signal-to-noise ratio in an image acquired by the image sensor, wherein the second number is larger than the first number.
6. The spectral chip-based image sensing method of claim 1, wherein determining the first plurality of physical pixels based on at least one of spatial resolution, image signal-to-noise ratio and spectral accuracy of the image sensor comprises:
determining a first number of physical pixel matrices in response to a first spatial resolution and/or a first spectral accuracy of the image sensor; and
a second number of physical pixel matrices is determined in response to a second spatial resolution of the image sensor that is less than the first spatial resolution and/or a second spectral accuracy that is greater than the first spectral accuracy, the second number being greater than the first number.
7. The method of claim 1, wherein at least one of the first plurality of physical pixels is used to restore an image.
8. The spectral chip-based image sensing method of claim 1, further comprising:
determining a second image parameter to be sensed;
determining a second plurality of physical pixels of an image sensor of the spectral chip based on the second image parameter to be sensed;
determining a response function of the second plurality of physical pixels and determining a light intensity reading for each physical pixel in the second plurality of physical pixels; and
The second image parameter to be sensed is calculated based on the light intensity reading of each physical pixel of the second plurality of physical pixels and the response function of the second plurality of physical pixels.
9. The spectroscopic chip based image sensing method as defined in claim 8 wherein determining the light intensity reading for each physical pixel in the second plurality of physical pixels comprises:
and acquiring light intensity readings of physical pixels corresponding to the first plurality of physical pixels in the second plurality of physical pixels based on the corresponding relation between the second plurality of physical pixels and the first plurality of physical pixels.
10. The method of claim 9, wherein the first image parameter is color data and the second image parameter is color temperature data.
11. The spectroscopic chip based image sensing method as defined in claim 10, wherein the first plurality of physical pixels is a first number of physical pixel matrices and the second plurality of physical pixels is a second number of physical pixel matrices, the second number being greater than the first number.
12. The spectral chip-based image sensing method of claim 10, wherein determining the response function of the second plurality of physical pixels comprises:
Determining a first region and a second region of the image sensor of color temperature data to be sensed based on the color temperature data to be sensed; and
response functions of a plurality of physical pixels of the first region and the second region are determined, respectively, to obtain response functions of the second plurality of physical pixels.
13. The spectroscopic chip based image sensing method as defined in any one of claims 1 to 12, wherein the spectroscopic chip is a spectroscopic chip for calculating light of a spectral device receiving a wavelength band in the range of 350 to 1000 nanometers.
14. An image sensing device based on a spectrum chip, comprising:
the parameter determining unit is used for determining a first image parameter to be sensed, wherein the first image parameter is color data or spectrum information;
a pixel determination unit for determining a first plurality of physical pixels of an image sensor of the spectrum chip based on the first image parameter to be sensed;
a data acquisition unit for determining a response function of the first plurality of physical pixels and measuring a light intensity reading of each physical pixel of the first plurality of physical pixels; and
a parameter calculation unit for calculating the first image parameter to be sensed based on the light intensity reading and the response function;
Wherein determining a first plurality of physical pixels of the spectral chip for image sensing based on the first image parameter to be sensed comprises:
the first plurality of physical pixels is determined based on at least one of spatial resolution, image signal-to-noise ratio, and spectral accuracy of the image sensor.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110281439.6A CN115086581B (en) | 2021-03-16 | 2021-03-16 | Image sensing method and device based on spectrum chip |
| PCT/CN2022/080327 WO2022194040A1 (en) | 2021-03-16 | 2022-03-11 | Spectral chip-based image sensing method and apparatus, spectral recovery method and apparatus, and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110281439.6A CN115086581B (en) | 2021-03-16 | 2021-03-16 | Image sensing method and device based on spectrum chip |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115086581A CN115086581A (en) | 2022-09-20 |
| CN115086581B true CN115086581B (en) | 2024-03-01 |
Family
ID=83245477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110281439.6A Active CN115086581B (en) | 2021-03-16 | 2021-03-16 | Image sensing method and device based on spectrum chip |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115086581B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111024229A (en) * | 2019-11-28 | 2020-04-17 | 天津津航技术物理研究所 | Single-chip integrated spectral imaging micro-system spectral data correction method |
| CN112018139A (en) * | 2020-08-14 | 2020-12-01 | 清华大学 | Method for generating micro-nano structure array in spectrum chip |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8203114B2 (en) * | 2009-05-14 | 2012-06-19 | Raytheon Company | Adaptive spatial-spectral processing (ASSP) |
| US9593982B2 (en) * | 2012-05-21 | 2017-03-14 | Digimarc Corporation | Sensor-synchronized spectrally-structured-light imaging |
| US10514335B2 (en) * | 2017-09-08 | 2019-12-24 | Arizona Board Of Regents On Behalf Of The University Of Arizona | Systems and methods for optical spectrometer calibration |
-
2021
- 2021-03-16 CN CN202110281439.6A patent/CN115086581B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111024229A (en) * | 2019-11-28 | 2020-04-17 | 天津津航技术物理研究所 | Single-chip integrated spectral imaging micro-system spectral data correction method |
| CN112018139A (en) * | 2020-08-14 | 2020-12-01 | 清华大学 | Method for generating micro-nano structure array in spectrum chip |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115086581A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11193830B2 (en) | Spectrocolorimeter imaging system | |
| Arad et al. | Sparse recovery of hyperspectral signal from natural RGB images | |
| US7313273B2 (en) | Automatic color constancy for image sensors | |
| JP3829238B2 (en) | Color reproduction system | |
| CN108020519B (en) | Virtual multi-light-source spectrum reconstruction method based on color constancy | |
| CN109963133B (en) | Color correction system and method | |
| CN113170028B (en) | Method for generating image data of machine learning based imaging algorithm | |
| CN108896499A (en) | In conjunction with principal component analysis and the polynomial spectral reflectance recovery method of regularization | |
| CN111024229B (en) | Single-chip integrated spectral imaging micro-system spectral data correction method | |
| Ji et al. | Compressive recovery of smartphone RGB spectral sensitivity functions | |
| CN106507065B (en) | Imaging device, imaging system, image generation device, and color filter | |
| KR20170102459A (en) | Spectral imaging method and system | |
| KR20190036459A (en) | Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor | |
| Xiao et al. | Reflectance spectra reconstruction from trichromatic camera based on kernel partial least square method | |
| US7557826B2 (en) | Method for device spectral sensitivity reconstruction | |
| Monno et al. | N-to-sRGB mapping for single-sensor multispectral imaging | |
| CN115086581B (en) | Image sensing method and device based on spectrum chip | |
| CN115479668A (en) | Spectrum analysis device and spectrum video recording method | |
| WO2022194040A1 (en) | Spectral chip-based image sensing method and apparatus, spectral recovery method and apparatus, and electronic device | |
| CN115791628A (en) | Spectrum recovery method, spectrum recovery device and electronic equipment | |
| CN115567784A (en) | Image sensor and color restoration method | |
| CN115147287B (en) | Spectrum recovery method and device of spectrum chip and electronic equipment | |
| Tominaga | CIC@ 20: multispectral imaging | |
| US20240163410A1 (en) | Color reconstruction using homogeneous neural network | |
| WO2022163327A1 (en) | Information processing device, imaging system, information processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |