CN112396670A - Image reconstruction method for novel binary image sensor - Google Patents
Image reconstruction method for novel binary image sensor Download PDFInfo
- Publication number
- CN112396670A CN112396670A CN201910752407.2A CN201910752407A CN112396670A CN 112396670 A CN112396670 A CN 112396670A CN 201910752407 A CN201910752407 A CN 201910752407A CN 112396670 A CN112396670 A CN 112396670A
- Authority
- CN
- China
- Prior art keywords
- image reconstruction
- binary data
- binary
- image
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
Abstract
The image reconstruction method for the novel binary image sensor comprises the steps of collecting optical signals, generating electric signals, generating binary data streams, collecting the binary data streams, configuring image reconstruction parameters, selecting an image reconstruction mode, processing the binary data streams, finishing image reconstruction and outputting reconstructed images; the image reconstruction method and the image reconstruction system are combined with the characteristics of a novel binary image sensor such as high resolution, low noise and high frame frequency, the operations of generating, collecting, processing and reconstructing binary data streams are completed under the appointed image reconstruction parameters and image reconstruction modes, and the reconstructed image is finally output.
Description
Technical Field
The invention relates to the field of image sensors and digital image processing, in particular to an image reconstruction method for a novel binary image sensor.
Background
An image sensor is a semiconductor light-sensing element that converts an optical signal into an electrical signal. Currently, the mainstream image sensor is a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor. The CMOS image sensor has the inherent advantages of being compatible with a standard CMOS process, high in integration level, low in power consumption, high in speed, low in cost, high in irradiation resistance and the like; meanwhile, with the continuous deepening of related scientific research, the performance indexes such as quantum efficiency, dark current, sensitivity, signal-to-noise ratio, dynamic range, linearity and the like are continuously improved. For the above two reasons, the CMOS image sensor has gradually replaced the conventional Charge Coupled Device (CCD) image sensor, and has gained dominance in various fields of consumer electronics, industrial electronics, automotive electronics, scientific imaging, and the like.
A binary image sensor is an image sensor that converts an optical signal into a binary data stream and outputs the binary data stream, and output data thereof includes both 0 and 1. Wherein, data 0 represents that the pixel unit of the image sensor corresponding to the data does not receive enough photons, and data 1 represents that the pixel unit corresponding to the data receives enough photons. The threshold for converting data 0 to data 1 may be generally configurable. For example, the threshold is configured to be 10 photons, and when the pixel unit receives 10 photons or more, data 1 is output, otherwise, data 0 is output.
In recent years, image sensors have gradually exhibited several new characteristics: with the development of semiconductor technology, the pixel size is gradually reduced to below 1um to reach submicron level; with the development of low-noise pixel-related research, pixel readout noise can be reduced to below 0.5 electron to reach a sub-electron level; with the research of high-speed readout circuits, the theoretical frame rate of the image sensor can exceed 1Mfps by combining technologies such as digital pixels and integrating storage nodes in the pixels.
In combination with the new characteristics, the novel binary image sensor has the following characteristics: first, high resolution. I.e. the pixel size is small enough to enable integration of a larger number of pixel cells, typically 10MP and above, on a smaller chip size. Secondly, low noise. That is, the image sensor has a capability of distinguishing one photon to a certain extent, and a single photon counting capability, with a typical value of 0.5 e-or less, when the readout noise reaches a sub-electronic level. The output data 0 represents no photon incidence, and the output data 1 represents at least 1 photon incidence. Third, high frame rate. I.e. a sufficient number of read frames per second, typically 1kfps and above.
Aiming at the novel binary image sensor and the characteristics thereof, an image reconstruction method and a system matched with the novel binary image sensor are required to be provided, the image reconstruction method and the system are required to complete the operations of binary data stream generation, acquisition, processing, reconstruction and the like, and finally output the reconstructed image.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an image reconstruction method for a novel binary image sensor. The image reconstruction method and the image reconstruction system are combined with the characteristics of a novel binary image sensor such as high resolution, low noise and high frame frequency, the operations of generating, collecting, processing and reconstructing binary data streams are completed under the appointed image reconstruction parameters and image reconstruction modes, and the reconstructed image is finally output.
The image reconstruction method for the novel binary image sensor has the flow shown in fig. 1, and the image reconstruction system includes the following flows: (1) collecting optical signals: the signal electrons generated by the incident photons are collected and stored by the pixels and their arrays. (2) Generating an electrical signal: the signal electrons are converted into signal voltage values through operations of charge transfer, correlated double sampling, amplification and the like. (3) Generating a binary data stream: the signal voltage value is quantized to a digital value of 0 or a digital value of 1 by a binary quantizer, such as a 1-bit analog-to-digital converter (1-bit ADC). Where a digital value of 0 represents no photon incident and a digital value of 1 represents at least one photon incident. In each frame, the individual pixel output results form a binary data matrix. (4) Collecting a binary data stream: and collecting and storing each frame of binary data matrix generated by the image sensor, namely collecting binary data stream. (5) And (3) configuring image reconstruction parameters: and configuring image reconstruction parameters for subsequent image reconstruction operation. (6) Selecting an image reconstruction mode: an image reconstruction mode is selected for a subsequent image reconstruction operation. (7) Processing a binary data stream: the binary data stream is data processed under specified image reconstruction parameters and modes. (8) And finishing image reconstruction and outputting the reconstructed image.
The image reconstruction method for the novel binary image sensor has a novel binary image sensor structure as shown in fig. 2. The novel binary image sensor consists of V rows and H columns of V x H pixel units (each square unit in FIG. 2); because of adopting the process technology with lower characteristic dimension, the typical value of the dimension of each pixel unit is 1um to 1um and below, and the typical value of the total number of the pixel units is 10MP and above; controlling V x H pixel units through a row addressing circuit, a column addressing circuit and a related driving circuit; quantizing the output result of each column of pixels through a column reading circuit, and outputting each column of data including data 0 or data 1; and carrying out parallel-serial processing on each column of data through a basic data processing circuit to form a binary data stream.
Fig. 3 shows a schematic diagram of a binary data flow of the image reconstruction method for a novel binary image sensor. The data stream comprises F frames of binary data matrixes, each frame of binary data matrix comprises V rows and H columns, and V x H data units in total correspond to V x H pixel units. As shown in fig. 3, each white square represents at least one photon incident, outputting data 1, and each black square represents no photon incident, outputting data 0. The binary data stream contains V × H × F bits of data in total.
The image reconstruction method for the novel binary image sensor has an image reconstruction parameter schematic diagram as shown in fig. 4. Before data processing and image reconstruction, configuring image reconstruction parameters as (k)V,kH,kF) (ii) a Extracting k from F-frame binary data matrix based on original binary data stream (V, H, F)FA frame; extracting k from binary data matrix of V row and H columnVLine kHA column binary data submatrix; and finally, extracting the binary data block with the specified size. At image reconstruction parameters (k)V,kH,kF) Then, the maximum value of the binary data submatrix per frame is kV*kHThe minimum value is 0; binary data block maximum value kV*kH*kFAnd the minimum value is 0. For example, as shown in FIG. 4, the image reconstruction parameters (k)V,kF,kF)=(3,3,3)，The maximum value of the binary data block extracted under the reconstruction parameters is 3 × 3= 27; the sum of the values of the binary data blocks with the specified size is 4+5+4=13, and if the binary data blocks are subjected to subsequent data processing and image reconstruction, the binary data blocks correspond to a pixel point with a gray value of 13/27 × 255=123 in the reconstructed image.
Fig. 5 shows a schematic diagram of an image reconstruction mode of the image reconstruction method for the novel binary image sensor. Before data processing and image reconstruction, a corresponding image reconstruction mode needs to be selected. As shown in fig. 5, k is extracted based on the original binary data stream (V, H, F)VLine, kHThe columns binary data sub-matrix and this sub-matrix is referred to as the image reconstruction window. As shown in fig. 5(a) and (b), the first type of image reconstruction mode is called a non-data-overlapped image reconstruction mode, and is characterized as follows: when horizontal scanning and vertical scanning are carried out, data do not overlap among the image reconstruction windows. As shown in fig. 5(c) and (d), the second type of image reconstruction mode is called a data-overlapped image reconstruction mode, and is characterized as follows: when horizontal scanning and vertical scanning are carried out, data overlap exists between the image reconstruction windows, and the overlap range can be adjusted according to actual conditions.
The image reconstruction method for the novel binary image sensor is combined with the novel binary image sensor, can configure image reconstruction parameters and select an image reconstruction mode, can completely complete a series of operations such as binary data stream generation, acquisition, processing, reconstruction and the like, and finally outputs a reconstructed image.
Drawings
FIG. 1 is a flow chart of an image reconstruction system of the present invention;
FIG. 2 is a diagram of the novel binary image sensor architecture of the present invention;
FIG. 3 is a schematic representation of the binary data stream of the present invention, depicting a particular form of the raw binary data stream (V, H, F) generated by the image sensor;
FIG. 4 is a schematic diagram of image reconstruction parameters of the present invention, depicting the specific significance of three image reconstruction parameters;
FIG. 5 is a schematic diagram of the image reconstruction mode of the present invention, depicting the basic principles of two types of image reconstruction modes;
fig. 6 shows an embodiment of simulation based on the image reconstruction method of the present invention.
Detailed Description
Fig. 6 shows an embodiment of simulation based on the image reconstruction method of the present invention. The following describes a specific embodiment of the present invention with reference to this example.
This embodiment is based on the original binary data stream (V, H, F) = (4096, 16), i.e. the original data stream of this embodiment contains 16 4096 rows and 4096 columns of binary data matrices, i.e. 268435456 data units, for a total of 268435456 bits of data. In practical application, the original binary data stream related to the present invention is output by the novel binary image sensor, i.e. four processes of collecting light signals, generating binary data streams and collecting binary data streams are performed. In this case, a 4096 row, 4096 column binary data matrix corresponds to a 4096 row, 4096 column pixel array, and 16 binary data matrices correspond to 16 frames of image data. Of the 268435456 data cells, data 0 indicates that the pixel cell corresponding to the data cell has no incident photons, and data 1 indicates that the pixel cell corresponding to the data cell has at least one incident photons.
The image reconstruction parameter (k) configured by the embodimentV,kH,kF) = 16,16,16, i.e. the embodiment selects all the binary data matrices of 4096 rows and 4096 columns of 16 frames based on the original binary data stream (4096, 16), selects the binary sub-matrices of 16 rows and 16 columns thereof, and completes the extraction of the binary data block of 16 × 16. The theoretical maximum value of the 16 × 16 binary data block is 16 × 1=4096 during subsequent data processing and image reconstruction. When the actual value of the 16 × 16 binary data block is S, the gray value of the corresponding pixel point in the reconstructed image is S/4096 × 255.
The image reconstruction mode selected by the embodiment is a non-data-overlapping image reconstruction mode, namely, when data processing and image reconstruction are carried out based on the image reconstruction parameters (16,16,16), data do not overlap among the image reconstruction windows. As can be seen, the number of pixels corresponding to the reconstructed image is (4096/16) × (4096/16) =256 × 256=65536, that is, the resolution of the reconstructed image is 256 × 256.
As shown in fig. 6, this embodiment generates an original binary data stream (4096, 16) by an analog simulation tool based on a standard test picture "lena" in digital image processing, configures image reconstruction parameters to (16,16,16), selects a data-free overlapped image reconstruction mode, and completes the grayscale image reconstruction with a resolution of 256 × 256 through data processing.
Claims (4)
1. The image reconstruction method for the novel binary image sensor is characterized by comprising the following steps: the method comprises the following steps: (1) collecting optical signals: collecting and storing signal electrons generated by incident photons through the pixels and the array thereof; (2) generating an electrical signal: converting the signal electrons into signal voltage values through operations of charge transfer, related double sampling, amplification and the like; (3) generating a binary data stream: quantizing the signal voltage value into a digital value 0 or a digital value 1 by a binary quantizer; wherein, a digital value of 0 represents no photon incidence, and a digital value of 1 represents at least one photon incidence; in each frame, the output results of each pixel form a binary data matrix; (4) collecting a binary data stream: collecting and storing each frame of binary data matrix generated by the image sensor, namely collecting binary data stream; (5) and (3) configuring image reconstruction parameters: configuring image reconstruction parameters for a subsequent image reconstruction operation; (6) selecting an image reconstruction mode: selecting an image reconstruction mode for a subsequent image reconstruction operation; (7) processing a binary data stream: under the appointed image reconstruction parameter and mode, processing the data of the binary data stream; (8) and finishing image reconstruction and outputting the reconstructed image.
2. The image reconstruction method for the novel binary image sensor according to claim 1, characterized in that: the binary data stream comprises F frames of binary data matrixes, each frame of binary data matrix comprises V rows and H columns, and V x H data units in total correspond to V x H pixel units.
3. The image reconstruction method for the novel binary image sensor according to claim 1, characterized in that: the image reconstruction parameters are specifically: before data processing and image reconstruction, configuring image reconstruction parameters as (k)V,kH,kF) (ii) a Extracting k from F-frame binary data matrix based on original binary data stream (V, H, F)FA frame; extracting k from binary data matrix of V row and H columnVLine, kHA column binary data submatrix; finally, extracting the binary data block with the specified size; at image reconstruction parameters (k)V,kH,kF) Then, the maximum value of the binary data submatrix per frame is kV*kHThe minimum value is 0; binary data block maximum value kV*kH*kFAnd the minimum value is 0.
4. The image reconstruction method for the novel binary image sensor according to claim 1, characterized in that: the image reconstruction mode specifically includes: based on the original binary data stream (V, H, F), k is extractedVLine kHColumn binary data submatrix, and the submatrix is called as an image reconstruction window; the first type of image reconstruction mode is called a non-data overlapping image reconstruction mode, and when horizontal scanning and vertical scanning are carried out, data do not overlap among image reconstruction windows; the second type of image reconstruction mode is called a data overlapping image reconstruction mode, when horizontal scanning and vertical scanning are performed, data overlapping exists between image reconstruction windows, and the overlapping range can be adjusted according to actual conditions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752407.2A CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752407.2A CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112396670A true CN112396670A (en) | 2021-02-23 |
| CN112396670B CN112396670B (en) | 2023-01-10 |
Family
ID=74602784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910752407.2A Active CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112396670B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09109461A (en) * | 1995-10-16 | 1997-04-28 | Seiko Epson Corp | Apparatus for restoring compressed two-value image data |
| KR20040061738A (en) * | 2002-12-31 | 2004-07-07 | 정보통신연구진흥원 | Method of decomposing and composing multimedia data based on discrete wavelet transformation and system for decomposing and composing the same |
| CN101218803A (en) * | 2005-07-08 | 2008-07-09 | 西门子公司 | Method for sending a media data stream and method for receiving and creating a reconstructed media data stream, and associated transmission apparatus and reception apparatus |
| DE102009013402A1 (en) * | 2009-04-01 | 2010-10-07 | Compugroup Services Gmbh | Method for transmitting binary data in stream of facsimile connection, involves processing images, and allowing insertion of to-be-transmitted meta data directly into binary image information by facsimile connection |
| CN102036023A (en) * | 2009-10-01 | 2011-04-27 | 索尼公司 | Image sensor, electronic system and method for controlling image sensor |
| US20130293702A1 (en) * | 2010-11-01 | 2013-11-07 | The Hong Kong Research Institute Of Textiles And Apparel Limited | Multispectral imaging color measurement system and method for processing imaging signals thereof |
| CN103975580A (en) * | 2011-12-09 | 2014-08-06 | 索尼公司 | Image pickup device, electronic apparatus, optically stimulated luminescence detection scanner, and image pickup method |
| CN104380340A (en) * | 2012-03-20 | 2015-02-25 | 保罗－里德－史密斯－吉塔尔斯股份合作有限公司 | Scoring and adjusting pixels based on neighborhood relationships for revealing data in images |
| CN106656466A (en) * | 2016-12-20 | 2017-05-10 | 中国科学院理化技术研究所 | Correlated encryption transmission device and method of underwater optical image |
| US20170280079A1 (en) * | 2016-03-25 | 2017-09-28 | Qualcomm Incorporated | Apparatus and method for generating local binary patterns (lbps) |
| CN109413407A (en) * | 2017-12-05 | 2019-03-01 | 西北工业大学 | High spatial resolution optical field acquisition device and image generating method |
| CN109672891A (en) * | 2018-12-25 | 2019-04-23 | 哈尔滨工业大学 | The lossless second-compressed method of jpeg image |
| CN109799731A (en) * | 2019-01-28 | 2019-05-24 | 北京环境特性研究所 | Multiband optics radiant type Hardware In The Loop Simulation Method and system |
| CN109816603A (en) * | 2018-12-30 | 2019-05-28 | 天津大学 | The imaging sensor image restoring method of single photon counting imaging |
-
2019
- 2019-08-15 CN CN201910752407.2A patent/CN112396670B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09109461A (en) * | 1995-10-16 | 1997-04-28 | Seiko Epson Corp | Apparatus for restoring compressed two-value image data |
| KR20040061738A (en) * | 2002-12-31 | 2004-07-07 | 정보통신연구진흥원 | Method of decomposing and composing multimedia data based on discrete wavelet transformation and system for decomposing and composing the same |
| CN101218803A (en) * | 2005-07-08 | 2008-07-09 | 西门子公司 | Method for sending a media data stream and method for receiving and creating a reconstructed media data stream, and associated transmission apparatus and reception apparatus |
| DE102009013402A1 (en) * | 2009-04-01 | 2010-10-07 | Compugroup Services Gmbh | Method for transmitting binary data in stream of facsimile connection, involves processing images, and allowing insertion of to-be-transmitted meta data directly into binary image information by facsimile connection |
| CN102036023A (en) * | 2009-10-01 | 2011-04-27 | 索尼公司 | Image sensor, electronic system and method for controlling image sensor |
| US20130293702A1 (en) * | 2010-11-01 | 2013-11-07 | The Hong Kong Research Institute Of Textiles And Apparel Limited | Multispectral imaging color measurement system and method for processing imaging signals thereof |
| CN103975580A (en) * | 2011-12-09 | 2014-08-06 | 索尼公司 | Image pickup device, electronic apparatus, optically stimulated luminescence detection scanner, and image pickup method |
| CN104380340A (en) * | 2012-03-20 | 2015-02-25 | 保罗－里德－史密斯－吉塔尔斯股份合作有限公司 | Scoring and adjusting pixels based on neighborhood relationships for revealing data in images |
| US20170280079A1 (en) * | 2016-03-25 | 2017-09-28 | Qualcomm Incorporated | Apparatus and method for generating local binary patterns (lbps) |
| CN106656466A (en) * | 2016-12-20 | 2017-05-10 | 中国科学院理化技术研究所 | Correlated encryption transmission device and method of underwater optical image |
| CN109413407A (en) * | 2017-12-05 | 2019-03-01 | 西北工业大学 | High spatial resolution optical field acquisition device and image generating method |
| CN109672891A (en) * | 2018-12-25 | 2019-04-23 | 哈尔滨工业大学 | The lossless second-compressed method of jpeg image |
| CN109816603A (en) * | 2018-12-30 | 2019-05-28 | 天津大学 | The imaging sensor image restoring method of single photon counting imaging |
| CN109799731A (en) * | 2019-01-28 | 2019-05-24 | 北京环境特性研究所 | Multiband optics radiant type Hardware In The Loop Simulation Method and system |
Non-Patent Citations (2)
| Title |
|---|
| JIANGTAO XU等: ""A Method of Biomimetic Visual Perception and Image Reconstruction Based on Pulse Sequence of Events"", 《 IEEE SENSORS JOURNAL 》 * |
| 徐江涛等: ""基于CMOS图像传感器列级ADC的数字双采样"", 《南 开 大 学 学 报（自然科学版)》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112396670B (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170310910A1 (en) | Oversampled image sensor with conditional pixel readout | |
| JP4485203B2 (en) | Noise elimination type CMOS image sensor | |
| CN207022104U (en) | Image sensor pixel and imaging system | |
| US20160118424A1 (en) | Split-gate conditional-reset image sensor | |
| EP2323361B1 (en) | Image sensor with interleaved image output | |
| CN203775318U (en) | Ultraviolet focal plane readout circuit based on pixel-level analog-to-digital conversion | |
| CN103856730A (en) | Ultraviolet focal plane readout circuit and method based on pixel level analog-to-digital conversion | |
| CN102625059B (en) | Dynamic range extension for CMOS image sensors for mobile applications | |
| JP2000516774A (en) | Image sensor with high dynamic range linear output | |
| CN101888492B (en) | Processor used for image sensor wtih multiple data readout scheme | |
| US9391632B1 (en) | Method and system for implementing an extended range approximation analog-to-digital converter | |
| CN116349240A (en) | Digital pixel sensor with reduced quantization operation | |
| EP3930312B1 (en) | Delta image sensor with digital pixel storage | |
| CN112396670B (en) | Image reconstruction method for novel binary image sensor | |
| CN101014098A (en) | Imaging device | |
| CN110336953B (en) | Image sensor with four-pixel structure and reading control method | |
| Fowler | High dynamic range image sensor architectures | |
| JP4311095B2 (en) | Solid-state imaging device and driving method thereof | |
| US11284031B1 (en) | Systems and methods for low-power image digitization | |
| CN111669526B (en) | CMOS image sensor for improving frame frequency high-speed all-digital data reading | |
| CN111182246B (en) | CMS-based CMOS image sensor reading circuit | |
| KR100737915B1 (en) | Image sensor, and test system and test method for the same | |
| CN112449125B (en) | Image sensor reading circuit based on self-adaptive threshold adjustment | |
| Ma et al. | Photon-counting imaging with multi-bit quanta image sensor | |
| CN113054976A (en) | SPAD array suitable for address coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |