CN111107273A

CN111107273A - High dynamic range imaging system and method

Info

Publication number: CN111107273A
Application number: CN201811248605.7A
Authority: CN
Inventors: 陈柏璋
Original assignee: Himax Imaging Ltd
Current assignee: Himax Imaging Ltd
Priority date: 2018-10-25
Filing date: 2018-10-25
Publication date: 2020-05-05
Anticipated expiration: 2038-10-25
Also published as: CN111107273B

Abstract

A high dynamic range imaging system includes a pixel array providing high sensitivity signals and low sensitivity signals; and a high dynamic range encoder for generating an encoding signal according to the high sensitivity signal and the low sensitivity signal. And when the high-sensitivity signal is not greater than the preset critical value, setting the flag bit as active, otherwise, setting the flag bit as inactive.

Description

High dynamic range imaging system and method

Technical Field

The present invention relates to an imaging system and method, and more particularly, to a High Dynamic Range (HDR) imaging system and method applicable to a global or rotary shutter.

Background

Complementary Metal Oxide Semiconductor (CMOS) image sensors are commonly used in mobile applications. Complementary Metal Oxide Semiconductor (CMOS) image sensors are also suitable for other applications, such as automotive or security applications. However, the requirements for automotive or security applications are different from those for mobile applications. For example, automotive or security applications require High Dynamic Range (HDR) so that very dark and very bright scenes can capture good quality images in the same frame.

The dynamic range of the pixel can be achieved by dividing the pixel into a large and a small photodiode. However, since the long exposure and the short exposure do not usually overlap, a harmful motion artifact (artifact) is generated. Motion artifacts can arise due to different exposure times between long-exposure pixels and short-exposure pixels. Long exposed pixels can cause motion blur, creating a dragging phenomenon at the edges of the object.

In view of this, high dynamic range imaging systems have been proposed that use pixel arrays arranged in space (spatial) or frames of multiple exposures in time (temporal). However, conventional high dynamic range imaging systems require more analog-to-digital conversion cycles or pixel data bits.

It is therefore desirable to provide a novel high dynamic range imaging mechanism to overcome the shortcomings of conventional high dynamic range imaging systems.

Disclosure of Invention

In view of the foregoing, it is an objective of the claimed invention to provide a High Dynamic Range (HDR) imaging system and method, which can enhance the dynamic range, effectively reduce the amount of data generated and transmitted and/or reduce the analog-to-digital conversion period.

According to an embodiment of the present invention, a high dynamic range imaging system includes a pixel array and a high dynamic range encoder. The pixel array provides high sensitivity signals and low sensitivity signals. The high dynamic range encoder generates an encoded signal based on the high sensitivity signal and the low sensitivity signal. And when the high-sensitivity signal is not greater than the preset critical value, setting the flag bit as active, otherwise, setting the flag bit as inactive.

Drawings

FIG. 1A shows a block diagram of a high dynamic range imaging system according to a first embodiment of the present invention;

FIG. 1B shows a flow chart of a high dynamic range imaging method of an embodiment of the invention;

FIG. 2A illustrates a schematic diagram of the pixel array of FIG. 1A;

FIG. 2B illustrates a schematic diagram of another pixel array of FIG. 1A;

FIG. 2C is a graph illustrating the relationship between the high-sensitivity signal generated by the high-sensitivity pixel and the low-sensitivity signal generated by the low-sensitivity pixel and the incident light intensity;

FIG. 2D illustrates a graph of the relationship between the incident light intensity and the high sensitivity signal with enhanced dynamic range;

FIG. 3 shows a block diagram of a high dynamic range imaging system in accordance with a second embodiment of the present invention; and

fig. 4 shows a block diagram of a high dynamic range imaging system according to a third embodiment of the present invention.

Detailed Description

Fig. 1A shows a block diagram of a high-dynamic-range (HDR) imaging system 100 according to a first embodiment of the invention, and fig. 1B shows a flowchart of a high-dynamic-range imaging method according to an embodiment of the invention. The high dynamic range imaging system (hereinafter referred to as an imaging system) 100 and the high dynamic range imaging method (hereinafter referred to as an imaging method) are applicable to a machine vision (machine vision) camera.

In the present embodiment, the imaging system 100 may include a pixel array (step 101) 10, which may include a plurality of high-sensitivity (sensitivity) pixels 10H and a plurality of low-sensitivity pixels 10L, wherein the sensitivity of the high-sensitivity pixels 10H is higher than that of the low-sensitivity pixels 10L. Fig. 2A is a schematic diagram illustrating the pixel array 10 of fig. 1A, wherein the high-sensitivity pixels 10H are spatially (spatially) disposed in odd-numbered rows, and the low-sensitivity pixels 10L are spatially disposed in even-numbered rows. Fig. 2B illustrates another pixel array 10 of fig. 1A, in which the high-sensitivity pixels 10H and the low-sensitivity pixels 10L are spatially crossed.

Fig. 2C illustrates the relationship between the high-sensitivity signal generated by the high-sensitivity pixel 10H and the low-sensitivity signal generated by the low-sensitivity pixel 10L and the incident light intensity. As shown, the high-sensitivity pixel 10H is more sensitive to light than the low-sensitivity pixel 10L, but the high-sensitivity pixel 10H quickly reaches saturation (saturation), as shown in fig. 2C. In this specification, the term signal may refer to either an analog electronic signal or a digital data signal, as may be readily understood from the context.

In addition to the spatial arrangement described above, the high sensitivity signal and the low sensitivity signal may be generated using other mechanisms, such as long/short exposure time, high/low gain, or large/small exposure-gain product. In another embodiment, the high-sensitivity signal and the low-sensitivity signal may be generated by a temporal (temporal) configuration. For example, a high sensitivity signal is generated from the current frame (frame) and a low sensitivity signal is generated from the next frame, where the next frame has a shorter exposure time (or lower gain or smaller exposure-gain product). If the generated high-sensitivity signal is in an analog form, the signal can be stored in a capacitor; if in digital form, it may be stored in memory. For details of the temporal configuration, see "64 × 64CMOS Image sensor On-Chip Moving Object Detection and Localization" proposed by Bo ZHao et al, published in the institute of Electrical and electronics Engineers (IEEE Transactions On Circuits and Systems for video Technology), volume 22, No. 4, April 2012; and "Tri-Mode Smart Vision Sensor With11 Transistors/pixels for Wireless sensing network (Tri-Mode Smart Vision Sensor With 11-transducers/Pixel for Wireless Sensor Networks" by dongso Kim et al, published in the institute of electrical and electronics engineers Sensor Journal (IEEE Sensors Journal), volume 13, No. 6, year 2013, the contents of which are considered part of this specification.

The high-sensitivity signal and the low-sensitivity signal can be generated by using an appropriate exposure mechanism, such as a global shutter (global shutter) or a rolling shutter (rolling shutter). In the global shutter mechanism, a pixel array with a first exposure time and a gain is used to capture a first frame. After reading out the first frame, a pixel array with a second exposure time and a gain is used for capturing a second frame, wherein the product of the second exposure time and the gain is larger than the product of the first exposure time and the gain. The high dynamic range image can be obtained by combining the first frame and the second frame. In the rotary shutter mechanism, instead of exposing the entire pixel array at once to capture a frame, the pixel array is scanned vertically or horizontally, line by line. Details of the exposure mechanism can be found in "comparison of high dynamic range complementary metal oxide semiconductor image sensors for automotive applications" (a contrast of high dynamic range CIS technologies for automatic applications) "by johannes solhusvik et al in 2013; and "high dynamic range Sensor of 1280 × 12804.2 micron Split diode pixels using 110 nm backside illuminated CMOS process" (A1280 × 10804.2 μm Split-diode Pixel HDR Sensor in 110 nm BSI CMOSPs) by Trygve Willassen et al, 2015, the contents of which are considered part of this specification.

In step 102, a sensitivity ratio (sensitivity ratio) of the high-sensitivity pixels and the low-sensitivity pixels is determined. In the present embodiment, the light sensitivity ratio of the high dynamic range image frame is a fixed value. The imaging system 100 of the present embodiment may include an analog-to-digital converter (ADC)11 for equivalently converting the signal (generated by the pixel array 10) from an analog form to a digital form. Wherein the analog-to-digital converter 11 can convert the analog high-sensitivity signal (generated by the high-sensitivity pixel 10H) into the digital high-sensitivity signal S_HAnd converts the analog low-sensitivity signal (generated by the low-sensitivity pixel 10L) into a digital low-sensitivity signal S_L。

The imaging system 100 of the present embodiment may include a memory 12, such as a Dynamic Random Access Memory (DRAM) or a Static Random Access Memory (SRAM), for temporarily storing the high-sensitivity signal S_HWith a low sensitivity signal S_LAt least one of (a). According to one feature of this embodiment, the imaging system 100 may comprise a high dynamic range encoder 13 (which includes a memory 12) receiving the high-sensitivity signal S from the ADC 11 and the memory 12_HWith a low sensitivity signal S_L(step 103) to generate a code signal S_E. At step 104, the signal S is compared to a higher sensitivity_HAnd a predetermined threshold value S_T. In the present embodiment, the threshold S is preset_TMay be equal to or less than the saturation level of the high and low luminance sensing signals. If signal S of high sensitivity_HIs not greater than a predetermined threshold S_T(i.e., S)_H<＝S_T) The flag bit is set to active (e.g. 1) and connected in series to the high sensitivity signal S_HTo generate a coded signal S_E(step 105). For example, an active flag bit (having a value of "1") is concatenated as the Most Significant Bit (MSB). If signal S of high sensitivity_HIs N bits, then the signal S is encoded_EIs an (N +1) bit whose Most Significant Bit (MSB) is "1" (i.e., S)_E＝2^N+S_H). On the contrary, if the signal S is high in sensitivity_HGreater than the preset valueBoundary value S_T(i.e., S)_H>S_T) The flag bit is set to inactive (e.g., a value of "0") and concatenated to the low sensitivity signal S_LTo generate a coded signal S_EE.g. an (N +1) bit with its Most Significant Bit (MSB) being "0" (i.e. S)_E＝S_L) (step 106).

The imaging system 100 of the present embodiment may include a transmitter 14 that receives and transmits the encoded signal S_E(step 107). On the other hand, the imaging system 100 may comprise a receiver 15 which receives the transmitted coded signal S_E(step 108). The transmitter 14 and the receiver 15 of the present embodiment may be operated in a wired manner or a wireless manner.

In the present embodiment, the imaging system 100 may include a high dynamic range decoder 16 that receives the encoded signal S_EAnd generates a decoding signal S according to the decoded signal S_D. In step 109, the received code signal S is determined_EWhether the flag bit (i.e., the most significant bit) of (a) is active (e.g., "1"). If the flag bit is active (e.g., "1"), the received encoded signal S is removed_EFlag bit to generate the decoding signal S_D(step 110). On the other hand, if the flag bit is inactive (e.g., "0"), the received encoded signal S is decoded_EMultiplying by the photosensitive ratio (of step 102) to generate a decoded signal S_DI.e. S_D＝R*S_E(step 111).

According to the above embodiments, the dynamic range of the pixel array 10 can be effectively increased, as shown in fig. 2D. More importantly, the present embodiment requires only one extra (flag) bit to be generated and transmitted. In contrast, in an imaging system that does not use the high dynamic range encoder 13 of the present embodiment, a plurality of extra bits need to be generated and transmitted. For example, when the high sensitivity signal and the low sensitivity signal are 10 bits of digital data and the sensitivity ratio is 16, 14 bits are required for the output signal. The higher the photo ratio, the more bits and power are required.

Fig. 3 shows a block diagram of a high dynamic range imaging system 200 according to a second embodiment of the present invention. In fig. 3, blocks similar to those in the first embodiment (fig. 1A) are denoted by the same reference numerals, details of which are not repeated, and the flowchart shown in fig. 1B is used.

Similar to the high dynamic range encoder 13 of FIG. 1A, the high dynamic range imaging system (hereinafter referred to as the imaging system) 200 of the present embodiment may include a comparator 131 for receiving and comparing the high-sensitivity signal S (generated by the high-sensitivity pixel 10H)_HAnd a predetermined threshold value S_T(step 104). If signal S of high sensitivity_HIs not greater than a predetermined threshold S_T(i.e., S)_H<＝S_T) The comparator 131 generates an active flag bit C_cmp(e.g., a value of "1"); otherwise comparator 131 generates an inactive flag bit C_cmp(e.g., a value of "0").

The imaging system 200 of the present embodiment may include a multiplexer 132 according to the flag bit C_cmpTo select a high sensitivity signal S_HWith a low sensitivity signal S_LAs a multiplexed signal S_M. If flag bit C_cmpIs active, then the signal S is multiplexed_MIs a high sensitivity signal S_H(ii) a Otherwise the signal S is multiplexed_MIs a low-sensitivity signal S_L。

The imaging system 200 of the present embodiment may comprise an analog-to-digital converter (ADC)11 for multiplexing the signal S_MIs equivalently converted from analog form to digital form, thereby generating a digital multiplex signal S_M。

In the present embodiment, the imaging system 200 may include a serial unit 133 that receives the digital multiplex signal S_MAccording to the flag bit C_cmpTo generate a coded signal S_E. If flag bit C_cmpFor initiative, the active flag bit C_cmp(having a value of "1" as the most significant bit) is concatenated to the multiplexed signal S_MI.e. high sensitivity signal S_H(step 105). For example, if the signal S has a high sensitivity_HIs N bits, then the signal S is encoded_EIs an (N +1) bit whose Most Significant Bit (MSB) is "1" (i.e., S)_E＝2^N+S_H). Conversely, if flag bit C_cmpIs inactive, the inactive flag bit C is set_cmp(having a value of "0" as the most significant bit) is connected in series to the multiplexed signal S_M(Low sensitivity signal)S_L) I.e. S_E＝S_L(step 106).

Compared to the first embodiment (fig. 1A), the encoding performed by the comparator 131 and the multiplexer 132 of the present embodiment is prior to the conversion performed by the adc 11. Therefore, the present embodiment can effectively reduce the analog-to-digital conversion period because only the high-sensitivity signal S_HWith a low sensitivity signal S_LPerforms analog-to-digital conversion. That is, the analog-to-digital conversion of the present embodiment requires only half of the previous embodiment at most. For the architect of FIG. 3 that does not use this embodiment, more analog-to-digital conversion cycles are required. For example, for an M × N pixel array configured in space (spatial), an M × N analog-to-digital conversion period is required, instead of only an M × N/2 conversion period as in the present embodiment. For the temporal high dynamic range example, such as interleaved high dynamic range, which exposes (and reads out) the pixel array Z times, an mxnxz analog-to-digital conversion cycle is required, rather than the mxnxz/2 conversion cycle as in the present embodiment.

Fig. 4 shows a block diagram of a high dynamic range imaging system 300 according to a third embodiment of the present invention. In fig. 4, blocks similar to those of the first embodiment (fig. 1A) or the second embodiment (fig. 3) are denoted by the same reference numerals, details of which are not repeated, and the flowchart shown in fig. 1B is used.

In the present embodiment, the high dynamic range imaging system (hereinafter referred to as the imaging system) 300 may include an Analog Signal Conditioner (ASC) 31 that receives and conditions (e.g., performs current-to-voltage conversion, Correlated Double Sampling (CDS), filtering, and/or amplifying) the signals collected by the high-sensitivity pixels 10H and the low-sensitivity pixels 10L, thereby generating the high-sensitivity signal S_HWith a low sensitivity signal S_L。

Similar to the previous embodiment, this embodiment only needs to generate and transmit one (flag) bit, rather than multiple bits as is typical for conventional imaging systems. In addition, compared to the conventional system, the present embodiment only requires half of the analog-to-digital conversion period.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the claims of the present invention; it is intended that all such equivalent changes and modifications be included within the scope of the following claims without departing from the spirit of the invention as disclosed.

Description of the symbols

100 high dynamic range imaging system

200 high dynamic range imaging system

300 high dynamic range imaging system

10 pixel array

10H high-sensitivity pixel

10L low-sensitivity pixel

11 analog-to-digital converter

12 memory

13 high dynamic range encoder

131 comparator

132 multiplexer

133 series unit

14 emitter

15 receiver

16 high dynamic range decoder

31 analog signal regulator

101 provide a high dynamic range pixel array

102 determine the photosensitivity ratio

103 receive the high sensitivity signal and the low sensitivity signal

104 comparing the high sensitivity signal with a predetermined threshold

105 coded signals equal to high sensitivity signals concatenated with most significant bits

106 coded signal equals low sensitivity signal

107 transmit coded signals

108 receive the transmitted encoded signal

109 determining whether the flag bit is active

110 removing the flag bit to generate a decoding signal

111 multiplying the encoded signal by the photosensitive ratio to generate a decoded signal

S_HHigh sensitivity signal

S_LLow sensitivity signal

S_EEncoding a signal

S_TPreset critical value

S_DDecoding a signal

R photosensibility ratio

S_MMultiplexing signals

Claims

1. A high dynamic range imaging system, comprising:

a pixel array providing a high sensitivity signal and a low sensitivity signal; and

a high dynamic range encoder for generating an encoded signal based on the high sensitivity signal and the low sensitivity signal;

when the high-sensitivity signal is not larger than a preset critical value, the flag bit is set to be active, otherwise, the flag bit is set to be inactive.

2. The high dynamic range imaging system of claim 1, wherein said predetermined threshold is equal to or less than a saturation level of said high sensitivity signal.

3. The high dynamic range imaging system of claim 1, further comprising:

an analog-to-digital converter disposed between the pixel array and the high dynamic range encoder for converting the high sensitivity signal and the low sensitivity signal from an analog form to a digital form;

wherein the active flag bit is concatenated to the digital high-sensitivity signal to generate the encoded signal when the digital high-sensitivity signal is not greater than the predetermined threshold; otherwise the flag that is not active is concatenated to the low sensitivity signal in digital form to generate the encoded signal.

4. The high dynamic range imaging system of claim 3, further comprising:

a high dynamic range decoder for generating a decoded signal according to the encoded signal;

wherein when the flag bit is active, the flag bit is removed to generate the decoding signal; otherwise, multiplying the coding signal by a sensitization ratio between the high-sensitivity pixel and the low-sensitivity pixel to generate the decoding signal.

5. The high dynamic range imaging system of claim 3, wherein said high dynamic range encoder further comprises:

and the memory temporarily stores at least one of the high-sensitivity signal and the low-sensitivity signal.

6. The high dynamic range imaging system of claim 1, wherein said high dynamic range encoder comprises:

a comparator for comparing the high sensitivity signal with the preset critical value;

when the high sensitivity signal is not greater than the preset critical value, the comparator generates the active flag bit; otherwise the comparator generates the flag bit that is not active.

7. The high dynamic range imaging system of claim 6, wherein said high dynamic range encoder further comprises:

a multiplexer for selecting one of the high-sensitivity signal and the low-sensitivity signal as a multiplexing signal according to the flag bit;

an analog-to-digital converter to convert the multiplexed signal from an analog form to a digital form; and

the serial unit receives the multiplexing signal in a digital form and generates the coding signal according to the flag bit;

wherein when the flag bit is active, the multiplexed signal is the high-sensitivity signal, and the active flag bit is concatenated to the high-sensitivity signal in digital form to generate the encoded signal; otherwise the multiplexed signal is the low luminance signal and the flag that is inactive is concatenated to the low luminance signal in digital form to generate the encoded signal.

8. A high dynamic range imaging method, comprising:

providing a pixel array to generate a high sensitivity signal and a low sensitivity signal;

determining the sensitization ratio between the high-sensitivity pixels and the low-sensitivity pixels; and

generating a coding signal according to the high sensitivity signal and the low sensitivity signal;

9. The high dynamic range imaging method of claim 8, wherein said predetermined threshold is equal to or less than a saturation level of said high sensitivity signal.

10. The high dynamic range imaging method of claim 8, further comprising:

converting the high sensitivity signal and the low sensitivity signal from an analog form to a digital form;

11. The high dynamic range imaging method of claim 10, further comprising:

generating a decoding signal according to the coding signal;

wherein when the flag bit is active, the flag bit is removed to generate the decoding signal; otherwise, multiplying the coded signal by the photosensitive ratio to generate the decoded signal.

12. The high dynamic range imaging method of claim 8, further comprising:

comparing the high sensitivity signal with the preset critical value;

wherein when the high sensitivity signal is not greater than the predetermined threshold, the active flag bit is generated; otherwise, the flag bit is generated to be inactive.

13. The high dynamic range imaging method of claim 12, further comprising:

selecting one of the high-sensitivity signal and the low-sensitivity signal as a multiplexing signal according to the flag bit;

converting the multiplexed signal from analog to digital form; and

generating the coding signal according to the flag bit;