KR20220164774A - Imaging device and method - Google Patents

Abstract

An imaging device comprising a 3D thermal imager, a signal processor, a laser, an optical scanner, a controller and a bimodal array operating a plurality of slow thermal detectors to generate 2D thermal images as inputs to the signal processor. The controller operates the bimodal array, laser, optical scanner, and thermal imaging camera in a predetermined sequence to derive distance data for delivery to a signal processor that interlaces the 2D thermal image with the distance data to create a 3D thermal image. do. The method uses a slow response thermal detector to derive an analog signal detected from an emitting laser pulse returned from a target, and asynchronously samples the detection signal to obtain a series of two time events corresponding to rising and decaying portions of the detection signal. derive Flight times and distances are calculated by using two time event series.

Description

Imaging device and method

The present disclosure relates to a 3D camera and a method for range finding, in particular, by performing thermal imaging, measuring distance, interlacing the distance with a 2D thermal image, and Apparatus utilizing slow response detectors within the thermal range of the optical spectrum to derive 2D and 3D imaging data from landscapes.

Imaging apparatus or imaging devices are well known in the art and are widely used for thermal imaging in 2D and for obtaining 3D imaging data, for various purposes, for example in the field of distance measurement. One drawback with the sophisticated devices available is their construction which requires high quality and therefore expensive signal detectors. Another disadvantage is the need to use cooled thermal detectors due to their high cost and low reliability. Another disadvantage is that these devices are dedicated to a specific application. For example, one distance finder to measure distance, one 2D thermal imager to distinguish temperature and one 3D imager to acquire 3D imaging are examples.

Accordingly, it would be advantageous to provide an apparatus capable of combining a plurality of imaging devices into one apparatus. For example, a 3D imaging camera that integrates and combines the features of a range finder with the features of a 2D thermal imager, and also uses an uncooled slow response detector, which can be afforded but delivers the same results as top-of-the-line equipment. It would be advantageous to provide

[Description of related technologies]

The equation derived below for the change in the amount of charge carriers on a photodetector as a response to incident light radiation of the flux φe(t) is given by E.L. Dereniak and G.D. Boreman, "Infrared Detectors and Systems", Chapter 5, para. Based on 5.6 pp.190-192.

Also, MAIMON Shimon, W02020/018600, dated January 23, 2020, which mentions an image detector for detecting objects, including selectively detecting meteorological phenomena, and Margarit, Josep Maria, et al. "A 2 kfps sub-μ W/Pix uncooled-PbSe digital imager with 10 bit DR adjustment and FPN correction for high-speed and low-cost MWIR applications." IEEE Journal of Solid-State Circuits 50.10 (2015): 2394-2405 is related.

An object of the embodiments described herein is to disclose an apparatus for 3D imaging that combines a distance measurement with a 2D thermal image and a method of implementing the apparatus. Devices include signal processors, laser devices, optical scanners and controllers.

A feature of the device according to the exemplary embodiment is the use of a slow response thermal detector for 3D imaging with high distance resolution. Slow response detectors are generally available and are advantageously priced. The device combines the 2D thermal image with the distance data and creates a 3D thermal image from it.

An additional feature of the device according to the exemplary embodiment is the use of a bimodal array of slow response thermal detectors (THRDTC) (BMDARR) to produce asynchronous pixel sampling for standard thermography or distance measurement. The bimodal array is commanded by a controller and includes a plurality of slow response thermal detectors. Also, the bimodal array is designed to generate 2D thermal images and accurate distance data by use of respective slow response thermal detectors. Both the 2D thermal image and distance data collected by each slow-response thermal detector in the array are passed to the signal processor.

The controller is further configured to operate each slow response thermal detector of the laser, optical scanner and thermal imager. This allows the controller to operate the laser, optical scanner and thermal detector of the thermal imager and derive distance data therefrom. The distance data is collected by use of a pixel sampling circuit dedicated to deriving a plurality of time event couples (t1i, t2i). These time event couples t1i and t2i are derived from the sampled detection signal DS which is passed to the signal processor.

Finally, the signal processor is configured to calculate the distance with an accuracy much better than the response time of the slow response thermal detectors in the array. The signal processor may interlace the 2D thermal image with the distance data to create a 3D thermal image by combining the distance data with the 2D thermal image.

It is noted that the bimodal array includes a plurality of slow response thermal detectors, each of which produces one pixel in a 2D thermal image and a 3D image.

Also provided is a method for constructing and implementing an apparatus for 3D imaging that combines distance measurements with 2D thermal imaging. The device includes a 3D thermal imager, signal processor, laser, optical scanner and controller. The device uses a slow response thermal detector (THRDTC) to derive data from the detected analog signal. The analog signal may be a response to an emitting laser pulse (LP) returned from the target. The apparatus is configured to asynchronously sample the detection signal and derive data therefrom. This derived data may include two series of time events denoted t1i and t2i, respectively.

The apparatus is further configured to derive from the data a length of time (Tbg) equal to a time of flight (TOF) by use of a formula. This equation includes one of Equation 3 and Equations 4.1 to 4.3. Then, the distance R to the target TRGT is calculated by use of equation (5).

[Technical Issues]

One problem relates to the implementation of an apparatus for distance measurement using a slow response detector device, cooled or uncooled, having a relatively slow time response for operation in the thermal region of the optical spectrum. Refers to the use of uncooled heat detectors. A slow time response means that it is slow relative to the desired time dependent resolution of the ranging, but serves as an accurate range finder for the desired ranging resolution. The thermal region of the optical spectrum is defined as the range of optical wavelengths from 2 μm to 14 μm.

Another problem is how to integrate the 3D imaging of the landscape observed in the device APP with the 2D thermal imaging, in order to obtain a 3D thermal image of the whole landscape or selected parts thereof. 2D means two dimensions, and 3D means three dimensions.

1A shows the principle of operation of a commonly used distance measuring device. In a general distance measuring device, a laser pulse LP emitted toward a target TRGT by a laser generating device LSR hits the target TRGT and a photodetector DTCT (or short detector DTCT) is detected therefrom. ) measures the time-of-flight (TOF), which is the time it takes to return to Then, the distance R can be calculated by multiplying the speed of light by half the derived time-of-flight (TOF).

Clearly, the accuracy of distance measurement depends on the measurement accuracy of time-of-flight (TOF).

To this end, such a general accurate distance finder has a time constant (τ) shorter than the desired time resolution (dt), which depends on the resolution of the distance measurement, and a time scale of the same order as the time constant (τ), as shown in FIG. Both laser pulses (LP) with a pulse width (PW) are required.

1 b shows the time sequence of short laser pulses LP emitted towards and returning from the target TRGT and from the laser pulse LP by a fast response photodetector DTCT operating as a distance measuring device according to the prior art. The derived detection analog signal DS is shown. The time-of-flight (TOF) is also shown in FIG. 1B.

The equation commonly used to derive the distance separating the target (TRGT) away from the range finder is:

R = c*TOF/2 (Equation 1)

where R is the distance, c is the speed of light, and the time measured from the emission time of the laser pulse to the start of generation of the detector signal DS travels in both directions to the target TRGT and back to the detector DTCT. TOF/2 is the time-of-flight (TOF) divided by 2, since it is equal to the time taken by light.

For example, to obtain a distance accuracy measurement of 5 m, the required temporal resolution (dt) is the time it takes for light to travel back and forth 5 m, which is approximately 33 ns. Therefore, a slow response detector (DTCT) with a time constant (τ) of less than 1 ns is required to measure the distance R with this accuracy. Since many slow response detectors (DTCTs), especially slow response thermal detectors (THRDTCs), have time constants (τ) in the range of a few μs to a few ms, if the same ranging accuracy can be achieved with a slow response detector (DTCT), It will be advantageous.

Another issue concerns the ability to use the device (APP) as a Light Imaging Detection And Ranging device (LIDAR) and derive a 2D thermal image from the observed landscape for interlacing with the distance measured by the LIDAR. In other words, it is the ability of the device (APP) to combine measured distances with 2D image pixels to form a 3D thermal representation or 3D thermal image of a targeted object. A 3D thermal representation is a mathematical representation of the surface and/or dimensions of an object including its length, width and depth as well as its relative temperature.

A typical LIDAR system uses a fast response detector (DTCT) and a short-pulse laser (LSR) as a single detector or array of detectors to create a 3D representation. LIDARs operating in the near-infrared region of the optical spectrum are commercially available, but cannot acquire 2D thermal images of the observed landscape in combination with 3D imaging of the same landscape.

Data included in the 2D thermal image representing the relative temperature between pixels and data included in the 3D image representing the different distances to each pixel complement each other. For advanced image processing, algorithms and artificial intelligence, it would be advantageous if both were represented as a single 3D thermal representation or interlaced as a thermal image. An apparatus (APP) for obtaining such a 3D thermal representation based on a slow response dual mode thermal detector array (or bimodal array (BMDARR) for short) may be more advantageous for cost effectiveness and simplicity of design.

To address the first problem, a method of measuring distance R by use of a slow response thermal detector (THRDTC) is now described.

Figure 1 schematically shows a graph of amplitude versus time of a laser pulse LP emitted at time t = 0. The amplitude of the laser pulse LP rises to a plateau through a rise time tr and then falls through a fall time tf, taking longer than the time constant τ of the slow response detector THRDTC. It has a sustained pulse width (PW). The time constant τ is not shown in the figure.

펄스(LP)로부터 저속 시간 응답 검출기(THRDTC)에 의해 도출된 검출 신호(DS)를 더 도시한다. 검출 신호(DS)가 비행 시간(TOF)의 마지막에서 시작하기 때문에, 검출 신호(DS)의 시작 시간은 이에 따라 시간 Tbg이다. 정의에 따르면, 시간 Tbg는 비행 시간(TOF), 즉 레이저 디바이스(LSR)에 의해 타겟(TRGT)을 향해 방출된 레이저 펄스(LP)가 타겟(TRGT)에 부딪히고 그로부터 복귀하는 데 걸리는 시간과 같다. 따라서 다음과 같다:1 shows a laser emitted toward and returning from a target TRGT.

It further shows the detection signal DS derived by the slow time response detector THRDTC from the pulse LP. Since the detection signal DS starts at the end of the time of flight TOF, the start time of the detection signal DS is thus time Tbg. By definition, the time Tbg is equal to the time of flight TOF, i.e. the time it takes for the laser pulse LP emitted by the laser device LSR towards the target TRGT to hit the target TRGT and return from it. . So:

TOF = Tbg (Equation 2)

As shown in FIG. 1, the detection signal DS rises exponentially from the end of the time of flight TOF to the time Tpeak reaching the maximum intensity of the detection signal DS, and then the detection signal DS rises geometrically. decrease exponentially Reference is now made to sampling of the detection signal DS, in particular to an asynchronous sampling method.

Fig. 2 shows an asynchronous sampling of the detection signal DS, and Fig. 3 shows a specific result of this sampling process.

According to the definition, the asynchronous sampling method samples both the rising part ta and the attenuating part td of the detection signal DS, and the sampling proceeds in sequential and successive amplitude steps of constant and equal amplitude level AMPLVLi. .

In Fig. 2, the amplitude interval separating the amplitude levels AMPLVLi is constant, but not the time interval separating successive time events of asynchronous sampling. For each amplitude level (AMPLVi), the asynchronous sampling method derives two sampled signals corresponding to two time events. The first time event t1i is for the rising part ta of the detection signal DS, and the second time event t2i is for the attenuating part td of the detection signal DS. Thus, according to index i for one equal amplitude level (AMPLVLi), index j = 1 for the occurrence of the time event during the rising part and index j = 2 for the decaying part of the sampled detection signal DS, the amplitude level is It can be denoted as AMPLVLi, and asynchronous time sampling can be denoted as tji. Index i is an integer from 1 to n, and index j is 1 or 2. In Fig. 2, for example, for an amplitude level of i = 3 or AMPLVL3, the corresponding values of time event tji are tl3 and t23.

Fig. 3 clearly shows that the amplitude interval separating the amplitude levels AMPLVLi is constant, whereas the distance separating the two time events tji depends on the shape of the detection signal DS. 3 also illustrates a specific case in which the sampled signal SIGSMP has three different amplitude levels AMPLVL. In a more general case, a plurality of amplitude levels (AMPLVLi) will be generated depending on the distance (R) of the target, a radiometric property of the target, such as emissivity, and an amplitude interval (INTVL) separating the amplitude levels (AMPLVLi). can

The two time event series, t1i and t2i, can be used to calculate the start time Tbg of the detection signal DS shown in Figs. 1, 2 and 3 by use of the following equations.

tf

0일 때, 검출 신호(DS)의 시작 시간(Tbg)은 다음의 수학식 3을 사용하여 계산될 수 있다:The rise time (tr) and fall time (tf) are approximately zero or close to zero, so that tr

tf

When 0, the start time Tbg of the detection signal DS can be calculated using Equation 3:

(수학식 3)

(Equation 3)

In other cases, the detection signal start time Tbg can be calculated by using the following three equations, Equations 4.1 to 4.3:

(수학식 4.1)

(Equation 4.1)

(수학식 4.2)

(Equation 4.2)

(수학식 4.3)

(Equation 4.3)

The time Tbg can be determined by selecting a pair of times (t1i, t2i) corresponding to a certain amplitude level (AMPLVLi) or alternatively by using several time events (t1i, t2i) belonging to different amplitude levels (AMPLVLi). It can be calculated by averaging the Tbg time values.

Now, the distance R to the target TRGT can be calculated by using Equation 2 and Equation 5:

R = c*Tbg/2 (Equation 5)

It is noted that the only variation in the calculation of distance R is a series of time events (t1i, t2i).

Thus, it can be concluded that a slow response detector (THRDTC), i.e. a cooled or uncooled thermal detector with a relatively slow time response, can be used to measure the distance R. The accuracy of the distance measurement R depends only on the accuracy of the time measurement of the time event series t1i and t2i.

The apparatus (APP) described herein has the technical advantage of using a slow response detector to derive a 3D thermal imaging that combines a distance measurement with a 2D thermal image. The description is insufficient to disclose or suggest many practical advantages in many fields of use.

Indeed, the advantages of the device APP include distance measurement and representation of the scenery in 3D in combination with selected 2D thermal images of the scenery.

The performance of the device APP may be enhanced for a particular use by calibration of distance and geometric dimension measurements, and by calibration for relative or absolute temperature measurements.

For public health purposes, the device (APP) is selected from among a first group of people whose body temperature is above an a priori selected level where the individual is individually targeted, located by distance, and identified by measurement of a biometric characteristic. Recognize the individual collectively. Further, the device APP recognizes an individual closer to an individual in the first recognized group by less than a specifically selected range of mutual separation distances, and puts the individual in the second group at risk of being infected by the first identified group. identify as having Thereby, the device APP replaces sequential testing and identification of sick or contaminated persons. For instance, in the case of an epidemic, the device (APP) enables the automatic designation and identification of specific individuals at risk of contracting an epidemic due to ignoring the minimum mutual separation distance in a crowd. Thereby, the isolation of at-risk individuals can prevent the spread of infectious diseases to people.

Lifesaving can be realized with a combination of the three technological achievements described herein and the economic advantages of simplicity of design and low-cost production. Combinations of these technological functions can be used in advanced driver assistance systems (ADAS), for example, when detecting and avoiding people and animals in poor visibility situations and when handling complex situational hazards to prevent accidents. The application is relatively available and cost effective compared to existing systems.

In addition, life saving can be realized by installing a device (APP) for detecting the driver's biometric information and providing a warning not to forget the infant in the parked vehicle inside the vehicle.

In case of a fire, the device (APP) can detect people and recognize their number and distance, even if they are hidden from view beyond walls of flame.

In manufacturing, for example, for a group of products placed within the field of view of the device APP, products that exceed predetermined critical dimensional tolerances and/or temperature ranges are detected and identified simultaneously instead of separate and separate inspection procedures.

In robotics for example, the device APP can be used as an improved artificial eye, whereby an accurate 3D image, including thermal mapping, provides additional data to the robot, thus providing an image of the surrounding area. accuracy can be increased.

In addition, the device (APP) is designed to improve the situational awareness of the control center and to add 3D thermal and Lidar imaging to existing optical cameras for increased public safety in a cost-effective manner, "Smart Seat ( Also known as “smart city”, it can be used in cities that incorporate advanced technologies to improve the experience and well-being of their residents.

In the drawings, like reference characters generally refer to the same parts throughout the different views. In addition, the drawings are schematic and not to scale, and focus instead on explaining the principles of the present invention generally. Various non-limiting embodiments of the present invention are described with reference to the following description of exemplary embodiments in conjunction with the following figures:
1A shows the principle of operation of a commonly used distance measurement method;
1b shows the time sequence of short laser pulses emitted to and returned from the target and analog signals detected therefrom by a fast-response photodetector, used with a distance measurement method according to the prior art;
1 shows a graph of a laser pulse (LP) and a signal (DS) derived therefrom captured by a slow response thermal detector (THRDTC);
Figure 2 shows an asynchronous sampling proceeding at constant amplitude intervals (INTVL) of an analog signal generated by a slow response detector in response to a received long laser pulse;
3 shows an exemplary structure of a signal SIGSMP with three amplitude levels AMPLVLi;
Fig. 4 is a block diagram of the main electrical circuit (MNCRCT) showing the electrical components of a distance measuring device (APP) by use of a slow response thermal detector (THRDTC) and asynchronous sampling;
5 shows a data storage pattern symmetric about the transition point (TRPNT) for use with an asynchronous sampling process;
6 shows a pixel sampling circuit PXSMP;
7 shows a thermal imaging detector array of standard design, referred to as a photoarray (PHTARR);
8 shows a bimodal array for thermal imaging and distance measurement (BMDARR) highlighting a selected line i;
9 shows a block diagram of an exemplary embodiment of an apparatus APP;
10 shows a part of a 2D image generated by a bimodal array (BMDARR) and a scanning pattern of a laser beam (LSRBM) projected thereon;
Figure 11 shows the successive stages of operation of the controller CNTRL of the apparatus APP for the generation of a 2D thermal image interlaced with a 3D image.

The distance measurement method described above using a slow response thermal detector (THRDTC) may be implemented in the device APP.

Figure 4 is a schematic block diagram of the main electrical circuit MNCRCT showing the electrical components of the Apparatus APP for distance R measurement by use of a low-cost detector and asynchronous sampling, and shows its operation. Low-cost detectors can cost at least one or even two orders of magnitude less than the cost of commercially available high-quality and high-speed detectors.

At time t = 0, a start signal (STRT) simultaneously initializes two elements of the device (APP): laser driver (LSRDR) and clock (CLK) (or time counter (CLK) which starts counting time). Provided. Also, the start signal STRT resets the intermediate memory INTMM coupled thereto. When initiated by the start signal STRT, the laser driver LSRDR commands the laser generating device LSR (or laser LSR) coupled thereto to emit a first laser pulse LP. This first laser pulse LP is then sent to the target TRGT and from there returns to the device APP. Upon return from the target TRGT, the laser pulse LP is received by a detector THRDTC, which may be a slow response thermal detector THRDTC.

Next, the thermal detector (THRDTC) converts the collected light from the laser pulse (LP) into an electrical signal referred to as a detection signal (DS). The thermal detector THRDTC, as shown above with respect to FIG. 1 b , measures from the emission time of the laser pulse LP starting at time t = Tbg to the time of the first laser pulse LP at time t = TOF + PW. Until the last return, the first laser pulse LP is received. As shown in FIG. 1B , TOF is the time of flight, PW is the pulse width, tr is the rise time, and tf is the fall time of the laser pulse LP. The slow response thermal detector (THRDTC) may be a single detector (THRDTC) or may be one pixel (PXL) of a slow bimodal array (BMDARR) described below. In response to the first laser pulse LP, the slow response thermal detector TRDTC outputs the detection signal DS to the preamplifier PA. A preamplifier (PA) is a preamplifier circuit according to the prior art.

A preamplifier PA is used to amplify the detection signal DS into an analog voltage signal Vo(t) which in turn serves as an input to the pixel sampling circuit PXSMP. The function of the pixel sampling circuit PXSMP is twofold. The first function of the pixel sampling circuit PXSMP is to determine and indicate whether the voltage signal Vo(t) is rising or falling. The flag signal indicated by R/F is set to LOGIC1 when the voltage signal Vo(t) is rising, and is set to LOGIC0 when the voltage signal Vo(t) is falling. A second function of the pixel sampling circuit PXSMP is to determine whether the signal Vo(t) has reached the next amplitude level AMPLVLi. If so, the pixel sampling circuit (PXSMP) sends the clock sampling signal (CLKSMP) to the intermediate memory (INTMM), which transmits the flag signal (R/F) and the current reading of the clock (CLK). make it save Clock (CLK) is a standard time counting component with a resolution derived from the required ranging resolution.

For example, if the required range resolution is 5 m, the time coefficient resolution of the clock CLK should be chosen to be 33 ns, as described above with reference to FIG. 1A. This pixel sampling circuit PXSMP, coupled to the intermediate memory INTMM, executes the process of generating the two time event series t1i and t2i described above with respect to FIGS. 2 and 3 . This process generates two series of time events (t1i, t2i) for storage in intermediate memory (INTMM), and for each sampled clock time (CLK) the flag signal (R/F) is belongs to the series t1i for the rising portion of the signal SIGSMP or to the series t2i for the falling portion thereof. The intermediate memory (INTMM) is coupled to a gate G1 which is further coupled to a standard field programmable gate array (FPGA) (or gate component (FPGA) for short).

The signal transmitted by the signal processor SGNPRC shown in Fig. 8 opens the gate G1, whereby the two time event series t1i and t2i are delivered as data files TMFL to the gate component FPGA. do. The gate component FPGA first finds the position of the transition point TRPNT in the data file TMFL by finding the position in the file where the flag signal R/F changes from LOGIC1 to LOGICO.

Accordingly, FIG. 5 shows that the transition point TRPNT is found by searching the data file TMFL for the location where LOGIC1 switches to LOGICO or the location where LOGICO changes to LOGIC1.

As shown in Fig. 5, the data file TMFL has a storage pattern symmetric with respect to the transition points TRPNT of the time event series t1i and t2i. Accordingly, the two time events t1i and t2i closest to the transition point TRPNT on both sides belong to the maximum amplitude level AMPLVL1. The next two second time events arranged symmetrically with respect to the transition point TRPNT belong to the next amplitude level AMPLVL2, and so on. Then, this pair of time events t1i, t2i, as described above with respect to FIGS. 1 and 3, to calculate the time Tbg of the detection signal DS and the distance to the target TRGT, Equation 3, Used by the gate component (FPGA) with Equations 4.1 to 4.3 and Equation 5. The implementation of these equations, Equations 3, 4.1 to 4.3 and 5, in gate components (FPGAs) is a common engineering practice. Other parameters required for the calculation of the distance (R) including the width (PW) of the laser pulse (LP), the rise time (tr) and fall time (tf), the time constant of the detector (τ) and the speed of light (c) are the factory System constants or physical constants that may be pre-stored in the memory of the gate component during assembly in .

Implementing a pixel sampling circuit

An exemplary and schematic embodiment of a pixel sampling circuit PXSMP is shown in FIG. 6 .

In the first stage of FIG. 6 , the first comparator COMP1 transmits the instantaneous time between the analog amplified voltage signal Vo(t) received from the preamplifier PA as shown in FIG. 4 and the previous amplitude level AMPLVLi. Calculate the difference. For the first sample of the sampling process, the first sampling amplitude (AMPLVLi) is set to zero.

Then, in the second stage, a pair of comparators coupled to the first comparator COMP1, that is, the second comparator COMP2 and the third comparator COMP3 receive the output of the first comparator COMP1, and 1 The instantaneous difference of the output of the comparator (COMP1) is compared with the positive and negative constant voltages, which are +INTVL and -INTVL, respectively. When the instantaneous difference between the signal Vo(t) and the amplitude level (AMPLVLi) is equal to +INTVL, the output of the comparator (COMP2) rises to the voltage level defined by LOGIC1 and thus, compared to the previous amplitude level (AMPLVLi), It represents the rise of the voltage Vo(t) by the amplitude level. When the instantaneous difference between the signal Vo(t) and the amplitude level (AMPLVLi) is equal to -INTVL, the output of the comparator (COMP3) rises to the voltage level defined by LOGIC1, and thus compared to the previous amplitude level (AMPLVLi), Represents a decrease in voltage Vo(t) by an amplitude level. Accordingly, the output of the second comparator COMP2 may serve as the output flag signal R/F of the pixel sampling circuit PXSMP, as described above with reference to FIG. 4 .

Next, in the third stage, the logic XOR gate LXOR receives the output of a pair of comparators, that is, the second comparator COMP2 and the third comparator COMP3 of the second stage. When the output of either the second comparator (COPM2) or the third comparator (COPM3) is set to LOGIC1 but the outputs of both are not set to LOGIC1, the logic XOR gate (LXOR) outputs the logic output clock sampling signal (CLKSMP). will create As described above with reference to FIG. 4 , the second logic output clock sampling signal CLKSMP will store the current reading values of the flag signal R/F and the clock CLK in the intermediate memory INTMM. Also, when clock sampling signal CLKSMP is LOGIC1, standard sample and hold circuit SMPHOL coupled to the output of logic XOR gate LXOR will sample the present value of signal Vo(t). This last current value of the signal Vo(t) will be taken as the new previous amplitude level AMPLVLi as a reference for the next iteration of the pixel sampling circuit PXSMP. A standard delay line (DL) is used to delay the input of this new amplitude level (AMPLVLi) to the first stage comparator (COMP1) to avoid the occurrence of possible internal conflicts due to the asynchronous nature of this process.

Imaging detector implementation

7 shows an exemplary schematic embodiment of a standard design of a slow response thermal imaging photodetector array (PHTARR) (or photoarray (PHTARR) for short). Figure 7 shows part of two arbitrary pixel lines PXLij of the thermal imaging light array PHTARR, namely lines LNOAi and LNOAi+1. Each pixel PXLij in these two lines contains a photodetector composed of a material sensitive to incident light at thermal wavelengths of the light spectrum. Readout electronics ROIC implemented on the substrate of the photoarray PHTARR connects the pixels PXL to lines and lines to frames, individually for each line and individually for the entire photoarray PHTARR. includes a gate that The entire electronic device is referred to as an integrated circuit (ROIC) for short. The main purpose of this integrated circuit (ROIC) is to convert the signal of the column optical array (PHTARR) into an amplified voltage signal and to synchronously transmit this amplified voltage signal in a standard international video format.

Thermal light arrays (PHTARR) are constructed of materials that are sensitive to thermal wavelengths. Some examples of these materials that react slowly and are also suitable for uncooled imaging include PbSe, VOx and amorphous silicon.

FIG. 8 shows an exemplary schematic embodiment showing how the pixel sampling circuit (PXSMP) described above with respect to FIG. 6 may be incorporated into a standard slow response bimodal array (BMDARR). This bimodal array (BMDARR) can perform standard thermal imaging or asynchronous pixel sampling for distance measurement purposes as described with respect to FIG. 4 .

In FIG. 8 , line LNOAi shows an exemplary embodiment of one randomly selected line i of bimodal array BMDARR, and thus one line LNOAi of a plurality of lines LNOA of bimodal array BMDARR.

In Fig. 8, each pixel PXLij of the bimodal array BMDARR includes a thermal detector THRDTC and a preamplifier PA, which can be used for standard imaging purposes or for distance measurement. The selected mode, either imaging or ranging, which is the selected operation in which the pixels PXLij of the bimodal array BMDARR are set, is commanded by the mode switch PXMOD, as described below with respect to FIGS. 9 and 11 . received from an external source. When the dynamic mode of the mode switch PXMOD is selected as LOGICO, the output signal of the preamplifier PA will be directly connected to the output of the bimodal array BMDARR for standard imaging. However, when the selected operating mode of the mode switch PXMOD is selected as LOGIC1, the output signal of the preamplifier PA will be coupled to the pixel sampling circuit PXSMP for the distance measurement described in connection with FIG. The simplicity of the pixel sampling circuit PXSMP allows its implementation to be included in each pixel PXLij.

Next, the pixel PXL of line LNOAi is connected to the output of the integrated circuit ROIC, via a gate, in the same way as in the standardly used imaging mode. The other components shown in Fig. 4, including the intermediate memory (INTMM), clock (CLK) and gate component (FPGA), are different from the bimodal array (BMDARR) and are assembled in a signal processor (SGNPRC) separate from it.

Alternatively, some of these components may be implemented as part of a bimodal array (BMDARR).

Implementation of 3D thermal imaging

As shown in Fig. 9, the device APP includes a thermal imager C1 (or camera C1) utilizing a bimodal array BMDARR and a signal processor SGNPRC. The signal processor SGNPRC, as shown in FIG. 8 , includes a clock CLK, an intermediate memory INTMM and a field programmable gate component FPGA. Apparatus APP further comprises a laser LSR controlled by a laser driver LSRDR. The laser LSR emits a laser beam LSRBM of a wavelength to which the bimodal array BMDARR is sensitive, and has a much narrower beam divergence than the field of view FOV of the camera C1.

Further components of the device APP include an optical scanner OPTSCN and a controller CNTRL for operating the device APP and other components thereof.

Optical scanner OPTSCN directs laser beam LSRBM to a selected area of the landscape viewed by camera C1. Assume that the optical scanner (OPTSCN) is aligned with the camera (C1) according to common engineering practice. It is further assumed that controller CNTRL coupled to camera C1 and optical scanner OPTSCN can direct laser beam LSRBM to a specific selected area of the landscape viewed by camera C1. This can be achieved by commanding the optical scanner (OPTSCN) to move at a specific predefined scan angle. That is, an area selected within the field of view (FOV) of the camera C1 may be uniquely mapped by a specific scan angle of the optical scanner OPTSCN.

Fig. 10 shows an exemplary sequence of operations of the controller CNTRL of the apparatus APP, which may be implemented to generate 3D imaging interlaced with 2D images. For clarity of explanation, it is accepted that the bimodal array (BMDARR) has 100 x 100 pixels (PXLij), ie 100 lines (LNOA) of 100 pixels (PXL) each. It is also acknowledged that the device APP produces images at a rate of 0.1 HZ, which is equal to a time sequence of 10 seconds.

In a first stage of operation STG1, controller CNTRL activates bimodal array BMDARR to create a 2D thermal image of a selected part of the landscape. To this end, the bimodal array BMDARR is set to imaging mode by the controller CNTRL by commanding the operation of the mode switch PXMOD to LOGICO which is transmitted to the pixels of the bimodal array BMDARR. After that, the 2D image is captured by use of common standard methods and transmitted by slow response bimodal array (BMDARR) to gate component FPGA included in signal processor SGNPRC, as shown in FIG. . In FIG. 10, the time range required to capture and transmit the 2D image indicated by “capture” and “transmit” is 3.4 ms. This time range is based on an exposure time of 3 ms of the bimodal array (BMDARR) to the landscape, followed by a read-out time of 400 μs of the pixel values and the transfer time of these values to the gate component (FPGA).

In the second operational stage STG following the previous first stage STG1, the controller CNTRL will operate the following components of the device APP to generate the 3D image: Bimodal array BMDARR , laser (LSR) through laser driver (LSRDR), optical scanner (OPTSCN) and signal processor (SGNPRC).

Still in the second stage of operation (STG), FIG. 11 shows an exemplary portion of 21 x 20 pixels of the 2D image generated by the bimodal array (BMDARR) in the first stage (STG1). 11, the scanning pattern of the laser beam LSRBM projected on the bimodal array BMDARR is indicated by an arrow marked A. In this example, the width of laser beam LSRBM is assumed to illuminate two pixels of bimodal array BMDARR at each instant when reflected back to camera C1 from landscape or target TRGT.

The controller CNTRL adjusts the scanning angle of the optical scanner OPTSCN according to a specific scanning pattern, so that the laser beam LSRBM captures the selected part of the landscape viewed by the camera C1 by two pixels PXLij at a time. command to scan.

As a first step, to illuminate the first set SET1 of the two pixels PXLij, the controller CNTRL shown in FIG. 9 starts by setting the scanning pattern and accordingly the scan angle of the optical scanner OPTSCN. . In Fig. 11, the first set SET1 is placed in the uppermost row at the top of the first and second columns, LNOA1 and LNOA2, respectively. After that, the controller CNTRL sets the mode switch PXMOD to LOGIC "1". In a second step, the controller CNTRL starts the distance measurement process by sending a start command to the clock CLK, the intermediate memory INTMM of the signal processor SGNPRC and the laser driver LSRDR. This is followed by the process described above with reference to FIGS. 4-6. In this example, the duration of this entire process is 2 ms.

Next, in the third stage STG3, to pass the file TMFL to the gate component FPGA for processing, the controller CNTRL emits a signal to open the gate G1 shown in FIG. 4. . In response, the gate component FPGA calculates the distance to the portion of the scenery associated with the first set of pixels SET1 as described in relation to FIG. 11 . In parallel, the controller CNTRL moves the optical scanner OPTSC to illuminate a second set SET2 of two pixels PXL arranged in a second row immediately below and adjacent to the first set SET1, Then the distance measurement process for these two pixels (PXL) of SET2 is repeated. This distance measurement process is repeated further iteratively until the laser beam LSRBM reaches the end of the first two lines of the bottom row (LNOA1 and LNOA2, respectively).

The laser beam LSRBM will then be directed by the controller CNTRL to the first two pixels of the next two lines (LNOA3 and LNOA4 respectively). This same scanning pattern is repeated repeatedly until the laser beam LSRBM has completed scanning of the whole part of the scenery, thereby generating the distance to each and every pixel PXLij of that part of the scenery. In this example, the duration for the entire operation of the bimodal array (BMDARR) is approximately 10 seconds. Adding the duration of 2D image generation, which is 3.4 ms, to the total duration of image generation gives 10 seconds, which correlates with the requirement for a speed of 0.1 HZ.

It should be noted that the various numerical examples provided herein are provided for illustrative purposes and do not represent the best obtainable performance achievable by operation of the apparatus APP. In the last stage, the gate component FPGA interlaces the 2D thermal image stored in its memory in pixel format with the distance data by adding the distance data of each pixel PXL to the stored 2D thermal image data. This creates a 3D thermal representation of the landscape.

The result of the three stages (STG) of the controller (CNTRL) is to actuate the 3D thermal imaging camera (C1) to image the captured landscape within its field of view, measure the distance to a selected object within its field of view, and determine the distance The ability to interlace 2D thermal images with distance data to create 3D thermal imaging that combines with 2D thermal images.

Although not shown as such in the drawings, other exemplary embodiments may operate differently (the operation sequence of the described stage may be different, the number of pixels illuminated momentarily and the scanning pattern may be different) means there is) should be noted.

Thus, an imaging device (APP) comprising a 3D thermal imaging camera (C1), a signal processor (SGNPRC), a laser (LSR), an optical scanner (OPTSCN) and a controller (CNTRL) has been described.

A feature of the device according to the exemplary embodiment is the use of a slow response thermal detector for 3D imaging with high distance resolution. Slow response detectors are generally available and are advantageously priced. The device combines the 2D thermal image with the distance data and creates a 3D thermal image from it.

Apparatus APP includes a bimodal array BMDARR which is an electronic circuit that operates a plurality of slow thermal detectors THRDTC. When used, the bimodal array (BMDARR) performs asynchronous pixel sampling for standard thermal imaging or distance measurement. The bimodal array is commanded by a controller and includes a plurality of slow response thermal detectors. Additionally, the bimodal array is designed to generate 2D thermal images and precise distance data by use of slow response thermal detectors. Both the 2D thermal image and distance data collected by each slow-response thermal detector in the array are passed to the signal processor.

The controller is further configured to operate each slow response thermal detector of the laser, optical scanner and thermal imager. This allows the controller to operate the laser, optical scanner and thermal detector of the thermal imager and derive distance data therefrom. The distance data is collected by use of a pixel sampling circuit dedicated to deriving a plurality of time event couples (t1i, t2i). These time event couples t1i and t2i are derived from the sampled detection signal DS which is passed to the signal processor.

Finally, the signal processor is configured to calculate the distance with greater accuracy than the response time corresponding to the slow response thermal detectors in the array. Thereby, the signal processor may interlace the 2D thermal image with the distance data to generate a 3D thermal image by combining the distance data with the 2D thermal image.

It is noted that the bimodal array includes a plurality of slow response thermal detectors, each of which produces a pixel in a 2D thermal image and/or a 3D thermal image.

Each detector (THRDTC) is coupled to a mode switch (PXMOD). The mode switch PXMOD is configured to set the bimodal array BMDARR to one of two operating modes: an operating mode, one of imaging mode, and a second mode, ranging mode.

The bimodal array BMDARR further includes a plurality of pixel sampling circuits PXSMP. Each of the pixel sampling circuits PXSMP is configured to recognize the rising portion ta of the detection signal DS and the attenuating portion td of the detection signal DS.

Signal processor SGNPRC further includes a clock CLK, which is coupled to intermediate memory INTMM. The pixel sampling circuit PXSMP is coupled to the intermediate memory INTMM. The pixel sampling circuit PXSMP is configured to transmit the clock sampling signal CLKSMP and the flag signal R/F to the signal processor SGNPRC. In response to these two signals, signal processor SGNPRC commands intermediate memory INTMM to store the current readings of clock (CLK) and flag signal (R/F).

Each of the pixel sampling circuits PXSMP is coupled to an intermediate memory INTMM and is configured to perform at least two operations. The first operation of the pixel sampling circuit PXSMP is to generate two time event series, a first time event series t1i and a second time event series t2i. The time event t1i belongs to the rising part ta of the detection signal DS, and the time event t2i belongs to the attenuating part td of the same detection signal DS. A second operation performed by the pixel sampling circuit PXSMP is to indicate which of the rising portion ta and the decaying portion td each time event belongs to.

Intermediate memory INTMM is coupled with gate G1 to component FPGA. The opening of gate G1 is commanded by a signal processor SGNPRC configured therewith. Thereby, the two time event series (t1i, t2i) stored in the intermediate memory (INTMM) can be transferred to the component (FPGA) as a data file (TMFL).

The gate component FPGA is configured to locate the transition point TRPNT in the data file TMFL. A transition point (TRPNT) separates between the two time event series (t1i, t2i). To find the transition point (TRPNT), the position at which the flag signal (R/F) switches from setting to LOGIC1 to setting to LOGIC0 or vice versa, when the flag signal (R/F) switches from setting to LOGIC0 to setting to LOGIC1 It is enough to find a location where

As described above, the pixel sampling circuit PXSMP is configured to derive a plurality of time event couples t1i and t2i from the sampled detection signal DS. This time event couple (t1i, t2i) can be used to calculate the distance to the target (TRGT).

The pixel sampling circuit PXSMP is further configured to deliver the flag signal R/F and derive a plurality of time event couples t1i and t2i. All of these are collected by asynchronous sampling of the detection signal DS. The detection signal DS has a rising portion ta and an attenuating portion td. The flag signal R/F is configured to indicate which of the rising portion ta and the decaying portion td each of the time events t1i and t2i relate to.

The pixel sampling circuit PXSMP is further configured to perform at least two more operations. One or more operations is to asynchronously sample the rising portion ta and the decaying portion td of the sampled detection signal DS. The purpose of asynchronous sampling is to derive two time event couples at constant and equal range sequential steps of interval (INTVL) and successive amplitude levels (AMPLVLi). The second operation is to transmit a flag signal R/F for the purpose of indicating which of the two time events t1i and t2i each time event belongs to. Next, the data file TMFL including the time event couples t1i and t2i and the flag signal R/F is stored in the intermediate memory INTMM according to the transmission of the clock sampling signal CLKSMP.

Each time event couple (t1i, t2i) has a first time event (t1i) and a second time event (t2i). At the first time event t1i, the first index is indicated as 1. At the second time event t2i, the first index is indicated as 2. The first index and the second index are correlated with the flag signal R/F. Each time event (t1i, t2i) is identified by a second index i. Its second index i is related to the amplitude level AMPLVLij. Each of the time events t1i and t2i is stored on the opposite side of the two-sided transition point TRPNT. The time event t1i is stored in the time file TMFL at the first side of the transition point TRPNT, and the time event t2i is stored in the time file TMFL at the second side of the transition point TRPNT. Time events with numerically lower index i are stored closer to the transition point TRPNT, while time events with numerically higher index i are stored farther to the transition point TRPNT.

The thermal detector (THRDTC) can be selected as an uncooled detector or as a cooled detector.

Accordingly, a method for constructing and implementing an imaging apparatus (APP) has also been described. The imaging device (APP) includes a 3D thermal imager (C1), a signal processor (SGNPRC), a laser (LSR), an optical scanner (OPTSCN) and a controller (CNTRL). The method uses a slow response thermal detector (THRDCT) to derive an analog detection signal (DS) from an emitting laser pulse (LP) returned from a target (TRGT). The method asynchronously samples the detection signal DS and derives from it two time event series, t1i and t2i, respectively. The method also derives time Tbg by use of (Equation 3) and use of (Equation 4.1), (Equation 4.2) and (Equation 4.3). Then, the method calculates the distance R to the target TRGT by use of (Equation 5).

The method uses a thermal detector (THRDTC) which can be selected as either an uncooled detector or a cooled detector.

In this method, the detection signal DS has a rising portion ta and an attenuating portion td. The detection signal DS is asynchronously sampled in its rising part ta and said attenuating part td.

In addition, the detection signal DS is asynchronously sampled in sequential and successive amplitude steps at constant and equal amplitude intervals INTVL.

In this method, the sampled signal SIGSMP is stored in a time file TMFL. The sampled signal SIGSMP is listed in the time file TMFL in a pattern symmetrical with respect to the transition point TRPNT.

The detection signal DS is amplified into an analog voltage signal Vo(t) before being input to the pixel sampling circuit PXSMP. The pixel sampling circuit PXSMP has a twofold function.

The detection signal DS is amplified into an analog voltage signal Vo(t) for input to the pixel sampling circuit PXSMP. The pixel sampling circuit PXSMP has at least two functions. A first function of the pixel sampling circuit PXSMP deals with determining and indicating whether the voltage signal Vo(t) is rising or decaying. A second function of pixel sampling circuit PXSMP deals with determining whether signal Vo(t) has reached its next amplitude level AMPLVLi.

The pixel sampling circuit PXSMP is an element of the main electrical circuit MNCRCT. The pixel sampling circuit PXSMP is configured to operate an asynchronous pixel sampling process.

The main electrical circuit (MNCRCT) includes a standard sample and hold circuit (SMPHOL). A sample and hold circuit SMPHOL is coupled to delay line DL. Delay line DL and sample and hold circuit SMPHOL are configured to prevent possible internal conflicts of the derived data.

Exemplary embodiments of the devices and methods described above are applicable to industries and appliances that use cameras, including, for example, public health, civil safety and protection, robotics, and guidance and navigation of vehicles.

Claims (15)

In an imaging device (APP) including a 3D camera (C1), a signal processor (SGNPRC), a laser (LSR), an optical scanner (OPTSCN) and a controller (CNTRL),
The device APP is configured to perform one of i) standard thermal imaging and ii) asynchronous pixel sampling for range finding and a bimodal array (BMDARR) commanded by the controller CNTRL. ),
The bimodal array (BMDARR) includes a plurality of slow response thermal detectors (THRDTC), and is configured to generate 2D thermal images and accurate distance data by using the slow response thermal detectors (THRDTC) and the 2D thermal image and distance data are transferred to the signal processor SGNPRC,
The controller (CNTRL) obtains distance data by use of a pixel sampling circuit (PXSMP) configured to derive a plurality of time event couples (t1i, t2i) from the sampled detection signal (DS) passed to the signal processor (SGNPRC). further configured to operate the laser (LSR), the optical scanner (OPTSCN) and the thermal imager (C1) to derive,
characterized in that the signal processor (SGNPRC) is configured to calculate the distance with an accuracy corresponding to the detector time response and superior to the detector time response and interlacing the 2D thermal image with the distance data to generate a 3D thermal image; To do, the device (APP). According to claim 1,
wherein the bimodal array (BMDARR) comprises a plurality of slow response thermal detectors (THRDTC) each generating one pixel in a 2D thermal image and a 3D image. According to claim 2,
The pixel sampling circuit PXSMP,
The rising part ta and the attenuating part td of the detection signal DS sampled to derive the two said time event couples at sequential steps of constant and equal range of interval INTVL and continuous amplitude level AMPLVLi. ) asynchronously sampled,
Derive a flag signal (R/F) to indicate which of the two time events (t1i, t2i) each time event relates to
configured to
A data file (TMFL) including the time event couple (t1i, t2i) and the flag signal (R/F) is stored in an intermediate memory (INTMM) according to transmission of a clock sampling signal (CLKSMP) Apparatus (APP) . According to claim 3,
Each of the time event couples t1i and t2i includes a first time event t1i with a first index of 1 and a second time event t2i with an index of 2, and 2 is correlated with the flag signal (R/F),
each time event (t1i, t2i) is identified by a second index i, which is associated with an amplitude level (AMPLVLi);
The time event t1i is stored in the time file TMFL at the first side of the two-sided transition point TRPNT, and the time event t2i is stored at the second side of the transition point TRPNT. stored,
A time event with a numerically lower index i is stored closer to the transition point TRPNT. According to claim 1,
Apparatus (APP), wherein the thermal detector (THRDTC) is one of an uncooled detector and a cooled detector. In a method for implementing an imaging device (APP) including a 3D camera (C1), a signal processor (SGNPRC), a laser (LSR), an optical scanner (OPTSCN) and a controller (CNTRL),
using a slow response thermal detector (THRDCT) to derive an analog detection signal (DS) from an emitting laser pulse (LP) returned from a target (TRGT);
asynchronously sampling the detection signal (DS) to derive two time event series therefrom, t1i and t2i, respectively;

(Equation 3)
and

(Equation 4.1),

(Equation 4.2),

(Equation 4.3)
deriving time Tbg by use of one of: τ is the time constant of the slow response detector (DTCT), t1i is the first time event, PW is the pulse width of the laser pulse (LP), and t2i is a second time event, tr is the rising time of the laser pulse LP, and tf is the falling time of the laser pulse LP;
R = c*Tbg/2 (Equation 5)
Calculating the distance (R) to the target (TRGT) using Tbg is the time of flight (TOF)
Characterized in that, the method comprising a. According to claim 6,
wherein the detector (THRDTC) is a slow response thermal detector (THRDTC) that costs at least one order of magnitude less than the cost of commercially available high-quality and high-speed detectors. According to claim 6,
The method of claim 1, wherein the detector (THRDTC) is an uncooled detector. According to claim 6,
wherein the detection signal (DS) has a rising portion (ta) and a decaying portion (td) and is sampled asynchronously in both the rising portion (ta) and the decaying portion (td). According to claim 6,
wherein the detection signal (DS) is asynchronously sampled in sequential and successive amplitude steps of constant and equal amplitude intervals (INTVL). According to claim 6,
The sampled signal (SIGSMP) is stored in a time file (TMFL) and listed therein in a pattern symmetrical with respect to transition points (TRPNT). According to claim 6,
The detection signal (DS) is amplified to an analog voltage signal (Vo(t)) before being input to a pixel sampling circuit (PXSMP) having a twofold function. According to claim 6,
The detection signal (DS) is amplified into an analog voltage signal (Vo(t)) for input to a pixel sampling circuit (PXSMP),
The pixel sampling circuit PXSMP,
a first function for determining and indicating whether the voltage signal (Vo(t)) is rising or decaying; and
A second function for determining whether the signal Vo(t) has reached the next amplitude level AMPLVLi
Having, how. According to claim 13,
wherein the pixel sampling circuit (PXSMP) is incorporated into a bimodal array (BMDARR) configured to perform asynchronous pixel sampling for thermal imaging and distance measurement. According to claim 14,
the pixel sampling circuit (PXSMP) is an element of the main electrical circuit (MNCRCT) and is configured to operate the asynchronous pixel sampling process;
wherein said main electrical circuit (MNCRCT) comprises a standard sample and hold circuit (SMPHOL) coupled to a delay line (DL), the last two being configured to prevent possible internal collisions of derived data.

Images