WO2021002822A1

WO2021002822A1 - Movable object detection device and method thereof

Info

Publication number: WO2021002822A1
Application number: PCT/TR2020/050580
Authority: WO
Inventors: Melih Ballikaya
Original assignee: Smarte Teknoloji Ve Enerji Sanayi Ticaret Anonim Sirketi
Priority date: 2019-07-04
Filing date: 2020-07-03
Publication date: 2021-01-07

Abstract

The movable object detection device (1) of the invention comprises a thermal camera (2), a distance sensor module (3) and a processor. Said thermal camera (2) is configured to constitute at least one thermal image of a physical field (A). Said distance sensor module (3) comprises a unit (31) comprising a TOF 2D SPAD array operating as a single pixel, and is configured to constitute at least one 1D distance data of at least one movable object in said physical field (A). A processor connected to the thermal camera (2) and the distance sensor module (3) matches said at least one thermal image and said at least one 1D distance data. The present invention also relates to a method to detect at least one movable object with assembly of the movable object detection device (1) onto the ceiling.

Description

MOVABLE OBJECT DETECTION DEVICE AND METHOD THEREOF

Field of the Invention

The present invention relates generally to a movable object detection device and method, and particularly to a thermal image acquisition in an infrared spectrum as well as a distance measurement for an object in said image with time of flight technology (TOF).

Backaround of the Invention

Qualitative and quantitative information acquired relating to a movable object within any image is used in fields such as automotive, robotics and safety engineering (plant supervision, people counting and pedestrian detection). A movable object has a role in a number of applications such as automated detection of people, energy system management, people traffic analysis and marketing analytical calculation. For example, HVAC systems in buildings with intense people traffic such as business centers and stores may be managed depending on the number of people therein and locations thereof, thereby acquiring remarkable energy savings. Moreover, it is also possible to provide security in said buildings in accordance with the number of people detected.

Different technologies and sensor platforms have been suggested for correct and effective people counting applications. It is known that visual cameras (e.g., CMOS, RGB cameras) are used for a number of human counting applications. However, data processing of high- resolution cameras is accompanied by complicated calculations. Thus, such applications requiring additional processing power are not time- or cost-efficient. Moreover, such cameras have high image qualities, thereby leading to personal security problems for people of whom images are recorded.

In order to overcome the problem of privacy caused by visual cameras, thermal cameras have been suggested that can acquire internal information such as temperature for the detection of movable objects (e.g., people). This kind of solution has almost no concern for privacy as resolution of said thermal cameras is low, and similar to human body temperature. However, depth information cannot be obtained in the thermal cameras acquiring 2- dimensional (2D) images just like the visual cameras due to perspective failures. In the prior art, the visual cameras and/or thermal cameras are used along with LIDAR devices to overcome the problem of depth in 2D images. However, pretty inefficient outcomes are obtained given the sensor fusion limitations and high costs of the devices used as well as the processing power required for calculation.

For instance, in a study conducted by Wim Abbeloos and Toon Goedeme (2013) relating to people detection, a time of flight (TOF) camera has been used. An approach commonly used for 3-dimensional (3D) distance measurements in TOF cameras constituting one class of LIDARs' consists of the measurement of the time between propagation and echoing of a measurement signal. The TOF is based on the principle of measuring the time and distance required for sending and receiving of a signal for a signal with a known propagation rate in a certain environment. In the system suggested by the researchers, there is a camera conducting the 3D distance measurement with the principle of TOF as well as a thermal camera. However, the fusion of 3D distance data and 2D thermal image data has led to calculation failures due to the spectral differences of the devices.

Hoegner et. al. (2014) has used a thermal camera comprising a TOF camera and a bolometer in an adapted experimental assembly. Nevertheless, the calibration method applied has led to incorrect outcomes due to different visibility ranges and occlusions.

The application numbered WO201 1 144458 (A1 ) has suggested a 3D imaging device for the detection of objects. Said device comprises a 3D distance measurement camera functioning with the principle of a TOF, a thermal sensor and also an RGB camera. However, the application does not have an original contribution to the prior art, and also industrial applicability of the invention requires inconvenient and expensive assembly.

Consequently, considering the suggestions mentioned above, the detection of movable objects (human, animal, etc.) constitutes a complicated problem due to factors such as failures in 3D and/or 2D data fusions. Hence, this phenomenon exists as a yet unsolved problem.

Summary of the Invention

The present invention relates to a movable object detection device comprising a thermal camera, a distance sensor module and a processor. Said thermal camera is configured to constitute at least one thermal image of a physical field. Said distance sensor module comprises at least one unit comprising a TOF 2D SPAD array operating as a single pixel, and is configured to constitute at least one 1 D distance data of at least one movable object in said physical field. A processor connected to the thermal camera and the distance sensor module matches said at least one thermal image and said at least one 1 D distance data.

In a preferred embodiment of the present invention, the thermal camera comprises a thermopile focal plane array.

In another preferred embodiment of the present invention, said thermal camera operates in a spectral wavelength ranging from 50 mm to1000 mm.

In another preferred embodiment of the present invention, said distance sensor module operates in a spectral wavelength ranging from 0.7 mm to 1 mm.

In another preferred embodiment of the present invention, the movable object detection device of the invention further comprises a wireless local area network module.

In another aspect, the present invention relates to a movable object detection method. By means of mounting of the movable object detection device onto a ceiling, said method comprises the steps of constituting a thermal image of a physical field (A) with a thermal camera, constituting at least one 1 D distance data of at least one movable object in a physical field with a distance sensor module, and matching said thermal image with said 1 D distance data.

Description of the Drawinqs

Fig. 1 illustrates the movable object device (1 ), the thermal camera (2) comprised by the device (1 ), the distance sensor module (3) and at least one unit (31 ) comprising TOF 2D SPAD array operating as a single pixel comprised by the distance sensor module (3) in a preferred embodiment of the invention.

Fig. 2 illustrates a first physical field (A^ corresponding to the field of view (FOV^ of the thermal camera (2) and a second physical field (A₂) corresponding to the field of view (FOV₂) of the distance sensor module (3), and a physical field (A) being an intersection of said first physical field (A^ and said second physical field (A₂) in a preferred embodiment of the invention. The fields exhibited as A₂’, A₂” and A₂”’ herein refers to the physical field (A₂’, A₂” or A₂”’) corresponding to the field of view of at least one unit (31 ) comprising TOF 2D SPAD array operating as a single pixel comprised by the distance sensor module (3). In this embodiment, 3 physical fields (A₂’, A₂”, A₂”’) are illustrated as 3 units (31 ) comprising the TOF 2D SPAD array operating as a single pixel are used.

Fig. 3 illustrates the SPAD pixels corresponding to the TOF 2D SPAD array opearting as a single pixel as a first zone (Z^ and a second zone (Z₂).

Detailed Description of the Invention

The present invention presents a device and method for the detection of movable objects. Devices and methods according to some embodiments of the present invention will be better understood with the explanations in this section as well as the drawings. However, it should be noted that the present invention is not limited to the embodiment explained below in detail, and the modifications thereof. In this respect, the drawings presented shall not limit the present invention, but only aims for a better understanding of some embodiments of said invention. Hence, the present invention has further embodiments within the scope of the claims. It is also noted that the terminology used here does not have any limiting purpose besides explanation. in one aspect, the present invention relates to a movable object detection device (1 ) comprising a thermal camera (2), a distance sensor module (3) and a processor. The term "movable object" herein is used to refer to an object with a certain movement direction and speed. In particular, said term comprises people, animals and the like with a constant body temperature.

Said thermal camera (2) comprised by the movable object detection device (1 ) of the present invention is configured to constitute at least one thermal image of a physical field (A). In other words, the present invention comprises thermal measurement. In accordance with the Stefan-Boltzmann law, each object produces an energy propagation and absorption from/into the environment thereof. The law is expressed as P =e^*o^*A^*(T₀ ⁴-T_c ⁴) as a formula. Wherein: P is the total propagation in a unit of time, A is the object surface area, T₀ is the temperature of the object, T_c is the temperature of the environment, e is a coefficient dependent on color and surface structure of the object, and s is the Stephan Boltzmann constant. A thermal camera consists of series of elements measuring the total propagation (P) value. Hence, it provides information relating to a hotter or colder presence of a certain object in an imaged physical field (A) with respect to the other surrounding components.

The term“physical field (A)” herein refers to a field of intersection of a first physical field (A^ corresponding to the field of view (FONA) of said thermal camera (2) and a second physical field (A₂) corresponding to the field of view (FOV₂) of the distance sensor module. The thermal camera (2) produces 2-dimensional (2D) images known as thermal images in the physical field (A). A thermal image is generally acquired by constituting a 2D temperature map of a surface. Said thermal image may have a visual form or a thermal information form corresponding to this visual form, or a combination thereof. For example, thermal information may be stated as a multiple-numerical set. Said n-dimensional (-tuple) set may contain the coordinates of each thermal camera as well as thermal information corresponding to the pixel of each thermal camera.

The term "thermal camera pixel" herein states each data point comprised by the thermal image. The thermal image generally consists of multiple thermal camera pixels. Each thermal camera pixel herein may provide visually expressed thermal information using a colorful or gray scale.

As may be understood from the statements above, the thermal information has a form of thermal image that can be convertible into visible signals.

The terms "thermal image" and "thermal information" are used interchangeably and without limiting the scope of the invention in any manner during the specification. Hence, the term thermal image is not limited to the transformation of the thermal information into the visual signals. For instance, the thermal image may be stored in a computer-readable environment as a multiple dimensional (n-tuple) set as mentioned above.

On the other hand, as the 2D thermal images of the thermal cameras have low temporal and spatial capacity, they are known to be insufficient for the detection of the movable objects in the art. Thus, the inventor has reasonably suggested a device (1 ) in which the thermal camera (2) and the distance module (3) operate simultaneously. Thus, a practical and cheap invention is presented without the requirement for any complicated sensor fusion calculations.

Hence, said distance sensor module (3) comprised by the object detection device (1 ) of the present invention comprises at least one unit (31 ) comprising a time of flight (TOF) 2D single photon avalanche diode (SPAD) array operating as a single pixel. The unit (31 ) comprising a TOF 2D SPAD array operating as a single pixel may also comprise a light source. In optical calculation systems, measurement signals are light waves. The term "light" as used herein is used - unless otherwise noted - with a meaning to cover the infrared light. In other words, the term "light" refers to an electromagnetic radiation with a spectral range from 0.7 mm to 1000 mm.

The TOF technology used herein is the measurement of the distance between a sensor and an object with calculation of times for sending of the photons from a light source to an object, and returning of the sent photons from said object to the sensor with echoing. This is defined as direct measurement of the distance. In an alternative scenario, the distance may be measured by calculating the phase shift between the photon signal coming from the light source and the photon signal sent to the object and detected by the sensor. This is the indirect measurement of the distance.

TOF 2D SPAD array operating as a single pixel comprises single photon avalanche diodes (SPAD). Use of photodiodes for distance measurements with the use of avalanche phenomenon in P-N junctions is known. The avalanche phenomenon may take place in the P-N junction of the diode when the diode is in a reverse current in a condition close the junction breakdown voltage. If the avalanche photodiode is in a reverse current right below the breakdown voltage, it produces an electrical current proportional to the density of photon flux received by the photodiode. The photodiodes may be used that are in reverse current below the breakdown voltage in order to detect the low density photon flux. Such photodiodes are called as single photon avalanche diodes (SPAD) or diodes functioning in the "Geiger" mode. When a photon arrives in such photodiode, an avalanche phenomenon takes place, producing a dense current in the P-N junction of the photodiode.

Said distance sensor module (3) comprised by the device of the invention is configured to constitute at least one 1 D distance data of at least one movable object in said physical field (A).

In a preferred embodiment of the present invention, the thermal camera (2) and the distance sensor module (3) are positioned side-by-side in such a manner to provide a space there between with a predetermined value to see the same physical field (A) (Fig. 1 ). With such arrangement, it is aimed to acquire the image and/or data of the same physical field (A) by the thermal camera (2) and the distance sensor module (3).

The field of view (FO A) of the thermal camera (2) corresponds to an array of thermal camera pixels. Moreover, field of view (FO\A) of the thermal camera (2) is always larger than the field of view (FOV₂) of the distance sensor module (3). For the purpose of providing a measurement within a broader physical field, said distance sensor module (3) may comprise more than one unit (31 ) comprising TOF 2D SPAD array operating as a single pixel. In Fig. 1 , the distance sensor module (3) comprised by said movable object detection device (1 ) comprises 3 units (31 ) comprising TOF 2D SPAD array operating as a single pixel. The number of said units (31 ) may be determined depending on the size of the detection area.

In order to realize a 1 D distance measurement for movable object detection, a distance sensor module (3) is used that comprises a unit (31 ) comprising a TOF 2D single photon avalanche diode operating as a single pixel.

The 1 D distance information relating to the movable object in the physical field (A) may be directly determined in accordance with the time of flight between the arrival of a photon sent to the physical field (A) to the object, and time of returning of the photon to the diode. Accuracy of the distance measurement is particularly based on the propagation continuity of the photon pulses, and thusly the fact that these pulses are short enables more accurate calculations. It has been established by the inventor that a measurement within at most 4 meter range of the movable object detection device (1 ) of the present invention presents fairly good results.

As seen, the SPADs are designed with a reverse current below the breakdown voltage. Flence, it differs from the "avalanche photons (APD)" measuring merely the density of the signals arrived. SPAD outputs are independent of the optical signal power. Due to these features, the inventor has concluded that the use of SPADs in the device of the present invention is surprisingly advantageous in terms of the distance measurement outcomes. The reason may be that the SPADs count the number of the returning photons rather than calculating the returning signal power. This may have role in enhancing the performance in low-lighted environments as well as enhancing the measurement speed and accuracy.

The 1 D distance information stated herein may be associated with the structure of the TOF 2D SPAD array. Said single photon avalanche diode array preferably consists of a 16x16 diode array. Said array may be operated in different modes. Flowever, particularly 1 D mode is preferred in the present invention. In the 1 D mode, the array functions as a large single photon avalanche diode, in other words a sum of data acquired from each single photon avalanche diode is provided. Thus, the distance of a movable object in a physical field (A) to the unit (31 ) comprising the TOF 2D SPAD array operating as a single photon (thereby to the distance module (3)) is independently measured from the location (e.g., coordinate data) of said object in the physical field (A). As may easily be understood by a person with ordinary skill in the art, the distance sensor module (3) used in the present invention is fairly different from LIDAR kind of TOF depth cameras used in the prior art. Due to the single point sensor structure comprised by the module, it provides only 1 D distance information of said physical field (A).

The device of the invention comprises a processor connected to the thermal camera (2) and the distance sensor module (3). Said processor matches said at least one thermal image and said at least one 1 D distance data.

The thermal image in the present invention comprises multiple thermal images. Frame rate of the thermal camera (2) is preferably between 0.5 and 32 Flz. Preference of such a range is to acquire movement information to enable monitoring of entrance and leaving of people with an average walking speed of, for instance, 1.38 m/s within an inner place.

Monitoring of entrance and leaving of people into/from an inner place is a problem more complicated than thought. Generally, other systems relating to the movable object detection assumes that the target object would move in a direction forward away from the entrance door. However, the target object may present various actions other than moving in a vertically or horizontally straight movement. For example, the object may just stand in the entrance path or move forwards/backwards. Hence, the direction of movement of a movable object cannot be detected from a single frame. Thus, the thermal image data consisting of multiple thermal images as well as the distance sensor module (3) provide the multiple 1 D distance data. The distance sensor module (3) contributes to the determination of entrance and exit directions. Correct acquisition of entry and exit information in the movable object detection is important for calculation of the number of objects. Another processor in the unit (31 ) comprising the TOF 2D SPAD array operating as a single pixel may acquire entrance- exit information with a proper algorithm. As shown in for example Fig. 3, the SPAD pixels corresponding to the TOF 2D SPAD array operating as a single pixel may be divided as a first zone (Z^ and a second zone (Z₂). In Fig. 3, the SPAD pixel dimension of each zone is given to be 16 x 8. Dimensions of said zones may differ. The direction of passage of a movable object from the SPAD pixel zone corresponding to the TOF 2D SPAD array operating as a single pixel in line with the axis (y) may be determined with the principle of chronogram in accordance with the ranking of the detection of said object in a first zone (Z^ and a second zone (Z₂). Consequently, a first set of data acquired from the thermal camera (2) and a second set of data acquired from the distance sensor module (3) are matched with a processor connected to the thermal camera (2) and the distance sensor module (3).

With matching of a thermal image (multiple thermal images) and 1 D distance information (multiple 1 D distance information), an easy-to-calculate outcome may be acquired. Thusly, the movable object detection may be quickly realized with a high accuracy without the need of any complicated calibration methods. in a preferred embodiment of the present invention, a movable object in a physical field (A) may be detected with a predetermined threshold value with the use of a thermal camera (2). For example, in case the object to be detected is a human, the threshold value may be determined as 36.8±0.4°C by making use of human body temperature. in another preferred embodiment of the present invention, a movable object in a physical field (A) may be detected with a predetermined distance threshold value. For example, in case the object to be detected is a human, the threshold value may be determined with respect to human height. This embodiment may be combined with the previous embodiment.

In another embodiment of the present invention, said thermal camera (2) comprises a thermopile focal plane array. The thermophile is an electronic device converting the thermal energy into the electric energy. The thermal camera may also comprise a lens to enable a more homogenous field of view.

In another preferred embodiment of the invention, the thermopile focal plane array has a pixel dimension of 32x24. Different dimensions of said array may also be used.

Infrared radiation operating the thermal camera (2) is also an important parameter for the measurement in a preferred embodiment of the present invention, the thermal camera (2) operates with a spectral wavelength range of 50-1000 mm. In an ISO 20473 scheme, this wavelength range expresses a far infrared spectrum.

In another preferred embodiment of the present invention, the distance sensor module (3) functions with a spectral wavelength range of 0.7 mm - 1 mm. In an ISO 20473 scheme, this wavelength range expresses a near infrared spectrum. This embodiment may be combined with an appropriate previous embodiment. More particularly, the distance sensor module (3) operates in a spectral wavelength of 0.94 mm. The movable object detection device presented with the present invention, in another preferred embodiment, also comprises a wireless local area network module. Said wireless local area network module comprises, but not limited to, Wi-Fi, Ethernet, ZigBee and Bluetooth. Thusiy, calculation data acquired with the processor may be sent to a computer environment. The sent data may be used for further analysis. As information such as the number of movable objects detected and entrance-exit activities at the end of the analysis, the systems within these places may be adjusted in accordance with this information. In another aspect, the present invention relates to a movable object detection method. Said method comprising the mounting of said movable object detection device (1 ) onto the ceiling as well as the following steps:

- Constitution of at least one thermal image of a physical field (A) by a thermal camera (2),

- Constitution of at least one 1 D distance data of at least one movable object in said physical field (A) by a distance sensor module (3),

- Matching of said at least one thermal image and said at least one 1 D distance data by a processor connected to the thermal camera (2) and the distance sensor module (3).

Said method of the invention is realized with the use of said movable object detection device (1 ). Said device may be mounted onto the ceiling of preferably an indoor environment for minimizing the perspective-related problems. Thusiy, a possible overlapping may be prevented, and the detection of more than one object simultaneously present in said physical field (A) may be realized.

Claims

1 . A movable object detection device (1 ) comprising a thermal camera (2), a distance sensor module (3) and a processor, characterized in that:

- said thermal camera (2) is configured to constitute at least one thermal image of a physical field (A),

- said distance sensor module (3) comprising at least one unit (31 ) comprising TOF 2D SPAD array operating as a single pixel, and is configured to constitute at least one 1 D distance data of at least one movable object in said physical field (A), and

- a processor connected to the thermal camera (2) and the distance sensor module (3) matches said at least one thermal image and said at least one 1 D distance data.

2. A movable object detection device (1 ) according to claim 1 , characterized in that the thermal camera (2) comprises a thermopile focal plane array.

3. A movable object detection device (1 ) according to claim 1 or 2, characterized in that said thermal camera (2) operates with a spectral wavelength ranges from 50 mm to1000 mm.

4. A movable object detection device (1 ) according to claim 1 , 2 or 3, characterized in that said distance sensor module (3) functions with a spectral wavelength ranges from 0.7 mm to 1 mm.

5. A movable object detection device (1 ) according to any preceding claim, characterized in that it also comprises a wireless local area network module.

6. A movable object detection device (1 ) according to any of the claims 1 to 5, arranged for detection of a movable object in a physical field (A), with a predetermined threshold value with the use of the thermal camera (2).

7. A movable object detection device (1 ) according to the claim 6, wherein the threshold value is 36.8±0. °C.

8. A method for detecting at least one movable object with mounting of the movable object detection device (1 ) according to claim 1 onto the ceiling, comprising the steps of:

i. Constituting a thermal image of a physical field (A) with a thermal camera (2), ii. Constituting at least one 1 D distance data of at least one movable object in a physical field (A) with a distance sensor module (3),

iii. Matching said thermal image with said 1 D distance data.

9. The method according to the claim 8, wherein the method includes arranging the device (1 ) for detection of a movable object in a physical field (A), with a predetermined threshold value with the use of the thermal camera (2).

10. The method according to the claim 9, wherein the method includes selecting the threshold value as 36.8±0.4°C as human body temperature, for detection of humans.