NL2024953B1

NL2024953B1 - Systems and methods for infrared sensing

Info

Publication number: NL2024953B1
Application number: NL2024953A
Authority: NL
Inventors: Narayana Sujay; Sathyanarayana Rao Vijay; Rao Venkatesha Prasad Ranga
Original assignee: Univ Delft Tech
Priority date: 2020-02-20
Filing date: 2020-02-20
Publication date: 2021-10-13
Also published as: WO2021165408A1

Abstract

An infrared sensing system for determining a position of a radiating body in an environment, the system comprising: . a passive infrared sensor, configured to sense an infrared radiation emitted by the radiating body and to generate an electric signal associated with the infrared radiation sensed by the passive infrared sensor; . an amplifier, connected with the passive infrared sensor, configured to amplify the electric signal; an infrared array sensor comprising a plurality of infrared sensing elements, wherein o each of the plurality of infrared sensing elements is configured to sense the infrared radiation emitted by the radiating body and to generate a respective element signal associated with the infrared radiation sensed by the respective infrared sensing element; and 0 the infrared array sensor further is configured to generate, based on a plurality of element signals, a thermal image; . a control unit, o connected with the amplifier to receive the amplified electric signal; and o connected with the infrared array sensor to receive the thermal image; and . wherein the control unit is configured to: 0 determine a displacement information from a difference between an imaged location of the radiating body in the thermal image at a first image time instance and an imaged location of the radiating body in the thermal iimage at a second time instance; o determine, from the amplified electric signal and the displacement information, the position of the radiating body in the environment.

Description

P34179NLO0 Title: Systems and methods for infrared sensing Field of the invention The present invention relates to systems and methods for infrared sensing. In particular, the present invention relates to the use of a passive infrared sensor and an infrared array sensor for determining positional information of a radiating body in an environment. Background of the invention In environments, for example in assisted living spaces, localization of people using low-power and privacy-aware technologies is useful. Common requirements for these technologies are related to determining spatio-temporal properties such as presence, count, location, track and identity. While identification would generally require vision based sensors, the other requirements have been attempted with infrared sensor based technologies, which are generally more suitable to preserve privacy. For example, Beltran, Alex, Varick L. Erickson, and Alberto E. Cerpa, disclose in "Thermosense: Occupancy thermal based sensing for hvac control.", Proceedings of the 5th ACM Workshop on Embedded Systems For Energy-Efficient Buildings. ACM, 2013., an occupancy monitoring system that utilizes thermal based sensing and PIR, or passive infrared, sensors. A low-power multi-sensor node is disclosed for measuring occupancy utilizing a thermal sensor array combined with a PIR sensor.

However, a drawback of the system proposed by Beltran et al. is that no position of a radiating body with respect to the system is determined. Summary of the invention It would be desirable to provide systems and methods for infrared sensing that may determine a position of a radiating body with respect to an infrared sensing system. Therefore, in various aspects of the present invention, systems and methods are provided that overcome the aforementioned drawback and overcome possible other drawbacks of known systems and methods.

2.

In a first aspect of the invention, an Infrared sensing system for determining a position of a radiating body in an envirenment is provided, wherein the infrared sensing system comprises: s a passive infrared sensor, configured to sense an infrared radiation emitted by the radiating body and to generate an electric signal associated with the infrared radiation sensed by the passive infrared sensor, wherein the electric signal depends on a position and a velocity of the radiating body with respect to the passive infrared sensor; = an amplifier, connected with the passive infrared sensor, configured to amplify the glactric signal; s an infrared array sensor comprising a plurality of infrared sensing elements, wherein o each of the plurality of infrared sensing elements is configured to sense the infrared radiation emitted by the radiating body and to generale a respective element signal associated with the infrared radiation sensed by the respective infrared sensing slement; and othe infrared array sensor further is configured to generate, based on a plurality of element signals, a thermal image; e 3 control unit, o connected with the amplifier to receive the amplified electric signal; and o connected with the infrared array sensor to receive the thermal image; and s wherein the control unit is configured to: o determine a displacement information from a difference between an imaged location of the radiating body in the thermal image at a first image time instance and an imaged location of the radiating body in the thermal image at a second time instance; o determing, from the amplified electric signal and the displacement information, the position of the radiating body in the environment.

The infrared sensing system, also briefly referred to as system, comprises various components, including the passive infrared sensor, the amplifier, the infrared array sensor and the control unit. The system may comprise other components as well. Components may be connected directly or indirectly with each other. In particular, components may be connected indirectly with each other via communication means or modules.

The system, in particular the control unit, is configured to determine the position of the radiating body in the environment. To do so, the control unit may process information on the

23. position of the radiating body. Said information may in particular be provided directly, or indirectly, by the passive infrared sensor and by the infrared array sensor.

The radiating body may refer to a body emitting infrared radiation. Infrared radiation may refer to electromagnetic radiation, wherein the wavelength of the radiation is within a spectrum extending from the nominal red edge of the visible spectrum at 700 nanometers to 1 millimeter. The radiating body may additionally emit electromagnetic radiation outside of the infrared spectrum.

The radiating body may be a human, an animal or any other body of which a position may be determined by the infrared sensing system. The radiating body may be a single body, but may refer to a plurality of bodies as well.

The environment may be any location within the field of view of the infrared sensing system, and in particular within the field of view of the passive infrared sensor and within the field of view of the infrared array sensor. The environment may be a room indoors or may alternatively be a location outside.

The position of the radiating body may refer to a location, with respect to at least part of the infrared sensing system, on a horizontal plane. The horizontal plane may be parallel to a floor the radiating body may be standing on or may be the floor itself. Alternatively, the position of the radiating body may refer the a location in three dimension with respect to at least part of the infrared sensing system. The location may be determined with respect to any of the components of the infrared sensing system. In particular, the position may be determined with respect to the passive infrared sensor and/or the infrared array sensor. The location with respect to a component of the infrared sensing system may be used to determine the position with respect to another component of the infrared sensing system in case the relative position of the two components is predetermined or determinable.

The passive infrared sensor may be a pyroelectric infrared sensor. Infrared radiation may enter through the front of the passive infrared sensor. The passive infrared sensor may be a solid state sensor or set of solid state sensors, e.g. made from pyroelectric materials. The passive infrared sensor may be manufactured as part of an integrated circuit.

The passive infrared sensor may also be a photodiode, which may be able to detect electromagnetic radiation, and in particular infrared radiation.

The passive infrared sensor may convert the electromagnetic radiation into current or voltage based on a mode of operation.

The passive infrared sensor may comprise optical filters, built-in lenses or other surface areas.

The passive infrared sensor may detect changes in the amount of infrared radiation impinging upon it, which may vary depending on the temperature and surface characteristics of the objects or bodies in front of the sensor. The passive infrared sensor may covert the

-4- resulting incoming infrared radiation into an electric signal such as a voltage signal or a current signal. The passive infrared sensor may also convert a change in the incoming infrared radiation into a change of the electric signal.

The generated electric signal may depend on a distance of the radiating body to the passive infrared sensor. The generated electric signal may further depend on a velocity or speed of the radiating body with respect to the passive infrared sensor. The generated electric signal may further depend the radiation intensity or temperature of the radiating body.

Pairs of passive infrared sensors or sensor elements may be wired as opposite inputs to a differential amplifier or a similar device. In such a configuration, some passive infrared measurements may cancel each other so that for example the average temperature of the field of view, or FoV, may be removed from the electric signal. Consequently, an increase of infrared energy across the entire passive infrared sensor may not result in an electric signal generated by the passive infrared sensor.

The amplifier may modify the electric signal generated by the passive infrared sensor.

In particular, the amplifier may increase the amplitude of the electric signal, resulting in an amplified electric signal. The signal may be amplified with a first gain. Additionally, the amplifier may further modify the electric signal. The amplified electric signal may be such that it may be further processed by the control unit. The amplified electric signal may be an analog signal, and the control unit may further process the analog amplified electric signal and may infer information on the radiating body associated with the analog amplified electric signal.

The infrared array sensor comprises a plurality of infrared sensing elements. The infrared sensing elements may be connected in series, in parallel or in a combination of both.

The infrared array sensor may be a thermopile array sensor, comprising several thermopiles. The infrared array sensor may be a thermopile array sensor, wherein each thermopile may comprise several thermocouples connected in series, in parallel or in a combination of both.

Thermopile array sensors may comprise a plurality of infrared sensing elements working together to generate the thermal image. The thermal image may comprise a plurality of pixels of which some of the pixels may be associated with the radiating body.

The infrared sensing elements comprised in the infrared array sensors may generate element signals which may be processed further to allow the infrared array sensor to determine absolute temperatures as well as temperature gradients, which may be further process to generate the thermal image. In case a plurality of thermal images are generated at different time instances, a movement or a displacement information associated with the radiating body may be extracted from the plurality of thermal images. An imaged location of the radiating body may be determined in each of the plurality of the thermal images. The

-5. displacement information may comprise information on the positon of the radiating body and/or on the velocity of the radiating body.

Generally, the amplified electric signal may not be unique to a particular position and/or movement of the radiating body.

For example, the amplitude of the electric signal and of the amplified electric signal may increase in case the radiating body is closer to the passive infrared sensor, or may increase in case the radiating body moves faster.

Consequently, a fast moving but further away radiating body may cause the passive infrared sensor to generate the same electric signal as a slow moving, close by radiating body.

Similarly, the displacement information associated with the plurality of thermal images may not be unique to a particular position of the radiating body.

In particular, the displacement information may be insufficient to determine a distance of the radiating body to the infrared array sensor, as a large but further away radiating body, may be represented in the thermal image the same as a small but closer radiating body.

However, the amplified electric signal and the displacement information may change independently from each other when the position of the radiating body may change.

For example, in case the radiating body moves towards the passive infrared sensor, the amplified electric signal may depend quadratic on the distance between the passive infrared sensor and the radiating body.

On the other hand, the displacement information extracted from the imaged location of the radiating body in the thermal body may depend linearly on the distance between the infrared array sensor and the radiating body.

Various components of the infrared system may be operated using battery power and/or solar power.

An insight of the present invention is that combining the information associated with the amplified electric signal with the displacement information associated with the plurality of thermal images may permit to determine the position of the radiating body.

Consequently, the control unit may determine, from the amplified electric signal and the displacement information, the position of the radiating body in the environment.

The infrared array sensor such as a thermopile array sensor may generate a signal due to a Seebeck effect.

On the other hand, the passive infrared sensor such as a pyroelectric infrared sensor may generate a signal due to a pyroelectric effect.

In an embodiment, the infrared system further is configured to track the radiating body by determining the position of the radiating body at consecutive time instances.

In an embodiment, the infrared sensing system further comprises a memory device having stored therein:

-6- e a plurality of possible values of the amplified electric signal, and per possible value of the amplified electric signal, at least one respective predetermined position; and wherein the control unit further is configured to: e retrieve from the memory device at least one predetermined position associated with a value of the amplified electric signal as received from the amplifier; e determine, from the displacement information and from the at least one predetermined position as retrieved from the memory device, the position of the radiating body in the environment.

The memory device may be any device capable of storing data, in particular digital data. Examples of memory devices include hard drives, solid state drives, random-access memories, read-only memories, etc.

The contral unit is connected with the memory device to retrieve or access data stored in the memory device. The control unit may further be configured to compare the amplified electric signal with the plurality of possible values of the amplified electric signals stored in the memory device. To compare the amplified electric signal with a respective possible value of the amplified electric signal, the control unit may determine a first similarity score between the amplified electric signal and the respective possible value of the amplified electric signal.

Based on a plurality of first similarity scores, the control unit may determine which possible value of the amplified electric signal stored in the memory device is most similar to the received amplified electric signal from the amplifier. Subsequently, the control unit may determine at least one predetermined position associated with the most similar possible value of the amplified electric signal. As explained above, various positions of the radiating body may give rise to a similar or same amplified electric signal. Consequently, with a respective possible value of the amplified electric signal, more than one predetermined position may be associated. Also, per possible value of the amplified electric signal, at least one respective predetermined position may be stored. Therefore, by comparing the amplified electric signal with the plurality of possible values of the amplified electric signals, the control unit may determine a plurality of possible positions of the radiating body. The control unit may further determine, based on the displacement information, which of the plurality of possible positions of the radiating body may be most likely to be correct.

In an embodiment, the infrared sensing system further comprises an image memory device having stored therein:

-7- e a plurality of possible values of the displacement information, and per possible value of the displacement value, at least one respective predetermined position; and wherein the control unit further is configured to: e retrieve from the image memory device at least one predetermined position associated with a value of the displacement information as determined by the control unit; e determine, from the amplified electric signal and from the at least one predetermined position as retrieved from the image memory device, the position of the radiating body in the environment.

Similar to having a plurality of possible values of the amplified electric signal stored, wherein per possible value of the amplified electric signal, at least one respective predetermined position is associated, it may be beneficial to have a plurality of possible values of the displacement information stored, wherein per possible value of the displacement value, at least one respective predetermined position is associated.

The image memory device may be any device capable of storing data, in particular digital data. Examples of the image memory devices include hard drives, solid state drives, random-access memories, read-only memories, etc.

The memory device and the image memory device may be the same device.

The control unit is connected with the image memory device to retrieve or access data stored in the image memory device. The control unit may further be configured to compare the displacement information with the plurality of possible values of the displacement information stored in the image memory device. To compare the displacement information with a respective possible value of the displacement information, the control unit may determine a second similarity score between the amplified electric signal and the respective possible value of the displacement information. Based on a plurality of second similarity scores, the control unit may determine which possible value of the displacement information stored in the image memory device is most similar to the determined displacement information. Subsequently, the control unit may determine at least one predetermined position associated with the most similar possible value of the displacement information. As explained above, various positions of the radiating body may give rise to a similar or same displacement information. Consequently, with a respective possible value of the displacement information, more than one predetermined position may be associated. Also, per possible value of the displacement information, at least one respective predetermined position may be stored. Therefore, by comparing the displacement information with the plurality of possible values of the displacement information, the control unit may determine a plurality of possible positions

-8- of the radiating body. The control unit may further determine, based on the amplified electric signal, which of the plurality of possible positions of the radiating body may be most likely to be correct. In an embodiment, the control unit further is configured to: e determine a plurality of first similarity scores between the value of the amplified electric signal and at least two of the plurality of possible values of the amplified electric signal, wherein each of the plurality of first similarity scores is associated with a respective predetermined position; e determine a plurality of second similarity scores between the value of the displacement information and at least two of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position; + determine, based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, wherein each of the plurality of joint similarity scores is associated with a respective predetermined position; e determine, based on a ranking of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment.

Based on comparing the amplified signal with a plurality of possible values of the amplified electric signal, the control unit may determine the plurality of first similarity scores, wherein a respective first similarity score may depend on the similarity between the value of the amplified electric signal and a respective possible value of the amplified electric signal.

Generally, a high first similarity score may indicate a high similarity between the value of the amplified electric signal and the respective possible value of the amplified electric signal. At least two first similarity scores may be determined by comparing the value of the amplified electric signal with at least two of the plurality of possible values of the amplified electric signal.

Similarly, based on comparing the value of the displacement information with a plurality of possible values of the displacement information, the control unit may determine the plurality of second similarity scores, wherein a respective second similarity score may depend on the similarity between the value of the displacement information and a respective possible value of the displacement information. Generally, a high second similarity score may indicate a high similarity between the value of the displacement information and the respective possible value of the displacement information. At least two second similarity

-9- scores may be determined by comparing the value of the displacement information with at least two of the plurality of possible values of the displacement information.

Various variations in the number of determined first similarity scores and the number of determined second similarity scores may be possible. Also the order of determining the first similarity scores and determining the second similarity scores may be varied. Furthermore, in an embodiment, second similarity scores may be only determined for predetermined positions associated with previously determined first similarity scores. Similarly, in an embodiment, first similarity scores may be only determined for predetermined positions associated with previously determined second similarity scores.

A first similarity score and a second similarity score associated with a predetermined position may be combined to determine a joint similarity score. For example, a respective joint similarity score may be a weighted average of the respective first similarity score and the respective second similarity score. By computing for a plurality of predetermined positions the first similarity score and the second similarity score, a plurality of joint similarity scores may be determined. By ranking the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment may be determined. Ranking the plurality of joint similarity scores, and/or similarity scores in general may comprise determining which similarity has the highest score.

In an embodiment, the control unit, to determing the imaged location of the radiating body in the thermal image at the first image time instance, further is configured to: e determine a first contour associated with the radiating body in the thermal image at the first image time instance; and wherein the control unit, to determine the imaged location of the radiating body in the thermal image at the second image tims instance, further is configured to: e determine a second contour associated with the radiating body in the thermal image at the second image time instance In the thermal image the radiating body may be associated with a contour. To determine the contour the control unit may determine which pixels in the thermal image are associated with the radiating body and which pixels are not.

By determining the contour in the thermal image, the imaged location of the radiating body may be determined in the thermal image more consistently, which may further improve the determining of the displacement information.

By determining the first contour and the second contour associated with the first image time instance and the second image time instance respectively, the displacement

-10 - information may be determined from the difference between the first contour and the second contour.

In an embodiment, from the first contour and/or the second contour information on the position of the radiating body may be obtained independently.

In an embodiment, the thermal image comprises an array of pixels, and wherein the displacement information comprises an amount of pixels transversed, associated with the radiating body, between the first image time instance and the second image tims instance. Typically, the thermal image comprises a plurality of pixels, wherein each pixel has a particular colour and/or brightness, Some of the plurality of pixels may be associated with the radiating body. A movement of the radiating body with respect to the infrared array sensor generally results in a change of pixels that are associated with the radiating body. Therefore, the change of pixels associated with the radiating body provides information on the displacement of the radiating body. Between the first image time instance and the second image time instance the imaged location associated with the radiating body may transversed an amount of pixels. In an embodiment, the amount of pixels transversed comprises: e a first amount of pixels transversed in a first direction; and e a second amount of pixels transversed in a second direction, wherein the second direction is not parallel to the first direction.

The thermal image comprises a two-dimensional array of pixels. A movement of the radiating body may subsequently result in changes of pixels associated with the radiating body in two directions. In case the first direction and the second direction are not parallel with each other, the first amount of pixels transversed and the second amount of pixels transversed may comprise different information on the displacement of the radiating body.

In an embodiment, the difference between the first image time instance and the second image time instance is less than 3 seconds, preferably less than 1 second.

To determine the position of the radiating body it may be beneficial that the movement of the radiating body is limited during the determination process in order to reduce the error in the determined position. Subsequently, the time interval between the first image time instance and the second image time instance should be limited. On the other hand, the time interval between the first image time instance and the second image time instance should be sufficiently long to permit the determining of the displacement

-11 - information when the radiating body is moving at a typical speed. The higher the image resolution of the thermal image is, the shorter the time between the first image time instance and the second image time instance may be, since smaller movements or displacements may be detected.

In an embodiment, a first gain of the amplifier is such that the amplified electric signal is within a dynamic range of the amplifier, and wherein the control unit further is configured to determine of the amplified electric signal at least one value selected from a group comprising: e a peak-to-peak value; e an amplitude value; e an integration value; and e a frequency value; and wherein the control unit further is configured to associate the at least one value with the position of the radiating body in the environment.

The amplifier may amplify the electric signal with the first gain. To preserve information contained in the electric signal, it may be beneficial to have a value of the first gain such that the amplified electric signal is within a dynamic range of the amplifier.

Consequently, a cut-off, clipping or another distortion of the amplified electric signal may be avoided.

The control unit may determine a value of the amplified electric signal which depends on the position and movement of the radiating body. The amplified electric signal may generally be a voltage or current signal which may change over time and may alternate or oscillate. Typically, the amplitude, peak-to-peak value and/or a frequency of the amplified electric signal will depend on the position and movement of the radiating body. Consequently, also a mathematical integration, or integration, of such value depends on the position and movement of the radiating body.

Obtained value may be associated with the position of the radiating body, for example by comparing the determined value with possible values of the amplified electric signal stored in the memory device.

In an embodiment, e the amplifier further comprises a variable gain; and e the control unit further is configured to

-12- co adjust the variable gain of the amplifier to be the first gain at a first time instance and a second gain at a second time instance, wherein the first gain is different from the second gain; o determine the at least one value when the variable gain is the first gain; and o determine the at least one value when the variable gain is the second gain.

Instead of a fixed gain, the amplifier may comprise a variable gain. The electric signal may therefore be amplified with a plurality of gains, possibly providing additional information on the position of the radiating body. It may be beneficial to amplify the electric signal with more than two different gains. The amplified electric signal associated with the second gain may be within the dynamic range of the amplifier, but it may be outside of the amplifier as well. It may be beneficial that the second gain is choses such that a maximal amplification is obtained, in order to detect the presence of the radiating body earlier. By varying the variable gain, a curve may be obtained relating a value of the variable gain with a respective value of the amplified electric signal.

In an embodiment, the control unit further is configured to: e associate the amplified electric signal with a provided predetermined position; and e store the provided predetermined position and a value of the amplified electric signal associated with the provided predetermined position in the memory device.

The infrared sensing system may be used to obtain a possible value of the amplified electric signal which may be stored in the memory device, i.e. the infrared sensing system may be trained. The radiating body may be in the fields of view of the passive infrared sensor and the infrared array sensor.

The contral system may be provided with the predetermined position of the radiating body, which may be predetermined by other means. Additionally, the control unit may also be provided with a predetermined velocity, a predetermined speed, and/or a predetermined displacement information associated with the radiating body.

Associated with the provided predetermined position of the radiating body, the infrared sensing system may generate an amplified electric signal. This amplified electric signal may be stored in the memory device as a possible value of the amplified electric signal, associated with provided predetermined position.

In an embodiment, the control unit further is configured to:

-13- e associate the displacement information with a provided predetermined position; and e store the provided predetermined position and a value of the displacement information associated with the provided predetermined position in the image memory device. Similarly as above, the infrared sensing system may be used to obtain a possible value of the displacement information which may be stored in the image memory device. The radiating body may be in the fields of view of the infrared array sensor and the infrared array sensor.

The control system may be provided with the predetermined position of the radiating body, which may be predetermined by other means. Additionally, the control unit may also be provided with a predetermined velocity, a predetermined speed, and/or a predetermined amplified electric signal associated with the radiating body.

Associated with the provided predetermined position of the radiating body, the infrared sensing system may generate thermal images and may determine the displacement information. This displacement information may be stored in the image memory device as a possible value of the displacement information, associated with provided predetermined position.

In a second aspect of the invention, a method for determining a position of a radiating body in an environment is provided, wherein the method for determining the position of the radiating body in the environment comprises: se sensing, by a passive infrared sensor, an infrared radiation emitled by the radiating body; « generating an electric signal associated with the infrared radiation sensed by the passive infrared sensor, wherein the electric signal depends on a distance and a velocity of the radiating body with respect to the passive infrared sensor; « amplifying, by an amplifier, the electric signal; s Sensing by each of a plurslity of infrared sensing elements comprised in an infrared array sensor, the infrared radiation and generating a respective element signal associated with the infrared radiation sensed by the respective infrared sensing element; s generating, based on a plurality of element signals, a first thermal image at a first image time instance; + generating, based on a plurality of element signals, a second thermal image at a second Image time instance;

-14 - es determining a displacement information from a difference between an image location of the radiating body in the first thermal image and an image location of the radiating body in the second thermal image; and s determining, from the amplified electric signal and the displacement information, the position of the radiating body in the environment. In an embodiment, the method for determining a position of a radiating body in an environment further comprises: e providing a plurality of possible values of the amplified electric signal, and per possible value of the amplified electric signal, at least one respective predetermined position; e providing a plurality of possible values of the displacement information, and per possible value of the displacement value, at least one respective predetermined position; e determining a plurality of first similarity scores between a value of the amplified electric signal and each of the plurality of possible values of the amplified electric signal, wherein each of the plurality of first similarity scores is associated with a respective predetermined position; e determining a plurality of second similarity scores between a value of the displacement information and each of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position; e determining, based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, wherein each of the plurality of joint similarity scores is associated with a respective predetermined position; e determining, based on a ranking of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment.

In an embodiment, the method for determining a position of a radiating body in an environment further comprises: + adjusting an amplification of the electric signal to be a first gain at a first time instance and a second gain at a second time instance, wherein the first gain is different from the second gain.

-15- In a third aspect of the invention, a method for training an infrared sensing system is provided, wherein the method for training the infrared sensing system comprises: e providing a predetermined position of a radiating body; es sensing, by a passive infrared sensor, an infrared radiation emitted by the radiating body; s generating an electric signal associated with the infrared radiation sensed by the passive infrared sensor, wherein the electric signal depends on a position and a velocity of the radiating body with respect to the passive infrared sensor; s amplifying, by an amplifier, the electric signal; e sensing by each of a plurality of infrared sensing slements comprised in an infrared array sensor, the infrared radiation emitled by the radiating body and generating a respective element signal associated with the infrared radiation sensed by the respecijve infrared sensing element; s generating, based on a plurality of element signals, a first thermal image at a first image time instance; s generating, based on a plurality of element signals, a second thermal image at a second image time instance; « determining a displacement information from a difference between an imaged location of the radiating body in the first thermal image and an imaged location of the radiating body in the second thermal image; and e storing, associated with the predetermined position of the radiating body, a value of the ampilifiad electric signal and the displacement information.

The methods described above may provide the same or similar advantages as the advantages of the first aspect of the invention.

These and other aspects of the invention will be more readily appreciated as the same becomes better understood by reference to the following detailed description and considered in connection with the accompanying drawings in which like reference symbols designate like parts.

Brief descriptions of the drawings Figure 1 depicts a first embodiment of an infrared sensing system according to the invention.

Figure 2 depicts a second embodiment of an infrared sensing system according to the invention, wherein the sensors are placed remotely from a control unit.

-16 - Figure 3 depicts a third embodiment of an infrared sensing system according to the invention, wherein an additional sensor is mounted within the same housing.

Figure 4 depicts a fourth embodiment of an infrared sensing system according to the invention, wherein the system further comprises another passive infrared sensor and another infrared array sensor.

Figure 5 depicts a lens in front of a passive infrared sensor.

Figure 6 illustrates a determining of a first contour associated with a radiating body.

Figure 7 depicts a first contour (7a) and a second contour (7b) associated with a radiating body.

Figure 8 schematically depicts another embodiment of an infrared sensing system according to the invention.

Figure 9 depicts a variation of an amplified electric signal over time. Figure 10 illustrates a relation between a speed of a radiating body, a distance of the radiating body with respect to a passive infrared sensor, a first gain and a peak-to-peak voltage associated with an amplified electric signal.

Figure 11 illustrates a relation between a varying first gain and a peak-to-peak voltage associated with an amplified electric signal.

Figure 12 depicts a flow diagram of an embodiment of a method for determining positional information of a radiating body in an environment.

Figure 13 depicts a flow diagram of an embodiment of a method of training an infrared sensing system.

Detailed description of embodiments Figure 1 depicts a first embodiment of an infrared sensing system 101 according to the invention. The infrared sensing system 101 comprises a passive infrared sensor 103, an infrared array sensor 105, an amplifier 121 and a control unit 111, mounted in a housing 119.

The infrared sensing system depicted in Figure 1 further comprises a memory device 113, an image memory device 123 and a control communication unit 115.

The contral unit 111 is connected with the amplifier 121, the infrared array sensor 105, the memory device 113, the image memory device 123 and the communication unit 115. The amplifier further is connected with the passive infrared sensor 103. The various connections may be wired or wireless.

According to the first embodiment, the passive infrared sensor 103 comprises pyroelectric materials and may then also be called a pyroelectric infrared sensor or PIR sensor. According to the first embodiment, the infrared array sensor 105 is a thermopile array sensor. However, other types of sensors may be used.

-17 - The relative position of the passive infrared sensor 103 and the infrared array sensor 105 is predetermined or determinable.

According to the first embodiment of the infrared sensing system 101, the passive infrared sensor 103 and the infrared array sensor are mounted in a single housing. The passive infrared sensor 103 is placed adjacent to the infrared array sensor 105. Preferably, the position of the passive infrared sensor 103 with respect to the infrared array sensor 105 is such that a field-of-view of the infrared array sensor 105 is not blocked by the passive infrared sensor 103. Similarly, it is preferred that a field-of-view of the passive infrared sensor 103 is not blocked by the infrared array sensor 105.

The field-of-view of the passive infrared sensor 103 and the field-of-view of the infrared array sensor 105 overlap, such that a radiating body 1 may be simultaneously in the field-of-view of the passive infrared sensor 103 and in the field-of-view of the infrared array sensor 105. Preferably, the overlap in the fields-of-view is large, covering a most of the environment in which the radiation body 1 may move.

In the above, a field-of-view of a respective sensor 103, 105, refers to an area from where radiation, in particular infrared radiation, may travel or propagate to the respective sensor 103, 105. In case a radiating body 1 emits radiation, in particular infrared radiation, and the radiating body 1 is in an area which is in the field-of-view of the passive infrared sensor 103 and in the field-of-view of the thermopile array sensor, then emitted radiation originating from the radiation body 1 may travel or propagate to the passive infrared sensor 103 and to the infrared array sensor 105. The infrared array sensor comprises a plurality of infrared sensing elements, wherein each of the plurality of infrared sensing elements is configured to sense the infrared radiation emitted by the radiating body 1 and to generate a respective element signal associated with the infrared radiation sensed by the respective infrared sensing element. The infrared array sensor 105 further is configured to generate, based on a plurality of element signals, a thermal image; An example of a suitable passive infrared sensor 103 is the ZSBG446671 from Zilog. However, the present invention is not limited to the use of a particular passive infrared sensor

103.

Typically, a passive infrared sensor 103 comprises two infrared sensitive elements generating a voltage when infrared energy is absorbed by the infrared sensitive elements. The passive infrared sensor generates a voltage, or another electric signal, which is proportional to the differential change in the generated charge on the infrared sensitive elements. When the passive infrared sensor 103 is idle or does not generated a voltage, both infrared sensitive elements detect the same amount of infrared radiation. For example, the detected infrared radiation may be caused by the ambient temperature of a location within the field-of-view of the passive infrared sensor 103. When a radiation body 1 passes in front of

-18- the passive infrared sensor 103, the radiation received by the first element may differ from the radiation received by the second element, causing a differential change between the two elements.

A generated electric signal by the passive infrared sensor 103 is amplified by an amplifier 121, becoming an amplified electric signal. In Figure 8 and the description thereof, a possible amplifier 121 is further detailed. However, various types of amplifiers 121 may be used to amplify the electric signal.

The control unit 111 is connected with the amplifier 121 to receive the amplified electric signal, and is connected with the infrared array sensor 105 to receive the thermal image. The control unit is configured to determine a displacement information from a difference between an imaged location of the radiating body 1 in the thermal image at a first image time instance and an imaged location of the radiating body In the thermal image at a second time instance. Furthermore, the control unit 111 is configured to determine from the amplified electric signal and the displacement information, the position of the radiating body 1 inthe environmsnt.

According to the first embodiment, the infrared array sensor 105 is a thermopile array sensor. However, the infrared array sensor 105 may also comprise photodiodes. Also any infrared camera may be used as an infrared array sensor 105. Essentially, the infrared array sensor may be any sensor that may generate a thermal image.

An example of a suitable thermopile array sensor 105 is the MLX90640ESF-BAA from Melexis. However, the present invention is not limited to the use of a particular thermopile array sensor 105. Typically, a thermopile array sensor 105 comprises a plurality of thermopile sensors. These thermopile sensors may be arranged in the form of a grid or array. The present invention is not limited to a particular arrangement of the plurality of thermopile sensors. Preferably, the thermopile array sensor 105 comprises sufficient thermopile sensors such that the thermopile array sensor 105 has a resolution which is sufficiently high. For example, a resolution of 32 x 24 = 768 pixels may be sufficient to localize a human. The resolution may be associated with the number of thermopile sensors on the thermopile array sensor 105. A single pixel may be associated with a single thermopile sensor or with a plurality of thermopile sensors.

The control unit 111 may comprise various hardware components such a processing unit. Also, the memory device 113, the image memory device 123 and/or the control communication unit 115 may be part of the control unit 111. The control communication unit 115 may be used to receive data from and to transmit data to a server. The control communication unit may communicate wireless with the server. An example of a processing

-19- unit is a low-power ARM Cortex MO microcontroller ATSAML21 from Atmel's pico power series microcontrollers, for retrieving signals or data from the infrared sensors 103, 105 and to process the signals or data. The data associated with detection, position, localization, and/or height of the radiating body 1 may be transmitted to a central server, e.g. using a WiFi module.

The passive infrared sensor 103, such as a pyroelectric infrared, PIR, sensor is generally inexpensive and consumes little energy. By placing the passive infrared sensor 103 adjacent, or at a predetermined or determinable position with respect to the thermopile array sensor 105, such as a thermopile array sensor, additional depth information or distance information of the radiating body may be obtained.

The refresh rate of the infrared array sensor 105 may be set to a value which is sufficiently high to accurately determine the movement of the radiation body 1, between two or more thermal images. For example, a refresh rate of 8 Hz may be sufficient. A further increase of the refresh rate may not aid in improving accuracy considering a typical human movement in indoors, whereas a higher refresh rate generally causes a higher power consumption.

The analog-to-digital converter, ADC, sampling rate of the passive infrared sensor 103 may be set to twice as that of infrared array sensor 105 to meet the Nyquist criterion.

The infrared array sensor 105, such as the MLX20640, may not output the absolute temperature values but raw values read by each pixel. These values correspond to the amount of infrared energy falling on each pixel. The raw values may be converted to temperature values. However, such conversion generally involves complex computation of multiple floating point numbers to turn the raw pixel data into temperature data. This may demand a relatively high processing power, causing higher power consumption. Alternatively, the calculations may be simplified by working with the relative difference between the raw pixel data rather than the absolute temperature data, reducing the amount of required processing, since the absolute temperature is not needed to identify the radiating body on the thermal image.

Figure 2 depicts a second embodiment of an infrared sensing system 101 according to the invention. The working of the second embodiment of an infrared sensing system 101 is substantially the same as the working of the firs embodiment described above. However, in contrast to the first embodiment, sensors 103, 105 are placed remotely from a control unit

111. The passive infrared sensor 103 and the infrared array sensor are connected to a sensor

-20- communication unit 215 to transmit and receive signals or information, from or to the control unit 111, via the control communication unit 115. Alternatively, each sensor 103, 105 is connected to a separate communication unit, wherein each is connected with the control communication unit.

In the second embodiment of the infrared sensing system 101, the sensors 103, 105 are not mounted in a single housing.

The connection 223 between the control communication unit 115 and the sensor communication unit 215 is wireless.

However, also a wired connection 223 is possible.

The position of the passive infrared sensor 103 and the infrared array sensor 105 is predetermined.

Components such as the memory device 113, the image memory device 123 are not depicted in Figure 2 and may be integrated in the control unit 111. Figure 3 depicts a third embodiment of an infrared sensing system according to the invention.

The third embodiment extends the first embodiment in that an additional sensor 303 is mounted within the housing 119 and connected with the control unit 111. Although for an infrared sensing system 101 according to the invention to be able to determine the position of the radiating body 1 in an environment, it is sufficient that the system 101 comprises as sensors the passive infrared sensor 103 and the infrared array sensor 105, the system could further use additional information gather by and associated with the additional sensor 303. The additional information may increase an accuracy of the system 101. The additional sensor 303 may be another passive infrared sensor or another infrared array sensor.

However, also using other types of sensors may be used.

Figure 4 depicts a fourth embodiment of an infrared sensing system 101 according to the invention, wherein the system 101 further comprises another passive infrared sensor 403 and another infrared array sensor 405. The fourth embodiment is similar to the embodiments depicted in Figures 1-3. The fourth embodiment comprises additionally e another passive infrared sensor 403, configured to sense an infrared radiation emitted by the radiating body 1 and to generale another electric signal associated with the infrared radiation sensed by the another passive infrared sensor 403, wherein the another electric signal depends on a position and a velocity of the radiating body 1 with respect to the another passive infrared sensor 403; e another amplifier 411, connected with the ancther passive infrared sensor 403, configured to amplify the another electric signal; e another infrared array sensor 405 comprising another plurality of infrared sensing elements, wherein o gach of the another plurality of infrared sensing elements is configured to sense the infrared radiation emitied by the radiating body 1 and tc

-21 - generaie another respective glement signal associated with the infrared radiation sensed by the respective another infrared sensing element, and o the another infrared array sensor 405 further is configured to generate, based on a plurality of another element signals, another thermal image.

The control unit 111 further is o connected with the another amplifier 411 to receive the amplified electric signal, and o connected with the another infrared array sensor 405 to receive the another thermal image. The control unit 111 further is configured to: o determine ancther displacement information from another difference between another imaged location of the radiating body 1 in the another thermal image at another first image lime instance and another imaged location of the radiating body 1 in the another thermal image at a another second time instance; o determine, also from the another amplified electric signal and the another displacement information, the position of the radiating body 1 in the environment.

The passive infrared sensor 103 and the infrared array sensor 105 may be placed remotely from the another passive infrared sensor 403 and the another infrared array sensor

405. In particular, the passive infrared sensor 103 and the infrared array sensor 105 may be placed in a housing 119, whereas the another passive infrared sensor 403 and the another infrared array sensor 405 may be placed in another housing 419.

In particular, the housing 118 and its components may be the same as the another housing 419 and its components. As a results, a plurality of housings may be produced efficiently.

The fourth embodiment shows only two housings and their components. It may be clear that the infrared sensing system 101 may be extend to three, four, etc. of such housings as well. Each housing may comprise its own sub control unit, wherein each sub control unit may be connected to the control Unit.

Figure 5 depicts a lens 503 placed in front of the passive infrared sensor 103. The lens may strengthen a signal received by the passive infrared sensor 103 by concentrating

22.

infrared radiation which is received by the lens 503 on the passive infrared sensor 103 or on the sensing elements of the passive infrared sensor 103.

Preferably, the focal point of the lens 503 coincides with the center of the passive infrared sensor 103.

Preferably, the field-of-view of the lens 503 is larger than the field-of-view of the passive infrared sensor 103.

The lens 503 may be a Fresnel lens.

The electric signal generated by the passive infrared sensor 103 may be altered by adjusting the configuration or moulding of the lens 503.

In an embodiment, the electric signal is independent from the direction of the incoming radiation emitted by the radiating body 1 with respect to the passive infrared sensor 103. In case a lens 503 is placed in front of the passive infrared sensor 103, the lens may be such that the electric signal remains independent or substantially independent from the direction of the incoming radiation. The lens 503 may be a generic golf ball lens. The golf ball lens may bea semi-sphere comprising multiple spot Fresnel lenses on its circumference.

Figure 8 depicts another embodiment of an infrared sensing system 101 according to the invention. The passive infrared sensor 103 in Figure 8 is a pyroelectric infrared sensor 103 or PIR sensor. The amplified electric signal may be at least one value selected from a group comprising: e a peak-to-peak value; e an amplitude value; e an integration value; and e a frequency value.

Further, the control unit 111 may be configured to associate the at least one value with the position of the radiating body 1 in the environment. The amplified electric signal may refer to a voltage signal and/or a current signal. In the below, the invention is further explained using a peak-to-peak voltage value, although other signals or values may be used as well following similar principles.

The electric signal may be the output voltage or electric signal, from the PIR sensor 103 which is low and is in the order of yV. Hence, the output voltage may need to be amplified several thousand times in order to get a reasonable signal that can be measured by a microcontroller or control unit 111. This amplification may be done using operational amplifiers in two stages. In Figure 8, Texas Instrument’s LPV802 dual channel nano-power amplifer is adopted as it consumes little power. However, other amplifiers 121 may be used as well. Instead of determining a binary output of the PIR sensor 103, i.e. determining whether there is a moving radiating body or not, the infrared sensing system according to the

- 93. invention utilizes the PIR output in analog form. The analog signals from a single PIR sensor 103 are read by the ADC pin of the microcontroller 111 to estimate the distance of the moving object from the sensor 103. To do so, the microcontroller may vary the gain of each amplifier stage using digital potentiometers that are controlled using 12C lines. The amplified output VO ofthe PIR sensor 103 read by the microcontroller 111 is proportional to the overall gain given as, VO=-Vin(1+Rf1/R1)* (Rf2/R2), (Equation 1) wherein Rf1 is fixed to 3 MQ, Rf2 is a 1 MQ dual channel digital potentiometer, D5242BRUZ1M from Analog Devices, whose resistance can be varied in 255 steps between 0 and 1 MQ. Both the resistor channels are connected in series to get a broader range of up to 2 MQ. Similarly, R1 and R2 are 512 kQ digital potentiometers, AD5272BRMZ from Analog Devices, that can be varied in 1024 steps between 0 and 512 kQ. By adjusting these resistor values dynamically, the overall gain of the PIR sensor output can be varied in about 537x106 steps, hence customizing the detection range of the PIR sensor. The analog output from the PIR sensor is similar to a sine wave and produces negative voltage. As the used microcontroller is not capable of measuring negative voltages on its ADC pins, fixed resistors R3, R4, and R5, R6 are introduced to scale and shift the, possibly partially amplified, PIR output before fed to the ADC pin of the microcontroller. The resistances are fixed as follows R3 =510kQ, R4 = 240 kQ, R3 = 10 kQ, and R3 = 2.4 kQ to get a full PIR output swing between 0.2 V and 3.25 V. However, in case the microcontroller may measure negative voltages such scaling and shifting of the voltage may not be needed. In any case, it may not be necessary that the gain of the amplifier 121 can be varied. The gain may be a first gain. The first gain of the amplifier 121 may be such that the amplified electric signal is within a dynamic range of the amplifier 121. In case the amplifier 121 comprises a variable gain, the control unit 111 may be configured to e adjust the variable gain of the amplifier 121 to be the first gain at a first time instance and a second gain at a second time instance, wherein the first gain is different from the second gain; e determine the at least one value, such as the peak-to-peak voltage, when the variable gain is the first gain; and s determine the at least one value, such as the peak-to-peak voltage, when the variable gain is the second gain.

A sample amplified output from the PIR sensor 103 is shown in Figure 9, wherein it is illustrated how the analog voltage varies as a result of a moving radiation body 1. When the

-24- sensor is idle, both sensing elements in the PIR sensor 103 detect the same amount of infrared radiation from the ambient temperature.

When the radiating body 1 moves across in front of the sensor 103, the infrared radiation is first intercepted at one of the elements of the PIR sensor, causing a positive differential change between the two elements.

When the radiating body leaves the sensing area, the sensor reports a negative differential change.

While the thermopile array sensor 105 can detect both static and moving radiating bodies, the PIR sensor 103 can detect only when the radiating bodies move.

Hence, a fusion of these sensors to identify radiating bodies such as humans from the rest of the background warm objects is used.

Generally, as the distance between the radiating or warm body and the thermopile array sensor 105 increases, the temperature read by the thermopile array sensor decreases.

For example, says T « 1/(d42} , where T is the temperature recorded by a pixel and d is the distance of the radiating body from the sensor.

As the radiating body moves away from the sensor 105, the number of pixels used to indicate the radiating body 1 decreases.

Hence, if the height of the radiating body is known, the distance between the body 1 and the sensor 105 can be estimated as the vertical field of view and the total pixel count in the vertical dimension is known.

Similarly, if the distance is known, the height of the radiating body 1 can be estimated.

One of the important observations is the spatial-temporal changes in the number of pixels covered by the moving radiating body 1. When a radiating body moves in front of the thermopile array sensor 105, in any direction, the number of pixels traversed horizontally and vertically in a specific duration is proportional to the speed of the movement and distance of movement from the sensor 105. This provides a new relation, P(t,h,v) «< s/d, where P(t,h,v) is the number of pixels traversed horizontally (h) and vertically (v) in time t, s is the speed of the movement, and d is the distance of person from the sensor.

The relation between the peak to peak voltage Vp-p generated by the PIR sensor and the amplifier gain G may be as follows: Vp-p «x (I*G)/(s"2*d"2), (Equation 2)

wherein | is the infrared energy from the moving radiating object incident on the PIR sensor 103, s is the speed, and d is the distance of the radiating body from the sensor.

In Figure 10, the peak to peak voltage generated by the PIR sensor for different gains at different speeds is depicted.

It must be observed that large gain is required to see the same radiating body 1 at a more far distance at a constant speed.

Similarly, as the movement speed increases at a constant distance, the gain has to be increased to observe the same levels of signal.

As the distance of the radiating body 1 from the PIR sensor 103 increases,

- 95. the peak to peak voltage generated by the amplifier 121 decrease, and vice versa, provided that the speed of the movement, body temperature and the amplifier gain remains approximately the same. Provided that a radiating body 1 moves at a specific distance with a constant speed, the peak to peak voltage generated by the amplifier output can be varied by changing the overall gain G of the amplifier stages. This can be observed in Figure 10a or Figure 10b, where speed is constant in both the cases. The peak to peak voltage output from the amplifier stages decrease as the speed of the movement at a specific distance increases. This is evident from Figure 10a and Figure 10b. The peak to peak voltage output from the amplifier stages for a radiating body 1 moving at distance d1 with speed s1 may be same as the output for the same radiating body moving with speed s2 at distance d2, where d1 < d2 and s1 > s2. This is because the duration and amount of IR rays falling on the sensor 103 reduces as the speed of the moving object increases. Similarly, when the radiating body 1 moves slowly at farther distance, the exposure time of the body 1 is longer, providing higher chance of absorbing all the infrared rays emitted by the radiating body 1, and thus, generating relatively higher amplitude. Static hot objects such as hot kettle, computer, light bulbs do generally not affect the output of the PIR sensor 103.

Below it is explained how the features from thermopile array sensor 105 and PIR sensor 103 presented may be exploited to perform localization and/or tracking. The principal idea behind the fusion of two sensors — thermopile array sensor 105 and PIR sensor 103 in the proposed infrared sensing system 101 is that the thermopile array sensor 105 can be used to estimate the location in two dimensions (across the field of view cone axis), and PIR sensor 103 can be used to estimate the range between the infrared sensing system 101 and the radiating body 1, thus providing the location information in the third dimension. The steps involved in achieving this are: obtaining training data, also referred to as possible values, that may be retrieved online computation in the operational phase. Table 1 below depicts possible training data, wherein the training data comprises possible values of the amplified electric signal and possible values of the displacement information. The online computation in the operational phase may involve a series of steps: (1) Detection of the movement, (2) Background and noise removal from thermopile data, (3) Interpolation of the thermopile data, (4) Estimate the position of the object in one dimension using thermopile data, (5) Machine learning or computational classification of PIR and thermopile data to estimate the location in another dimension (distance of the object from the infrared sensing system 101), (6) Tracking the movement of the radiating body 1. These steps are explained in detail in the below:

- 96 - The training data comprises a plurality of possible values of the amplified electric signal, and per possible value of the amplified electric signal, at least one respective predetermined position. The training data further comprises a plurality of possible values of the displacement information, and per possible value of the displacement value, at least one respective predetermined position.

In case per predetermined position a plurality of possible electric signals are provided, each associated with a respective gain, such collection of data may be referred to as a V-G curve. An example of a V-G curve is given in Figure 11. Essentially, a V-G curve is a plurality of peak-to-peak voltage — gain pairs.

To obtain possible values of the amplified electric signal to be stored in the memory device, peak-to-peak voltage measurements Vp-p, see also Equation 2, may be determined against all dependent parameters such as possible amplifier gains G, various speeds s, and at different distances d or positions. This is accomplished by varying the amplifier gain from the maximum, which may be the second gain, to the value at which there will not be any detection (peak to peak voltage equivalent to the noise level) when a radiating body 1 is moving at a constant speed at a given distance. The obtained peak to peak values Vp-p for various gains G may be stitched together to form Voltage - Gain curve (V-G curve). A sample curve for movement with speed 0.5 m/s at 2 m in front of the sensor is shown in Figure 11. Similar such curves are recorded for different radiating bodies moving with speeds between

0.1 m/s and 2 m/s (it is assumed that 2 m/s is the maximum walking speed of human in indoor scenarios, however, this can be extended to higher speeds if needed). Hence, a single V-G curve spans over three dimensions with speed, distance and different radiating bodies and clothing, that addresses all the variables in Equation 2. The training data from PIR sensor may be represented as Vp-p = f(s,d,l }(G), (Equation 3) where gain G is varied from maximum to the minimum detection level, | represents infrared energy from radiating bodies. Instead of or additional to the distance, the position of the radiating body may be used.

In the case of thermopile sensor, there are two factors - movement speed and distance — that affect the number of pixels traversed by the moving radiating body as indicated by the relation P(t,h,v) « s/d. Hence, the dataset from thermopile array comprises of Pixel Traversed data (P-T data) in horizontal and vertical direction recorded for different speeds s at distances d. Similarly as above, instead of or additional to the distance, the position of the radiating body may be used.

-27 -

Hence, each P-T data is of four dimensions.

The datasets from both PIR sensor 103 and thermopile sensor 105 are recorded concurrently so that both the datasets represent the same event.

The training data from thermopile sensor may be represented as pt(h,v} = (i, {sd}, wherein pt (h,v) is the pixels traversed in horizontal and vertical direction.

Table 1 below illustrates possible values of the amplified electric signal (peak-to-peak voltage) and possible values of possible displacement information (P-T data). Five columns are provided where for each column a position (p1, p2 or p3), a speed (in meter per second, m/s), and an intensity of the radiation body 11, 12 or I3 is indicated.

Furthermore, for various gains the determined peak-to-peak voltage is shown, as well as the displacement information (P-T data). Training pi, 0.55, 11 p2,0.5m/s,12 p3,im/s,13 Mg Gain) veep Gain | Veep Gain 3,3 :800000000 3,3 :8E+08 1,3 : 7,5E+10 ei {330240000000 | BE AEO | 109 2408 : 3 :100000000 3,3 1E+08 0,6 ‚1E+08 22 75000000 | 28 7500000] 2 75000000 eet }46 50000000 | ]205 50000000) 02 50000000 11 :100000 15 1000000 0,2 100000 eee E00] 100 011000 0,2 :900 0,8 5000 0,2 :1000 P-T data (Horizontal v ; | | oo... DATAA | |... DATAS p3Am/sil Ps 25 SE || SEMO 2,2 7,5E+10 2,6 8E+09 1,7 :2,4E+08 2,1 2,4E+08 1,4 ‚1E+08 1,8 ‚1E+08 1,05 75000000 1,3 75000000 06 50000000; 10,9 50000000 02 ‘100000 | 106 100000 02 lo | [04 1000 021900 | 1000 3x1 7x2 1 Table 1

-28- The P-T data may be obtained by comparing two contours associated with the radiating body. An illustrative example is provided in Figure 7, wherein the contour associated with the radiation body 1 is moved to the right, indicating a movement of the radiating body. Although Table 1 shows associated with a distance, a speed and an intensity, these values may not be stored in the memory device, or image memory device.

Once the training data is available, localization and tracking can be performed with the following steps.

Initially, the amplifier gain of the PIR sensor 103 may be set to the highest to detect the movement of the radiating body 1 or human in the FoV. Simultaneously, snapshots of a frame from the thermopile sensor 105 are taken every second (as the refresh rate is 8 Hz, a frame snapshot is the pixel to pixel average of 8 frames) that forms the background frame. This frame will be subtracted from the detection frame (frame in which a human is detected) in later stages. This helps to eliminate static warm objects such as monitors and light bulbs from the detection frame. The human presence is indicated when Vp —p > 0.02, as 0.02 is the mean noise amplitude at the highest possible gain. When there is a movement detection in the PIR sensor, the background estimation process is stopped. Figure 6a (Background frame) shows the background frame. When there is a human presence (shown in Figure 6b (Actual image)), the background frame is subtracted from each thermopile frame that is being read.

Figure 6c (Thermal frame from thermopile} shows the frame from thermopile corresponding to the scenario shown in Figure Sb. The resultant background removed frame is shown in Figure 6d (Background subtracted image) wherein the human presence is persistent. Even though the static warm bodies and background noise are removed from the data, there may be a few pixels present that do not represent the human. Such pixels can be neutralized using two- dimensional Gaussian filter or another type of filter. The filtered data is shown in Figure 6e (Image obtained after applying Gaussian filter). The filtered data is interpolated by 2 times to get better accuracy. The interpolated image is shown in Figure Sf (Interpolated image). The next step is to identify the position of the person perpendicular to the field of view axis of the sensor platform. To perform this, the outline of the pixels representing the person has to be detected. For example a contour associated with the person may be determined. This can be done using Canny edge detector as this involves low complexity processing compared to other edge detection techniques. This is a multi-step technique that detects edges as well as suppresses noise at the same time. Figure 6g (Image after edge detection}shows the position of the human in the frame, i.e. Figure 6g shows a contour of a radiating body 1. However, this position cannot be mapped onto the physical location unless the depth of the person from the sensor platform is known as the thermopile frame is the rectangular projection that enlarges away from the sensor. Hence, this position may be called the virtual position.

- 99. To determine the displacement information or P-T data, at least two contours associated with the person or radiating body may be determined.

Therefore, the control unit or microprocessor 111 may determine the imaged location of the radiating body or person 1 inthe thermal image at the first image time instance, and may determine a first contour associated with the radiating body in the thermal image at the first image tims instance.

Also, the control unit 111 may determine the imaged location of the radiating body 1 in the thermal image al the second image time instance, and may further be configured to determine a second contour associated with the radiating body in the thermal image at the second image time instance.

Referring to Figure 7, the first contour may be on the left image, Figure 7a, and the second contour may be on the right image, Figure 7h.

In particular, the displacement information may comprise an amount of pixels transversed, associated with the radiating body, between the first image time instance and the second image time Instance, which may be determined by comparing the location of the first contour and the location of the second contour.

The amount of pixels transversed may comprise a first amount of pixels transversed in a first direction (e.g. horizontal) and a second amount of pixels transversed in a second direction (e.g. vertical}, wherein the second direction is not parallel to the first direction.

The next step is to determine a possible distance of the person from the infrared sensing system 101. This is done using the variable gain.

As soon a human is detected the gain of the amplifier 121 is reduced from the maximum to the level at which the peak to peak amplitude of the output is just above the detection level.

Note that the gain can be reduced to the minimum but the outputs for gains set below the detection level contain only noise.

For each gain set in the range between detection level and the maximum, the Vp-p is recorded to form a V-G curve.

This has to be performed as soon as possible (within a second or two), before the movement speed and/or distance of the person or radiating body is changed substantially.

Table 2 shows a possible measurement.

-30- en 3,3 :800000000 3240000000 2,15 75000000 î 177 50000000 ; 0,19 900 Vertical} for 1 sec : 4x1 | Table 2 Here, various amplified electric signal are obtained corresponding to various gains of the amplifier 121. However, also obtaining an amplified electric signal at a single gain may be sufficient, if the amplified electric signal is within the dynamic range of the amplifier 121. Also the pixels transversed, i.e. displacement information, is obtained. The control unit may be configured to compare the measurement as shown in Table 2, with training data as shown in Table 1. In particular the control unit 111 may determine a plurality of first similarity scores between the value of the amplified electric signal and at least two of the plurality of possible values of the amplified electric signal, wherein each of the plurality of first similarity scores is associated with a respective predetermined position. For example, the control unit 111 may determine which V-G curve in Table 1 is closest to the V-G curve of Table 2, by determining a weighted average of the pointwise distances.

Similarly, the control unit may determine a plurality of second similarity scores between the value of the displacement information and at least two of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position. For example, the amount of pixels transversed as shown in Table 2, may be compared with the plurality of possible values of pixels transversed shown in Table 1.

Based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, wherein each of the plurality of joint similarity scores is associated with a respective predetermined position. The joint similarity score may be a weighted average of the first similarity score and the second similarity score. Generally, many possible ways exist to compare pluralities of numerical data. Which particular method works best or is most efficient may depend on the particular application and environment at hand. Based on a ranking of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment may be

-31- determined. Within the present example, the control unit may determine whether the measurement according to Table 2 is closest to training data associated with position p1, position p2 or position p3.

Since each point in the PIR training dataset represents a V-G curve, K-Nearest neighbors (KNN) classification for machine learning can be used for the classification of measurement V-G curve in the training dataset. An Euclidean distance may be employed to find the distance of each point in testing V-G curve and each V-G curve in training data. In this case K = 5, where 5 V-G curve instances in the training dataset are most similar to the test V-G curve. However, also other values of K are possible. Similarly, the test data (P-T data) from thermopile is recorded at the same time when the test V-G curve from PIR is computed. This P-T test data is classified for 5 nearest neighbors in thermopile training dataset using KNN classification. Each V-G curve in PIR training dataset has its pair in P-T data in thermopile dataset. Hence, out of the resultant 5 nearest neighbors in both PIR and P- T dataset, there is a unique V-G curve in PIR dataset that pairs with corresponding P-T data in thermopile dataset. It is observed from Equation 2 that, as speed and/or distance increases, Vp-p decreases. However, from P(t,h,v) « s/d it follows that P(t,h,v) increases as speed increases. Further, at constant speeds, Vp-p varies quadratically and P(t,h,v) varies linearly with distance. Hence, there cannot be multiple pairs in K-nearest neighbours (where k=5 in our case) classified for the test data in PIR - Thermopile training dataset.

Now that the absolute position of the person is known, the height can be estimated using the number of pixels in a frame in vertical direction, representing the person in the vertical FoV of 75°. Assuming that the persan is perpendicular to the FoV cone axis, the height H of the person is given by H = Ph x Number of pixels representing the person vertically, (11) where Ph = Y tan ((37.5)/12), wherein Y is a known distance of the person or radiating body to the sensor.

In an embodiment, the infrared system further is configured to detect a falling of the radiating body. For example, the falling may be detected by determining a change of the height of the radiating body.

Figure 12 depicts a flow diagram of an embodiment of a method for determining positional information of a radiating body 1 in an environment. In particular the method may be carried out using the infrared sensing system 101 described above.

Step 1201 comprises sensing, by a passive infrared sensor, an infrared radiation emitled by the radiating body.

-32.

Step 1203 comprises generating an electric signal associated with the infrared radiation sensed by the passive infrared sensor, wherein the electric signal depends on a distance and a velocity of the radiating body with respect to the passive infrared sensor.

Step 1205 comprises amplifying, by an amplifier, the electric signal.

Step 1211 comprises sensing by each of a plurality of infrared sensing elements comprised in an infrared array sensor, the infrared radiation Step 1213 comprises generating a respective element signal associated with the infrared radiation sensed by the respective infrared sensing element.

Step 1215 comprises generating, based on a plurality of element signals, a first thermal image at a first image time instance and generating, based on a plurality of element signals, a second thermal image al a second image time instance.

Step 1217 comprises determining a displacement information from a difference between an image location of the radiating body In the first thermal image and an image location of the radiating body In the second thermal image.

Step 1221 comprises determining, from the amplified electric signal and the displacement information, the position of the radiating body in the environment.

Step 1205 may further comprise the step of adjusting an amplification of the electric signal to be a first gain at a first time instance and a second gain at a second time instance, wherein the first gain is different from the second gain.

Step 1221 may further comprise the following steps: e providing a plurality of possible values of the amplified electric signal, and per possible value of the amplified electric signal, at least one respective predetermined position; e providing a plurality of possible values of the displacement information, and per possible value of the displacement value, at least one respective predetermined position; e determining a plurality of first similarity scores between a value of the amplified electric signal and each of the plurality of possible values of the amplified electric signal, wherein each of the plurality of first similarity scores is associated with a respective predetermined position; e determining a plurality of second similarity scores between a value of the displacement information and each of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position;

-33.

e determining, based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, wherein each of the plurality of joint similarity scores is associated with a respective predetermined position; and e determining, based on a ranking of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment.

Figure 13 depicts a flow diagram of an embodiment of a method of training an infrared sensing system 101. In particular, Figure 13 depicts an embodiment of a method to train an infrared sensing system 101 as described above. Furthermore, some steps in the method of training are similar to the steps depicted in Figure 12 describing an embodiment of a method for determining positional information of a radiating body 1 in an environment. A more detailed description of steps 1201-1217 is provided in the description of Figure 12.

However, with respect to the method shown in Figure 12, the method of training the infrared sensing system 101 comprises the following two steps: Step 1301 comprises providing a predetermined position of a radiating body. Step 1321 comprises storing, associated with the predetermined position of the radiating body, a value of the amplified eleciric signal and the displacement information. In essence, the system is now provided with the predetermined position of the radiating body, and stores the associated value of the amplified electric signal and the displacement information which are determined by the infrared sensing system in the memory device and in the image memory device respectively. As explained in detail above, an infrared sensing system for determining a position of a radiating body in an environment is disclosed, the system comprising: s 3 passive infrared sensor, configured to sense an infrared radiation emitted by the radiating body and to generate an electric signal associated with the infrared radiation sensed by the passive infrared sensor, wherein the electric signal depends on a position and a velocity of the radiating body with respect to the passive infrared sensor; e an amplifier, connected with the passive infrared sensor, configured to amplify the siectric signal;

-34- es an infrared array sensor comprising a plurality of infrared sensing elements, wherein o each of the plurality of infrared sensing elements is configured to sense the infrared radiation emitted by the radiating body and io generate a respective element signal associated with the infrared radiation sensed by the respective infrared sensing element; and o the infrared array sensor further is configured to generate, based on a plurality of element signals, a thermal image; sz control unit, o connected with the amplifier to receive the amplified slectric signal; and o connected with the infrared array sensor to receive the thermal Image; and » wherein the control unit is configured tc: o determing a displacement information from a difference between an imaged location of the radiating body In the thermal image at a first image time instance and an imaged location of the radiating body in the thermal image at a second time instance; o determing, from the amplified electric signal and the displacement information, the position of the radiating body in the environment.

As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention, which can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present invention in virtually any appropriately detailed structure. Further, the terms and phrases used herein are not intended to be limiting, but rather, to provide an understandable description of the invention.

The terms "a" or "an", as used herein, are defined as one or more than one. The term plurality, as used herein, is defined as two or more than two. The term another, as used herein, is defined as at least a second or more. The terms including and/or having, as used herein, are defined as comprising (i.e., open language, not excluding other elements or steps). Any reference signs in the claims should not be construed as limiting the scope of the claims or the invention.

-35- The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

- 36 -

CONCLUSIONS

An infrared detection system for determining a position of an radiating body in an environment, the system comprising: a passive infrared sensor configured to detect an infrared radiation emitted from the radiating body and generate an electrical signal associated with the infrared radiation detected by the passive infrared sensor, the electrical signal being dependent on a position and a speed of the radiating body relative to the passive infrared sensor; - an amplifier, connected to the passive infrared sensor, configured to amplify the electrical signal; - an infrared array sensor comprising a plurality of infrared detection elements, wherein each of the plurality of infrared detection elements is configured to detect the infrared radiation emitted from the radiating body and generate a respective element signal associated with the infrared radiation which is sensed by the respective infrared detecting element; and o the infrared array sensor is further configured to generate a thermal image based on multiple element signals; - a control unit connected to the amplifier to receive the amplified electrical signal; and o connected to the infrared array sensor to receive the thermal image; and - wherein the controller is configured to: determine a displacement information from a difference between an image location of the radiating body in the thermal image at a first image time instant and an image location of the radiating body in the thermal image at a second image time instant; o determine the position of the radiating body in the environment from the amplified electrical signal and the displacement information.

Infrared detection system according to claim 1, further comprising a memory device which stores: - a plurality of possible values of the amplified electrical signal, and per possible value of the amplified electrical signal, at least one respective predetermined position; and wherein the control unit is further configured to: - request from the memory device at least one predetermined position associated with a value of the amplified electrical signal received from the amplifier;

-37- - determine the position of the radiating body in the environment from the displacement information and from the at least one predetermined position as retrieved from the memory device.

Infrared detection system according to one or more of the preceding claims, further comprising an image memory device in which is stored: - a plurality of possible values of the displacement information, and per possible value of the displacement value, at least one respective predetermined position; and wherein the controller is further configured to: - request from the image memory device at least one predetermined position associated with a value of the displacement information determined by the controller; - determining from the amplified electrical signal and from the at least one predetermined position retrieved from the image memory device the position of the radiating body in the environment.

Infrared detection system according to claims 2 and 3, wherein the control unit is further configured for: - determining a plurality of first similarity scores between the value of the amplified electrical signal and at least two of the plurality of possible values of the amplified electrical signal, each of the plurality of first similarity scores is associated with a respective predetermined position; - determining a plurality of second similarity scores between the value of the displacement information and at least two of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position; - determining, based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, wherein each of the plurality of joint similarity scores is associated with a respective predetermined position; - determining, on the basis of an arrangement of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment.

An infrared detection system according to any one of the preceding claims, wherein the control unit to determine the image location of the radiating body in the thermal image at the first image time instant is further configured for

-38- - determining a first contour associated with the radiating body in the thermal image at the first image time instant; and wherein the control unit is further configured to: determine a second contour associated with the radiating body in the thermal image at the second image time instant in order to determine the image location of the radiating body in the thermal image at the second image time instant

An infrared detection system according to any one of the preceding claims, wherein the thermal image consists of an array of pixels, and wherein the displacement information includes a plurality of pixels traversed, associated with the radiating body, between the first image time instant and the second image time instant.

The infrared detection system of claim 6, wherein the plurality of pixels traversed comprises: - a first plurality of pixels traversed in a first direction; and - a second plurality of pixels traversed in a second direction, the second direction not being parallel to the first direction.

An infrared detection system according to any one of the preceding claims, wherein a difference between the first image time instant and the second image time instant is less than 3 seconds, preferably less than 1 second.

An infrared detection system according to any one of the preceding claims, wherein a first gain of the amplifier is such that the amplified electrical signal is within a dynamic range of the amplifier, and wherein the controller is further configured to have at least one value of the amplified electrical signal selectable from a group consisting of: - a peak-to-peak value; - an amplitude value; -an integration value; and - a frequency value; and wherein the controller is further configured to associate the at least one value with the position of the radiating body in the environment.

The infrared detection system of claim 9, wherein - the amplifier further comprises a variable gain; and - the control unit is further configured to

-39.- o set the variable gain of the amplifier as a first gain at a first time instant and a second gain at a second instant, the first gain being different from the second gain; o determine the at least one value when the variable gain is the first S gain; and o determine the at least one value when the variable gain is the second gain.

An infrared detection system according to any one of claims 2 to 10, wherein the control unit is further configured to: - associate the amplified electrical signal with a given predetermined position; and - storing the supplied predetermined position and a value of the amplified electrical signal, coupled to the supplied predetermined position, in the memory device.

An infrared detection system according to any one of claims 3 to 11, wherein the control unit is further configured to: - associate the displacement information with a given predetermined position; and - storing the provided predetermined position and a value of the displacement information related to the provided predetermined position in the image memory device.

A method for determining a position of an radiating body in an environment, the method comprising: - detecting, by a passive infrared sensor, an infrared radiation emitted by the radiating body; - generating an electrical signal associated with the infrared radiation detected by the passive infrared sensor, the electrical signal being dependent on a distance and a speed of the radiating body relative to the passive infrared sensor; - amplifying, by an amplifier, the electrical signal; - detecting by each of a plurality of infrared sensor elements included in an infrared array sensor, the infrared radiation and generating a respective element signal associated with the infrared radiation detected by the respective infrared sensor element; - generating, based on a plurality of element signals, a first thermal image at a first frame time instant;

- 40 - generating, based on a plurality of element signals, a second thermal image at a second frame time; - determining a displacement information from a difference between an image location of the radiating body in the first thermal image and an image location of the radiating body in the second thermal image; and - determining the position of the radiating body in the environment from the amplified electrical signal and the displacement information.

The method of claim 13, further comprising: providing a plurality of possible values of the amplified electrical signal, and per possible value of the amplified electrical signal, at least one respective predetermined position; - providing a plurality of possible values of the displacement information and per possible value of the displacement value, at least one respective predetermined position; - determining a plurality of first similarity scores between a value of the amplified electrical signal and each of the plurality of possible values of the amplified electrical signal, wherein each of the plurality of first similarity scores is associated with a respective predetermined position; determining a plurality of second similarity scores between a value of the displacement information and each of the plurality of possible values of the displacement information, wherein each of the plurality of second similarity scores is associated with a respective predetermined position; - determining, based on the plurality of first similarity scores and the plurality of second similarity scores, a plurality of joint similarity scores, each of the plurality of joint similarity scores being associated with a respective predetermined position; - determining, on the basis of an arrangement of the plurality of joint similarity scores, from the associated predetermined positions, the position of the radiating body in the environment.

A method according to claim 13 or 14, further comprising: - adjusting a gain of the electrical signal as a first gain at a first time instant and a second gain at a second time, wherein the first gain is different from the second gain.

18. A method of training an infrared detection system, the method comprising:

-41 - - providing a predetermined position of a radiating body; - detecting, by a passive infrared sensor, an infrared radiation emitted from the radiating body; - generating an electrical signal related to the infrared radiation sensed by the passive infrared sensor, the electrical signal being dependent on a position and a speed of the radiating body relative to the passive infrared sensor; - amplifying, by an amplifier, the electrical signal; - detecting, by each of a plurality of infrared sensor elements in an infrared array sensor, the infrared radiation emitted by the radiating body and generating a signal associated with the infrared radiation detected by the respective infrared sensor element; - generating, based on a plurality of element signals, a first thermal image at a first frame time instant;

generating, based on a plurality of element signals, a second thermal image at a second frame time; - determining a displacement information from a difference between an imaged location of the radiating body in the first thermal image and an imaged location of the radiating body in the second thermal image; and

- storing a value of the amplified electrical signal and the displacement information related to the predetermined position of the radiating body.