CN111435562A

CN111435562A - Method of processing ambient radio frequency data for activity identification

Info

Publication number: CN111435562A
Application number: CN201910026329.8A
Authority: CN
Inventors: 林均仰; 贾真; 奚杰; T.陈; Y.孙
Original assignee: Carrier Corp
Current assignee: Carrier Corp
Priority date: 2019-01-11
Filing date: 2019-01-11
Publication date: 2020-07-21
Also published as: US11077826B2; EP3680817A1; US20200223393A1

Abstract

A method of operating an activity recognition system includes capturing environmental Radio Frequency (RF) data by an RF sniffer. The ambient RF data is then received by the processor. The processor reduces noise content of the ambient RF data. Background is then subtracted from the ambient RF data by the processor. The processed ambient RF data is then converted into an image by the processor. The system generates successive images for each of a plurality of time intervals. Applying an image processing algorithm stored in a storage medium and executed by the processor to the plurality of consecutive images to determine activity recognition.

Description

Method of processing ambient radio frequency data for activity identification

Background

The present disclosure relates to activity recognition systems, and more particularly, to a method of processing ambient radio frequency data by a system for activity recognition.

Presence detection, intrusion detection and other activity recognition are typically performed by motion detectors in a variety of forms, including optical and thermal/infrared cameras, passive/active infrared motion detectors, acoustic sensors, vibration sensors, window magnetic sensors and/or glass break sensors. The most common motion sensor for intrusion detection is a passive infrared sensor (PIR), which relies on sensing heat radiated by a human body. PIRs may be deployed at entrances or transition points in buildings through which intruders may enter.

More recently, research and advancements have developed motion and/or presence sensing technologies that utilize changes in radio frequency electromagnetic fields (i.e., commonly referred to as RF fields) generated by wireless devices. Some systems include multiple wireless nodes/transceivers, where each node may determine a change in signal strength and/or link quality of a particular encoded or general RF signal received from other nodes. Then, the decision logic determines motion/presence. Other systems are based on a single transmitter and receiver to determine motion and/or presence in a region using unidirectional or bidirectional measurements. Unfortunately, these systems rely on the deployment of specific devices to generate and sample the RF field. Such deployment may result in deployment costs. Furthermore, it is desirable to improve the pre-processing of the radio frequency data stream to increase the detection confidence.

Disclosure of Invention

A method of operating an activity recognition system according to one non-limiting exemplary embodiment of the present disclosure includes: capturing, by a Radio Frequency (RF) sniffer, environmental RF data; receiving, by a processor, ambient RF data; reducing, by a processor, noise content of ambient RF data; subtracting, by the processor, the background from the ambient RF data; converting, by a processor, the noise-reduced and background-subtracted ambient RF data into an image; generating, by a processor, a continuous image for each of a plurality of time intervals; and applying an image processing algorithm stored in the storage medium and executed by the processor to each successive image to determine the activity recognition.

In addition to the foregoing embodiments, the noise content of the ambient RF data is reduced by averaging the multi-Channel State Information (CSI) subcarriers at the same time index to subtract out the common mode noise.

Alternatively or additionally, in the foregoing embodiment, subtracting the background includes converting the ambient RF data to a first derivative of time.

Alternatively or additionally, in the foregoing embodiments, converting to an image includes combining RF data from multiple antenna channels.

Alternatively or additionally, in the foregoing embodiments, the plurality of time intervals are associated with characteristics of a building area containing the RF sniffer.

Alternatively or additionally, in the foregoing embodiments, the image processing algorithm applies a deep learning network.

Alternatively or additionally, in the foregoing embodiments, the deep learning network is a Convolutional Neural Network (CNN).

Alternatively or additionally, in the foregoing embodiments, the environmental RF data is environmental WiFi data.

Alternatively or additionally, in the foregoing embodiments, the ambient WiFi data is Channel State Information (CSI) data.

A building system according to another non-limiting embodiment includes: a radio comprising a transmitting component configured to transmit Radio Frequency (RF) and a receiving component configured to receive RF to accomplish a primary task; and an activity recognition system configured to perform an activity recognition task, the activity recognition system comprising: a sniffer configured to sample and measure ambient RF signals over time; control circuitry including one or more processors and one or more storage media; RF background data stored in at least one of the one or more storage media and indicative of inactivity; computer instructions stored in at least one of the one or more storage media and executed by at least one of the one or more processors, wherein the computer instructions are configured to process the measured ambient RF signals, convert the processed ambient RF signals into a plurality of sequential images, and apply an image-based algorithm to compare the plurality of sequential images to RF background data to determine activity recognition.

In addition to the foregoing embodiments, the transmitting device, the receiving device and the sniffer are located in a building.

Alternatively or additionally, in the aforementioned embodiment, the sniffer is one of a plurality of sniffers each located in a respective one of a plurality of regions of the building.

Alternatively or additionally, in the aforementioned embodiments, the radio is one of a plurality of radios that each transmit a respective RF signal sampled by the sniffer.

Alternatively or additionally, in the foregoing embodiments, the radio is a WiFi device.

Alternatively or additionally, in the foregoing embodiment, the activity recognition system is an intruder alert system.

The foregoing features and elements may be combined in various configurations, without exclusivity, unless expressly stated otherwise. These features and elements and their operation will become more apparent from the following description and the accompanying drawings. It is to be understood, however, that the following description and the accompanying drawings are intended to be illustrative in nature and not restrictive.

Drawings

Various features will become apparent to those skilled in the art from the following detailed description of the disclosed non-limiting embodiments. The drawings that accompany the detailed description can be briefly described as follows:

FIG. 1 is a schematic diagram of a building system utilizing an activity recognition system, as one non-limiting exemplary embodiment of the present disclosure;

FIG. 2 is a schematic diagram of an activity recognition system;

FIG. 3 is a flow diagram of a method of operating an activity recognition system; and is

Fig. 4 is a flow chart of a method of pre-processing an ambient RF signal by a system.

Detailed Description

In the present disclosure, activity recognition detection relies on existing wireless sensors previously deployed in a building. Due to the penetration of wireless IoT devices, Radio Frequency (RF) signals are increasingly available, particularly in indoor building automation, the present disclosure proposes environmental RF fields generated with previously deployed devices that are not specifically used for intrusion detection purposes. A decision system is presented that determines devices suitable for motion, intrusion and/or activity recognition detection purposes. The system is also configured to facilitate novel preprocessing of RF fields (e.g., WiFi) to improve detection confidence.

In addition, more conventional systems may require deployment of wireless nodes around an area of interest (e.g., a room or house perimeter). However, these systems may not address the false positive problem caused by movement outside the region of interest. In the present disclosure, such problems are addressed by a method of unambiguously determining regions of interest in any arbitrary deployment. Further, the present disclosure includes machine learning and/or neural network routines that learn changes in the RF field corresponding to movement within the region of interest. Thus, machine learning and/or neural network routines may reject false positives caused by movement outside the region of interest.

Referring to fig. 1, an exemplary embodiment of a building system 20 (e.g., a wireless communication system) includes one or more commodity radios or links 22 (i.e., two shown in fig. 1) and one or more activity recognition systems 24 (i.e., two shown in fig. 1) that apply radio frequency sensing. The commodity radio 22 and the activity recognition system 24 are typically located in or near a building 26. Each commodity radio 22 is generally stationary and may include a transmitting component 28 (see arrow 30) configured to transmit Radio Frequency (RF) signals and a receiving component 32 configured to receive RF signals 30. Non-limiting examples of radios 22 and associated RF signals 30 include WiFi devices, Zigbee devices, iBeacon, and the like. Non-limiting applications for commodity radios 22 may include wireless telephones, entertainment systems, television systems, and any other type of wireless RF system commonly used in or near buildings 26. One non-limiting example of an activity recognition system 24 may be an activity recognition system.

Each respective commodity radio 22 is configured to perform a respective primary task, and the respective RF signal 30 enables such primary task to be accomplished. For example, a wireless television system may stream movies from a transmitting component 28 (e.g., a router) and to a receiving component 32 (e.g., a smart television). In another example, the telephone system may transmit the verbal communication as an RF signal 30, and from the transmitting component 28 (e.g., a power charger base) and to the receiving component 32 (e.g., a handheld telephone). All RF signals 30 acquired together in a given space constitute an ambient RF signal 33 having various characteristics such as signal strength, Channel State Information (CSI), and so forth. CSI typically represents the combined effects of, for example, scattering, fading, and power attenuation over distance. In one embodiment, the plurality of commodity radios 22 are networks configured to communicate in one of a mesh topology and a star topology.

The activity recognition system 24 is configured to utilize the ambient RF signal 33 by generally detecting changes in prescribed characteristics of the ambient RF signal that indicate, for example, the presence of movement 34. That is, the ambient RF signal 33 is typically utilized for dual purposes: a primary task when applied to one or more radios 22 (as previously described with respect to signal 32) and an activity recognition alert task when applied to activity recognition system 24. In one non-limiting example, the presence 34 may be a human intruder and the activity recognition system 24 may be an intrusion detection system.

Referring to fig. 1 and 2, the activity recognition system 24 includes one or more RF sniffers 36 (i.e., two shown), control circuitry 38, RF data 40 (e.g., RF background data), and prescribed instructions 42 (i.e., software programs). The RF sniffer 36 may be an RF device that measures physical characteristics of the received RF signals, such as signal strength, CSI, and the like. Non-limiting examples of the RF sniffer 36 are a Wi-Fi network interface card, a CSI monitor, and the like. Each RF sniffer 36 is located in a respective area 44 of the building 26. For example, zone 44 may be a separate room, or the first zone may be a zone proximate a first entrance door and the second zone may be a zone proximate a second entrance door of the same building 26. Each RF sniffer 36 is configured to sample and measure characteristics of the ambient RF signal 33 in the respective region 44.

It should be appreciated that the RF signal strength of the same RF signal 32 may vary from one area 44 to the next due to, for example, attenuation (i.e., passing through an object such as a wall) and/or distance from the transmission member 28. The region 44 is defined and configured during system commissioning. In one embodiment, the installer may pass through a corner of the area and cause the RF sniffer 36 to collect measurements of characteristics of the ambient RF signal 33. This may be stored in a site-specific database and the machine learning algorithm infers whether a change in a characteristic of the ambient RF signal 33 indicates an activity and/or movement presence 34 within the configured region 44. The characteristics of the ambient RF signal 32 are further measured over time as such measurements may differ over time depending on, for example, the use of the radio 22.

In one embodiment and as shown in fig. 1, the control circuit 38 may be located in each sniffer 36 as a single independent unit. In another embodiment, each sniffer 36 may be in communication with a single control circuit 38, and the control circuit 38 may be located in the building 26, or remotely. Upon detecting the presence of movement, control circuitry 38 may output a notification signal (see arrow 46 in fig. 1) to notification device 48 to notify a user, government agency, and/or the like.

The control circuit 38 may include one or more processors 50 (e.g., microprocessors) and one or more storage media 52 (e.g., non-transitory storage media) that may be writable and readable by a computer. The RF data 40 and instructions 42 are stored in a storage medium 52. In operation, the RF data 40 is used by the processor 50 along with input signals (see arrows 54 in fig. 2) indicative of the ambient RF signals 33 measured when the instructions 42 are executed to determine the presence of the moving presence 34 within the region of interest 44. In one embodiment, the RF data 40 may include RF background data indicating that no movement exists. The RF background data may be learned by the processor 50 as part of the instructions 42 via, for example, a machine learning algorithm. The RF data may include features that allow for determining whether the ambient RF signal 33 was generated by the fixed transmission component 28 by matching MAC addresses or by looking at temporal variations of the RF data 40, or both. The RF data 40 may also include extracted features associated with signal features due to motion of the presence 34 that are invariant to RF background variations (i.e., temporal variations of the ambient RF signal 33 due to motion).

Referring to FIG. 3, and in operation, activity recognition system 24 may be in an automatic configuration at block 100 upon initialization. At block 102, control circuitry 38 may determine whether the system is up. If not, system 24 loops back to apply block 102 again. If the system is started, and at block 104, the control circuitry 38 applies an auto-calibration that requires self-learning of the previously described RF background data and extracted features. At block 106, the system 24 may continuously monitor for a moving presence 34 within the region of interest 44 by comparing the measured ambient RF signal 33 to RF background data and/or extracted features stored as part of the RF data 40. At block 108, and based on the comparison, the control circuitry 38 determines whether activity (e.g., movement presence 34) is detected. If not, the system loops back to block 106. If movement presence 34 is detected, control circuitry 38 may effect triggering of an alarm via notification device 48 at block 110.

Referring to fig. 4, a method of pre-processing ambient RF signals or data 33 prior to applying machine learning or other data analysis algorithms is shown. At block 200, the environmental RF data 33 is captured by the RF sniffer 36 of the activity recognition system 24 and sent as an input signal 54 to the control circuitry 38 (see fig. 2). In one example, the captured ambient RF data may be WiFi CSI data and the RF sniffer 36 may be a CSI monitor. At block 202, noise is reduced as part of the ambient RF data 33. In one example, such noise reduction of data 33 is facilitated by removing an average of multiple CSI subcarriers at the same time index to subtract out common mode noise. The operations of

blocks

200 and 202 may be part of the prescribed instructions 42 stored in the storage medium 52 of the control circuit 38 and executed by the processor 50.

At block 204, the background is subtracted. In one example, the background subtraction is facilitated by converting the noise-reduced ambient RF data 33 into the first time derivative of the data. This step "flattens" the environmental background data to help detect signals that may be directly attributable to activity recognition. At block 206, the background-subtracted ambient RF data 33 is converted into an image 60 (see FIG. 2) that may be stored in the storage medium 52. In one example, the conversion to the image 60 is facilitated by combining data from multiple antenna channels, which are then converted to an image format. As an image, the processed ambient RF data 33 may be analyzed by the image processing algorithm 62 as part of the instructions 42. In this manner, the algorithm 62 may take advantage of the coherence between adjacent RF signal subcarriers, which are not typically considered by other more conventional algorithms. The operations of

blocks

204 and 206 may be part of the prescribed instructions 42 stored in the storage medium 52 of the control circuit 38 and executed by the processor 50.

At block 208, a successive image 60 is generated for each of a plurality of successive images for each of a plurality of time intervals 64 that are pre-programmed and stored in the storage medium 52. The duration of each time interval 64 is established by and associated with the characteristics of the particular building region 44 in which the ambient RF data 33 is detected. The plurality of sequential images 60 facilitates the use of a deep learning network to train shift invariance. One example of a network is a Convolutional Neural Network (CNN). The operations of block 208 may be part of the prescribed instructions 42 stored in the storage medium 52 of the control circuit 38 and executed by the processor 50. At block 210, the image processing algorithm 62 is applied to the plurality of consecutive images 60.

An advantage and benefit of the method for pre-processing the ambient RF signal 33 is that the noise level of the ambient signal is reduced and the activity information is maintained only by subtracting the background. In addition, the method combines different channels into an image data format, which enriches the level of information in the training data to achieve optimal recognition results.

The various functions described above may be implemented or supported by a computer program that is formed from computer readable program code and that is embodied in a computer readable medium. The computer readable program code may include source code, object code, executable code, and the like. The computer-readable medium may be any type of medium that can be accessed by a computer and may include Read Only Memory (ROM), Random Access Memory (RAM), a hard disk drive, a Compact Disc (CD), a Digital Video Disc (DVD), or other forms.

As used herein, terms such as component, module, system, and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. It is to be understood that an application running on a server and the server can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.

Advantages and benefits of the present disclosure include an RF activity recognition system configured to sense and utilize pre-existing RF signals. Another advantage is the plug and play functionality of the system, with little effort required by the user. Another advantage can supplement an installed intrusion detection system by providing full building coverage with transmissions for data/voice communications.

While the disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the spirit and scope of the disclosure. In addition, many modifications may be made to adapt a particular situation, application, and/or material to the teachings of the disclosure without departing from the essential scope thereof. Therefore, the present disclosure is not to be limited to the particular examples disclosed herein, but includes all embodiments falling within the scope of the appended claims.

Claims

1. A method of operating an activity recognition system, comprising:

capturing, by a Radio Frequency (RF) sniffer, environmental RF data;

receiving, by a processor, the ambient RF data;

reducing, by the processor, noise content of the ambient RF data;

subtracting, by the processor, background from the ambient RF data;

converting, by the processor, the noise-reduced and background-subtracted ambient RF data into an image;

generating, by the processor, a continuous image for each of a plurality of time intervals; and

an image processing algorithm stored in a storage medium and executed by the processor is applied to each successive image to determine an activity recognition.

2. The method of claim 1, wherein the noise content of the ambient RF data is reduced by removing averages of multi-Channel State Information (CSI) subcarriers at a same time index to subtract common mode noise.

3. The method of claim 1, wherein the subtracting the background comprises converting the ambient RF data to a first derivative of time.

4. The method of claim 2, wherein the subtracting the background comprises converting the ambient RF data to a first derivative of time.

5. The method of claim 1, wherein the converting into an image comprises combining RF data from multiple antenna channels.

6. The method of claim 4, wherein said converting into an image comprises said combining RF data from a plurality of antenna channels.

7. The method of claim 1, wherein the plurality of time intervals are associated with characteristics of a building area that includes the RF sniffer.

8. The method of claim 7, wherein the image processing algorithm applies a deep learning network.

9. The method of claim 8, wherein the deep learning network is a Convolutional Neural Network (CNN).

10. The method of claim 1, wherein the environmental RF data is environmental WiFi data.

11. The method of claim 10, wherein the ambient WiFi data is Channel State Information (CSI) data.

12. A building system, comprising:

a radio comprising a transmitting component configured to transmit a Radio Frequency (RF) and a receiving component configured to receive the RF to accomplish a primary task; and

an activity recognition system configured to perform an activity recognition task, the activity recognition system comprising: a sniffer configured to sample and measure ambient RF signals over time; control circuitry including one or more processors and one or more storage media; RF background data stored in at least one of the one or more storage media and indicative of inactivity; computer instructions stored in at least one of the one or more storage media and executed by at least one of the one or more processors, wherein the computer instructions are configured to process the measured ambient RF signals, convert the processed ambient RF signals into a plurality of sequential images, and apply an image-based algorithm to compare the plurality of sequential images to the RF background data to determine activity recognition.

13. The building system of claim 12, wherein the transmitting device, the receiving device, and the sniffer are located in a building.

14. The building system of claim 13, wherein the sniffer is one of a plurality of sniffers each located in a respective one of a plurality of regions of the building.

15. The building system of claim 13, wherein the radio is one of a plurality of radios that each transmit a respective RF signal sampled by the sniffer.

16. The building system of claim 12, wherein the radio is a WiFi device.

17. The building system of claim 12, wherein the activity recognition system is an intruder alert system.