WO2015102713A2 - Suivi de mouvement via des réflexions radioélectriques sur des corps - Google Patents

Suivi de mouvement via des réflexions radioélectriques sur des corps Download PDF

Info

Publication number
WO2015102713A2
WO2015102713A2 PCT/US2014/059856 US2014059856W WO2015102713A2 WO 2015102713 A2 WO2015102713 A2 WO 2015102713A2 US 2014059856 W US2014059856 W US 2014059856W WO 2015102713 A2 WO2015102713 A2 WO 2015102713A2
Authority
WO
WIPO (PCT)
Prior art keywords
detecting
reflections
pattern
gesture
motion
Prior art date
Application number
PCT/US2014/059856
Other languages
English (en)
Other versions
WO2015102713A3 (fr
Inventor
Fadel ADIB
Zachary Edward KABELAC
Dina Katabi
Original Assignee
Massachusetts Institute Of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Massachusetts Institute Of Technology filed Critical Massachusetts Institute Of Technology
Priority to CN201480067276.XA priority Critical patent/CN105814450B/zh
Priority to EP14877116.5A priority patent/EP3055708A2/fr
Priority to JP2016521320A priority patent/JP6796485B2/ja
Publication of WO2015102713A2 publication Critical patent/WO2015102713A2/fr
Publication of WO2015102713A3 publication Critical patent/WO2015102713A3/fr

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
    • G01S5/02—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using radio waves
    • G01S5/0247—Determining attitude
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/05—Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves 
    • A61B5/0507—Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves  using microwaves or terahertz waves
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1116—Determining posture transitions
    • A61B5/1117—Fall detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/113—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb occurring during breathing
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/003—Bistatic radar systems; Multistatic radar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/06—Systems determining position data of a target
    • G01S13/08—Systems for measuring distance only
    • G01S13/32—Systems for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
    • G01S13/34—Systems for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/66—Radar-tracking systems; Analogous systems
    • G01S13/72—Radar-tracking systems; Analogous systems for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/87—Combinations of radar systems, e.g. primary radar and secondary radar
    • G01S13/878—Combination of several spaced transmitters or receivers of known location for determining the position of a transponder or a reflector
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
    • G01S5/02—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using radio waves
    • G01S5/0294—Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
    • G01S5/02—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using radio waves
    • G01S5/14—Determining absolute distances from a plurality of spaced points of known location
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/415—Identification of targets based on measurements of movement associated with the target
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/16—Details of sensor housings or probes; Details of structural supports for sensors
    • A61B2562/17—Comprising radiolucent components
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7253—Details of waveform analysis characterised by using transforms
    • A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms

Definitions

  • This invention relates to tracking people via body radio reflections.
  • Alternative technologies for detecting presence of and/or tracking people in an indoor environment include camera-based approaches in which captured images or video is processed, and ultrasonic or passive infrared motion sensors. Each of these alternative technologies suffers from limited accuracy, computational complexity, privacy and/or cost of deployment.
  • multiple radio frequency antennas makes use of multiple radio frequency antennas.
  • localization data is determined by concurrently transmitting signals from multiple antennas, and receiving reflections from the bodies as well as fixed objects (e.g., walls, desks, etc.) at multiple antennas.
  • a method for indoor localization uses signal reflections from a moving body.
  • the method includes emitting a transmitted signal comprising repetitions of a transmitted signal pattern from a transmitting antenna, and receiving, at each receiving antenna of a set of receiving antennas, a received signal comprising reflections of the transmitted signal.
  • the received signal is processed to form successive patterns of reflections of the transmitting signal pattern from moving bodies, and the successive patterns of reflections are processed by retaining effects of direct reflections from said moving body and excluding at least some effects of multipath reflections from said moving body that are also reflected from at least one static object.
  • a location of the body is determined using a result of processing the successive patterns of reflections.
  • excluding multipath reflections includes identifying a direct reflection from the moving body, and identifying indirect reflections from the moving body that also reflect from the at least one static object.
  • excluding multipath reflections includes identifying a first round-trip time from the transmitting antenna to the receiving antenna, said first round-trip time representing a shortest round-trip propagation time of a signal pattern reflecting from the moving body.
  • Determining the location of the body includes using the identified first round-trip times for each received signal to eliminating multipath reflections corresponding to round-trip times greater than the identified first round-trip time.
  • Using the identified first round-trip times comprises tracking said times over successive patterns of reflections.
  • the set of receiving antennas includes at least three antennas, with at least two antennas having different elevations.
  • Determining the location of the body comprises determining an elevation of the body.
  • the method further comprises detecting a pattern of motion in the tracked locations of the body.
  • Detecting a pattern of motion comprises detecting a temporal pattern of a trajectory of locations of the body.
  • Detecting a pattern of motion comprises detecting a pattern of motion in an environment of the localization.
  • Detecting the pattern of motion comprises detecting a gesture.
  • Detecting the gesture comprises detecting a whole-body gesture.
  • Detecting the gesture comprises detecting a limb gesture.
  • Detecting the gesture comprises detecting a pointing gesture.
  • Detecting a pointing gesture includes detecting a direction of a pointing gesture.
  • the method further comprises controlling an electrical or electronic device according to the detecting of the gesture.
  • Controlling the electrical or electronic device comprises controlling at least one of a gaming device, an appliance, and a lighting device.
  • Detecting the pattern of motion comprises detecting falling activity.
  • Detecting falling activity comprises monitoring at least one of a change in elevation, a duration of changing elevation of the body, and a proximity of a final elevation to a floor level.
  • the signal pattern comprises a narrow band signal at a carrier frequency swept from a first frequency to a second frequency.
  • the first frequency and the second frequency are both in a range of 5.46 to 7.25 gigahertz.
  • Receiving the received signal includes receiving a reflection of the transmitted signal after propagation through at least one obstruction.
  • the body is a person.
  • the body is a limb of a person.
  • a method for interaction with a device uses gestures made by a person.
  • the method includes tracking a motion of a body, including emitting a radio frequency transmitted signal comprising repetitions of a transmitted signal pattern from a transmitting antenna, receiving, at each receiving antenna of a set of receiving antennas, a received signal comprising reflections of the transmitted signal, tracking the location using successive reflections of the transmitted signal pattern.
  • a pattern of motion is detected using the tracked motion of the body. The detected pattern of motion is used to interact with the device.
  • the tracking of the motion includes using propagation time of the transmitted signal.
  • the transmitted signal pattern comprises a Frequency Modulated Carrier Wave (FMCW) signal pattern.
  • FMCW Frequency Modulated Carrier Wave
  • the reflections of the transmitted signal comprise reflections passing through an obstruction.
  • Detecting the pattern of motion comprises detecting a gesture.
  • Detecting the gesture includes detecting a two-dimensional gesture.
  • Detecting the gesture includes detecting a three-dimensional gesture.
  • Detecting the gesture comprises detecting a whole-body gesture.
  • Detecting the gesture comprises detecting a limb gesture.
  • Detecting the limb gesture includes distinguishing a reflection from a limb from a reflection of a whole body of a person according to a size of the reflection.
  • Detecting the gesture comprises detecting a pointing gesture.
  • the method further comprises controlling an electrical or electronic device according to the detecting of the gesture.
  • a method is directed to detecting fall of a person.
  • the method includes tracking a motion of the person, including emitting a radio frequency transmitted signal comprising repetitions of a transmitted signal pattern from a transmitting antenna, receiving, at each receiving antenna of a set of receiving antennas, a received signal comprising reflections of the transmitted signal, tracking the location using successive reflections of the transmitted signal pattern.
  • a pattern of motion is detected using the tracked motion of the body, and the detected pattern of motion is used to detect a fall of the person.
  • the reflections of the transmitted signal comprise reflections passing through an obstruction
  • Detecting the pattern of motion comprises monitoring at least one of a change in elevation, a duration of changing elevation of the body, and a proximity of a final elevation to a floor level.
  • Detecting the pattern of motion comprises monitoring a change of elevation includes detecting a change of elevation of at least a threshold value.
  • Detecting the pattern of motion comprises monitoring the duration of changing elevation includes detecting a duration less than a threshold value.
  • Detecting the pattern of motion comprises monitoring proximity of the final elevation includes detecting a final elevation less than a threshold value.
  • Detecting the pattern of motion comprises detecting when each condition of the group of conditions consisting of a change of elevation of at least a fist threshold value, a duration less than a second threshold value, and a final elevation less than a third threshold value, are satisfied.
  • a system in another aspect, in general, includes a transmitting antenna and a set of receiving antenna.
  • a transmitter is coupled to the transmitting antenna and configured to generate a transmitting signal comprising repetitions of a transmitting signal pattern.
  • a receiver is coupled to the set of receiving antennas, for receiving signals comprising reflections of the transmitting signal.
  • a processor is coupled to the transmitter and to the receiver. The processor is configured to cause the system to perform all the steps of any one of the methods set forth above.
  • the signals transmitted the multiple antennas have narrow frequency band (e.g., sinusoidal waves) with the frequency varying in a fixed pattern between a first frequency and a second frequency, for example, swept from the first frequency to the second frequency linearly changing frequency over time. This swept pattern may be repeated.
  • the signals transmitted from different antennas are offset in time from one another (or alternatively offset in frequency), thereby resulting in a received signal at a receiving antenna having components from different transmitters reflected from a body at distinct frequencies.
  • a reference transmit signal is distributed to each of the transmit antennas, and delayed by different times at different transmit antennas.
  • the reference transmit signal is also used at each of the receive antennas to determine propagation (time of flight) times reflecting from bodies from each of the transmit antennas to the receive antenna. In some examples, these propagation times are determined by modulating the received signals by the reference transmit signal.
  • the demodulated received signals are averaged (either as complex signals with magnitude and phase, or alternatively averaging magnitude of squared magnitude alone) over a number of sweep cycles to obtain an averaged received signal.
  • a reference timing signal is distributed to the stations, and the reference transmit signal is synthesized separately at each station.
  • Locations of multiple bodies are determined from the determined propagation times between multiple pairs for transmit antennas and receive antennas.
  • aspects can include one or more of the following features.
  • the transmit and/or receive antennas are directional antennas.
  • the antennas are arranged in a fixed arrangement along a line, in a two- dimensional pattern (e.g., in a "T" pattern on a face of a region in which bodies are tracked), or in a three-dimensional arrangement.
  • the fixed arrangement along a line, in a two- dimensional pattern (e.g., in a "T" pattern on a face of a region in which bodies are tracked), or in a three-dimensional arrangement.
  • the fixed arrangement along a line, in a two- dimensional pattern (e.g., in a "T" pattern on a face of a region in which bodies are tracked), or in a three-dimensional arrangement.
  • the fixed arrangement along a line, in a two- dimensional pattern (e.g., in a "T" pattern on a face of a region in which bodies are tracked), or in a three-dimensional arrangement.
  • the fixed arrangement along a line, in a two- dimensional pattern (e.g., in a "T" pattern on
  • the geometric relationship of the antenna locations is inferred using radio propagation time between locations.
  • the antennas are arranged in a room, for example, along a wall or a ceiling. In other examples, the antennas are arranged over multiple rooms of a multi- room space (e.g., a hospital wing or a retail store).
  • a multi- room space e.g., a hospital wing or a retail store.
  • Locations of multiple bodies are determined by iteratively (a) determining a location of a body based on determined propagation times between multiple transmitter- receiver pairs, and (b) having determined a location, effectively removing the effect of reflections from that location from the remaining signals.
  • the step of removing the effect of reflections from a location includes removing the effect of reflections from a region around the locations, where that region is determined according to a spatial model of the object causing the reflections (e.g., a model of a human body). In some examples, this iteration is repeated after having identified N bodies in a region by, at each iteration, determining a location for the n th body having removed the effect of the other N-l bodies in the received signals.
  • the delays of the reference transmit signal at the various transmitters may be fixed, or may be adapted according to the detected locations of bodies in the
  • Breathing of relatively motionless bodies is detected using an oscillation in reflected power from a location of a detected body at a breath frequency.
  • the difference between a reference power for a body at a reference time and subsequent power for that body is used to detect the breathing related oscillations.
  • Complex bodies are detected in three-dimensional space.
  • a human body with limbs extending from a torso may be detected according to a localization of the torso of the body, which may provide a relatively strong reflected energy, and according to a localization of a limb of the body, which may provide a relatively weaker reflected energy, in proximity and/or an appropriate geometric relationship of the detected torso location.
  • localization of complex bodies may be based on localization of elements of the complex body in geometric relationship based on a model or constraint of the complex body.
  • a device in another aspect, is used to track three-dimensional motion of one or more humans and body parts from the radio signals reflected off his/her body.
  • the approach works even if the person is occluded from the device or in a different room.
  • the device has one or more antenna for transmission and one or more antennas for receiving.
  • the device transmits a radio signal and uses its reflections to estimate the time it takes the signal to travel from the transmitting antenna to the reflecting object and back to each of the receiving antennas.
  • the device uses its knowledge of the position of the antennas to create a geometric reference model, which maps the round trip delays observed by the receive antennas to a two-dimensional (or three-dimensional) position of the reflecting body.
  • FMCW Multi-Shift Frequency-Modulated Carrier Wave
  • the technique allows distinguishing between FMCW signals generated from different sources by shifting them in time or in frequency. This distinguishability can be achieved by shifting in time, frequency, or code (e.g., multiplying the signals transmitted from the different antennas by different spreading codes).
  • Successive Silhouette Cancellation a technique referred to as Successive Silhouette Cancellation.
  • the technique iteratively finds the location of different people in the scene by finding the strongest reflector(s) - i.e., person (or set of persons), removing their effect on the received signals, and then proceeding to find weaker reflectors (persons).
  • the algorithm is described in the context of localizing people, it can be generalized to localizing any kind of moving or static object.
  • a method and apparatus is used for distinguishing between frequency modulated carrier wave (FMCW) signal transmitted by different sources.
  • the different sources may be connected to a same or to multiple different devices.
  • distinguishability is achieved by shifting the FMCW signals with respect to each other in the frequency domain.
  • shifting in the frequency domain is achieved by mixing with a signal while in other
  • the shifting in the frequency domain is achieved by adding a delay line.
  • distinguishability is achieved by shifting the FMCW signals with respect to each other by multiplying them with different codes.
  • the approach to two-dimensional and/or three-dimensional localization uses the signal reflections off one or more bodies or objects.
  • Time-of-flight (TOF) measurements from one or more transmit antennas to each of the receive antennas are computed.
  • Knowledge of the placement of the transmit and receive antennas is used to create a spatial reference that maps time-of-flight to a function of the position of the reflectors.
  • the method includes successively identifying and eliminating the reflections of different objects to focus on weaker or stronger reflectors in the environment in situations in which there are multiple reflectors in the environment.
  • the antennas may be arranged in various patterns, and in general can be placed in arbitrary locations.
  • static multipath is eliminated/mitigated by background subtraction.
  • "Fictitious" targets are eliminated by combining the signals from different transmit-receive antenna pairs. For example, the system first finds the strongest reflector, removes it, then re-encodes and proceeds to other reflectors. In some implementations, after finding some (or all) the reflectors, the system proceeds to a focusing step whereby it eliminates interference from unwanted reflectors and focuses on the reflector of interest. This focusing step is used to obtain a better estimate of a reflector's location.
  • estimated locations are processed to disentangles crossing paths by projecting that targets would move along the same direction after crossing as they did before crossing
  • the system can track body parts, for example to track hand movements. Such tracking can be done in two or in three dimensions.
  • the body part motion may be interpreted as a gesture, and that gesture may be used to control devices or communicate commands.
  • the system can track movement when multiple people are moving their body parts or where a person or multiple people are moving multiple body parts
  • breathing motion of humans is used to eliminate static reflectors.
  • multiple subtraction windows are used to identify and/or localize reflectors with different speeds.
  • One or more persons can be localized based on their breathing.
  • the system counts the breaths of one or more people, and can detect occasions when one or more people hold their breaths.
  • additional sensors of different modality e.g., cameras, infrared sensors, etc. are combined to improve the accuracy or add functionality.
  • the detected motion of a person or a body part is used to detect patterns, for example, by comparing against known patterns (e.g., a known gesture).
  • the multi-body localization can provide a count of bodies in a region as compared to mere detection of presence of one or more bodies. Furthermore, accurate trajectories of the bodies are also available, which may not be easily determined using alternative technologies.
  • radio frequency reflections can provide increased privacy. For example, monitoring presence/traffic in a public bathroom could be accomplished without privacy concerns associated with video monitoring.
  • Detection of breathing provides a way of detecting a relatively stationary body, which could otherwise be mistaken for background. For example, in an application that monitors presence of people in a room, an ultrasonic motion detector may miss the motionless person whereas their breathing would nevertheless be detected.
  • Concurrent transmission from all (or sets of multiple fewer than all) transmit antennas can be less affected by motion of a body (e.g., less "smearing" of the estimated location) than sequential transmission from single antennas where the change in location may be significant between transmit times.
  • a narrow band signal i.e., a swept tone
  • a narrow band signal provides a direct way of separating the transmissions from different transmitters in a receive signal, and can provide robustness to phase noise.
  • Energy efficiency approaches based on detected people and longer-term patterns of traffic of people in a building (e.g., energy usage can be based on the number of people in a room rather than mere presence of a person);
  • Traffic tracking in a retail space for example, to determine traffic patterns of individual people as they shop in a store.
  • FIG. 1 is a schematic block diagram of a first embodiment of a motion tracking system.
  • FIG. 2A is a plot of transmit and receive frequency over time; and FIG. 2B is a plot received energy over frequency corresponding to FIG. 2A.
  • FIG. 3A is a spectrogram (spectral profile versus time) for one transmit and antenna pair;
  • FIG. 3B is spectrogram after background subtraction; and
  • FIG. 3C is a plot of estimates of a first round trip time corresponding to the spectrogram in FIG. 3B.
  • FIG. 4 is a schematic block diagram of a second embodiment of a motion tracking system with concurrent transmission from multiple transmitting antennas.
  • FIG. 5 A is a plot of transmit frequencies from two transmit antennas and received frequency at one antenna; and FIG. 5B is a plot received energy over frequency corresponding to FIG. 5A.
  • FIG. 6A is a time of flight (TOF) profile; and FIG. 6B is a difference TOF profile corresponding to FIG. 6A
  • FIGS. 7A-E are "heat maps" formed from difference TOF profiles for one through five transmit-receive antenna pairs, respectively.
  • FIG. 8A is an initial heat map
  • FIGS. 8B-D are heat maps after successive iterations of a silhouette cancellation procedure.
  • FIGS. 9A-D are heat maps generated by a focusing procedure using locations estimated using the heat maps shown in FIGS. 8A-D, respectively.
  • FIG. 10 is a plot of estimated elevation versus time for walking, sitting (in a chair and on the ground), and falling to the ground.
  • FIG. 11 is a spectrogram of a pointing gesture.
  • FIG. 12 is a schematic diagram on an antenna array.
  • a first embodiment of a motion tracking system 100 makes use of a number antennas 150 to transmit and receive radio frequency signals that are reflected from objects (e.g., people) in the environment of the system, which may include one or more rooms of a building, the interior of a vehicle, etc., and may be partitioned, for example, by substantially radio-transparent barriers, for instance building walls or cloth sheets.
  • objects e.g., people
  • the objects in the environment include both fixed objects, such as chairs, walls, etc., as well as moving objects, such as but not limited to people.
  • the system can track people, who may be moving around a room, or may be relatively stationary, for instance, sitting in a chair of lying in bed, but may nevertheless exhibit breathing motion that may be detected by one or more embodiments described below.
  • the system provides motion analysis results 130 based on the radio frequency signals. In various embodiments, these results include locations of one or more people in the environment, detected body or limb gestures made by the people in the environment, and detected physical activity including detection of falls.
  • TOF time-of- flight
  • round-trip time information derived for various pairs of antennas 150.
  • three paths 185A-C reflecting off a representative point object 180 are shown between a transmitting antenna and three receiving antenna, each path generally having a different TOF.
  • the TOF from an antenna at coordinates (x t , y t , z t ) reflecting from an object at coordinates (x 0 , y 0 , z 0 ) and received at an antenna at coordinates (x r , y r , z r ) can be expressed as
  • the TOF for example associate with path 185 A, constrains the location of the object 180 to lie on an ellipsoid defined by the three-dimensional coordinates of the transmitting and receiving antennas of the path, and the path distance determined from the TOF.
  • a portion of the ellipsoid is depicted as the elliptical line 190A.
  • the ellipsoids associated with paths 185B-C are depicted as the lines 190B-C.
  • the object 180 lies at the intersection of the three ellipsoids.
  • the system 100 includes a signal generator that generates repetitions of a signal pattern that is emitted from the transmitting antenna 150.
  • the signal generator is an ultra wide band frequency modulated carrier wave (FMCW) generator 120. It should be understood that n other embodiments other signal patterns and bandwidth than those described below may be used while following other aspects of the described embodiments.
  • FMCW ultra wide band frequency modulated carrier wave
  • TOF estimates are made using a Frequency-Modulated Carrier Wave (FMCW) approach.
  • FMCW Frequency-Modulated Carrier Wave
  • a transmit frequency is swept from over a frequency range as shown by solid line 210.
  • the frequency range is 5.46-7.25 GHz (i.e., a frequency range of approximately 1.8 GHz) with a sweep duration and repetition rate of 2.5 milliseconds.
  • the receiving antenna receives the signal after a TOF 222 (i.e., reflected from a single object), with frequency as shown in the dashed line 220.
  • the TOF 222 corresponds to a difference 224 in transmitted and received frequencies, which is a product of the TOF and the rate of frequency change of the swept carrier for the transmit antenna.
  • a frequency shift component 160 implements the frequency shifting, for example, including a modulator that modulates the received signal with the transmitted signal and retaining a low frequency range representing TOF durations that are consistent with the physical dimensions of the environment.
  • the output of the frequency shifter is subject to a spectral analysis (e.g., a Fourier
  • the output of the frequency shifter is samples and a discrete time
  • FFT Fast Fourier Transform
  • the system addresses the first multipath effect, referred to as static multipath, using a time differencing approach to distinguish a moving object's reflections from reflections off static objects in the environment, like furniture and walls.
  • static multipath using a time differencing approach to distinguish a moving object's reflections from reflections off static objects in the environment, like furniture and walls.
  • reflections from walls and furniture are much stronger than reflections from a human, especially if the human is behind a wall. Unless these reflections are removed, they would mask the signal coming from the human and prevent sensing her motion. This behavior is called the "Flash Effect'".
  • these reflections include both signals that travel directly from the transmitting antenna to the moving body (without bouncing off a static object), reflect of the object, and then travel directly back to the receiving antenna, as well as indirect paths that involve reflection from a static object as well as form a moving object.
  • these indirect reflections we refer to these indirect reflections as dynamic multi-path. It is quite possible that moving object reflection that arrives along an indirect path, bouncing off a side wall, is stronger than her direct reflection (which could be severely attenuated after traversing a wall) because the former might be able to avoid occlusion.
  • FIGS. 3A-C the horizontal axis of each figure represents a time interval of approximately 20 seconds, and the vertical axis represents a frequency range corresponding to zero distance/delay at the bottom to a frequency corresponding to a range of approximately 30 meters at the top.
  • FIGS 3A and 3B show FFT power before background subtraction (FIG. 3A) and after background subtractions (FIG. 3B).
  • FIG. 3C shows the successive estimates of the shortest distance (or equivalently time) of flight for successive sweeps, as well as a "denoised” (e.g., smoothed, outlier eliminated, etc.) contour.
  • Outlier Rejection the system rejects impractical jumps in distance estimates that correspond to unnatural human motion over a very short period of time. For example, in FIG. 2C , the distance from the object repeatedly jumps by more than 5 meters over a span of few milliseconds. Such changes in distance are not possible over such small intervals of time, and hence the system rejects such outliers.
  • Interpolation the system uses its tracking history to localize a person when she stops moving.
  • the background-subtracted signal would not register any strong reflector.
  • the system uses a Kalman Filter to smooth the distance estimates.
  • the system After contour tracking and de -noising of the estimate, the system obtains a clean estimate of the distance travelled by the signal from the transmit antenna to the moving object, and back to one of the receive antennas (i.e., the round trip distance). In this embodiment that uses one transmitting antenna and three receiving antenna, at any time, there are three such round trip distances that correspond to the three receive antennas. The system uses these three estimates to identify the three-dimensional position of the moving object, for each time instance. [0123] The system leverages its knowledge of the placement of the antennas.
  • the antennas are placed in a "T" shape, where the transmitting antenna is placed at the crosspoint of the "T” and the receiving antennas are placed at the edges, with a distance of 1 meter between the transmitting antenna and each of the receiving antennas.
  • the z axis refers to the vertical axis
  • the x axis is along the horizontal
  • the "T" shaped antenna array mounted to a wall the y axis extends into the room. Localization in three dimensions uses the intersection of the three ellipsoids, each defined by the known locations of the transmitting antenna and one of the receiving antennas, and the round-trip distance.
  • only two receiving antennas may be used, for example with all the antennas placed along a horizontal line.
  • a two dimensional location may be determined using an intersection of ellipses rather than an intersection of ellipsoids.
  • more than three receiving antennas i.e., more than three transmitting-receiving antenna pairs
  • more than three ellipsoids do not necessarily intersect at a point, various approaches may be used to combine the ellipsoids, for example, based on a point that is closest to all of them.
  • a second embodiment uses concurrent transmissions from multiple transmitting antennas, and also introduces techniques that are applicable to tracking multiple bodies. It should be understood that the approach to using concurrent transmissions is independent of the other techniques.
  • the multiple concurrent transmissions may be used as described in the first embodiment, for example, to track the first time of flight for each antenna pair to mitigate dynamic multipath as described above.
  • antennas 150 on the left are illustrated as transmit antennas and three antennas 150 on the right are illustrated as receiving antennas.
  • antennas may be configured in operation to at times act as transmitting antennas and at times act as receiving antennas. Therefore, for N antennas, up to N(N - 1) / 2 different pairs of antennas may be used to determine separate TOF information for a particular object.
  • the system is operated in a succession of transmit intervals, generally with multiple antennas emitting radio signals concurrently in each interval, and generally multiple antennas receiving reflections of those signals concurrently.
  • the set of transmitting antennas is generally varied from transmit interval to transmit interval, permitting sampling of all the different antenna pairs.
  • m antennas are used to transmit in each interval, and
  • (N - m) antennas are used to receive, and up to m(N - m) paths can be evaluated in one time interval.
  • m N
  • at least N paths are evaluated per
  • m and up to N / paths are evaluated per interval.
  • FIGS. 5A concurrent transmission from two transmit antennas is illustrated over a portion of an interval 212 (i.e, FIG. 5 A shows a shorter duration on the time axis than shown in FIG. 2A).
  • a first antenna 150 (see FIG. 4) transmits with a sweep frequency as shown by solid line 51 OA, which corresponds to line 210 in FIG. 2A.
  • a second antenna transmits the same swept signal with a delay 510 (referred to as ⁇ ), with this delayed signal is illustrated with solid line 5 IOC.
  • a receiving antenna receives two different reflected signals from the object, one from the first transmitting antenna and one from the second. The reflection of the signal from the first transmit antenna has a TOF 522A as discussed above with reference to FIG. 2A.
  • the reflection of the signal from the second transmit antenna has an apparent TOF of 522C, which is made up of the delay ⁇ plus the actual TOF from the second transmit antenna to the receive antenna.
  • the energy of the reflection of the first signal is at a frequency 524A corresponding to the TOF 522A from the first transmit antenna
  • the energy of the reflection from the second antenna is at a frequency 524C, corresponding to a time 522C equal to the sum of the TOF from the second transmit antenna plus the delay ⁇ 5310.
  • delay elements 440 each with a different delay (i.e., an integer multiple of ⁇ ) is introduced between a swept carrier signal generated by a FMCW generator 120 and the corresponding antennas selected for transmission to introduce different delays for each of the selected transmitting antennas.
  • a different delay i.e., an integer multiple of ⁇
  • FIG. 2B it should be recognized that the distribution of energy over frequency (and equivalently over TOF), is not generally concentrated as shown in the diagram. Rather, there is a distribution of energy resulting from the superposition of reflections from all the reflective objects in the environment. A more typical energy distribution as a function of TOF is shown in FIG.
  • TOF profile 4A which is referred to as a "TOF profile.”
  • One way of computing the TOF profile is to band-limit the output of the frequency-shift 160 (i.e., modulator) to the bandwidth of the frequency sweep, and then time sample the band- limited signal, followed by computation of an FFT.
  • the magnitude squared of the (complex) FFT values provides the TOF profile, which each FFT sample corresponding to a different frequency range and therefore to a different TOF range.
  • an approach to mitigating the effect of reflections from stationary objects is to identify time variation in the reflected energy, which generally corresponds to moving objects.
  • One approach to identifying this time variation is to compare a the TOF profile (e.g., the complex output of an FFT) obtained at a current time with a delayed version of the TOF profile.
  • a delay of 12.5 milliseconds e.g., a delay of five 2.5 millisecond sweeps
  • This (complex) difference between a current TOF profile and a delayed TOF profile may be averaged over a sequence of sweeps, for example, over five sweeps (e.g., 12.5 milliseconds).
  • an example of application of such a subtraction approach shows an energy peak 620 in a difference TOF profile.
  • each transmit-receive pair of antennas yields one TOF profile.
  • Each point of a TOF profile corresponds to one ellipsoid in three dimensions, or equivalently an ellipse in two dimensions (e.g., if the antennas and the objects are assumed to be in one plane). Therefore, an entire TOF profile may be mapped into a spatial distribution of estimated location, referred to below as a "heat map," by setting the intensity along each ellipse (or ellipsoid) according to each point of the TOF profile.
  • FIG. 7A difference TOF profile using one transmit-receive antenna pair is shown in FIG. 7A.
  • the heat map exhibits variation as a set of half-ellipses, with the foci of the ellipses being at the locations of the antennas used to form the TOF profile.
  • FIGS. 7B-7E show forming of combined heat maps by summing heat maps from two through five antenna pairs, respectively. Note that with five antenna pairs, as shown in FIG. 7E, locations of the two bodies are evident in the combined heat map.
  • This example uses a two-dimensional heat map approach (e.g., essentially assuming that the antennas and all the object are at the same height).
  • a three-dimensional point cloud analogous to a two- dimensional heat map (the terms “heat map”, “point cloud”, and “point map” are generally used interchangeably below), can be computed using the same approach taking into account the three-dimensional coordinates of the antennas.
  • the combination (i.e., superposition) of heat maps from different antenna pairs as described above may not be sufficient to accurately distinguish the location of bodies (i.e., people ) in the environment of the system.
  • bodies i.e., people
  • One reason is that different users may exhibit a "near-far" problem. Specifically, the reflections of a nearby person are generally much stronger than the reflections of a faraway person or a person behind an obstruction.
  • FIG. 8A illustrates the challenge of the near-far problem, showing the two- dimensional heat map obtained in the presence of four persons in the environment.
  • the heat map allows us to localize only two of these persons: one is clearly visible at (0.5, 2), and another is fairly visible at (-0.5, 1.3). The other two people, who happen to be further away from the antenna array, are completely overwhelmed by the power of the first two persons.
  • SSC Successive Silhouette Cancellation
  • Detection find the location of the strongest reflector by overlaying the heat maps derived from the difference TOF profiles for multiple (e.g., all) transmit- receive antenna pairs.
  • Re -mapping map the detected location to TOF (or a range of TOF) that could have generated that location for each transmit-receive antenna pair.
  • the location of the highest power reflector in the two- dimensional heat map of FIG 8A is at (0.5, 2), indicating that there is a person in that location.
  • Remapping step we map this location back to a corresponding TOF range separately for each transmit-receive pair.
  • the heat map is computed in two dimensions, each person is not a point reflector and rather has an extent in the vertical (z) direction.
  • the effect of reflections off his entire body on the TOF profile of each transmit-receive pair is estimated.
  • the shortest TOF for an object at (J , y) is with a reflection from the three-dimensional point (x, y, (z t - z r ) / 2) .
  • This TOF(min) is computed using the propagation speed c .
  • the greatest TOF for an object at (x, y) is assumed to correspond to a reflection from the three-dimensional point (JC, y, 0) , essentially assuming the point at floor level is farthest from the antennas.
  • This TOF(max) is also computed using the propagation speed c .
  • the range [TOF(min),TOF(max)] for each antenna pair is the product of the Remapping step.
  • other approaches to mapping and (x, y) into a range or other distribution over TOF for each antenna pair may be used, for example, using an estimated mean and standard deviation estimate of the TOF.
  • the Cancellation step is performed for each antenna pair.
  • the current TOF profile for an antenna pair is zeroed (or otherwise attenuated) in the estimated range of TOF corresponding to the estimates (x, y) of the person.
  • the combined heat map is then calculated by summing the heat maps determined from the TOF profiles after cancellation.
  • FIG. 7B illustrates the result after performing the procedure on the first person estimated to be at location (0.5, 2.0).
  • step 1 using the heat map of FIG. 8B a second person is located at position (-0.5, 1.3) and the steps are repeated yielding the heat map of FIG. 8C.
  • a third person is located in this heat map at location (0.8,2.7) and the steps are repeated yielding the heat map of FIG. 8D, in which a four person is located at (1.0,4.0).
  • a further Refocusing step is performed to improve the location estimates of the people found in the iteration described above as illustrated in FIGS. 8A- D.
  • each original TOF profile i.e., prior to any cancellation in the SSC procedure
  • TOFmin, TOFmax] range corresponding to the estimated lication of the object
  • time sequences of estimated (x, y) locations, or sets of locations may be further processed.
  • trajectory smoothing and/or assignment of locations to individuals may be performed, for example, using Kalman filtering or other model based or heuristic techniques.
  • the system can differentiate a hand motion from a whole-body motion (like walking) by leveraging the fact that a person's hand has a much smaller reflective surface than his entire body.
  • the system can track gestures even when they are simultaneously performed by multiple users.
  • the system focuses on each person individually and track his/her gestures.
  • a pointing gesture where different users may point in different directions at the same time, can be detected.
  • the system determines the direction in which each of the users is pointing. Note that the users perform these pointing gestures in three-dimensional and the system tracks the hand motion by using the TOFs from the different transmit-receive antenna pairs to construct a three dimensional point cloud in the refocusing step rather than a two-dimensional heat map.
  • the approach described above is effective at detecting and estimating the location of moving bodies.
  • as second approach is used instead of or in addition to detect relatively stationary people (e.g., sitting in a chair or lying in bed) based on their periodic breathing motion.
  • a difference TOF profile for each antenna pair is determined by taking difference over a relatively small time interval, for example, over 12.5 milliseconds. Breathing occurs at a much slower rate, for example, with a breath time of a few seconds.
  • an approach to detection of breathing people in the environment is based on detecting modulation of the TOF profile over time with a characteristic frequency consistent with breathing.
  • a variety of approaches may be used to detect such modulation for different TOF, for example, by analysis of a series of TOF profiles over a sufficient time scale to detect the modulation.
  • One way to do this is to apply the approach described above, but rather than can computing the different TOF profile with a short time difference (e.g., 12.5 milliseconds), a longer time difference (e.g., over a second, for example, 2.5 seconds) is used.
  • the SSC focusing step allows us to focus on each person while eliminating interference from all other people in the scene (as shown in FIGS. 9A-D).
  • it allows us to focus on each person individually, and monitor the max power of each person's focused heat map as a function of time.
  • the maximum power from each person's heat map goes up and down periodically. This is because breathing is a rhythmic motion that alternates between inhaling and exhaling.
  • the maximum power is lowest when the person's chest returns to its location in the initial subtraction frame, and is highest when it is at the furthest position from its initial location.
  • one person's peak-to-peak signal may be multiple orders of magnitudes higher than that of another person.
  • the SSC procedure produces a sequence of heat maps shown in FIGS. 9A-D that demonstrates the utility of the procedure in detecting weaker reflections and eliminating interference from other persons in the environment allowing focusing on each person individually and monitoring his/her breathing.
  • This approach allow detection of periods of time when a user stops breathing (e.g., holds his or breath, or as a result of sleep apnea).
  • One potential application relates to energy saving and building automation.
  • the system can identify the number of people in a room or a house and their locations.
  • the system can be leveraged as an element of a building an automation system that proactively adjusts the temperature in a room based on the number of people or turns off lights/HVAC for empty rooms. This can help in saving energy as well as increase awareness to how people roam around buildings. It could also learn when people come in and when they leave and predicatively turn on, off, or adjust light/HVAC/temperature systems.
  • the system can be used for a gesture-based interface whereby a person automates different household appliances by performing a gesture.
  • the system can track the gesture and interpret it as a command; this will allow the person to control appliances such as: turn on and off the TV, write a letter or a word in the air, turn off other household appliances, etc.
  • the system can learn people's everyday actions and automate tasks. For example, it can learn when a person wakes up (or detect when he/she gets out of bed) and immediately turn on the shower, then when she gets out of the shower, turn on the coffee machine. It can also know when a person is present and hence can determine when she comes into her house. So the system can learn the times at which a person typically returns home (by tracking these patterns across different days) and automatically adjust heating/HVAC/lighting based on previous patterns.
  • tracking customers can be used to improve understanding of purchasing behavior.
  • the system can track shoppers in retail stores with very high accuracy providing deep insights into their behavior. It localizes shoppers by relying on radio reflections off their bodies, without requiring them to have a smartphone. It generates wide-ranging in- store analytics such as the number of people who enter/exit the store, the paths they follow, the locations they stop at, and the time they wait in lines to pay. It also tracks hand movements and identifies when a user picks up or returns items to the shelves.
  • These insights provide stores with deep awareness into customer behavior; for example, stores can analyze customer's reactions to their promotion campaigns, and adapt these campaigns in real-time. Further, by understanding customers' trajectories, stores can make products more accessible and more efficiently distribute their staff, thus increasing customer satisfaction, and hence retention.
  • these analytics enable stores to identify opportunities to increase sales, enhance customer experience and loyalty, and optimize in-store marketing.
  • the second set of Retail Analytics customers are the consumer goods suppliers (CPG companies, primarily).
  • CPG companies Today, Market Research Directors within CPG companies spend approximately $5.6Bn per year on market research, including insights from point- of-sale data and customer panel surveys, compiled by research firms such as ACNielsen.
  • Location Analytics insights can help CPG Brand Managers (consumers of the Market Research) to be more efficient with sales promotion spend at retailers.
  • the system can track patients as they move about in the hospital. For example, patients fall in their rooms and bathrooms, and it is important to detect when they fall. It may also be important to detect when a patient leaves their room and track them as they move around in the hospital. This becomes even more important in psychiatric sections or psychiatric institutions. Further, because the system can monitor people in general, it can also provide information into visitor waiting times. In additional, it can track if a nurse or another caregiver came into the room and did their responsibility (e.g., checked/changed the patient's vitals)
  • Junior housing and assisted living facilities strive to offer freedom to their residents and to create an atmosphere focused on enjoying life, rather than being afraid of, so they tend to be wary of adopting any monitoring technology.
  • the system can be used to track people (mainly the elderly) in senior housing facilities. It can be used to detect falls, track how many times they used the bathroom or went out of their room or bed, whether they came near the window and received enough sunshine. It can also track the visitors (e.g., their number and actions) when they come into the facility, and the caregivers as they take care (or not take care) of the residents.
  • the system can also be used for home or enterprise security. Specifically, because it can track human motion, it can know if an intruder enters the building, or even if he is roaming outside the house. For example, it can automatically turn off an alarm or send an alert when someone enters the building when the security system is turned on. Further, even if there are residents inside the house or enterprise and it senses a person roaming around outside the facility, it can detect their presence, track their movements, and/or alert about their presence.
  • FIG. 10 shows the system's estimate of the elevation along the vertical (z) dimension for four activities: a person walking, sitting on a chair, sitting on the ground, and (simulated) falling on the ground.
  • to distinguish a fall on the ground from a sitting on the ground one has to exploit that during a fall the person changes her elevation faster than when she voluntarily sits on the floor.
  • the system can estimate a pointing angle, which can then be used to control devices such as electrical or electronic devices.
  • the system first provides coarse estimation of a body part motion, and then further processes that coarse estimate to determine the pointing angle.
  • the user starts from a state where her arm is rested next to her body. She raises the arm in a direction of her choice with the intention of pointing toward a device or appliance, and then drops her hand to the first position. The user may move around and at a random time perform the pointing gesture. We require however that the user be standing (i.e., not walking) when performing the pointing gesture. The goal is to detect the pointing direction.
  • FIG. 11 illustrates the difference between the spectrogram (spectral profile versus time) for one antenna pair when there is a whole body motion and that of an arm pointing.
  • the human was moving then stopped and performed the pointing gesture.
  • the two bright spots around 18 seconds and 21 seconds represent the arm being lifted and dropped, respectively.
  • the figure shows that the signal variance along the vertical axis is significantly larger when the reflector is the entire human body than when it is just an arm motion. If the reflector is large, its parts have slightly different positions from each other; hence, at any point in time the variance of its reflection along the y-axis is larger than that of an arm movement.
  • the system uses this spatial variance to detect body part motion from a whole body motion.
  • the system estimates the direction of the motion to identify the pointing direction, which involves the following steps:
  • FIG. 11 shows how the system segments the round trip distance spectrogram obtained from each receive antenna.
  • Denoising As is the case for a whole body motion, the contour of the segmented spectrogram is denoised and interpolated to obtain a clean estimate of the round trip distance of the arm motion as a function of time, for each receive antenna.
  • Examples of systems that estimate pointing direction can be to enable a user to control household appliances by simply pointing at them. Given a list of instrumented devices and their locations, the system can track the user's hand motion, determine the direction in which she points, and commands the device to change its mode (e.g., turn on or off the lights, or control our blinds).
  • the mode e.g., turn on or off the lights, or control our blinds.
  • the antenna array may form a linear array, multiple linear arrays (for example with a horizontal array and a vertical array in a "T' or inverted "T" shape, of an "H” shape).
  • multiple linear arrays may be disposed about the environment.
  • FIG. 12 shows an "H" shaped antenna array 1500 in which antennas 1510 are disposed along a set of linear supports 1520, which are mounted in a plane (e.g., on a wall) above the floor level.
  • the antennas are directional, for example, being mounted so that their sensitivity is directed into the room or region being monitored.
  • the relative locations of the antennas, and the location of the antenna array in the environment are known to the system when it is configured by the installer of the system.
  • the system monitors direct inter-antenna TOF to infer the relative locations of the antennas, and may monitor static reflections in infer the location of the array within the environment.
  • the system monitors propagation time to also estimate delays in electronic paths (e.g., wires, amplifiers, etc.) to that such delays can be subtracted from measured delays to yield the over-the-air propagation times used to estimate locations of reflecting bodies.
  • the approaches described above are generally implemented as part of the signal analysis module 170, shown in FIG. 1, and module 470 in FIG. 4. Operation of the signal analysis module is controlled by a controller 110 or 410, which coordinates the selection of transmit and receive antennas, generation of the swept carrier signal by the FMCW generator 120, in addition to coordinating operation of the signal analysis module 170 or 470.
  • the signal analysis module and/or the controller are implemented in software, for example, including one or more programmable processors that execute instructions stored on non-transitory media (e.g., semiconductor memory) in the system.
  • Implementation of the frequency shift modules 160 may be in hardware, for example, using analog modulation units, or may be implemented using components of a software-defined radio (again controlled by stored software instructions).
  • the analysis may be distributed, with some of the processing being performed on one or more processors in electronics associated with a particular antenna array, while other processing, for example, fusing information from multiple arrays, may be performed at a separate (e.g., central server) computer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Biophysics (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Veterinary Medicine (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physiology (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Electromagnetism (AREA)
  • Human Computer Interaction (AREA)
  • Radar Systems Or Details Thereof (AREA)
  • User Interface Of Digital Computer (AREA)
  • Emergency Alarm Devices (AREA)
  • Position Fixing By Use Of Radio Waves (AREA)

Abstract

La présente invention concerne un système de suivi de mouvement qui fait usage d'un certain nombre d'antennes pour émettre et recevoir des signaux de radiofréquence qui sont réfléchis par des objets (par exemple des personnes) dans l'environnement du système, qui peut comporter une ou plusieurs salles d'un immeuble, l'intérieur d'un véhicule, etc., et peut être partitionné, par exemple, par des murs ou des feuilles en toile. En général, les objets dans l'environnement comprennent des objets fixes, tels que des chaises, des murs, etc., ainsi que des objets mobiles, tels que, mais sans y être limité, des personnes. Le système peut suivre des personnes qui peuvent se déplacer autour d'une salle ou qui peuvent être relativement immobiles, par exemple, assises sur une chaise ou couchées sur un lit, mais peuvent néanmoins montrer des mouvements respiratoires qui peuvent être détectés. Le système peut détecter des gestes du corps ou de membres faits par les personnes dans l'environnement, et détecter une activité physique comprenant une détection de chutes.
PCT/US2014/059856 2013-10-09 2014-10-09 Suivi de mouvement via des réflexions radioélectriques sur des corps WO2015102713A2 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201480067276.XA CN105814450B (zh) 2013-10-09 2014-10-09 通过身体无线电反射的运动跟踪
EP14877116.5A EP3055708A2 (fr) 2013-10-09 2014-10-09 Suivi de mouvement via des réflexions radioélectriques sur des corps
JP2016521320A JP6796485B2 (ja) 2013-10-09 2014-10-09 体の無線反射によるモーショントラッキング

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201361888662P 2013-10-09 2013-10-09
US61/888,662 2013-10-09
US201461943957P 2014-02-24 2014-02-24
US61/943,957 2014-02-24
US201461985066P 2014-04-28 2014-04-28
US61/985,066 2014-04-28

Publications (2)

Publication Number Publication Date
WO2015102713A2 true WO2015102713A2 (fr) 2015-07-09
WO2015102713A3 WO2015102713A3 (fr) 2016-04-07

Family

ID=53494193

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/059856 WO2015102713A2 (fr) 2013-10-09 2014-10-09 Suivi de mouvement via des réflexions radioélectriques sur des corps

Country Status (4)

Country Link
EP (1) EP3055708A2 (fr)
JP (1) JP6796485B2 (fr)
CN (1) CN105814450B (fr)
WO (1) WO2015102713A2 (fr)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106764960A (zh) * 2015-11-19 2017-05-31 宏达国际电子股份有限公司 侦测方法及其照明装置
EP3193189A1 (fr) * 2016-01-15 2017-07-19 Panasonic Intellectual Property Management Co., Ltd. Capteur de positionnement et procédé d'estimation de direction
WO2017175604A1 (fr) * 2016-04-06 2017-10-12 Ntn株式会社 Dispositif de surveillance
WO2018009774A1 (fr) * 2016-07-08 2018-01-11 Wal-Mart Stores, Inc. Détection de présence de clients
WO2018009786A1 (fr) * 2016-07-08 2018-01-11 Wal-Mart Stores, Inc. Mesure de poids à distance d'un élément de support de produit
CN107576957A (zh) * 2016-07-05 2018-01-12 松下知识产权经营株式会社 传感器以及推定方法
CN107797105A (zh) * 2016-09-01 2018-03-13 现代自动车株式会社 运动检测装置、运动检测方法及运动检测天线
EP3300662A1 (fr) * 2016-09-30 2018-04-04 Nokia Technologies OY Détermination d'une activité intime par un dispositif de détection
WO2018159949A1 (fr) * 2017-03-02 2018-09-07 Samsung Electronics Co., Ltd. Procédé de reconnaissance de geste et dispositif électronique associé
JP2019513985A (ja) * 2016-04-15 2019-05-30 コグニティヴ システムズ コーポレイション 反復無線送信に基づく動き検出
WO2019192947A1 (fr) * 2018-04-04 2019-10-10 Valeo Schalter Und Sensoren Gmbh Procédé pour identifier au moins un schéma de mouvement d'au moins un objet cible, détecté par un système radar d'un véhicule, système radar et système d'aide à la conduite
WO2020037399A1 (fr) 2018-08-21 2020-02-27 Moonshot Health Inc. Systèmes et procédés pour cartographier un environnement donné
US10765357B2 (en) 2015-12-17 2020-09-08 Industrial Technology Research Institute System and method for detecting muscle activities
JP2020535861A (ja) * 2017-10-06 2020-12-10 テルース ユー ケア インコーポレーションＴｅｌｌｕｓ Ｙｏｕ Ｃａｒｅ， Ｉｎｃ． 高齢者介護のための非接触活動検出ネットワークによるバイタルサイン
US11050827B1 (en) 2019-12-04 2021-06-29 Motorola Solutions, Inc. Method and device for identifying suspicious object movements based on historical received signal strength indication information associated with internet-of-things devices
CN113456061A (zh) * 2021-06-16 2021-10-01 南京润楠医疗电子研究院有限公司 一种基于无线信号睡眠姿势监测方法及系统
CN113655881A (zh) * 2021-08-16 2021-11-16 苏州触达信息技术有限公司 基于声波控制的快捷激活系统及方法
US20220091231A1 (en) * 2015-07-17 2022-03-24 Sai Deepika Regani Method, apparatus, and system for human identification based on human radio biometric information
US11733350B2 (en) 2018-01-25 2023-08-22 Nec Corporation Object identification apparatus, object identification method, and object identification program
WO2024187157A1 (fr) * 2023-03-08 2024-09-12 Locationdas Inc. Systèmes et procédés d'intégration de mesure et de création précise de jumeau numérique

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106530619A (zh) * 2016-12-16 2017-03-22 北京囡宝科技有限公司 智能终端以及用其进行人体位置识别和跌倒检测的方法
CN108814615B (zh) * 2018-04-12 2021-07-13 东软熙康健康科技有限公司 一种在床监测设备及方法
CN108968970A (zh) * 2018-05-24 2018-12-11 厦门精益远达智能科技有限公司 一种多普勒毫米波雷达检测人体跌倒的方法、装置和雷达系统
JP7319827B2 (ja) * 2018-06-22 2023-08-02 旭化成エレクトロニクス株式会社 センサ装置およびシステムならびに生体センシング方法およびシステム
CN109239706A (zh) * 2018-08-27 2019-01-18 苏州矽典微智能科技有限公司 一种基于毫米波的人体检测方法及装置
CN111134685B (zh) * 2018-11-02 2022-08-09 富士通株式会社 跌倒检测方法和装置
KR102455345B1 (ko) * 2019-11-28 2022-10-18 디노플러스 (주) 합성곱 신경망 기반 한류 모션 교육시스템
JP7428876B2 (ja) 2019-12-26 2024-02-07 ミツミ電機株式会社 センサー装置
DE112020006507T5 (de) * 2020-03-13 2022-10-20 Mitsubishi Electric Corporation Radarvorrichtung
US11575452B2 (en) * 2021-04-13 2023-02-07 Qualcomm Incorporated Passive positioning with radio frequency sensing
JP7373754B2 (ja) * 2021-05-07 2023-11-06 フィンガルリンク株式会社 生体状態測定装置、生体状態測定方法、プログラム及び生体状態測定システム
JP7327868B1 (ja) * 2022-08-17 2023-08-16 フィンガルリンク株式会社 生体状態測定装置、生体状態測定方法、プログラム及び生体状態測定システム
WO2024038629A1 (fr) * 2022-08-17 2024-02-22 フィンガルリンク株式会社 Dispositif de mesure d'état biologique, procédé de mesure d'état biologique, programme et système de mesure d'état biologique

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002024958A (ja) * 2000-07-07 2002-01-25 Toto Ltd 浴室内人体検知装置
JP2003185735A (ja) * 2001-10-12 2003-07-03 Sony Corp 侵入物検出システムおよびその方法と受信装置
JP2003159298A (ja) * 2001-11-27 2003-06-03 Omron Corp 入浴者挙動検出装置および入浴者挙動検出方法
JP4258328B2 (ja) * 2003-09-12 2009-04-30 オムロン株式会社 ２周波ドップラ測距装置およびその装置を備えた検出システム
ATE479115T1 (de) * 2004-01-20 2010-09-15 Bae Systems Information Verfahren und vorrichtung zur erkennung durch die wand unter verwendung von cw-radar
JP4572305B2 (ja) * 2005-02-08 2010-11-04 広島市 人物在圏検知システム
JP2006255141A (ja) * 2005-03-17 2006-09-28 Tdk Corp センサー装置
US9063232B2 (en) * 2005-04-14 2015-06-23 L-3 Communications Security And Detection Systems, Inc Moving-entity detection
US7916066B1 (en) * 2006-04-27 2011-03-29 Josef Osterweil Method and apparatus for a body position monitor and fall detector using radar
JP2008145237A (ja) * 2006-12-08 2008-06-26 Fuji Heavy Ind Ltd 電波式物体検出装置
US20090017910A1 (en) * 2007-06-22 2009-01-15 Broadcom Corporation Position and motion tracking of an object
GB0719995D0 (en) * 2007-10-12 2007-11-21 Qinetiq Ltd Radar method and apparatus suitable for use in multipath environments
US8169362B2 (en) * 2009-08-03 2012-05-01 Raytheon Company Mobile sense through the wall radar system
JP5700940B2 (ja) * 2010-03-16 2015-04-15 ダイハツ工業株式会社 物体認識装置
JP5620232B2 (ja) * 2010-11-11 2014-11-05 アルプス電気株式会社 距離測定装置
CN103229071B (zh) * 2010-11-16 2015-09-23 高通股份有限公司 用于基于超声反射信号的对象位置估计的系统和方法
WO2012098962A1 (fr) * 2011-01-18 2012-07-26 シャープ株式会社 Dispositif d'entrée de coordonnées, dispositif d'affichage comprenant un dispositif d'entrée de coordonnées, et procédé d'entrée de coordonnées
JP5377689B2 (ja) * 2011-09-21 2013-12-25 斎藤 光正 定在波レーダー内蔵型ｌｅｄ照明器具
JP2013160705A (ja) * 2012-02-08 2013-08-19 Mitsubishi Electric Corp レーダ断面積計測システム
CN102866398B (zh) * 2012-09-21 2014-03-19 中国航天空气动力技术研究院 一种利用调频连续波雷达进行动目标识别的方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220091231A1 (en) * 2015-07-17 2022-03-24 Sai Deepika Regani Method, apparatus, and system for human identification based on human radio biometric information
US12066572B2 (en) * 2015-07-17 2024-08-20 Origin Research Wireless, Inc. Method, apparatus, and system for human identification based on human radio biometric information
CN106764960B (zh) * 2015-11-19 2019-11-05 宏达国际电子股份有限公司 侦测方法及其照明装置
CN106764960A (zh) * 2015-11-19 2017-05-31 宏达国际电子股份有限公司 侦测方法及其照明装置
US10203407B2 (en) 2015-11-19 2019-02-12 Htc Corporation Illumination device and detection method thereof
US10765357B2 (en) 2015-12-17 2020-09-08 Industrial Technology Research Institute System and method for detecting muscle activities
EP3193189A1 (fr) * 2016-01-15 2017-07-19 Panasonic Intellectual Property Management Co., Ltd. Capteur de positionnement et procédé d'estimation de direction
WO2017175604A1 (fr) * 2016-04-06 2017-10-12 Ntn株式会社 Dispositif de surveillance
JP2019513985A (ja) * 2016-04-15 2019-05-30 コグニティヴ システムズ コーポレイション 反復無線送信に基づく動き検出
CN107576957A (zh) * 2016-07-05 2018-01-12 松下知识产权经营株式会社 传感器以及推定方法
CN107576957B (zh) * 2016-07-05 2023-01-06 松下知识产权经营株式会社 传感器以及推定方法
WO2018009786A1 (fr) * 2016-07-08 2018-01-11 Wal-Mart Stores, Inc. Mesure de poids à distance d'un élément de support de produit
US10359309B2 (en) 2016-07-08 2019-07-23 Walmart Apollo, Llc Remote weight measurement of a product support member
GB2566414A (en) * 2016-07-08 2019-03-13 Walmart Apollo Llc Remote weight measurement of a product support member
WO2018009774A1 (fr) * 2016-07-08 2018-01-11 Wal-Mart Stores, Inc. Détection de présence de clients
CN107797105A (zh) * 2016-09-01 2018-03-13 现代自动车株式会社 运动检测装置、运动检测方法及运动检测天线
EP3300662A1 (fr) * 2016-09-30 2018-04-04 Nokia Technologies OY Détermination d'une activité intime par un dispositif de détection
US10684693B2 (en) 2017-03-02 2020-06-16 Samsung Electronics Co., Ltd. Method for recognizing a gesture and an electronic device thereof
WO2018159949A1 (fr) * 2017-03-02 2018-09-07 Samsung Electronics Co., Ltd. Procédé de reconnaissance de geste et dispositif électronique associé
JP2020535861A (ja) * 2017-10-06 2020-12-10 テルース ユー ケア インコーポレーションＴｅｌｌｕｓ Ｙｏｕ Ｃａｒｅ， Ｉｎｃ． 高齢者介護のための非接触活動検出ネットワークによるバイタルサイン
EP3691527A4 (fr) * 2017-10-06 2021-07-28 Tellus You Care, Inc. Réseau de détection d'activité sans contact pour soins de personnes âgées
JP7281457B2 (ja) 2017-10-06 2023-05-25 テルース ユー ケア インコーポレーション 高齢者介護のための非接触活動検出ネットワークによるバイタルサイン
US11733350B2 (en) 2018-01-25 2023-08-22 Nec Corporation Object identification apparatus, object identification method, and object identification program
WO2019192947A1 (fr) * 2018-04-04 2019-10-10 Valeo Schalter Und Sensoren Gmbh Procédé pour identifier au moins un schéma de mouvement d'au moins un objet cible, détecté par un système radar d'un véhicule, système radar et système d'aide à la conduite
WO2020037399A1 (fr) 2018-08-21 2020-02-27 Moonshot Health Inc. Systèmes et procédés pour cartographier un environnement donné
US11050827B1 (en) 2019-12-04 2021-06-29 Motorola Solutions, Inc. Method and device for identifying suspicious object movements based on historical received signal strength indication information associated with internet-of-things devices
CN113456061A (zh) * 2021-06-16 2021-10-01 南京润楠医疗电子研究院有限公司 一种基于无线信号睡眠姿势监测方法及系统
CN113655881A (zh) * 2021-08-16 2021-11-16 苏州触达信息技术有限公司 基于声波控制的快捷激活系统及方法
WO2024187157A1 (fr) * 2023-03-08 2024-09-12 Locationdas Inc. Systèmes et procédés d'intégration de mesure et de création précise de jumeau numérique

Also Published As

Publication number Publication date
CN105814450A (zh) 2016-07-27
EP3055708A2 (fr) 2016-08-17
WO2015102713A3 (fr) 2016-04-07
CN105814450B (zh) 2018-08-28
JP2016540960A (ja) 2016-12-28
JP6796485B2 (ja) 2020-12-09

Similar Documents

Publication Publication Date Title
US9753131B2 (en) Motion tracking via body radio reflections
JP6796485B2 (ja) 体の無線反射によるモーショントラッキング
Adib et al. 3D tracking via body radio reflections
Yang et al. See through walls with COTS RFID system!
Adib et al. See through walls with WiFi!
Adib et al. {Multi-Person} Localization via {RF} Body Reflections
Palipana et al. Recent advances in RF-based passive device-free localisation for indoor applications
Ngamakeur et al. A survey on device-free indoor localization and tracking in the multi-resident environment
Jiang et al. Design and evaluation of a wireless magnetic-based proximity detection platform for indoor applications
Shi et al. mPose: Environment-and subject-agnostic 3D skeleton posture reconstruction leveraging a single mmWave device
Chen et al. Robust dynamic hand gesture interaction using LTE terminals
KR20120045929A (ko) 로봇의 리로케이션 장치 및 방법
KR20210089144A (ko) 수동 객체들의 레이더 이미지들을 추적 또는 생성하기 위한 레이더
Lian et al. EchoSpot: Spotting your locations via acoustic sensing
Xie et al. Embracing lora sensing with device mobility
Konings et al. Improved distance metrics for histogram-based device-free localization
Surie et al. proxemics awareness in kitchen As-A-Pal: tracking objects and human in perspective
Schröder et al. InPhase: Phase-based ranging and localization
US11832933B2 (en) System and method for wireless detection and measurement of a subject rising from rest
Hu et al. A novel remote-tracking heart rate measurement method based on stepping motor and mm-Wave FMCW radar
Huang et al. An indoor FMCW localization system with the application of the extended Kalman filter
Hillyard et al. Through-wall person localization using transceivers in motion
Wengrowski A survey on device-free passive localization and gesture recognition via body wave reflections
Hamie Contributions to cooperative localization techniques within mobile wireless bady area networks
Amendolare et al. Transactional array reconciliation tomography for precision indoor location

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2016521320

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2014877116

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2014877116

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14877116

Country of ref document: EP

Kind code of ref document: A2