WO2020219692A1 - Suivi d'objets à l'intérieur avec des mesures de capteur inertiel - Google Patents

Suivi d'objets à l'intérieur avec des mesures de capteur inertiel Download PDF

Info

Publication number
WO2020219692A1
WO2020219692A1 PCT/US2020/029549 US2020029549W WO2020219692A1 WO 2020219692 A1 WO2020219692 A1 WO 2020219692A1 US 2020029549 W US2020029549 W US 2020029549W WO 2020219692 A1 WO2020219692 A1 WO 2020219692A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
hop
nodes
localization
localizing
Prior art date
Application number
PCT/US2020/029549
Other languages
English (en)
Inventor
Ayon Chakraborty
Karthikeyan Sundaresan
Sampath Rangarajan
Ashutosh DHEKNE
Original Assignee
Nec Laboratories America, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/568,747 external-priority patent/US11089432B2/en
Priority claimed from US16/855,179 external-priority patent/US11169518B2/en
Application filed by Nec Laboratories America, Inc. filed Critical Nec Laboratories America, Inc.
Publication of WO2020219692A1 publication Critical patent/WO2020219692A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/029—Location-based management or tracking services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W64/00—Locating users or terminals or network equipment for network management purposes, e.g. mobility management
    • H04W64/006—Locating users or terminals or network equipment for network management purposes, e.g. mobility management with additional information processing, e.g. for direction or speed determination
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U10/00—Type of UAV
    • B64U10/10—Rotorcrafts
    • B64U10/13—Flying platforms
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/10—UAVs characterised by their flight controls autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/20—Remote controls
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U30/00—Means for producing lift; Empennages; Arrangements thereof
    • B64U30/20—Rotors; Rotor supports
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U50/00—Propulsion; Power supply
    • B64U50/30—Supply or distribution of electrical power
    • B64U50/31—Supply or distribution of electrical power generated by photovoltaics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W84/00—Network topologies
    • H04W84/18—Self-organising networks, e.g. ad-hoc networks or sensor networks

Definitions

  • the present invention relates to object tracking and more particularly object tracking using an unmanned aerial vehicle.
  • Unmanned aerial vehicles are aircraft that do not have an onboard human pilot. UAVs can be included, along with a ground-based controller, in an unmanned aircraft system. The UAV and ground-based controller can communicate with a wireless communications system.
  • a method for localizing and tracking mobile objects.
  • the method includes determining an initial location of a node in a multi-hop network based on multi-lateration from an unmanned aerial vehicle.
  • the method also includes applying an adaptive aperture to address a non-uniform velocity of the node based on a turning and a velocity vector.
  • a determination whether localization for the node can be implemented using first hop nodes in the multi-hop network is made.
  • inertial sensor measurements associated with the node are accessed, and the inertial sensor measurements are integrated with the adaptive aperture to improve localization accuracy.
  • a system for localizing and tracking mobile objects.
  • the system includes a processor device operatively coupled to a memory device.
  • the processor device determines an initial location of a node in a multi-hop network based on multi-lateration from an unmanned aerial vehicle.
  • the processor device applies an adaptive aperture to address a non-uniform velocity of the node based on the turn and a velocity vector.
  • the processor device also includes determines whether localization for the node can be implemented using first hop nodes in the multi-hop network. In response to a determination that localization cannot be implemented using the first hop nodes, inertial sensor measurements associated with the node are accessed and the inertial sensor measurements are integrated with the adaptive aperture to improve localization accuracy.
  • FIG. 1 is a block diagram illustrating a high-level system for localizing and tracking from an unmanned aerial vehicle (UAV), in accordance with an embodiment of the present invention
  • FIG. 2 is a block/flow diagram illustrating a scenario for implementing the system for indoor object tracking from a UAV, in accordance with an embodiment of the present invention
  • FIG. 3 is a block/flow diagram illustrating a scenario of the localizing and tracking system in action along with its various architectural components, in accordance with an embodiment of the present invention
  • FIG. 4 is a block diagram illustrating components of master node and location solver, in accordance with an embodiment of the present invention.
  • FIG. 5 is a block diagram illustrating a scenario of resetting an aperture to limit a loss in localization accuracy, in accordance with an embodiment of the present invention
  • FIG. 6 is a block diagram illustrating scenario of multi-hop tracking for extended coverage, in accordance with an embodiment of the present invention.
  • FIG. 7 is a block diagram illustrating a bitmap constructed by the UAV and sent in a FINAL message, in accordance with an embodiment of the present invention
  • FIG. 8 is a block diagram illustrating a controller for a localizing and tracking system, in accordance with an embodiment of the present invention.
  • FIG. 9 is a flow diagram illustrating a method for localizing and tracking an object, in accordance with an embodiment of the present invention.
  • FIG. 10 is a block diagram illustrating a localization and tracking system with inertial sensors, in accordance with an embodiment of the present invention.
  • FIG. 11 is a block diagram illustrating an indoor localization system, in accordance with an embodiment of the present invention.
  • FIG. 12 is a flow diagram illustrating a method for localizing and tracking an object, in accordance with an embodiment of the present invention.
  • systems and methods are provided to/for (accurately) localizing and tracking mobile objects (for example, persons, responders, etc.) in real-time in (for example, large) indoor environments (such as, for example, a building, a structure, etc.) with use of inertial sensors integrated into the localization process.
  • the systems and methods leverage a mobile virtual infrastructure provided by unmanned aerial vehicles (UAVs), with wireless technology for localization (for example, radio frequency (RF) communications technology such as ultra- wide band (UWB), which provides a balanced penetration and accuracy tradeoff, to localize and track objects directly from outside, without relying on access to any indoor infrastructure.
  • UAVs unmanned aerial vehicles
  • RF radio frequency
  • UWB ultra- wide band
  • the systems incorporate inertial sensor measurements to localize nodes in a multi-hop network.
  • Example embodiments of the system include a UAV-UWB based system that can localize and track mobile responders to within predetermined (for example, within a range of about/ approximately 1 to 2 meters) accuracy from a single UAV operating outside of an indoor environment in real-time, such as deep indoor environments (for example, in excess of 40 to 50 meters indoors from a building’s periphery).
  • the system can deploy the UAVs on different sections of a structure (for example, building, edifice, etc.) for wider, simultaneous coverage.
  • the system incorporates localization and tracking elements, including trajectory tracking, a multi-hop localization paradigm, concurrent ranging protocol, and reverse location look-up.
  • Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements.
  • the present invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
  • Embodiments may include a computer program product accessible from a computer- usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system.
  • a computer-usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, apparatus, or device.
  • the medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium.
  • the medium may include a computer-readable storage medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk, etc.
  • Each computer program may be tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein.
  • the inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
  • a data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus.
  • the memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution.
  • I/O devices including but not limited to keyboards, displays, pointing devices, etc.
  • I/O controllers may be coupled to the localizing and tracking system 100 either directly or through intervening I/O controllers.
  • Network adapters may also be coupled to the localizing and tracking system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks.
  • Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
  • the localizing and tracking system 100 includes an unmanned aerial vehicle (UAV) 102 including a radio frequency localization master node device (RFL MN) 125, a (for example, moving) node 132 (for example, a responder) with a RFL client node device (CN) 130 (for example, a UWB helmet for responders) and a ground station 150.
  • UAV unmanned aerial vehicle
  • RFL MN radio frequency localization master node device
  • CN RFL client node device
  • the UAV 102 includes a propulsion system 106.
  • the propulsion system 106 can include propellers or other engines for flying the UAV 102.
  • the propulsion system 106 may include a jet engine or hover technology.
  • the propulsion system 106 may include one or more motors 108, which may include an internal combustion engine, electric motor, etc.
  • the UAV 102 may include a power source 120.
  • the power source 120 may include or employ one or more batteries, liquid fuel (e.g., gasoline, alcohol, diesel, etc.) or other energy source.
  • the power source 120 may include one or more solar cells or one or more fuel cells.
  • the power source 120 may include combustive gas (e.g., hydrogen).
  • the UAV 102 may be equipped with computing functions and controls.
  • the UAV 102 can include a processor 114.
  • the UAV 102 can include a transceiver 116.
  • the transceiver 116 may be coupled to a global position system (GPS) 118 to generate and alert of a position of the UAV 102 relative to other vehicles and a common coordinate system.
  • GPS global position system
  • the transceiver 116 may be equipped to communicate with a cellular network system. In this way, the UAV’s 102 position can be computed based on triangulation between cell towers base upon signal strength or the like.
  • the transceiver 116 may include a Wi-Fi or equivalent radio system.
  • the UAV 102 can include one or more sensors 112 for making different measurements.
  • the measurements may include pressure, temperature, wind speed, wind direction, speed, acceleration, etc.
  • the sensor 112 can include an optical sensor, an infrared (IR) sensor or a camera.
  • the sensor 112 can include an accelerometer or other motion sensor. It should be understood that the different systems of the UAV 102 may be employed in combination to take or make different measurements needed for the control of the UAV 102 or locating the UAV 102 from the ground using telemetry or other techniques.
  • the UAV 102 can include memory storage 110.
  • the memory storage 110 may include solid state or soft storage and work in conjunction with other systems on the UAV 102 to record data, run algorithms or programs, control the vehicle, etc.
  • the memory storage 110 may include a Read Only Memory (ROM), random access memory (RAM), or any other type of memory useful for the present applications.
  • UAV 102 can include RFL MN 125 (for example, a UWB master node) that collects range information from the client nodes (RFL CNs 130) inside the building.
  • a module in the UAV 102 can be programmed as the RFL MN 125 and is responsible for collecting the range measurements from the RFL CNs 130.
  • the RFL MN 125 can continuously range the RFL CNs 130 (for example, at about 6 Hz) and forward the information through a serial port.
  • the UAV 102 uses radio frequency (RF) communications technology such as ultra-wide band (UWB), which provides a balanced penetration and accuracy tradeoff wireless technology for localization of the RFL CNs 130.
  • RF radio frequency
  • UWB ultra-wide band
  • the UAV 102 can be controlled remotely. When controlled remotely, the UAV 102 can be controlled by a remote-control device over a wireless channel 140. This permits the user to dynamically position the UAV 102 where needed.
  • the UAV 102 can be positioned relative to an RFL CN device 130.
  • RFL CNs 130 can include a (for example, single-chip, multiple chip) wireless transceiver based on ultra-wideband techniques and a reduced instruction set computing (RISC) architectures based microcontroller that is programmed with a multi-hop ranging protocol.
  • a controller in the RFL CN 130 can include a system on a chip (SoC) with an integrated central processing unit (CPU) and an on-chip graphics processing unit (GPU) that reads from the port (for example, through a universal serial bus (USB)) and forwards the information (including range measurements) to the ground station 150 through a Wi-Fi interface.
  • SoC system on a chip
  • CPU central processing unit
  • GPU graphics processing unit
  • USB universal serial bus
  • the RFL CNs 130 can detect their corresponding hop memberships and either directly range with the RFL MN 125 (hopi) or other RFL CNs 130 (hopi nodes (hop2)). According to an embodiment, RFL CNs 130 can be mounted on hard hats worn by responders.
  • the ground station 150 can include a processor 152, a transceiver 154, a storage 156, remote control software (SW) / hardware (HW) 158, a tracking mechanism 160 and a positioning mechanism 162.
  • the transceiver 154 can communicate 145 with the UAV 102.
  • the transceiver 154 may communicate with a local area network (LAN), cellular system, Wi-Fi system, etc.
  • the positioning mechanism 162 may receive axial adjustment commands from the ground station 150 through the connection 145. The axial commands along with the vertical and horizontal movements allow the positioning mechanism 162 to track or follow the UAV 102 as it travels.
  • the positioning mechanism 162 may have a tracking mechanism 160 to provide axial commands, which permit the ground station 150 to follow an object vertically and/or horizontally.
  • the ground station 150 can provide multiple services.
  • the ground station 150 can continually locate the UAV 102 with the help of RF client nodes (for example, four UWB client nodes) that the ground station 150 is equipped with.
  • RF client nodes for example, four UWB client nodes
  • the inter-node distances are fixed and known in advance.
  • One or more of the nodes is equipped with a GPS receiver, which provides highly accurate stationary estimates over time.
  • the ground station 150 is also responsible for localizing the responder or client nodes (RFL CNs 130).
  • the RFL CNs 130 can be mounted on (or otherwise positioned on/with) an object (for example, personnel, a responder, etc.) that is to be tracked inside an indoor environment.
  • the RFL CNs 130 include UWB transmitters (and/or receivers) for communicating with the RFL MNs 125.
  • the tracking mechanism 142 may include a laser tracker.
  • the tracking mechanism 142 can allow a user to fly the UAV 102 and the positioning mechanism 162 can autonomously handle the tracking and gathering of positional information and optical data.
  • the positioning mechanism 162 may send telemetry data to the ground station 150.
  • the telemetry data may include the position of the UAV 102.
  • the telemetry data can include a three-axis location of the UAV 102 to show where the vehicle 102 was in relation to the positioning mechanism 162 and how high above the ground the UAV 102 traveled.
  • the telemetry data can be used in creating a map of the locations the UAV 102 is positioned.
  • the remote-control software and hardware 158 can include a wire or an antenna for a wireless connection to the UAV 102, and the software to modulate the output commands as a signal along the connection.
  • the remote-control software and hardware 158 may be employed to guide the UAV 102 by a user input using the ground station 150.
  • the remote-control software and hardware 158 may send preprogrammed commands to the UAV 102 to position the UAV 102 within a predetermined distance from an RFL CN device 130 (for example, attached to a helmet 132 of a responder).
  • the remote-control software and hardware 158 may send preprogrammed commands to maintain a predetermined distance from the location of other UAVs 102 in an aerial configuration.
  • FIG. 2 shows a scenario of indoor object tracking from a UAV using localizing and tracking system 100 in action along with its various architectural components, in accordance with example embodiments.
  • scenario 200 includes a building 202 with a UAV 102 deployed outside (outdoor 204).
  • UAV 102 for example, carrying a UWB MN 125
  • UWB MN 125 flying outside
  • the UAV 102 can fly (or hover, or otherwise position itself) outside the building 202 to create a synthetic aperture of anchor points, from where the UAV 102 can range to each of the indoor nodes (UWB CNs 130, shown as nodes 1 to 7) using UWB, thereby allowing for subsequent localization of the UWB CNs 130 through multi-lateration. As shown in FIG.
  • the nodes function as direct or one-hop nodes (UWB CNs 130-1) within an indoor 208 range of the UAV 102 or indirect of multi -hop nodes (UWB CNs 130-2) that can be reached beyond direct range 208 through transmissions from other nodes (for example, node 2, UWB CN 130-2 transmits to UAV 102 via node 1, UWB CN 130-1).
  • the UAV 102 flying outside the building’s 202 periphery is equipped with a UWB MN 125 that collects range information from the UWB CNs 130 inside the building 202.
  • the UWB CNs 130 are possibly worn by personnel (for example, firefighters, military troops, emergency responders etc.) who can be tracked through the localizing and tracking system 100.
  • UWB CNs 130 that are directly reachable from the RFL MN 125 (of UAV 102) are designated as hopi nodes (UWB CNs 130-1), additional nodes are referred to as hop2 nodes (UWB CNs 130-2), hop3 nodes (not shown in FIG. 2) and so on based on subsequent reachability.
  • a location of the UAV 102 can be computed by multilateration.
  • the UAV’s location along with the remaining range information can be input to location solver 220 that localizes the client nodes (RFL CNs 130).
  • the UAV 102 directly ranges the nodes in hopi (UWB CNs 130-1) which in turn ranges the nodes in hop2 and relays the information back to the UAV 102.
  • the UAV 102 can send the range information to a ground control station 150 that can (for example, substantially) simultaneously solve for the locations (using location solver 220) of all client nodes (UWB CNs 130).
  • the localizing and tracking system 100 can provide a visualization component 225 (for example, a mobile application) that can be used to visualize the tracking information (of the CNs 130) on a map (for example, with sub-second latency).
  • the map can include architectural details of a building in which a responder is located (for example, walls, doors, windows, etc.) including those determined by the UAV 102 andVor based on diagrams or maps received from secondary sources.
  • the localizing and tracking system 100 described herein can efficiently deploy UAVs 102 to cover (localize) responders in different sections of the building simultaneously, rather than to serve as stationary APs/BSs.
  • the localizing and tracking system 100 localizes and tracks indoor responders in real-time by a single UAV 102 using a key degree of freedom of the UAV (for example, mobility).
  • the localizing and tracking system 100 (for example, via ground station 150) can be used to perform multiple functions (and/or provide multiple capabilities) to facilitate tracking and localization of RFL CNs 130 including accounting (and/or adjusting, compensating, etc.) for the mobility of responders, providing indoor coverage, performing real-time tracking and/or providing absolute location fix.
  • the localizing and tracking system 100 can include a UAV-UWB based system that is capable of localizing and tracking mobile responders to within a predetermined accuracy (for example, within 1-2 m accuracy) from a single UAV 102 located outside of a building 202 in real-time, including in deep indoor environments. When multiple UAVs 102 are available, the localizing and tracking system 100 can deploy the multiple UAVs 102 on different sections of the building 202 for wider, simultaneous coverage.
  • the mobility of the UAV 102 can be used to create a synthetic aperture over time, which can serve to provide reference points for localizing an indoor nodel (for example, RFL CN 130) through multi-lateration.
  • multi-lateration approaches lave limited accuracy (for example, accuracy of a range of about/approximately 10 meters), when the indoor node (for example RFL CN 130) is also mobile.
  • the UAV’s 102 speed is increased compared to RFL CN’s 130 (for example, a responder’s) mobility (to alleviate the impact of responder mobility)
  • the localizing and tracking system 100 determines a lesser number of measurements with increased ranging errors. Instead of alleviating the impact of responder mobility, the system directly estimates the trajectory of the mobile responder (for example, RFL CN 130), rather than just a position of the RFL CN 130.
  • the localizing and tracking system 100 (for example, via ground station 150) provides trajectory tracking to account for the mobility of responders.
  • the localizing and tracking system 100 can analytically instrument a multi-alteration formulation that estimates the position of the RFL CN 130 and the velocity (and direction) of the RFL CN 130.
  • the localizing and tracking system 100 incorporates mechanisms for adaptively varying the size and choice of the synthetic aperture (anchor points used for localization) to address responders with non-uniform velocity (for example, those turning corners, etc.).
  • the localizing and tracking system 100 implements a multi-hop localization of the RFL CNs 130.
  • Localizing and tracking system 100 implements multi-hop localization for extended indoor coverage, where those RFL CNs 130 (for example, personnel, responders, etc.), directly reachable by the UAV 102 (hopi, 130-1), are localized first. Then, the hopi nodes can be used to serve as anchors/relays for localizing additional nodes (for example, responders (hop2, 130-2) that are reachable by the hopi nodes but not directly by the UAV 102. RFL CNs 130 can dynamically estimate their own hop status based on factors such as reachability to the UAV 102 and/or overheard ranging messages from neighboring RFL CNs 130.
  • Localizing and tracking system 100 can account (or compensate, alleviate, etc.) for the deterioration in accuracy over hops (due to iterative localization), by analytically selecting (for example, only) upstream responders with accurate location estimates (for example, within a predetermined minimum confidence) as anchors for the downstream RFL CNs 130 (for example, responders, persons, etc.). Localizing and tracking system 100 implements this selection process based on a localization mechanism that helps identify responder location estimates with high (static, or uniform velocity) or low confidence (non- uniform velocity, turning, etc.).
  • Localizing and tracking system 100 can thereby compensate for limitations in UWB’s penetration capabilities, which, while better than mmWave, are limited to tens of meters and hence cannot guarantee reachability to all nodes. Localizing and tracking system 100 can thereby ensure reachability to those that are deep inside the building without deploying multiple UAVs outside different sections of the building to alleviate coverage issues. Localizing and tracking system 100 can thereby cover the entire indoor environment (for example, building 202).
  • Localizing and tracking system 100 can implement a (for example, UWB) protocol for real-time tracking of RFL CNs 130.
  • the UWB protocol provides a framework of two-way ranging between two peer UWB nodes (UAV 102 and responder in our case).
  • Localizing and tracking system 100 can compensate for a lack (or limitations) of scalability of a time division multiple access (TDMA) operational structure for collecting enough ranging measurements to all RFL CNs 130 (for example, UWB nodes) from the UAV 102 in a real-time tracking in a large network.
  • TDMA time division multiple access
  • Localizing and tracking system 100 implements a concurrent ranging protocol to enable real-time tracking of a (for example, large, spread out, etc.) multi-hop network of responders (for example, dispersed throughout vertically and/or horizontally large buildings).
  • Localizing and tracking system 100 can transform UWB’s sequential ranging protocol into an efficient, concurrent protocol.
  • Localizing and tracking system 100 can leverage the broadcast nature of the wireless medium to parallelize the ranging measurements within each hop, and/or efficiently multiplex ranging measurements between hops by allowing RFL CNs 130 to access upstream messages, while also eliminating redundant message transmissions.
  • Localizing and tracking system 100 can thereby achieve an increase in speed (for example, in the range of three time (3X), resulting in a location update frequency that allows for real-time tracking (for example, frequency of 6 Hz or higher).
  • Localizing and tracking system 100 can determine an absolute location fix for the RFL CNs 130. Since the UAV 102 localizes the RFL CNs 130 with respect to its own position, to get the absolute location fix of the RFL CNs 130, localizing and tracking system 100 determines (for example, accurately estimates) the UAV’s 102 position. Localizing and tracking system 100 implements a reverse location look-up to (for example, accurately) estimate the location of the UAV 102 (for example, to within a meter) when the UAV 102 is mobile. According to an example embodiment, localizing and tracking system 100 estimates the location of the UAV 102 by itself, without using the UAV 102 as an anchor, by leveraging UWB.
  • localizing and tracking system 100 can use (for example, four) static RFL (for example, UWB) beacons 252, deployed, for example, on the roof corners of a responder service vehicle 250, as anchors.
  • RFL beacons 252 can also be fitted with a global positioning system (GPS) receiver (whose stationary estimates over time can be highly accurate).
  • GPS global positioning system
  • Localizing and tracking system 100 can use the GPS information coupled with known inter-beacon distances to determine accurate localization of the UAV 102.
  • Localizing and tracking system 100 can thus be implemented, in some example embodiments, with UAVs 102 with a single GPS receiver, thereby lowering the cost of implementation (for example, without multiple GPS receivers and/or inertial sensor fusion to provide position accuracy).
  • localizing and tracking system 100 can use lower-end UAVs 102 with a single GPS receiver that provides accuracy limited to around 2-3 meters owing to the mobility of the UAV 102 and the resulting limited measurements (at a location).
  • Localizing and tracking system 100 can implement reverse lookup for UAV location fix. Obtaining the UAV’s 102 precise GPS location is central to localizing and tracking system’s 100 end-end accuracy. In cases, where precise UAV location estimates are not possible, localizing and tracking system 100 leverages UWB to also localize the UAV 102.
  • localizing and tracking system 100 can include static UWB nodes as anchors at known locations on the ground. One of these anchors can be fitted with a GPS receiver. The latter’s GPS measurements along with the known, exact, pairwise distances between the anchors, enables localizing and tracking system 100 to accurately determine the GPS coordinates of these static anchors. These static anchors in turn allow for accurate localization of the UAV itself. In example embodiments these anchors can be permanently mounted at the four comers of a service vehicle (for example, at different heights to provide vertical diversity).
  • FIG. 3 shows a scenario of the localizing and tracking system 100 in action along with its various architectural components, in accordance with example embodiments.
  • multiple UAVs 102 can be deployed on different sections of the building 102 and/or at different altitudes of the same section (for example, for tall buildings) for wider, simultaneous coverage.
  • Each UAV 102 can execute localizing and tracking in parallel (with other UAVs 102).
  • the ensuing discussion with respect to a single UAV’s 102 operation can be applied to multiple UAVs 102 operating in parallel. Also, for easier exposition, the discussion is directed towards the UAV 102 localizing RFL CNs 130 (for example, on helmets 132 of responders) in a single floor (horizontal plane) by fixing its altitude appropriately.
  • the UAV 102 can scan floors and identify an appropriate altitude as described with respect to FIG. 4 herein below.
  • the ⁇ location; range > tuples measured by the UAV 102 do not uniquely map to a single location, resulting in poor localization accuracy of multilateration solvers.
  • localizing and tracking system 100 directly estimates the trajectory (speed and heading) of the mobile node, rather than just a position of the RFL CN 130. To accomplish this, localizing and tracking system 100 analytically instruments the multi- alteration formulation to estimate both the initial location (x;y) as well as the velocity vector v i t- U ./j
  • localizing and tracking system 100 can trace the node’s traversed path. Localizing and tracking system 100 can operate on an assumption that human mobility can be approximated with uniform velocity, which is reasonable within the short time-scales (for example, a few seconds) of the UAV’s 102 synthetic aperture. Localizing and tracking system 100 also accounts (or compensates, calculates, etc.) for non-uniform mobility (e.g., turning comers, etc.).
  • non-uniform mobility e.g., turning comers, etc.
  • localizing and tracking system 100 receives ranging measurements from n consecutive UAV 102 locations.
  • n is referred to as the aperture size and is a moving window of n historical measurements.
  • the UAV 102 records the mapping ⁇ (locationi; rangei >, where locationi is the UAV’s 102 three dimensional (3D) location and rangei is the distance estimate of the mobile node from the UAV.
  • the mobile node is located at an unknown location (x, ⁇ ; y,; z,).
  • Localizing and tracking system 100 can expand the UAV’s 102 3D-location to (Cx, ⁇ ; Cy,; Czi).
  • the measured range is given by: Eqn. (1)
  • Localizing and tracking system 100 can determine a minimization to find the best fitting location and velocity; effectively, minimizing the error function:
  • Localizing and tracking system 100 can obtain the various (xi; yi; zi) coordinates from the initial (xl; yl) and velocity (Vx; Vy) based on kinematic equations.
  • ⁇ T denotes the time between measurements
  • i [2...n ⁇ .
  • the responders can be equipped with inertial measurement units (IMUs) 320.
  • the IMUs 320 can provide information from compass 340, accelerometer, and gyroscope (each, for example, incorporated into the IMU 320, accelerometer and gyroscope not separately shown for clarity of illustration) that can be used to compute relative displacement using dead reckoning algorithms.
  • the IMUs 320 can provide inertial measurements that include user direction and user velocity to the system 100, which can be used, for example, in a manner as described with respect to FIGS. 10 and 11 herein below.
  • the responders can be equipped with radio frequency imaging units (RFIs) 330.
  • the RFIs 330 can implement RF imaging capabilities to enable localized nodes to image objects that are visually occluded, such as say objects on the other side of a wall.
  • an RFL MN 125 can include drone locations 405, hopi ranges 410 and hop2 ranges 415.
  • Location solver 220 includes a velocity -based solver 420 and an instantaneous solver 450.
  • Velocity based solver 420 includes a per node aperture 425, and a non-linear least squares (NL-LS) solver 430 that generates high confidence hop-1 locations 445 or low confidence hop- 1 nodes. If NL-LS solver 430 identifies low confidence nodes, a reset aperture command 440 is provided to the per node aperture 425.
  • Instantaneous solver 450 determines a location at an instant (for example, a few seconds) of turning of the nodes.
  • Instantaneous solver 450 includes NS-LS solver 455 that determines hop2 locations 460. The hop2 locations 460 are used to re- fix turning hopi nodes 465.
  • the static UWB anchors on the ground start localizing the UAV 102 to determine a (for example, precise) location estimate for the UAV 102.
  • the UAV 102 performs a flight trajectory and thereby creates a continuous, moving window of synthetic apertures (for example, of a predetermined duration, such as 4 secs each).
  • the UAV 102 performs the following:
  • the UAV 102 (via localizing and tracking system 100) executes a concurrent ranging protocol to (for example, help, determine, etc.) classify nodes (for example, RFL CNs 130) into various hops based on a reachability of each of the hops from the UAV 102.
  • the localizing and tracking system 100 then localizes the hopi nodes first using velocity-based solver 420 that estimates both the location and velocity of the hopi nodes.
  • the UAV 102 uses the error residuals of the velocity-based solver 420 (particularly, of NL-LS solver 430), the UAV 102 employs only the high-confidence (uniform velocity) hopi nodes 445 as anchors for localizing the hop2 nodes (via instantaneous solver 450).
  • localizing and tracking system 100 employs a multi- lateration solver (NL-LS solver 455) to obtain the location of hop2 nodes 460.
  • localizing and tracking system 100 uses a spatial aperture of hop2 nodes (anchors), along with an adaptive (temporal) aperture from the UAV 102, to refine the location estimate of the hopi nodes that have non-uniform velocity (or direction).
  • UAV 102 can also adjust for altitude considerations. As discussed herein above, the discussions of the determinations by the UAV 102 focused on the horizontal plane and assume that localizing and tracking system 100 is aware of the nodes’ altitude. However, in cases when localizing and tracking system 100 does not have information regarding the horizontal plane (for example, the floor) where the nodes (for example, service personnel) are currently located, the horizontal localization error (in instances that does not account for variances in the vertical coordinate) can be significant (since, in these instances, the localizing and tracking system 100 will not take into account the additional vertical offset the signals have to travel). Localizing and tracking system 100 can mitigate such situations by detecting the appropriate altitude (and hence floor) through a mitigating maneuver of the UAV 102.
  • the horizontal localization error in instances that does not account for variances in the vertical coordinate
  • Localizing and tracking system 100 can mitigate such situations by detecting the appropriate altitude (and hence floor) through a mitigating maneuver of the UAV 102.
  • Localizing and tracking system 100 can move the UAV 102 up and down through a (for example, short, limited) vertical distance that spans the target floors. During this movement, as the UAV 102 approaches the horizontal plane of the nodes 130, range estimates to the nodes 130 decreases, reaching a minimum when the UAV 102 is on the horizontal plane, and increasing when the UAV 102 moves away from the plane. Localizing and tracking system 100 can record the altitude as that corresponding to the minimum range estimates and determine the floor of interest. Localizing and tracking system 100 can then use this altitude to execute a localization process for the RFL CNs 130 on the target floor.
  • a (for example, short, limited) vertical distance that spans the target floors.
  • FIG. 5 illustrates a scenario 500 of resetting an aperture to limit a loss in localization accuracy, in accordance with example embodiments.
  • Localizing and tracking system 100 can analyze the improvement in localization accuracy achieved by incorporating velocity vectors over simple multilateration.
  • FIG. 5 shows the impact of sudden turns on the localization error (y-axis, 505) over time (x-axis, 510, measured in seconds) for a node without an adaptive aperture (straight line 530) as compared to use of an adaptive aperture (dotted line 540) (see key 515).
  • FIG. 5 illustrates measurements of a node moving in a straight line, then taking a 90° turn (at turning instant 520) and continuing again in a straight line.
  • a predetermined aperture for example, an aperture of 4 seconds
  • UAV 102 locations and the corresponding ranges of the past aperture for example, 4 seconds
  • the localization error without use of an adaptive aperture starts to increase from the point where the aperture’s head crosses over the turning position (first dashed vertical line 520) and falls back to its pre-turn values after the aperture’s tail has crossed over the turning position (second dashed vertical line 525).
  • Localizing and tracking system 100 can implement an adaptive aperture to address turns. If localizing and tracking system 100 detects turns, localizing and tracking system 100 can eliminate historic measurements that have occurred preceding the turn and restart building up the aperture. At the time of the turn, localizing and tracking system 100 removes all history and restarts estimating location after a short history has built up. Just after resetting history, the localization error can be high (dotted line 540 in FIG. 5) but quickly recovers and becomes acceptable once the aperture fills up with relevant measurements after the turn. In comparison, without an adaptive aperture, the effects of a turn last for the entire duration of the aperture (straight line 530).
  • Localizing and tracking system 100 can trigger an adaptive aperture based on a mechanism that detect turns.
  • the core reason for the worsening of localization error during turns is that the velocity-based solver 420 is unable to provide a sufficient answer because no single velocity estimate can represent all the measurements. This inherently results in larger residual errors after solving Eqn. 2.
  • the residuals are highly correlated with localization error. Thus, a sudden increase in the residuals helps identify non-uniform velocity situations such as turns.
  • Localizing and tracking system 100 can implement Gaussian mixture models on the residuals to identify low solver confidence. Localizing and tracking system 100 can use any unsupervised clustering mechanism that can differentiate the higher solver errors from the lower solver errors.
  • localizing and tracking system 100 can localize mobile nodes, including those with non-uniform velocity, through a combination of joint location- velocity solving and by adaptively resetting the aperture size.
  • different nodes (RFL CNs 130) in the topology can utilize different aperture sizes, and the n thus becomes a node specific value.
  • FIG. 6 illustrates a scenario 600 of multi-hop tracking for extended coverage, in accordance with example embodiments.
  • Localizing and tracking system 100 can implement multi-hop tracking for extended coverage. Localizing and tracking system 100 includes a framework that is adaptable to situations in which some nodes (RFL CNs 130-2 and further nodes) are not directly reachable from the UAV 102.
  • Localizing and tracking system 100 can allow (or facilitate, etc.) such unreachable nodes (RFL CNs 130-2) to range with other nodes in the vicinity (for example, RFL CNs 130-1) which can in-turn reach the UAV 102 and/or have already been localized.
  • RFL CNs 130-1 unreachable nodes
  • a multi-hop topology is dynamically created with nodes belonging to different hops based on their reachability characteristics (from the UAV 102).
  • the UAV’s 102 synthetic aperture localizes first hop (directly reachable) nodes (RFL CNs 130-1, such as 1, 7, 5 in FIG. 6) using the location solver 220.
  • Location solver 220 can iterate this process across hops (for example, to a hop3 or hop4 node, etc.).
  • Localizing and tracking system 100 can implement one or more of several mechanisms to ensure that mobile nodes (RFL CNs 130) can be accurately localized (including across multiple hops).
  • RFL CNs 130 can perform dynamic estimation of hop membership.
  • Localizing and tracking system 100 can intelligently select anchors for iterative localization. Two components contribute to the localization error of hop m nodes: (a) relative localization error of hop m nodes with respect to hopm-1 nodes, and (b) localization error of hop m-i nodes. Without loss of generality, hop m-i nodes can be assumed to be spaced far apart compared to the synthetic aperture formed by the UAV 102. This increased spacing between anchors allows for better localization of hop m nodes with respect to hop m-i nodes, compared to that of hopi nodes (with respect to the UAV 102). However, the localization error of the hop m- 1 nodes and recursively that of upstream hop nodes, will cumulatively contribute to the error of hopm nodes.
  • hop m-i has a cascading impact on the localization accuracy of downstream hop nodes (hops > m). Nodes that experience uniform velocity in hop m-i inherently have better localization accuracy compared to those with non-uniform velocity.
  • localizing and tracking system 100 leverages the location solver’s 220 ability to identify nodes with low confidence (in its location estimate) by avoiding nodes with non- uniform velocity as anchors for localizing hop m nodes. Localizing and tracking system 100 can thereby curtail the cascading of localization error across hops.
  • Localizing and tracking system 100 can decouple nodes from the UAV 102 for better mobility tracking across hops.
  • hop m (m > 1) nodes can be localized through a spatial aperture formed from a diverse placement of hop m-i nodes. This decoupling (from UAV’s 102 mobility), allows for instantaneous localization of hop m nodes from previously obtained hop m-i locations. The time scale of such localization can be milliseconds within which the nodes move a negligible distance.
  • hop m (m > 1) nodes can use conventional multilateration approaches (without need for velocity vectors) including in instances when they are mobile.
  • Localizing and tracking system 100 can address hopi nodes with non-uniform velocity using downstream spatial apertures. Unlike hopi nodes, mobility is not of concern for hop m (m > 1) nodes as these nodes can be instantaneously (for example, within a second) localized using a spatial aperture formed from high-confidence hop m-i nodes. Hence, hop2 nodes can in turn, form a spatial aperture (for example, serve as anchors) that can be used to correct (or determine) the location of hopi nodes, which are currently experiencing non- uniform velocity (low confidence). Localizing and tracking system 100 can eliminate most of (for example, reduce significantly) the impact of nonuniform velocity of hopi nodes.
  • localizing and tracking system 100 can implement the downstream spatial aperture technique in an opportunistic manner based on whether enough hop2 nodes exist.
  • the mechanism of adaptive (temporal) aperture of the UAV 102 provides benefits even when no other hop2 nodes exist in the topology.
  • localizing and tracking system 100 incorporates both these techniques to address non uni form velocity of hopi nodes.
  • Localizing and tracking system 100 can leverage multiple UAVs 102 vs. multiple hops. While multi-hop localization allows for coverage of even large buildings using a single UAV 102, the localization error of its downstream hop nodes (e.g., m > 2) will increase and in some instances may not satisfy a target confidence level (for example, a target of l-2m). Hence, localizing and tracking system 100 can leverage the multi-hop paradigm to primarily reach deep interiors of buildings (where even multiple UAVs 102 cannot help), and employs multiple UAVs 102 to provide non-overlapping, peripheral coverage for large buildings. Thus, using a combination of multiple UAVs 102 and multi-hops, localizing and tracking system 100 covers (for example, even large) buildings with two hops.
  • localizing and tracking system 100 can implement concurrent ranging for real-time tracking localizing and tracking system 100 uses a fast and reliable ranging protocol to implement a real-time localization system. Since the UWB’s (802.15.4) ranging protocol is designed for ranging between a pair of nodes, UWB’s (802.15.4) ranging protocol does not broadcast messages. This leads to a sequential ranging of every pair of nodes within a hop, which is not scalable for real-time operation, especially in a multi-hop network. Localizing and tracking system 100 leverages the broadcast nature of wireless signals to communicate and hence concurrently ranges with multiple nodes using a single transmission. Localizing and tracking system 100 can implement a protocol, such as shown in FIG. 6 that includes a concurrent ranging scheme. Although FIG. 6 and the discussion is directed with respect to the first two hops, the procedure is similar for other hops.
  • localizing and tracking system 100 performs concurrent ranging within hopi.
  • the UAV 102 simultaneously initiates ranging with all reachable hopi nodes by broadcasting a POLL 605.
  • Each node that receives this message takes turns (for example, based on the node’s hard-coded node identifier (NodelD)) to send a response (RESP) 610 message.
  • the UAV 102 After collecting the timings from all the RESPs 610, the UAV 102 sends a single FINAL message 615 containing information for all hopi nodes.
  • all hopi nodes have enough information to calculate their distance from the UAV 102.
  • the nodes again take turns to send their distance estimates to the UAV 102 (DIST EST message 620).
  • Hop2 nodes keep listening to the channel in a similar manner to hopi nodes. However, since hop2 are not in direct communication range of the UAV 102, the hop2 nodes cannot receive the POLL message 605. Instead, hop2 nodes only receive the messages sent by nearby hopi nodes in response to the UAV’s 102 POLL. After all hopi nodes have completed sending their DIST EST messages, hop2 nodes take turn in initiating a full sequence of POLL-RESPs-FINAL (625- 630-635) simultaneously with all hopi nodes in the vicinity. The nodes follow the same protocol as the UAV 102 with one key difference to reduce overhead in some example embodiments.
  • Hopi nodes do not communicate distance estimates back to hop2 nodes (no DIST EST is sent).
  • the hopi node calculates and locally stores the hopi to hop2 distance for every hop2 node that initiated ranging. These distances are included as additional data (for example, piggybacked) on the next DIST EST message sent to the UAV 102 (that is being sent independently of the timing of collection of distance information). This saves unnecessary network overhead, speeding up the collection of range estimates.
  • FIG.7 illustrates a bitmap 700 constructed by the UAV and sent in a FINAL message, in accordance with example embodiments.
  • Localizing and tracking system 100 can implement efficient multiplexing of ranging between hops.
  • the UAV 102 initiates a POLL message
  • the UAV 102 waits for all the nodes in the network to send a RESP.
  • the UAV 102 creates a bitmap 700 indicating which nodes are deemed to be in hopi (setting the bit to one) based on who responded.
  • the UAV 102 sends this bitmap in its FINAL message.
  • An example bitmap is shown in FIG.
  • node IDs 710 for example, 7 to 1 corresponding to different nodes, and a corresponding bit 720 (0 or 1) are included. If a particular node had responded to the UAV’s 102 POLL 605, but finds its own bit cleared in the received bitmap (for example node 6, 130-2), the node marks itself as being a hop2 node and does not send the DIST EST. All nodes, which were deemed to be in hopi by the UAV 102 (their corresponding bit was set in the bitmap, for example nodes 1, 5 and 7 130-1), calculate their distance from the UAV 102 and send a DIST EST message that includes a copy of the same bitmap.
  • the DIST EST sent by hopi nodes piggybacks their distance from all hop2 nodes obtained in the previous round, along with their own range estimates to the UAV 102.
  • the DIST EST messages are variable-sized to accommodate any number of distance estimates without incurring extra overhead.
  • the UAV 102 creates an adjacency matrix from the information contained in all the DIST EST messages it receives. The UAV 102 forwards this matrix to a ground control center (such as ground station 150) for further processing.
  • an exemplary control system 800 is shown which may represent the ground station 150 (FIG. 1).
  • the control system 800 includes at least one processor (CPU) 114 operatively coupled to other components via a system bus 802.
  • a first storage device 822 and a second storage device 824 are operatively coupled to system bus 802 by the I/O adapter 820.
  • the storage devices 822 and 824 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, and so forth.
  • the storage devices 822 and 824 can be the same type of storage device or different types of storage devices.
  • the first storage device 822 or the second storage device 824 may include remote control software 823 to control the UAV.
  • the remote control software 823 may be able to access the hardware necessary to get user inputs and output a control command to the remote control vehicle, like a joystick or an antenna.
  • the remote control software 823 may be able to take a user input from an input device, like a joystick, and output a command to control the movement of the remote control vehicle in the direction intended by the movement of the joystick by the user.
  • the first storage device 822 or the second storage device 824 may include map generating programs 825.
  • the map generating programs 825 may be able to take GPS, telemetry, or path information to plot in a three-dimensional axis to form a map.
  • the map generating programs 825 may take a produced map an overlay it on existing satellite or topographic maps to show the locations traveled gathering the data in relation to the physical world.
  • the first storage device 822 or the second storage device 824 may include localizing and tracking programs.
  • the localizing and tracking programs may localize and track mobile objects (for example, persons, responders, etc.) in real-time in (for example, large) indoor environments (such as, for example, a building, a structure, etc.) by leveraging a mobile virtual infrastructure provided by unmanned aerial vehicles (UAVs), with wireless technology for localization.
  • UAVs unmanned aerial vehicles
  • a speaker 832 may be operatively coupled to system bus 802 by the sound adapter 830.
  • the speaker 832 may be used to give the user an auditory warning of low fuel or power.
  • a transceiver 116 is operatively coupled to system bus 802 by network adapter 840.
  • a display device 862 is operatively coupled to system bus 802 by display adapter 860.
  • a first user input device 852, a second user input device 854, and a third user input device 856 are operatively coupled to system bus 802 by user interface adapter 850.
  • the user input devices 852, 854, and 856 can be any of a keyboard, a mouse, a keypad, a joystick, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present principles.
  • the user input devices 852, 854, and 856 can be the same type of user input device or different types of user input devices.
  • the user input devices 852, 854, and 856 are used to input and output information to and from system 800.
  • control system 800 may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements.
  • various other input devices and/or output devices can be included in control system 800, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art.
  • various types of wireless and/or wired input and/or output devices can be used.
  • additional processors, controllers, memories, and so forth, in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art.
  • FIG. 9 is a flow diagram illustrating a method 900 of localizing and tracking objects, in accordance with the present invention.
  • localizing and tracking system 100 performs trajectory tracking of an object (for example, a node RFL CN 130 corresponding to personnel such as a first responder) using the UAV 102.
  • Localizing and tracking system 100 analytically instruments a multi- lateration formulation to estimate the position and the velocity of the RFL CN 130.
  • localizing and tracking system 100 also adaptively varies a size and choice of a synthetic aperture to address RFL CNs 130 with non-uniform velocity.
  • localizing and tracking system 100 determines an initial location of an RFL CN 130 based on multi-lateration.
  • localizing and tracking system 100 determines a velocity vector associated with the RFL CN 130 based on multi-lateration, and detects a turning of the RFL CN 130 (at block 925). Localizing and tracking system 100 can determine the initial location initial location based on ultra- wideband signaling along with input from inertial sensors. Localizing and tracking system 100 also applies an adaptive aperture to determine a location of the RFL CN 130 based on the turning of the RFL CN 130 and the velocity vector (at block 930).
  • localizing and tracking system 100 implements a multi-hop localization process in which RFL CNs 130 directly reachable by the UAV 102 are localized first.
  • RFL CNs 130 dynamically estimate their hop status based on reachability from the UAV as well as overheard ranging messages from neighboring RFL CNs 130.
  • localizing and tracking system 100 parallelizes the ranging measurements within each hop, and efficiently multiplexes ranging measurements between hops by allowing responders to access upstream messages, while also eliminating redundant message transmission.
  • localizing and tracking system 100 performs reverse location lookup to estimate a location of the UAV 102.
  • FIG. 10 a localizing and tracking system 1000 with inertial measurements is illustratively shown.
  • localizing and tracking system 1000 can incorporate and enhance the operations and components described with respect to localizing and tracking system 100 described herein above.
  • the localizing and tracking system 1000 can receive and process measurements from inertial measurement units (IMUs) 320 deployed on users.
  • the measurements can include IMU user direction 1005 and IMU user velocity 1010.
  • the localizing and tracking system 1000 can incorporate these measurements with a synthetic aperture 1020 for localization 1030 (for example, implemented by location solver 220).
  • the temporal aperture formed by the UAV 102 during its flight comprises of location-stamped range information that is used to localize indoor RFL CNs 130 (for example, UWB nodes), specifically the ones in the first hop.
  • the second hop nodes are localized using multiple first hop nodes (at least three) where the latter form a spatial aperture.
  • inertial sensors such as IMUs 320, can be used.
  • the inertial measurement units (IMUs) 320 provide information from accelerometer, compass and gyroscope that can be used to compute relative displacement using dead reckoning algorithms.
  • IMUs inertial measurement units
  • a single RFL CN 130 for example, UWB node
  • inertial sensors localizing and tracking system 1000 can estimate its trajectory which in turn can be used as a synthetic aperture to localize other UWB nodes.
  • the RFL CNs 130 for example, UWB nodes
  • the RFL CNs 130 will be localized relative to other nodes.
  • an indoor localization system 1100 is illustratively shown.
  • Indoor localization system 1100 (similarly as described with respect to systems 100, and 1000) enables indoor localization without any reliance on pre-deployed indoor infrastructure. Indoor localization system 1100 can enable applications geared towards but not limited to first responders. Indoor localization system 1100 can also enable applications for academic, recreational and public safety implementation.
  • indoor localization system 1100 can support geo-tagged sensory data streams 1110.
  • Indoor localization system 1100 can transmit geo-tagged data streams in real-time from an indoor location.
  • the streams can comprise of photos, videos or other sensory data relevant to a given location.
  • the streams can also include data relevant to the localized node, e.g., biometric data from a first responder or thermal data from a robot, etc.
  • Indoor localization system 1100 can implement, based on localization, delivery of geo-tagged data including geo-tagged imagery and geo-tagged sensory data.
  • indoor localization system 1100 can support through-the-wall imaging 1120.
  • Localized nodes can be equipped with RF imaging capabilities (for example, RFI 330 in FIG. 3) that the nodes can use to image objects that are visually occluded.
  • the synthetic aperture formed by the mobility of the localized node emulates a larger antenna array acting in a similar manner as a radar’s aperture. Identification of humans in this way can be used to provide information to help first responders to plan, prioritize and execute their search and rescue operations.
  • FIG. 12 is a flow diagram illustrating a method 1200 of localizing and tracking objects incorporating inertial sensor measurements, in accordance with the present invention.
  • localizing and tracking system 1000 determines an initial location of nodes in a multi-hop network (such as described with respect to FIG. 6) based on multi- lateration from a UAV 102.
  • localizing and tracking system 1000 applies an adaptive aperture to address a non-uniform velocity of a node in the multi-hop network based on the turning of the node and a velocity vector.
  • localizing and tracking system 1000 determines whether localization can be implemented using first hop nodes. For example, localizing and tracking system 1000 can determines whether a first hop node is not reachable from the UAV 102 or whether there are not enough first hop nodes to localize.
  • localizing and tracking system 100 uses inertial sensors from the second hop nodes to localize the RFL CN 130.
  • the IMU 320 includes a compass, used to indicate the direction of motion which in turn helps to resolve the velocity vector more accurately. While the personnel is in motion, the compass reported direction correlates highly to the direction of the personnel’s motion. Hence the direction of the velocity vector can be directly inferred from the compass. Note, that the estimation of the velocity vector involves both, estimation of the magnitude of the velocity (speed) as well as the direction. The inferred estimate from the compass can be directly plugged in to the solver to make the estimate much better that compared to without a compass.
  • localizing and tracking system 100 in response to a determination that localization can be implemented using the first hop nodes (1220 -YES), performs localization based on the first hop nodes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Position Fixing By Use Of Radio Waves (AREA)

Abstract

L'invention concerne des systèmes et des procédés pour localiser et suivre des objets mobiles. Un procédé consiste à déterminer (1210) un emplacement initial d'un nœud dans un réseau à sauts multiples en fonction d'une multilatération à partir d'un véhicule aérien sans pilote. Le procédé consiste aussi à appliquer (1220) une ouverture adaptative pour traiter une vitesse non uniforme du nœud en fonction de la rotation et d'un vecteur de vitesse. Une détermination (1230) pour savoir si une localisation pour le nœud peut être mise en œuvre en utilisant des nœuds de premier saut dans le réseau à sauts multiples est effectuée. En réponse à une détermination que la localisation ne peut pas être mise en œuvre en utilisant les nœuds de premier saut, l'accès à des mesures de capteur inertiel associées au nœud est effectué (1240) et les mesures de capteur inertiel sont intégrées (1240) avec l'ouverture adaptative pour améliorer la précision de localisation.
PCT/US2020/029549 2019-04-25 2020-04-23 Suivi d'objets à l'intérieur avec des mesures de capteur inertiel WO2020219692A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201962838358P 2019-04-25 2019-04-25
US62/838,358 2019-04-25
US16/568,747 US11089432B2 (en) 2018-09-20 2019-09-12 Tracking indoor objects from an unmanned aerial vehicle
US16/568,747 2019-09-12
US16/855,179 2020-04-22
US16/855,179 US11169518B2 (en) 2018-09-20 2020-04-22 Tracking indoor objects with inertial sensor measurements

Publications (1)

Publication Number Publication Date
WO2020219692A1 true WO2020219692A1 (fr) 2020-10-29

Family

ID=72941294

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/029549 WO2020219692A1 (fr) 2019-04-25 2020-04-23 Suivi d'objets à l'intérieur avec des mesures de capteur inertiel

Country Status (1)

Country Link
WO (1) WO2020219692A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114608580A (zh) * 2022-03-10 2022-06-10 吉林大学 一种室内移动目标定位方法及系统
WO2022131775A1 (fr) 2020-12-18 2022-06-23 Samsung Electronics Co., Ltd. Procédé et dispositif électronique d'extension de couverture pour localisation de dispositifs par télémétrie collaborative
WO2023050385A1 (fr) * 2021-09-30 2023-04-06 深圳市大疆创新科技有限公司 Procédé et appareil de commande de véhicule aérien sans pilote, véhicule aérien sans pilote et support de stockage lisible par ordinateur
US12137388B2 (en) 2021-11-04 2024-11-05 Samsung Electronics Co., Ltd. Coverage extension for device localization through collaborative ranging

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016096014A1 (fr) * 2014-12-18 2016-06-23 Siemens Aktiengesellschaft Système de drone distribué et drone
KR101645722B1 (ko) * 2015-08-19 2016-08-05 아이디어주식회사 자동추적 기능을 갖는 무인항공기 및 그 제어방법
US20170088261A1 (en) * 2015-09-29 2017-03-30 Tyco Fire & Security Gmbh Search and Rescue UAV System and Method
US20180158197A1 (en) * 2016-12-01 2018-06-07 Skydio, Inc. Object tracking by an unmanned aerial vehicle using visual sensors

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016096014A1 (fr) * 2014-12-18 2016-06-23 Siemens Aktiengesellschaft Système de drone distribué et drone
KR101645722B1 (ko) * 2015-08-19 2016-08-05 아이디어주식회사 자동추적 기능을 갖는 무인항공기 및 그 제어방법
US20170088261A1 (en) * 2015-09-29 2017-03-30 Tyco Fire & Security Gmbh Search and Rescue UAV System and Method
US20180158197A1 (en) * 2016-12-01 2018-06-07 Skydio, Inc. Object tracking by an unmanned aerial vehicle using visual sensors

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ASHUTOSH DHEKNE† ∗ AYON CHAKRABORTY? KARTHIKEYAN SUNDARESAN? †UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN, ?NEC LABS AMERICA, INC. : "TrackIO: Tracking First Responders Inside-Out", USENIX, USENIX, THE ADVANCED COMPUTING SYSTEMS ASSOCIATION, 26 February 2019 (2019-02-26), Usenix, the Advanced Computing Systems Association, pages 758 - 771, XP061031763 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022131775A1 (fr) 2020-12-18 2022-06-23 Samsung Electronics Co., Ltd. Procédé et dispositif électronique d'extension de couverture pour localisation de dispositifs par télémétrie collaborative
EP4241105A4 (fr) * 2020-12-18 2023-09-27 Samsung Electronics Co., Ltd. Procédé et dispositif électronique d'extension de couverture pour localisation de dispositifs par télémétrie collaborative
WO2023050385A1 (fr) * 2021-09-30 2023-04-06 深圳市大疆创新科技有限公司 Procédé et appareil de commande de véhicule aérien sans pilote, véhicule aérien sans pilote et support de stockage lisible par ordinateur
US12137388B2 (en) 2021-11-04 2024-11-05 Samsung Electronics Co., Ltd. Coverage extension for device localization through collaborative ranging
CN114608580A (zh) * 2022-03-10 2022-06-10 吉林大学 一种室内移动目标定位方法及系统

Similar Documents

Publication Publication Date Title
US11169518B2 (en) Tracking indoor objects with inertial sensor measurements
US11378985B2 (en) Distributed localization systems and methods and self-localizing apparatus
CN108521670B (zh) 面向多机紧密编队飞行基于uwb通信与定位方法及一体化系统
Dhekne et al. {TrackIO}: Tracking First Responders {Inside-Out}
WO2020219692A1 (fr) Suivi d'objets à l'intérieur avec des mesures de capteur inertiel
CN107003378B (zh) 便携式电子装置和确定便携式电子装置的地理位置的方法
JP5767731B1 (ja) 空撮映像配信システムおよび空撮映像配信方法
CN110199566A (zh) 无人机用户设备指示
CN102859901B (zh) 地理位置调节点波束交叠
Liu et al. HAWK: An unmanned mini-helicopter-based aerial wireless kit for localization
US20210116941A1 (en) Positioning method using unmanned aerial robot and device for supporting same in unmanned aerial system
US20200413267A1 (en) Ue modem for drones with flight path and 3d wireless environment signal quality information
US11002824B2 (en) Ultra-wideband assisted precise positioning system and method
JP6533983B2 (ja) 追尾アンテナシステムおよび追尾アンテナ装置
JP2018517139A (ja) 位置特定信号の伝送をスケジュールし、自己位置特定装置を動作する方法およびシステム
WO2015114572A1 (fr) Système mis en œuvre par ordinateur et procédé servant à fournir des liaisons de communication robustes à des véhicules aériens sans pilote
CN111034286B (zh) 用于报告定位数据的方法
EP2854463A1 (fr) Appareil de recherche de position pour une station mobile, véhicule, procédé et programme informatique associé
Tian et al. HiQuadLoc: A RSS fingerprinting based indoor localization system for quadrotors
JP2017183875A (ja) 追尾アンテナシステムおよび追尾アンテナ装置
Teng et al. Holes-in-the-Sky: A field study on cellular-connected UAS
US11089432B2 (en) Tracking indoor objects from an unmanned aerial vehicle
Petitjean et al. Fast localization of ground-based mobile terminals with a transceiver-equipped UAV
Mendes et al. Small UAVs-supported autonomous generation of fine-grained 3D indoor radio environmental maps
GB2593121A (en) Deployable location system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20794704

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20794704

Country of ref document: EP

Kind code of ref document: A1