WO2023182989A1 - Système de détection multimodale et de vérification d'adhésion - Google Patents

Système de détection multimodale et de vérification d'adhésion Download PDF

Info

Publication number
WO2023182989A1
WO2023182989A1 PCT/US2022/021559 US2022021559W WO2023182989A1 WO 2023182989 A1 WO2023182989 A1 WO 2023182989A1 US 2022021559 W US2022021559 W US 2022021559W WO 2023182989 A1 WO2023182989 A1 WO 2023182989A1
Authority
WO
WIPO (PCT)
Prior art keywords
hash
sensor data
query
index
modal
Prior art date
Application number
PCT/US2022/021559
Other languages
English (en)
Inventor
Rahul Vishwakarma
Ravigopal Vennelakanti
Prasun Singh
Ramyar Saeedi
Malarvizhi SANKARANARAYANASAMY
Shouchun Peng
Masato Tamura
Original Assignee
Hitachi America, Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi America, Ltd. filed Critical Hitachi America, Ltd.
Priority to PCT/US2022/021559 priority Critical patent/WO2023182989A1/fr
Publication of WO2023182989A1 publication Critical patent/WO2023182989A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2255—Hash tables
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F17/00—Coin-freed apparatus for hiring articles; Coin-freed facilities or services
    • G07F17/24—Coin-freed apparatus for hiring articles; Coin-freed facilities or services for parking meters

Definitions

  • the present disclosure is generally directed to multi-modal sensing data system.
  • curbside space in urban areas has allowed for digital transformation of curbside from a parking spot into an extensively monitored dynamic economic hub.
  • a large city’s curb space can run to several miles and may be used for parking, commercial loading, last-mile docking, mobility, drive-through, and/or retail services. This transformation has propelled the use of a variety of sensors to parse information about congestion, occupancy, usage, and above all safety and security of this public zone.
  • the number of cameras and/or sensors for observing a large area is large and may include multiple imaging and/or sensing modalities (e.g., Night vision, light detection and ranging (LiDAR), and radio detection and ranging (RADAR)), observing multiple locations.
  • This sensor fusion or multi-modal system adds robustness to existing field-of-view (FoV) coverage at the cost of high bandwidth.
  • Such data streams pose a challenge in terms of analytics, variety, volume, and velocity considering city-scale sensing infrastructure.
  • FIG. 1 illustrates an example architecture of a current data analysis system.
  • a set of sensors in a sensor network 110 may transfer images 115 and/or their features over a network to a feature database server 120 on a private network that maintains the images 115 and/or their features for analysis and/or searching.
  • a feature database server 120 is typically queried, e.g., by a client computing device 130, based on a set of image data and/or query features 125 transmitted over an intervening network for similarity and/or retrieval using a nearest-neighbor- and/or neural-network-based matching algorithm that may be storage- and compute-intensive.
  • an image-based query, retrieval, or matching task transfers the image data (e.g., the set of image data and/or query features 125) over the network to image databases (e.g., feature database server 120).
  • This retrieval query can be done by nearest neighbor or a deep neural network to compute similarity of image features. This causes large loads on the network by transferring large numbers of image files and on the compute infrastructure that processes each received image and many stored images to identify similar images for even a low fidelity membership or count queries.
  • Example implementations described herein include an innovative method for multimodal perceptual hash-based indexing of data for fast membership verification queries.
  • the method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the method may further include storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • Example implementations described herein include an innovative computer-readable medium storing computer executable code for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • the computer executable code may include instructions for receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the computer executable code may also include instructions for computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the computer executable code may further include instructions for storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • Example implementations described herein include an innovative apparatus for multimodal perceptual hash-based indexing of data for fast membership verification queries.
  • the apparatus may include a memory and at least one processor configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the at least one processor may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the at least one processor may further be configured to store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • Example implementations described herein include an innovative method for an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries.
  • the method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the method may further include transmitting, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • Example implementations described herein include an innovative computer-readable medium storing computer executable code for an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries.
  • the computer executable code may include instructions for receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the computer executable code may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the computer executable code may further include transmitting, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • Example implementations described herein include an innovative apparatus that is an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries.
  • the apparatus may include a memory and at least one processor configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the at least one processor may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the at least one processor may further be configured to transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • Example implementations described herein include an innovative method for one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries.
  • the method may include receiving, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the method may also include storing the received plurality of hash indexes in a probabilistic data structure.
  • the method may further include receiving, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index.
  • the method may additionally include verifying an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • Example implementations described herein include an innovative computer-readable medium storing computer executable code for one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries.
  • the computer executable code may include instructions for receiving, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the computer executable code may also include instructions for storing the received plurality of hash indexes in a probabilistic data structure.
  • the computer executable code may further include instructions for receiving, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index.
  • the computer executable code may additionally include instructions for verifying an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • Example implementations described herein include an innovative apparatus that is one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries.
  • the apparatus may include a memory and at least one processor configured to receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the at least one processor may also be configured to store the received plurality of hash indexes in a probabilistic data structure.
  • the at least one processor may further be configured to receive, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index.
  • the at least one processor may additionally be configured to verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • FIG. 1 illustrates an example architecture of a current data analysis system.
  • FIG. 2 is a diagram illustrating a high-level view of a system for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 3 is a diagram illustrating elements of a system for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • FIG. 4 is a diagram illustrating a set of components in an example multi-modal deep hash composer, in accordance with some aspects of the disclosure.
  • FIG. 5 is a diagram illustrating an example process for processing data captured in accordance with some aspects disclosed herein.
  • FIG. 6 is a diagram illustrating a set of calculated indexes being transmitted to, and received by, a distributed multi-modal hash registry.
  • FIG. 7 is a diagram illustrating a process for verifying occupancy and/or a membership in the distributed hash index registry.
  • FIG. 8 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 9 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 10 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 11 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 12 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 13 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 14 illustrates an example computing environment with an example computer device suitable for use in some example implementations.
  • Example implementations described herein relate to an innovative method for multimodal perceptual hash-based indexing of data for fast membership verification queries.
  • the method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the method may further include storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • the presented invention exchanges the hashed index rather than the dense data frames from sensor(s) data stream and persists it to (e.g., stores it in) a distributed optimized data structures on an index registry.
  • This index registry is made available for membership query based on a shared set of hash functions.
  • the discussion below relates to data management for information verification in a cityscale scenario (e.g., for a city like New York or London with approximately 1.5 Million closed circuit television (CCTV) sensors).
  • the discussion specifically relates to (1) sensor fusion for multiple modalities of sensors deployed at a city-scale (e.g., fusing data received from a sensor network including two or more types of sensors such as CCTV, IR/Night Camera, RADAR, and LiDAR), (2) imaging data types captured by sensors in the sensor network such as an RGB image, a point cloud, or error codes and reports, (3) distributed infrastructure including, e.g., an edge device, a gateway system, cloud nodes, a private network, and TCP/HTTP communication systems.
  • a multi-modal sensor network distributed indexing and data membership verification for curbside monitoring system e.g., for a city like New York or London with approximately 1.5 Million closed circuit television (CCTV) sensors.
  • sensor fusion for multiple modalities of sensors deployed at a city-scale (e.g.,
  • more than one billion images may be associated with a city-scale membership or association query.
  • a membership verification algorithm with quasi-linear complexity, e.g., O(nlogn), or sub-linear complexity, e.g., O(logn)is desired.
  • a membership verification algorithm may be used to check if particular information (e.g., parking occupancy, bike dock occupancy, curbside loading zone usage) exists, is valid, or to retrieve statistics such as a count associated with the particular information from one or more distributed index and probabilistic data structures.
  • the presented query can be based on a visual data type or an associated data type like text to determine membership and/or association in one or more of a uni-modal and/or cross-modal sensor fusion space.
  • low fidelity perceptual hashes are used to index information in a large distributed data system.
  • Using the low-fidelity perceptual hashes involves propagating only the hashed index from multi-modal sensor(s) data stream and persisting it to one or more search-optimized data structures on a distributed registry and metadata store.
  • an index may contain low fidelity information across data types and is referred to as cross-modal, multi-modal, or sensor-fusion indexes.
  • the data system described herein benefits from two core technologies, (a) sensor fusion or multi-modal distributed sensor(s) data indexing and registry that relies on a family of perceptual deep hash composer functions that produce similar results if features in the source material are similar and (b) fast multi-modal distributed verification for membership or an association test across modalities & distributed data index on the deployment network.
  • a trade-off is made between dense image feature transfer over the network and low-latency m-bit binary coded perceptual hashes to enable fast membership checks over a large distributed city-scale sensor network.
  • the verification query may be capable of fast membership tests in a large distributed index registry.
  • there are four components that enable this workflow (1) a distributed multi-modal deep hash composer, (2) a set of perceptual hashes and fusion hashes, (3) a multi-modal hash index registry, and (4) a membership verification and/or an association module.
  • a system in accordance with some aspects of the disclosure may (1) enable a city-wide fast membership or information existence check due to sub-linear query complexity in probabilistic data structure, (2) reduce network bandwidth by leveraging perceptual hash code to represent and transmit sensing data, and (3) identify associations between image, point-cloud, and event-sensing modalities by using hash composition and metadata.
  • FIG. 2 is a diagram 200 illustrating a high-level view of a system for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • Diagram 200 illustrates that, as opposed to the system described in FIG. 1, the system for multi-modal perceptual hash-based indexing of data for fast membership verification queries may include a local multi-modal sensor network 210 including a set of sensors 250 that produce sensor data sets 255 and an edge computing device 270 that produce hash indexes 215.
  • the system may further include a multi-modal index registry 220 that interacts with at least one client computing device 241 in a verification-client local network 240.
  • the local multi-modal sensor network 210 may include a set of sensors, e.g., sensor 250 and sensor 260, that may each capture one of an image data set, a point cloud data set, a processed information, or an inferred data set.
  • the sensor 250 or the sensor 260 of the local multi-modal sensor network 210 may include one or more of a CCTV, an IR/Night Camera, a RADAR, or a LiDAR Sensor Network which may generate imaging data types such as an RGB image, a point cloud, an error code, and/or reports.
  • the local multi-modal sensor network 210 may monitor a monitored zone 280 that includes, e.g., a set of parking spaces 281, 282, and 283.
  • the local multi-modal sensor network 210 may also include an edge computing device 270 that receives sensor data sets from the sensors, e.g., sensors 250 and 260 in the local multimodal sensor network 210.
  • the edge computing device 270 may receive sensor data sets 255 (e.g., an image data set, a point cloud data set, a processed information data set, or an inferred data set) over a local network 265.
  • the edge computing device 270 may compute a hash index for each received sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the edge computing device 270 may also compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • the edge computing device 270 may transmit the computed indexes, e.g., as indexes 215, to the multi-modal index registry 220.
  • the multi-modal index registry 220 may store the transmitted indexes for fast membership verification queries.
  • the multi-modal index registry 220 in some aspects, may be a distributed index registry, while in other aspects, the multi-modal index registry 220 may be a centralized index registry or a combination of distributed and centralized components.
  • the index may be stored in a probabilistic data structure such as a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure.
  • the multi-modal index registry may also store a set of perceptual hash functions that may be retrieved by the local multi-modal sensor network 210 , e.g., an edge computing device 270, and applied to received sensor data sets.
  • a client device may retrieve the set of perceptual hash functions in order to generate an index-based query.
  • the client device may be provided with a single- mode data set or a multi-modal data set and apply the retrieved perceptual hash functions to the provided data set(s) to generate and/or compute a set of indexes, e.g., indexes 225, for an index-based query that is then transmitted to the multi-modal index registry 220 to perform a search (a membership verification) based on the received set of indexes.
  • the multi-modal index registry 220 may also include a data processing module 221 for data ingestion from edge computing devices that process input sensor data sets (e.g., edge computing device 270 of the local multi-modal sensor network 210).
  • the data processing module 221 may provide the processed data to a distributed data storage 223 of the multi-modal index registry 220.
  • the multi-modal index registry 220 also includes a query-processing module 222 for interacting with an edge computing device 242 associated with a verification-client local network 240.
  • the distributed multi-modal index registry 220 and the edge computing devices 270 and 242 may be considered a system 230 for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • FIG. 3 is a diagram 300 illustrating elements of a system 330 for multi-modal perceptual hash-based indexing of data for fast membership verification queries.
  • the diagram 300 illustrates includes a set of client-side applications 310 that may access a distributed data storage 340 including a distributed multi-modal hash index registry 341 and a metadata registry 342 stored in a set of databases 343 of the system 330 through interface(s) 331.
  • the set of client-side applications 310 may include a verification module 311 and a set of data analytics 312.
  • the verification module 311 may provide an image or data set to the system via the interface(s) 331.
  • the interface(s) 331 may include one or more edge computing devices that include and/or implement a query generator 360.
  • the query generator 360 may receive an image or other data set for the verification operation and may generate a membership query 365.
  • a deep hash composer 362 may calculate a set of hash indexes for inclusion in a hash exchange format.
  • the routing and evaluation module 361 may generate the membership query 365 including the set of hash indexes generated by the deep hash composer 362.
  • a set of sensor-side applications 320 may include a sensor network 321 a set of external systems 322 and a set of external data stores 323 (e.g., local data stores).
  • the sensor-side applications 320 may provide data to the system 330, and more specifically to the distributed data storage 340, via the interface(s) 332.
  • the interface(s) 332, in some aspects, may include one or more edge computing devices that include and/or implement a distributed multi-modal deep hash composer 350 in a same local network as one or more sensor devices in the sensor network 321.
  • a distributed multi-modal deep hash composer 350 may be distributed and shared across the system 330 for client-side and/or sensor-side (which may collectively be referred to as client-side) index generation using perceptual hashing.
  • the distributed multimodal deep hash composer 350 uses ⁇ -perceptual hash functions to encode low fidelity image information analogous to image or object features in an m-bit binary hash code.
  • calculated uni-modal and cross-modal perceptual hashes may be embedded in an exchange format (a format of hash exchange and metadata 355) including metadata that is persisted and/or stored in an index registry, e.g., distributed multimodal hash index registry 341 or a metadata registry 342.
  • the distributed (across a network) data storage 340 in some aspects, enables low-latency membership queries.
  • the structure of the registry includes a plurality of spatial, domain, and mode-specific databases that may use probabilistic data structures such as one or more scalable bloom filters, cuckoo filters, or other probabilistic data structures.
  • probabilistic data structures such as one or more scalable bloom filters, cuckoo filters, or other probabilistic data structures.
  • bloom and cuckoo filters in some aspects, may be used to determine (with a given degree of certainty) whether an item is present or absent from a collection.
  • the data included in the one or more probabilistic data structures may also be stored in, and/or transformed to, a tree structure for distribution, search, comparison, and version management.
  • diagram 300 may represent a deployment diagram for a multi-modal membership verification system (e.g., the system 330) that is capable of ingesting sensor data from a plurality of sensors (e.g., sensors in the sensor network 321) and code the frame data using the &-hash functions.
  • the perceptual hash functions may be orchestrated by a replicated hash/filter escrow system 351 that may be managed remotely across the system 330.
  • the hash functions run on client-side edge systems (e.g., edge computing devices) and share/transmit the binary code (e.g., in hash exchange and metadata 355) to the distributed data storage 340 including the distributed multi-modal hash index registry 341.
  • the parameters (k and ni) and hash models and/or filters may be chosen and distributed across the device network.
  • a verification test system (e.g., as implemented using verification module 311) can present an image or event to check for membership and/or the presence of corresponding or associated information in the distributed data storage 340.
  • Th verification test system in some aspects, may be deployed as a service and may communicate with the distributed data storage 340 including the distributed multi-modal hash index registry 341 to perform membership checks.
  • Verification in some aspects, is conducted as a fast hash-based membership check and, once the membership and or the presence of corresponding data is verified, corresponding metadata may be retrieved for additional information.
  • the system in some aspects, provides the ability to provide membership verification at a city-scale and may be integrated as a layer on a data bus and other middleware platforms.
  • the system 330 may be targeted towards application of multi-modal perceptual hashes-based indexing of low-fidelity data for fast membership query in a large cityscale camera sensor network to monitor and manage curbside information. This can cater to short term parking monitoring and just-in-time existence check. Some aspects may incorporate versions to verify existence over a larger time frame such as for the case with long-term parking at facilities like airport and railway station. In some aspects, the system may reduce a need for manual intervention and may provide a large computational benefit in search or verification operations over a large infrastructure.
  • Some specific monitoring functions include but are not limited to monitoring any one of a vehicle parked in a parking area, a repeat violator vehicle on a commercial curbside, docked bikes, sensor data fingerprint for validity, equipment (e.g., a barrier, cone, etc.) in a curbside.
  • the system may also be configured to retrieve associated night vision index based on daytime imaging data or the other way round.
  • Additional applications may include monitoring systems deployment by city or transit authorities for real-time zone regulation, people flow by video analytics for dynamic curbside management and productivity estimation. These applications may contribute to realizing new ROI (Return on Investment) for transforming a cityscape by effectively monitoring, regulating, and allocating access to urban curbside space.
  • Some aspects may incorporate versions of the distributed data storage and sets of perceptual hash functions for monitoring long-term airport parking.
  • the long-term airport parking monitoring in some embodiments, may include incorporating temporal information in the structures and an ability to compute existence in those temporal or even geo-spatial segments.
  • Additional aspects may include a combination of image hash index and metadata to compute a metadata hashing for reducing collision and increasing parameter awareness, security assessments and audits for the sensor network and information quality, and/or blockchain extensions for single source of truth for various assessments and manipulations.
  • the discussion below focuses on an example workflow associated with a short-term curbside parking occupancy and/or a loading zone occupancy use-case.
  • the discussion below may relate to a system as illustrated in FIG. 2.
  • the camera network setup in some aspects, is such that one or more CCTV cameras constantly monitors at least one zone (e.g., monitored zone 280), where the at least one zone may include multiple parking spots (e.g., parking spaces 281-283).
  • the event information may be captured into the system and the hashes index workflow is initiated.
  • the query to this index registry can be used to test occupancy, membership, and/or an association in a distributed multi-modal sensor index management system.
  • FIG. 4 is a diagram 400 illustrating a set of components in an example multi-modal deep hash composer, in accordance with some aspects of the disclosure.
  • the multi-modal deep hash composer may be included in an edge computing device, e.g., edge computing device 270 of FIG. 2.
  • the multi-modal deep hash composer may be shared or distributed across the system to distribute algorithms, hash functions, and orchestrate its operations.
  • the multi-modal deep hash composer in some aspects, may be executed on an edge client or device and enable analytics on a hash-based index in our system.
  • the multi-modal deep hash composer may interact with a hash escrow component (not shown) of a multi-modal index registry (e.g., the multi-modal index registry 220) that may perform identification, verification, orchestration, and generation beyond providing the sets of k perceptual hash functions (e.g., k-hash functions) for computing a set of one or more hash indexes from input data sets to the various client-side hashing modules that may be executed in an edge computing device (e.g., edge computing device 270).
  • a hash functions can rotate based on escrow orchestration.
  • a family of perceptual hash functions include existing technologies like a uni-modal locality-sensitive perceptual hash generating a hash code for a particular mode for inclusion in a uni-mode sensor indexing structure (e.g., point structure 437), or cross-modal (or multi-modal) hash leveraging Deep Neural Network (DNN) that is capable of generating an n-wise cross-modal sensor fusion indexing structure (e.g., triplet structure 434 or pair structure 436).
  • DNN Deep Neural Network
  • Such an n-wise cross-modal sensor fusion indexing structure can be used to encode pair, triplet, or n-mode sensor data into a common subspace.
  • a device-optimized cross-modal neural network such as a vision- language model, in some aspects, may be used to generate perceptual hashes enabling cross- modal membership verification.
  • the multi-modal deep hash composer may receive one or more datatypes (e.g., image, point cloud, text events) in sensor data 415 from a sensor network (e.g., the local multi-modal sensor network 210 of FIG. 2).
  • the multi-modal deep hash composer may include a multi-modal data stream handler 420 that may evaluate a data format (or type), may register and index the received data for local storage and identify, or associate, metadata for, or with, the received sensor data 415.
  • the multi-modal data stream handler in some aspects may provide the processed data 425, including register and index data 425a and metadata 425b, to a multi-modal hash calculation module 430.
  • the multi-modal hash calculation module 430 may retrieve sets of uni-modal hash functions (sets of k perceptual hash functions) corresponding to data types received from hash function escrow 440.
  • the multi-modal hash calculation module 430 may retrieve sets of cross-modal hash functions (sets of k' cross- modal perceptual hash functions) corresponding to combinations of data types received from the hash function escrow 440.
  • the sets of uni-modal and cross-modal hash functions may be associated with a mode aware &-hash models module 431 (e.g., implemented as a DNN).
  • the multi-modal hash calculation module 430 may include a unimodel hash calculation module 432 that calculates the uni-modal hashes based on the set of uni-modal hash functions for a particular type of received data.
  • the multi-modal hash calculation module 430 may also include, in some aspects, a cross-modal hash calculation module 433 that calculates one or more cross-modal hashes associated with one or more combinations of data sets.
  • the cross-modal hash calculation module 433 may calculate the cross-modal hashes based on calculated hashes from the uni-model hash calculation module 432 and/or processed data sets from the multi-modal data stream handler 420.
  • the multi-modal hash calculation module 430 may store the calculated hashes in one or more data structures such as point structure 437 (e.g., for storing uni-modal hash indexes), triplet structure 434, and/or pair structure 436.
  • the multi-modal deep hash composer e.g., via a registry encoder 450, may generate a data set 455 for transmission over a network to a multimodal hash index registry. For example, in an aspect using a bloom filter probabilistic data structure a vector or binary code of m-bits per image data frame for each perceptual hash function may be generated.
  • the binary code of m bits may initially be set to 0, for each of a set of k independent perceptual hash functions (e.g., [/i 1( h 2 , ..., h k ]). For each element x G X , the bits at positions h 2 x 1 ), .... h ⁇ x ⁇ ] in the vectors associated with the set of k perceptual hash functions [/i 1( h 2 , ... , h k ], respectively, may be set to 1.
  • the binary code of m bits is associated with the set of k perceptual hash functions such that for each perceptual hash h L of the k perceptual hashes are associated with a set of bits in the m bits.
  • the generated data set 455 may be formatted in an exchange format that, in some aspects, includes one or more of a registry target 455a (as an example of registry data), metadata 455b (e.g., including parameters (m, n, k) indicating a size ‘m’ of a hash index, a number ‘n’ of data types, and a number ‘k’ of hashes used in the hash index calculation), a security watermark (e.g., watermark fingerprint 455c), multi-modal hash exchange data 455d, an index structure identification in a common standard persisted on a distributed index and metadata registry server (e.g., multi-modal hash index registry 220).
  • a registry target 455a as an example of registry data
  • metadata 455b e.g., including parameters (m, n, k) indicating a size ‘m’ of a hash index, a number ‘n’ of data types, and a number ‘k’ of hashes used in the hash index calculation
  • FIG. 5 is a diagram 500 illustrating an example process for processing data captured in accordance with some aspects disclosed herein.
  • the process may be applied to a short-term curbside parking monitoring with low-latency just-in-time information management at a city-scale with multiple observation areas in each camera coverage zone.
  • the data processing may be used to verify occupancy or membership of a query image based on the hash index.
  • a set of sensors 510 may capture a set of N data points 515 (e.g., data sets from ‘N’ sensors).
  • the sets of data points may be associated with a same time period and a same or overlapping set of areas.
  • the set of N data points 515 may include image data from a CCTV and a night vision and/or IR camera.
  • the captured sets of data points may be stored locally based on metadata associated with a captured data point (e.g., a time associated with the data point, a coverage area of the data point, one or more zones associated with the data point, a data point type, and/or a capture device associated with the data point).
  • a detector component 520 performs a zone-based object and/or event detection and registration test. In some aspects, the detector component 520 may perform the zone-based object and/or event detection and registration test frequently to reduce redundant downstream operations on image.
  • a hash family e.g., a set of perceptual hashes
  • the deep hash composer 530, or an edge device including the deep hash composer 530 may transmit (or distribute) the calculated hash indexes to a distributed multi-modal hash registry.
  • FIG. 6 is a diagram 600 illustrating a set of calculated indexes being transmitted to, and received by, a distributed multi-modal hash registry.
  • the set of calculated indexes may be one of a plurality of sets of calculated indexes (e.g., a multi-modal hash index 615) for a plurality of different data sets (e.g., sets of captured data associated with different times or different coverage areas) that may be formatted in a standardized exchange format.
  • the set(s) of calculated indexes in some aspects, may be transmitted in accordance with a batching operation. The batching operation may collect sets of calculated hash indexes until some threshold is met.
  • the threshold in some aspects, may be one or more of a threshold time (for periodic transmission) and/or a threshold amount of data for transmission. For example, every few minutes an edge computing device may share an index (or set of calculated hash indexes) with hash exchange information and metadata about a set of observed parking spots (or coverage areas).
  • the transmitted, and received, set(s) of calculated hash indexes and any additional data received from an edge computing device may be persisted in a probabilistic data structure and distributed so as to be available for responding to a query.
  • a distributed multi-modal hash registry may allocate and register the received set(s) of calculated hash indexes such that the hash indexes in the set(s) of calculated hash indexes are added to the distributed multi-modal hash registry if not already included in the distributed multi-modal hash registry or may be used to update an existing version of a corresponding (or equivalent) hash index stored in the distributed multi-modal hash registry.
  • the distributed multi-modal hash registry 620 may store the set(s) of calculated hash indexes in multiple different data structures and/or formats. For example, in some aspects, each a uni -modal hash index 615a and a cross-modal hash index 615b may be stored in a linear index registry 621 of the distributed multi-modal hash registry 620.
  • the linear index registry 621 may be configured as a cuckoo or counting bloom filter structure used for a membership and/or verification query (or test).
  • a linear index registry 621 in some aspects, may be persisted in a probabilistic data structure such as a bloom filter or a cuckoo filter which may be used to verify membership and/or determine whether a given index exists in a set.
  • a count-min-sketch data structure or a Top-k data structure may identify a frequency of events and/or matching data in a data sample.
  • the membership and/or verification query may be associated with a configured matching criteria or threshold. For example, for a uni-modal match, the matching criteria may specify that each bit, or a threshold number that is less than the number of bits in a uni-modal hash index, match a set of bits in a hash index-based query.
  • a threshold number of bits that may be different from, or the same as, the threshold number of bits for the uni -modal hash index match criteria, may be configured to determine an association between a uni-modal hash index-based query or a cross-modal hash index-based query with stored information in the linear index registry 621.
  • a tree-based index registry 622 may be created to allow for quick search and comparison of an index, metadata, and its versions (e.g., snapshots of a linear index registry 621 for different time periods).
  • the tree-based index registry 622 may include pointers to a metadata registry 623 that may be used to store metadata associated with the data included in the tree-based index registry 622.
  • the metadata registry 623 may include data relating to a registration, an event, a sensor location, a zone identifier, a hash family (e.g., a set of k hash functions), or other information relating to data (e.g., multi-modal hash index 615) received from an edge computing device.
  • a hash index distribution within the distributed multi-modal hash registry 620 may be done based on a geo-spatial sensor network, observation time, and registry type (e.g., a data like zonal parking occupancy).
  • the various stores may be available for membership information query.
  • FIG. 7 is a diagram 700 illustrating a process for verifying occupancy (e.g., of a parking spot) and/or a membership in the distributed hash index registry.
  • occupancy e.g., of a parking spot
  • membership in the distributed hash index registry For example, for a low latency check for membership of a car image across an indexed parked-car-information registry the system may verify membership information to verify if a presented car image (F) exists in a short-term parking occupancy registry.
  • F presented car image
  • a deep hash composer 720 at an edge computing device may calculate a uni-modal hash index based on a same family of k perceptual hash functions 721, e.g., [/i 1( h 2 , — , h k ], as used to process captured data from a sensor network.
  • the family of hash of k perceptual hash functions 721, in some aspects, may be received, or retrieved, from a distributed hash escrow that provides families of perceptual hash functions to edge computing devices throughout the system.
  • the deep hash composer 720 may also generate an associated query.
  • the generated query may include or indicate a set of bits at positions [ ⁇ (K), h 2 (K), ..., h k (Y ] and, in some aspects, may indicate a particular hashed index registry (e.g., a hashed index registry maintained for a particular application).
  • the generated query may be passed to a routing & evaluation module 730 for evaluation.
  • the routing & evaluation module 730 may be responsible for mapping, routing, and executing the query (e.g., performing a membership evaluation operation) on an identified hashed index registry in the distributed index registry server 740.
  • Executing the query may include identifying at 731 whether bits in the identified hashed index registry at positions [h 1 (K), h 2 (Y), ... , i k (K)] are each evaluated to be 1. If any of the bits at positions [h 1 Y), h 2 Y), ..., h k (Y ] are evaluated to be 0, in some aspects, the image associated with the query may be determined to not match an image captured by the sensor network and stored in the distributed hash registry. If all of the bits at the positions [hi(K), /I 2 (K), ... , h k (Yy] are evaluated to be 1, the image associated with the query may be determined to likely match an image captured by the sensor network and stored in the distributed hash registry.
  • the determination may be erroneous based on the possibility and/or the probability of a hash collision.
  • the parameters k and m may be chosen such that the probability of a false positive (and hence a false hit) is calculated to be below a threshold (acceptable) probability. For example, a check for membership may be applied to a set of indices in a zonal parking occupancy registry.
  • the deep hash composer 720 may also apply a set of cross-modal hash functions to create a pairwise hash composition for cross-modal hash index evaluation operation.
  • the deep hash composer 720 may also select the type of registry a query in the distributed index registry server 740 should target based on a mode and data type associated with the query regarding image 715 and the type of analysis for each type of registry (e.g., identification 731 or a check 732).
  • a verification and/or membership query may include a check 732 for an association between the image 715 (i . e. , a first type of captured data) and data captured by other types of sensors in the distributed index registry server 740.
  • a query related to a daytime image may additionally include a check 732 for an association with another type of captured data (e.g., a nighttime image or a point cloud) in a cross-modal registry.
  • the results 745 e.g., a status or an index
  • the identification 731 and the check 732 may then be provided to the client computing device from which the query regarding image 715 was received.
  • a parking system may, continuously, periodically, or on an event-based basis, observe occupation of a target zone (e.g., monitored zone 280 including the set of parking spaces 281-283).
  • Various computer vision models may be used to track, detect, extract a region of interest, and encode features for uni-modal or cross-modal hash index generation.
  • a multimodal hash registry in some aspects, maintains multi-modal indices for each object that occupies a set of observed regions. Based on the maintained multi-modal indices, a specific image verification operation may be performed over the occupancy membership registry for the target zone or a set of target zones in a region of interest (e.g., a city, zip code, block, etc.).
  • Performing the specific image verification operation may reduce an overhead associated with frequently looking for parked vehicles in a specific locale or tracking a specific vehicle continuously.
  • the specific image verification operation may also include an association check to determine if alternate modality sensing data exists. It can serve as a security verification, fingerprinting, or validation for image frames that are served by the sensor network. This invention can be applied for consolidated parking revenue collection in curbside management scenario.
  • FIG. 8 is a flow diagram 800 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • the method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270).
  • the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • a system 230 or 330 may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions.
  • the set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions.
  • the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530).
  • the system may further compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433.
  • the cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing.
  • the hash indexes may be computed at an edge computing device. After computing the hash indexes, the computed hash indexes may be transmitted to a centralized data storage or a distributed data storage.
  • the system may store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623).
  • the stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • storing the information includes storing additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the system may also receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set as described in relation to FIGs. 3 and 7. Based on receiving the query, the system may perform a search of the probabilistic data structure based on the query-related hash index and may transmit a result of the search in response to the received query.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIG.
  • the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.
  • FIG. 9 is a flow diagram 900 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • the method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270).
  • the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • a system 230 or 330 may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions.
  • the set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions. For example, referring to FIGs.
  • the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530). [0075] At 930, the system may compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. For example, referring to FIGs. 3-5, the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433. The cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing. The hash indexes may be computed at an edge computing device.
  • the computed hash indexes may be transmitted, at 940, to a centralized data storage or a distributed data storage.
  • the transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes.
  • the information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage.
  • a distributed multi-modal deep hash composer 350, a multi-modal hash calculation module 430, or a deep hash composer 530 may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.
  • the system may store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623).
  • the stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • the system may store, at 960, additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the additional information may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry.
  • the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623.
  • the system may receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set.
  • the set of perceptual hash functions used to generate the query may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage.
  • the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query may further include an indication of a domain or the type of sensor associated with the query.
  • the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.
  • the system may perform a search of the probabilistic data structure based on the query-related hash index.
  • the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index.
  • the probabilistic data structure in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure.
  • the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index.
  • the system may transmit a result of the search in response to the received query.
  • the result may include a membership existence status, a storage location associated with one of the sensor data identified by the search, or a storage location of metadata associated with the sensor data.
  • the routing & evaluation module 730 may receive a query-related hash index generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.
  • FIG. 10 is a flow diagram 1000 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • the method may be performed by an edge computing device (e.g., edge computing device 270).
  • the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • an edge computing device of the system 230 or 330 may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions.
  • the set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions.
  • the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530).
  • the system may further compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433.
  • the cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing.
  • the hash indexes may be computed at an edge computing device. After computing the hash indexes, the computed hash indexes may be transmitted to a centralized data storage or a distributed data storage.
  • the edge computing device may transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • the transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes.
  • the information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage. For example, referring to FIGs.
  • a distributed multi-modal deep hash composer 350 may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.
  • the system may store the transmitted information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623).
  • the stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • storing the information includes storing additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the system may also receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set as described in relation to FIGs. 3 and 7. Based on receiving the query, the system may perform a search of the probabilistic data structure based on the query-related hash index and may transmit a result of the search in response to the received query.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIG.
  • the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.
  • FIG. 11 is a flow diagram 1100 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • the method may be performed by an edge computing device (e.g., edge computing device 270).
  • the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • a system 230 or 330 may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the edge computing device may retrieve the set of perceptual hash functions for the plurality of types of sensors from a one of a centralized hash escrow or a distributed hash escrow.
  • the centralized hash escrow server or the distributed hash escrow is an orchestrator for a family of k hash functions.
  • the centralized or distributed hash escrow may ensure that the same set of k perceptual hash functions are being used across the system so that the hash indexes produced based on sensor data and hash indexes produced based on test data are consistent (e.g., produce a same, or similar, hash for the same, or similar, image feature). For example, referring to FIGs.
  • the edge computing device 270, an edge computing device implementing distributed multi-modal deep hash composer 350, or an edge computing device implementing multi-modal hash calculation module 430 may retrieve, from a hash function escrow 440, one or more sets of perceptual hash functions to populate the replicated hash/filter escrow system 351.
  • the edge computing device may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions.
  • the set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be a set of perceptual hashes retrieved, at 1120, from the centralized hash escrow or the distributed hash escrow that is an orchestrator for the families of k perceptual hash functions.
  • an edge computing device implementing distributed multi-modal deep hash composer 350, an edge computing device implementing multi-modal hash calculation module 430, or an edge computing device implementing deep hash composer 530 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350, multi-modal hash calculation module 430, or deep hash composer 530, respectively.
  • the edge computing device may compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • an edge computing device implementing distributed multi-modal deep hash composer 350, an edge computing device implementing multi-modal hash calculation module 430, or an edge computing device implementing deep hash composer 530 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433.
  • the cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing.
  • the edge computing device may transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • the transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes.
  • the information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage. For example, referring to FIGs.
  • a distributed multi-modal deep hash composer 350 may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.
  • the edge computing device may store, at least in part, the received plurality of sensor data sets in a local data storage.
  • a local data storage For example, referring to FIGs. 3 and 6, an edge computing device may store the received plurality of sensor data sets in an external data store 323 (e.g., a local data store).
  • storing, at least in part, the raw data locally and transmitting the hash indexes allows for a search for matching data to be performed quickly and efficiently at a central (or distributed) server while maintaining the accessibility of the sensor data sets based on the results of the search for matching data.
  • the system may store the transmitted hash index data and associated metadata in a distributed data storage 340, e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342 or a distributed multi-modal hash registry 620, e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623.
  • a distributed data storage 340 e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342 or a distributed multi-modal hash registry 620, e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623.
  • the stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • the edge computing device may receive a request for at least one particular sensor data set of the received plurality of sensor data sets stored in the local data storage based on a query received at one of the centralized or the distributed server.
  • the request may be based on a storage location identified in the result of a search based on the query received at one of the centralized or the distributed server.
  • the set of perceptual hash functions used to generate the query (e.g., at a client-side edge computing device), in some aspects, may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage.
  • the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs. 3 and 7, an edge computing device in the system 330 may receive an indication (e.g., in a set of results 745) of a storage location in an external data store 323.
  • the edge computing device may transmit the requested at least one particular sensor data set to one of the centralized server or the distributed server.
  • an edge computing device in the system 330 may retrieve the requested at least one particular sensor data and transmit the requested at least one particular sensor data.
  • FIG. 12 is a flow diagram 1200 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270).
  • an edge computing device e.g., system 230, 330; edge computing device 270.
  • the system may receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • a system 230 or 330 may receive a plurality of hash indexes generated by a distributed multimodal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530) based on sensor data sets collected by the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the system may store the received plurality of hash indexes in a probabilistic data structure.
  • the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623).
  • the stored plurality of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • the system may store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the additional information may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry.
  • the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623.
  • the system may receive, from a client computing device, one of a membership query or an association query.
  • the membership query or the association query may include at least one hash index.
  • the set of perceptual hash functions used to generate the query may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage.
  • the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs.
  • the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.
  • an image e.g., image 715
  • a query generator 360 or a deep hash composer 362 or 720
  • the system may verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • the verification includes performing a search of the probabilistic data structure based on the query-related hash index.
  • the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index.
  • the probabilistic data structure in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. For example, referring to FIGs.
  • the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index to verify and/or determine whether matching sensor data set or an associated sensor data set is stored in a searched distributed data storage.
  • the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745.
  • the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query. [0101 ] FIG.
  • FIG. 13 is a flow diagram 1300 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries.
  • the method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270).
  • the system may receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs.
  • a system 230 or 330 may receive a plurality of hash indexes generated by a distributed multimodal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530) based on sensor data sets collected by the local multi-modal sensor network 210, the sensor network 321, or the same network 410.
  • the plurality of sensor data sets are associated with a same time period and a same area of interest.
  • the plurality of sensor data sets in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.
  • the system may store the received plurality of hash indexes in a probabilistic data structure.
  • the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623).
  • the stored plurality of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.
  • the system may store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the additional information may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry.
  • the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623.
  • the system may receive, from a client computing device, one of a membership query or an association query.
  • the membership query or the association query may include at least one hash index.
  • the set of perceptual hash functions used to generate the query may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage.
  • the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index.
  • the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure.
  • the query in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs.
  • the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.
  • an image e.g., image 715
  • a query generator 360 or a deep hash composer 362 or 720
  • the system may verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • the verification includes performing a search of the probabilistic data structure based on the query-related hash index.
  • the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index.
  • the probabilistic data structure in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. For example, referring to FIGs.
  • the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index to verify and/or determine whether matching sensor data set or an associated sensor data set is stored in a searched distributed data storage.
  • the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.
  • the system may identify, based on verifying the existence of the matching sensor data set or the associated sensor data, a location of the matching or the associated sensor data set. After identifying the location at 1350, the system may, at 1360, retrieve the matching or the associated sensor data set from the identified location.
  • the identified location may be in an external storage (e.g., a local storage to the sensor network)
  • the system 330 may identify, based on the search of the distributed multi-modal hash index registry 341 or the distributed index registry server 740, a location associated with a set of matching or associated data in a local storage (e.g., external data store 323). For example, referring to FIG. 3 the system 330 may retrieve, from the location identified at 1350, data from the external data store 323.
  • the system may transmit the retrieved matching or the retrieved associated sensor data set to the client computing device.
  • the retrieved data may include the matching or associated sensor data set and/or metadata associated with the matching or associated sensor data set.
  • the system 330 may transmit a sensor data set retrieved from an external data store 323 via interface(s) 332 to a verification module 311 via interface(s) 331.
  • FIG. 14 illustrates an example computing environment with an example computer device suitable for use in some example implementations.
  • Computer device 1405 in computing environment 1400 can include one or more processing units, cores, or processors 1410, memory 1415 (e.g., RAM, ROM, and/or the like), internal storage 1420 (e.g., magnetic, optical, solid-state storage, and/or organic), and/or IO interface 1425, any of which can be coupled on a communication mechanism or bus 1430 for communicating information or embedded in the computer device 1405.
  • IO interface 1425 is also configured to receive images from cameras or provide images to projectors or displays, depending on the desired implementation.
  • Computer device 1405 can be communicatively coupled to input/user interface 1435 and output device/interface 1440. Either one or both of the input/user interface 1435 and output device/interface 1440 can be a wired or wireless interface and can be detachable.
  • Input/user interface 1435 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch-screen interface, keyboard, a pointing/cursor control, microphone, camera, braille, motion sensor, accelerometer, optical reader, and/or the like).
  • Output device/interface 1440 may include a display, television, monitor, printer, speaker, braille, or the like.
  • input/user interface 1435 and output device/interface 1440 can be embedded with or physically coupled to the computer device 1405.
  • other computer devices may function as or provide the functions of input/user interface 1435 and output device/interface 1440 for a computer device 1405.
  • Examples of computer device 1405 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and/or coupled thereto, radios, and the like).
  • highly mobile devices e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like
  • mobile devices e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like
  • devices not designed for mobility e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and/or coupled thereto, radios, and the like.
  • Computer device 1405 can be communicatively coupled (e.g., via IO interface 1425) to external storage 1445 and network 1450 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configuration.
  • Computer device 1405 or any connected computer device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.
  • IO interface 1425 can include but is not limited to, wired and/or wireless interfaces using any communication or IO protocols or standards (e.g., Ethernet, 1402.1 lx, Universal System Bus, WiMax, modem, a cellular network protocol, and the like) for communicating information to and/or from at least all the connected components, devices, and network in computing environment 1400.
  • Network 1450 can be any network or combination of networks (e.g., the Internet, local area network, wide area network, a telephonic network, a cellular network, satellite network, and the like).
  • Computer device 1405 can use and/or communicate using computer-usable or computer readable media, including transitory media and non-transitory media.
  • Transitory media include transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like.
  • Non- transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid- state storage), and other non-volatile storage or memory.
  • Computer device 1405 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments.
  • Computer-executable instructions can be retrieved from transitory media, and stored on and retrieved from non-transitory media.
  • the executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).
  • Processor(s) 1410 can execute under any operating system (OS) (not shown), in a native or virtual environment.
  • OS operating system
  • One or more applications can be deployed that include logic unit 1460, application programming interface (API) unit 1465, input unit 1470, output unit 1475, and inter-unit communication mechanism 1495 for the different units to communicate with each other, with the OS, and with other applications (not shown).
  • API application programming interface
  • Processor(s) 1410 can be in the form of hardware processors such as central processing units (CPUs) or in a combination of hardware and software units.
  • API unit 1465 when information or an execution instruction is received by API unit 1465, it may be communicated to one or more other units (e.g., logic unit 1460, input unit 1470, output unit 1475).
  • logic unit 1460 may be configured to control the information flow among the units and direct the services provided by API unit 1465, the input unit 1470, the output unit 1475, in some example implementations described above.
  • the flow of one or more processes or implementations may be controlled by logic unit 1460 alone or in conjunction with API unit 1465.
  • the input unit 1470 may be configured to obtain input for the calculations described in the example implementations
  • the output unit 1475 may be configured to provide an output based on the calculations described in example implementations.
  • Processor(s) 1410 can be configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the processor(s) 1410 may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the processor(s) 1410 may further be configured to store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.
  • the processor(s) 1410 may further be configured to compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • the processor(s) 1410 may also be configured to transmit, from the edge computing device over an intervening network, the information regarding the computed set of hash indexes to the centralized data storage or the distributed storage.
  • the processor(s) 1410 may also be configured to store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.
  • the processor(s) 1410 may also be configured to receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set.
  • the processor(s) 1410 may further be configured to perform a search of the probabilistic data structure based on the query-related hash index.
  • the processor(s) 1410 may further be configured to transmit a result of the search in response to the received query.
  • Processor(s) 1410 can be configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors.
  • the processor(s) 1410 may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set.
  • the processor(s) 1410 may further be configured to transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.
  • the processor(s) 1410 may further be configured to compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets.
  • the processor(s) 1410 may further be configured to retrieve the set of perceptual hash functions for the plurality of types of sensors from a one of a centralized hash escrow or a distributed hash escrow.
  • the processor(s) 1410 may further be configured to store, at least in part, the received plurality of sensor data sets in a local data storage.
  • the processor(s) 1410 may further be configured to receive, from one of a centralized or a distributed server, a request for at least one particular sensor data set of the received plurality of sensor data sets stored in the local data storage based on a query received at one of the centralized or the distributed server.
  • the processor(s) 1410 may further be configured to transmit the requested at least one particular sensor data set to one of the centralized server or the distributed server.
  • Processor(s) 1410 can be configured to receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors.
  • the processor(s) 1410 may further be configured to store the received plurality of hash indexes in a probabilistic data structure.
  • the processor(s) 1410 may further be configured to receive, from a client computing device, one of a membership query or an association query.
  • the processor(s) 1410 may further be configured to verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.
  • the processor(s) 1410 may further be configured to identify, based on verifying the existence of the matching sensor data set or the associated sensor data, a location of the matching or the associated sensor data set.
  • the processor(s) 1410 may further be configured to retrieve the matching or the associated sensor data set from the identified location.
  • the processor(s) 1410 may further be configured to transmit the retrieved matching or the retrieved associated sensor data set to the client computing device.
  • Example implementations may also relate to an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs.
  • Such computer programs may be stored in a computer readable medium, such as a computer readable storage medium or a computer readable signal medium.
  • a computer readable storage medium may involve tangible mediums such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid-state devices, and drives, or any other types of tangible or non-transitory media suitable for storing electronic information.
  • a computer readable signal medium may include mediums such as carrier waves.
  • the algorithms and displays presented herein are not inherently related to any particular computer or other apparatus.
  • Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.
  • the operations described above can be performed by hardware, software, or some combination of software and hardware.
  • Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application.
  • some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software.
  • the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways.
  • the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un procédé d'indexation multimodale de données basée sur un hachage perceptif pour des interrogations de vérification d'adhésion rapide. Le procédé peut consister à recevoir une pluralité d'ensembles de données de capteur à partir d'une pluralité de capteurs comprenant une pluralité de types de capteurs. Le procédé peut également consister à calculer un indice de hachage pour chaque ensemble de données de capteur particulier dans la pluralité d'ensembles de données de capteur sur la base d'un ensemble de fonctions de hachage perceptif pour un type de capteur associé à l'ensemble de données de capteur particulier. Le procédé peut en outre consister à stocker des informations concernant l'ensemble calculé d'indices de hachage dans une structure de données probabiliste pour les interrogations de vérification d'adhésion rapide.
PCT/US2022/021559 2022-03-23 2022-03-23 Système de détection multimodale et de vérification d'adhésion WO2023182989A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2022/021559 WO2023182989A1 (fr) 2022-03-23 2022-03-23 Système de détection multimodale et de vérification d'adhésion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2022/021559 WO2023182989A1 (fr) 2022-03-23 2022-03-23 Système de détection multimodale et de vérification d'adhésion

Publications (1)

Publication Number Publication Date
WO2023182989A1 true WO2023182989A1 (fr) 2023-09-28

Family

ID=88101541

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/021559 WO2023182989A1 (fr) 2022-03-23 2022-03-23 Système de détection multimodale et de vérification d'adhésion

Country Status (1)

Country Link
WO (1) WO2023182989A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120274482A1 (en) * 2011-04-26 2012-11-01 National Chin-Yi University Of Technology Parking lot management system
US8576283B1 (en) * 2010-01-05 2013-11-05 Target Brands, Inc. Hash-based chain of custody preservation
US8762396B2 (en) * 2011-12-22 2014-06-24 Sap Ag Dynamic, hierarchical bloom filters for network data routing
US20220014467A1 (en) * 2021-09-23 2022-01-13 Sunil Cheruvu Information centric network routing

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8576283B1 (en) * 2010-01-05 2013-11-05 Target Brands, Inc. Hash-based chain of custody preservation
US20120274482A1 (en) * 2011-04-26 2012-11-01 National Chin-Yi University Of Technology Parking lot management system
US8762396B2 (en) * 2011-12-22 2014-06-24 Sap Ag Dynamic, hierarchical bloom filters for network data routing
US20220014467A1 (en) * 2021-09-23 2022-01-13 Sunil Cheruvu Information centric network routing

Similar Documents

Publication Publication Date Title
US10776665B2 (en) Systems and methods for object detection
US20210103616A1 (en) Short-term and long-term memory on an edge device
US8837906B2 (en) Computer assisted dispatch incident report video search and tagging systems and methods
US10037607B2 (en) Topology determination for non-overlapping camera network
Zabłocki et al. Intelligent video surveillance systems for public spaces–a survey
RU2632473C1 (ru) Способ обмена данными между ip видеокамерой и сервером (варианты)
US20160092473A1 (en) Multi-query privacy-preserving parking management system and method
Mallela et al. Detection of the triple riding and speed violation on two-wheelers using deep learning algorithms
Giyenko et al. Application of convolutional neural networks for visibility estimation of CCTV images
KR101773127B1 (ko) 효율적인 영상 분석 및 검색이 가능한 영상 분석 시스템, 이를 포함하는 통합 관제 시스템 및 그 동작방법
Pazho et al. Ancilia: Scalable intelligent video surveillance for the artificial intelligence of things
US11594043B1 (en) People and vehicle analytics on the edge
Yang et al. Effective contexts for UAV vehicle detection
Youssef et al. Automatic vehicle counting and tracking in aerial video feeds using cascade region-based convolutional neural networks and feature pyramid networks
CN113378616A (zh) 视频分析方法、视频分析的管理方法及相关设备
Liu et al. AutoVAPS: An IoT-enabled public safety service on vehicles
Jung et al. Real-time car tracking system based on surveillance videos
WO2023182989A1 (fr) Système de détection multimodale et de vérification d'adhésion
US20220415051A1 (en) Object re-identification
Ahmed et al. Multi-aspect detection and classification with multi-feed dynamic frame skipping in vehicle of internet things
Sooraj et al. Real-time traffic counter using mobile devices
Priymak et al. Real-Time Traffic Classification through Deep Learning
Huang et al. Technical and economic feasibility assessment of a cloud-enabled traffic video analysis framework
Biswas et al. Dynamic strategy to use optimum memory space in real-time video surveillance
US20220101018A1 (en) Systems and methods for efficient data communications in traffic monitoring

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22933828

Country of ref document: EP

Kind code of ref document: A1