US11907293B2 - Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs - Google Patents
Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs Download PDFInfo
- Publication number
- US11907293B2 US11907293B2 US17/120,349 US202017120349A US11907293B2 US 11907293 B2 US11907293 B2 US 11907293B2 US 202017120349 A US202017120349 A US 202017120349A US 11907293 B2 US11907293 B2 US 11907293B2
- Authority
- US
- United States
- Prior art keywords
- spatiotemporal
- proximity
- graph
- tracks
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 230000000694 effects Effects 0.000 claims abstract description 105
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000004590 computer program Methods 0.000 claims description 5
- 230000015654 memory Effects 0.000 description 27
- 238000013528 artificial neural network Methods 0.000 description 24
- 238000001514 detection method Methods 0.000 description 17
- 238000010801 machine learning Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000002093 peripheral effect Effects 0.000 description 7
- 230000001537 neural effect Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 5
- 230000004913 activation Effects 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 210000004027 cell Anatomy 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 229910001416 lithium ion Inorganic materials 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- QELJHCBNGDEXLD-UHFFFAOYSA-N nickel zinc Chemical compound [Ni].[Zn] QELJHCBNGDEXLD-UHFFFAOYSA-N 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000000638 stimulation Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- HBBGRARXTFLTSG-UHFFFAOYSA-N Lithium ion Chemical compound [Li+] HBBGRARXTFLTSG-UHFFFAOYSA-N 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 241000700159 Rattus Species 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 210000003050 axon Anatomy 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- OJIJEKBXJYRIBZ-UHFFFAOYSA-N cadmium nickel Chemical compound [Ni].[Cd] OJIJEKBXJYRIBZ-UHFFFAOYSA-N 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 210000003169 central nervous system Anatomy 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229910052987 metal hydride Inorganic materials 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 229910052759 nickel Inorganic materials 0.000 description 1
- PXHVJJICTQNCMI-UHFFFAOYSA-N nickel Substances [Ni] PXHVJJICTQNCMI-UHFFFAOYSA-N 0.000 description 1
- -1 nickel metal hydride Chemical class 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000000946 synaptic effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/787—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/75—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
- G06F16/784—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content the detected or recognised objects being people
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
- G06F16/786—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content using motion, e.g. object motion or camera motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/55—Rule-based translation
- G06F40/56—Natural language generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
- G06V20/47—Detecting features for summarising video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Definitions
- This application is generally related to methods and apparatuses for monitoring, summarizing, and searching surveillance video footage.
- each frame of video may contain a great deal of information (e.g., audio, visuals, timestamps, metadata, etc.), users must typically choose between speed and capacity, particularly when archiving video data. Accordingly, a need exists to improve the storage and retrieval of full motion video data in memory.
- a scene may include one or more tracks, where each track may correspond to an entity.
- the scene may be received from a database or an application.
- scenes may be developed in real-time based on recorded or live-streamed video.
- one or more spatiotemporal proximity graphs may be built based on the received track(s). For example, a node may be identified for each entity and each spatiotemporal proximity graph may include one or more nodes. Moreover, each node may include one or more attributes associated with the corresponding entity, e.g., an attribute indicating a time interval in which the node was present in the scene or an attribute indicating a track identifier associated with the corresponding track. In some embodiments, the spatiotemporal proximity graph(s) may be transmitted to a downstream application.
- one or more spatiotemporal relationships may be identified between the entities.
- one or more proximity edges may be identified based on each spatiotemporal proximity graph and each proximity edge may include an attribute indicating a span of frames in which one or more entities are within a defined spatial proximity (e.g., of each other or an object of interest).
- one or more bounding boxes may be associated with each track and a defined spatial proximity may be based on an overlap of bounding boxes.
- the spatiotemporal relationships may be transmitted to a downstream application.
- one or more activities of the entities may be identified based on the spatiotemporal relationships.
- one or more complex activities may be identified based on one or more identified activities.
- FIG. 1 is a schematic representation of an architecture of a system for activity detection in videos based on a spatiotemporal graph representation according to an aspect of the application.
- FIG. 2 is a diagram illustrating a graphic user interface of a spatiotemporal proximity graph on a computer monitor display according to an aspect of the application.
- FIG. 3 is a diagram illustrating a graphic user interface of an activity graph on a computer monitor display according to an aspect of the application.
- FIG. 4 illustrates an exemplary flowchart of a method 400 to detect activities in videos based on a spatiotemporal graph representation in accordance with the present disclosure.
- FIG. 5 illustrates a system diagram of an exemplary communication network node.
- FIG. 6 illustrates a block diagram of an exemplary computing system.
- references in this application to “one embodiment,” “an embodiment,” “one or more embodiments,” or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure.
- the appearances of, for example, the phrases “an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
- various features are described which may be exhibited by some embodiments and not by the other.
- various requirements are described which may be requirements for some embodiments but not by other embodiments.
- the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must).
- the words “include,” “including,” and “includes” and the like mean including, but not limited to.
- the singular form of “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.
- the term “number” shall mean one or an integer greater than one (i.e., a plurality).
- ANNs Artificial neural networks
- ANNs are models used in machine learning and may include statistical learning algorithms conceived from biological neural networks (particularly of the brain in the central nervous system of an animal) in machine learning and cognitive science.
- ANNs may refer generally to models that have artificial neurons (nodes) forming a network through synaptic interconnections (weights) and acquires problem-solving capability as the strengths of the interconnections are adjusted, e.g., at least throughout training.
- the terms ‘artificial neural network’ and ‘neural network’ may be used interchangeably herein.
- An ANN may be configured to detect an activity associated with an entity based on input image(s) or other sensed information.
- An ANN is a network or circuit of artificial neurons or nodes. Such artificial networks may be used for predictive modeling.
- the prediction models may be and/or include one or more neural networks (e.g., deep neural networks, artificial neural networks, or other neural networks), other machine learning models, or other prediction models.
- the neural networks referred to variously herein may be based on a large collection of neural units (or artificial neurons).
- Neural networks may loosely mimic the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons).
- Each neural unit of a neural network may be connected with many other neural units of the neural network. Such connections may be enforcing or inhibitory, in their effect on the activation state of connected neural units.
- neural networks may include multiple layers (e.g., where a signal path traverses from input layers to output layers).
- back propagation techniques may be utilized to train the neural networks, where forward stimulation is used to reset weights on the front neural units.
- stimulation and inhibition for neural networks may be more free-flowing, with connections interacting in a more chaotic and complex fashion.
- Disclosed implementations of artificial neural networks may apply a weight and transform the input data by applying a function, this transformation being a neural layer.
- the function may be linear or, more preferably, a nonlinear activation function, such as a logistic sigmoid, hyperbolic tangent (Tan h), or rectified linear activation function (ReLU) function.
- Intermediate outputs of one layer may be used as the input into a next layer.
- the neural network through repeated transformations learns multiple layers that may be combined into a final layer that makes predictions. This learning (i.e., training) may be performed by varying weights or parameters to minimize the difference between the predictions and expected values.
- information may be fed forward from one layer to the next.
- the neural network may have memory or feedback loops that form, e.g., a neural network. Some embodiments may cause parameters to be adjusted, e.g., via back-propagation.
- Each of the herein-disclosed ANNs may be characterized by features of its model, the features including an activation function, a loss or cost function, a learning algorithm, an optimization algorithm, and so forth.
- the structure of an ANN may be determined by a number of factors, including the number of hidden layers, the number of hidden nodes included in each hidden layer, input feature vectors, target feature vectors, and so forth.
- Hyperparameters may include various parameters which need to be initially set for learning, much like the initial values of model parameters.
- the model parameters may include various parameters sought to be determined through learning. And the hyperparameters are set before learning, and model parameters can be set through learning to specify the architecture of the ANN.
- the hyperparameters may include initial values of weights and biases between nodes, mini-batch size, iteration number, learning rate, and so forth.
- the model parameters may include a weight between nodes, a bias between nodes, and so forth.
- the ANN is first trained by experimentally setting hyperparameters to various values, and based on the results of training, the hyperparameters can be set to optimal values that provide a stable learning rate and accuracy.
- FIG. 1 illustrates a schematic representation of an architecture of a system 100 for activity detection in videos (e.g., overhead surveillance video) based on a spatiotemporal graph representation according to an aspect of the application.
- Some embodiments employ machine learning for computer vision and automated reasoning.
- the system 100 may receive object tracks from a computer vision-based multi-object tracker and uncover spatiotemporal relationships among tracked entities in video.
- the system 100 may infer high-level activities (e.g., entering/exiting a vehicle, people exchanging objects, loading/unloading objects from a vehicle/facility, etc.) from the structure of the spatiotemporal relationships.
- the uncovered relationships and inferred activities may support intelligence applications like monitoring and alerting, semantic search, video summarization and anomaly detection.
- a modular architecture may allow for integration of different detection and feature extraction methods into the system 100 .
- the system 100 may receive one or more video frame(s) 102 , e.g., archival footage or live video streams.
- the video frame(s) 102 may be received via a wired or wireless network connection from a database (e.g., a server storing image data) or an imaging system.
- an imaging system may include an aerial vehicle (e.g., a manned or unmanned aerial vehicle), a fixed camera (e.g., a security camera, inspection camera, traffic light camera, etc.), a portable device (e.g., mobile phone, head-mounted device, video camera, etc.), or any other form of electronic image capture device.
- the system 100 may receive the one or more video frames 102 via a wired or wireless network connection.
- video frames 102 may include ground based footage, aerial, or near-aerial footage.
- other sensors such as LiDAR or radar, or a combination of cameras and other sensors may provide data.
- a multi-object tracking engine 104 may receive the one or more video frames 102 , identify one or more objects, and perform tracking of the objects.
- the multi-object tracking engine 104 may generate a track associated with each object.
- a track generated by the multi-object tracking engine 104 may include a label or object identifier associated with an object.
- a track associated with the object identifier may include a class identifier associated with the object of interest.
- the multi-object tracking engine 104 may process a video by using computer vision to detect objects of interest and generates tracks for those objects, e.g., associating detections in multiple video frames to the same real-world entity.
- tracks generated by the multi-object tracking engine 104 may consist of a track identifier, a class label (e.g., vehicle, person), a sequence of bounding boxes representing the location of the tracked object in each frame of the video in which it was present, and/or a set of fine grain attribute predictions for each object (e.g., make, model, color for vehicle tracks or gender, clothing type and age range for personnel tracks).
- a graph generator 106 may receive tracks generated by the multi-object tracking engine 104 and build one or more spatiotemporal proximity graphs (e.g., spatiotemporal proximity graph 110 ) based on the received tracks. For example, the graph generator 106 may use the tracks to automatically build a spatiotemporal proximity graph (e.g., spatiotemporal proximity graph 110 ) including one or more nodes.
- each node may represent a distinct tracked entity (e.g., a specific truck) and the edges between nodes may represent a proximity relationship or a significant period of close spatial proximity between two entities.
- nodes may also have one or more “active” edges (e.g., self-loops) with attributes indicating a start and end time of each interval in which a corresponding entity was present in the scene.
- both nodes and edges may contain attributes.
- “proximity edges” may have attributes indicating a span of frames in which two entities were in close proximity and nodes may contain an attribute for a track identifier.
- nodes and proximity edges may have additional attributes for any fine grain attribute predictions, which may be used for search refinement in some applications.
- track proximity may be computed in different ways, e.g., based on closeness or overlap of track bounding boxes in pixel-space.
- a defined spatiotemporal proximity may be based on a threshold distance (e.g., in pixel space or coordinate space) between track bounding boxes.
- pixel coordinates may be translated to a coordinate frame (e.g., birds-eye view) and distances may be computed in that coordinate frame.
- system 100 may account for a change in viewpoint or perspective of the one or more video frames 102 .
- video footage from an aerial vehicle e.g., plane, drone, etc.
- the graph generator 106 may account for a change in viewpoint by predicting a path or trajectory associated with the one or more video frames 102 and the spatiotemporal proximity graph 110 may be built based on the predicted path or trajectory (e.g., a flight path of an aerial vehicle).
- the spatiotemporal proximity graphs (e.g., spatiotemporal proximity graph 110 ) generated by the graph generator 106 may be stored in graph database system 108 .
- the graph database system 108 may encompass storage of spatiotemporal proximity graphs (e.g., spatiotemporal proximity graph 110 ), an activity extractor 112 , or storage of high-level activity graphs (e.g., high-level activity graph 116 ).
- an activity extractor 112 may receive the spatiotemporal proximity graph 110 generated by the graph generator 106 . Based on the spatiotemporal proximity graph 110 , the activity extractor 112 may detect higher level activities through subgraph matching queries. For example, higher level activities detected by the activity extractor may include entering/exiting a vehicle or building, exchanging objects, etc.
- the activity extractor 112 may receive ontology data 114 and may use the received ontology data 114 to generate a high-level activity graph (e.g., high-level activity graph 116 ).
- ontology data 114 may include information such as class hierarchies (e.g., associated with spatiotemporal proximity graphs) or activity template queries to be used in identifying high-level activities.
- the activity extractor 112 may generate a high-level activity graph (e.g., high-level activity graph 116 ).
- detected activities may be represented in the high-level activity graph by “activity edges” between nodes participating in the activities.
- construction of the activity graph e.g., an activity detection process
- construction of the activity graph may be performed by the activity extractor 112 in an incremental manner.
- some complex activities may be detected as a combination of previously added activity edges and underlying proximity edges.
- the activity “loading an object into a vehicle” may be defined as an object entering a vehicle (e.g., represented as an “entered” activity edge between an object node and a vehicle node) and a proximity edge between a person and the object node with overlapping temporal attributes.
- the activity extractor 112 may employ machine learning based classifiers (e.g., a three dimensional convolutional neural networks) to extract activities that are not otherwise detectable from the graph structure.
- the activity extractor 112 may employ a machine learning classifier to partition a node's “active edge” into different unary activities for personnel nodes (e.g., walking, running, lying prone, and gesturing) or for vehicle nodes (e.g., turning left, turning right, making a U-turn, moving, or stationary).
- the activity extractor may extract the cropped video sequence representing the interaction for proximity edges that are not otherwise part of a detected activity and classify it using machine learning.
- proximity edges that are not otherwise part of a detected activity may be defined by the bounding boxes of the participating tracks and it may be determined that the proximity edge between two personnel nodes represents two individuals fighting, talking or shaking hands.
- the system 100 may enable users to automatically summarize, a long, complex video as a compact spatiotemporal semantic graph that can be consumed by an analyst in minutes (e.g., instead of hours), Moreover, the spatiotemporal semantic graph may be searched over or exploited by a variety of other downstream applications.
- activity graphs may feed a wide-variety of applications (e.g., API 118 ), including summarization, exploration, alerting and monitoring, semantic search, anomaly detection, textual report generation and more.
- FIG. 2 is a diagram illustrating a physical environment 200 including a display 210 .
- the display 210 may present (e.g., via a graphic user interface) a spatiotemporal proximity graph 220 according to an aspect of the application.
- spatiotemporal proximity graph 220 may be a directed, attributed multigraph.
- spatiotemporal proximity graph 220 may include one or more nodes representing distinct tracked entities (e.g., a specific person, object, or vehicle).
- the edges between entities may represent a proximity relationship or a significant period of close spatial proximity that is expressed by an edge attribute as a range of contiguous frame numbers.
- a spatiotemporal proximity graph 220 may be generated from video footage.
- Each entity may be represented by a node and each node may include one or more active edges (e.g., loops) that specify a range of frames that the node was active in the scene.
- each node may include one or more proximity edges (e.g., connections to other nodes) that specify a range of frames in which connected nodes are in close spatial proximity during the video.
- nodes may have one or more active edges (e.g., self-loops) that may represent intervals in which a track was active (e.g., visible in the scene). For example, a vehicle leaving a scene and then returning later may be captured by two active edges.
- active edges e.g., self-loops
- tracks generated by the multi-object tracking engine may include sub-classifications or attributes including various fine gran properties of an entity corresponding to each track. Accordingly, generated nodes may also include attributes indicating various fine grain properties of one or more corresponding entities. For example, nodes may include make, model and color for vehicles, or sex, age range and clothing type for personnel. According to some embodiments, attributes associated with the nodes may allow for more precise querying and alerting by downstream applications.
- proximity of one entity to another entity may be determined based on a closeness of track bounding boxes in a pixel space of a video. For example, close proximity in the pixel space of overhead video (e.g., drone, satellite, or flyover footage) or approximately overhead video (e.g., from elevated security cameras) may indicate true spatial proximity in a real-world environment.
- overhead video e.g., drone, satellite, or flyover footage
- approximately overhead video e.g., from elevated security cameras
- two tracks may be considered to be in close proximity in a frame X if (1) the intersection-over-union (IoU) of their two bounding boxes is greater than a first threshold, or (2) the minimum distance between any pair of points on the edges of the bounding boxes is less than a second threshold.
- IoU intersection-over-union
- determining whether the minimum distance between any pair of points on the edges of the bounding boxes is less than a threshold may capture proximity between objects (e.g., personnel) that may communicate or interact without being close enough for their boxes to overlap. Moreover, tracks may be required to be in close-proximity for a minimum number of frames (e.g., user-specified) in order for a proximity edge to be created in the graph. According to some embodiments, proximity intervals may contain brief gaps subject to a maximum number of frames (e.g., user-specified). Moreover, proximity intervals that are sufficiently close together may be merged.
- FIG. 3 is a diagram illustrating a physical environment 300 including a display 310 .
- the display 310 may present (e.g., via a graphic user interface) an activity graph 320 according to an aspect of the application.
- the activity extractor 112 may perform subgraph queries over one or more spatiotemporal proximity graphs (e.g., spatiotemporal proximity graph 220 ) to detect higher level activities.
- the activity extractor 112 may detect activities including “entered” (e.g., a person or object entered a vehicle or building), “exited” (e.g., a person or object exited a vehicle or building), “loaded” (e.g., a person loaded an object into a vehicle or building), “unloaded” (e.g., a person unloaded an object from a vehicle or building), “exchanged” (e.g., a person exchanged an object with another person), “abandoned” (e.g., a person abandoned an object), “picked up” (e.g., a person picked up an (abandoned) object), etc.
- activities including “entered” (e.g., a person or object entered a vehicle or building), “exited” (e.g., a person or object exited a vehicle or building), “loaded” (e.g., a person loaded an object into a vehicle or building), “unloaded” (e.g.,
- activities may be detected using subgraph queries, e.g., provided to the system 100 in the form of activities templates expressed in a human-readable data-serialization language format (e.g., YAML).
- a human-readable data-serialization language format e.g., YAML
- new types of activities may be automatically detected by adding a new template to a configuration file (e.g., ontology data 114 ).
- an “entered activity” may be identified using the following query template:
- the query is specified in Cypher syntax (e.g., used by Neo4j graph database system).
- the query template for “entered activity” may be interpreted as follows: find all people or objects interacting with vehicles, where the person/object disappeared at the same time that their interaction with the vehicle ended, where the vehicle persisted beyond when the person/object disappeared (e.g., to account for camera movement), and where the person/object did not reappear in the next 120 frames ( ⁇ 4 seconds). Create an “Entered” edge between the person/object and the vehicle.
- new activities may be defined in terms of previously defined activities.
- a “loaded activity” e.g., finds instances of people loading objects into vehicles
- the “entered” activity type may be defined using the “entered” activity type:
- query templates for other activity types may be defined in a similar fashion.
- an object ontology tree may be defined that allows templates to refer to large classes of objects by a common ancestor node in the tree.
- FIG. 3 shows an example of an activity graph 320 produced automatically from the spatiotemporal proximity graph 220 of FIG. 2 . As shown in FIG. 3 , the following observations can be made from activity graph 320 :
- Person 1 entered Vehicle 3 at frame 3749 and exited again at frame 9635 . Person 1 subsequently entered and exited the same vehicle two more times during the video and the vehicle left the scene and returned each time.
- Person 1 unloaded two more objects from the same vehicle over the course of the video: Object 7 was unloaded after the vehicle left and returned the first time and Object 17 after the second time. Person 1 was aided by person 4 in unloading Object 7 (at frame 10038 ).
- Persons 20 , 21 and 23 were present on the scene as spectators, not involved in any high-level activities.
- machine learning models may operate on video clips to detect other types of activities that are not evident from the graph structure or that cannot be easily expressed as a graph query.
- unary activities e.g., which can also be thought of as states
- these certain types of unary activities may include a person running, walking, lying prone, gesturing, etc.
- classifiers may be used to break active edges into subintervals based on detected states and machine learning models may be used to verify activities detected by the subgraph queries.
- the tracking engine may make mistakes that lead to false activity detections in a graph.
- the problem of false activity detections may be reduced to one of classification as opposed to detection.
- the problem of classification may be more tractable than end-to-end machine learning based activity detectors that operate on video.
- FIG. 4 illustrates an exemplary flowchart of a method 400 to detect activities in videos based on a spatiotemporal graph representation.
- the method 400 may be performed at a network device, UE, desktop, laptop, mobile device, server device, or by multiple devices in communication with one another.
- the method 400 is performed by processing logic, including hardware, firmware, software, or a combination thereof.
- the method 400 is performed by a processor executing code stored in a computer-readable medium (e.g., memory).
- each track of the plurality of tracks may correspond to on entity (e.g., an object of interest).
- entity e.g., an object of interest
- the tracks may be received from a computer vision-based multi-object tracker.
- each track may include a number of attributes (e.g., a label or an object identifier) associated with the track or the corresponding entity.
- a spatiotemporal proximity graph may be built based on the received plurality of tracks.
- the spatiotemporal proximity graph may include one or more nodes, e.g., where each node represents a distinct tracked entity, and edges between nodes may represent a proximity relationship or a significant period of close spatial proximity between two entities.
- nodes may have one or more “active” edges with attributes indicating a span of start and end time of each interval in which a corresponding entity was present in the scene.
- nodes may include “proximity” edges with attributes indicating a span of frames in which two entities were in close proximity. For example, proximity (e.g., of tracks or entities) may be based on closeness or overlap of track bounding boxes in pixel-space.
- a spatiotemporal relationship between a plurality of the entities may be Identified based on the spatiotemporal proximity graph.
- method 400 may detect higher level activities by performing subgraph matching queries on one or more spatiotemporal proximity graphs.
- one or more subgraph matching queries may be associated with an activity (e.g., entering or exiting a vehicle or building, exchanging objects, etc.).
- an activity graph may be constructed based on the detected activities.
- detected activities may be represented in the activity graph by “activity edges” between nodes participating in the activities.
- activity edges may be represented in the activity graph by “activity edges” between nodes participating in the activities.
- some complex activities may be detected as a combination of previously added activity edges and underlying proximity edges.
- activity graphs may be used by downstream applications to generate natural language reports summarizing a video feed.
- generated natural language reports may be used by analysts to rapidly take note of key events or to determine whether analysts need to view the video.
- activity graph could also be used in a sematic search application to issue powerful queries over large numbers of videos. For example, a semantic search query could include “find an adult male and a child wearing a blue hat entering a red pickup truck”.
- alerting and monitoring may be seen as an extension of search.
- alerts may be expressed as graph queries that run whenever a graph is generated or updated, e.g., triggering internal events or notifications.
- an alert could be set to send a notification when objects are loaded or unloaded from a vehicle in a particular parking lot at a specific time of day.
- pattern-of-life and anomaly detection may also be possible based on alerting and monitoring.
- an interactive visualization tool may enable rapid exploration and understanding of a video based on its activity graph. For example, an analyst may be able to playback an original input video with various overlays and interactive features.
- all tracks may be drawn on top of the video as an overlay (e.g., including a track ID, class label and bounding box).
- a user may select a displayed track and, based on the selection, see a list of all of the upcoming activities in which the selected track was involved.
- selecting an activity in the listing may advance the video playback to the specific frame in which that clicked activity begins. For example, whenever an activity occurs, an alert may pop up on the screen in the general area where the activity is occurring.
- keyboard shortcuts may allow a user to easily jump to next/previous activity, jump to a specific frame number, pause, fast forward or rewind playback.
- an interactive web-based dashboard may render different views of an underlying activity graph or proximity graph (e.g., full graph, person-to-person proximity graph, activities only, a subgraph surrounding a particular entity, etc.). For example, a user may select a specific frame of a video and see a snapshot of an activity graph or spatiotemporal proximity graph for that frame.
- a dashboard may contain a side bar where a user may view an animated gif for each tracked entity in the video or filter the displayed animations by track ID or entity type.
- FIG. 5 is a block diagram of an exemplary hardware/software architecture of a node 500 of a network, such as clients, servers, or proxies, which may operate as an server, gateway, device, or other node in a network.
- the node 500 may include a processor 502 , non-removable memory 504 , removable memory 506 , a speaker/microphone 508 , a keypad 510 , a display, touchpad, and/or indicators 512 , a power source 514 , a global positioning system (GPS) chipset 516 , and other peripherals 518 .
- GPS global positioning system
- the node 500 may also include communication circuitry, such as a transceiver 520 and a transmit/receive element 522 in communication with a communications network 524 .
- the node 500 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.
- the processor 502 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like.
- the processor 502 may execute computer-executable instructions stored in the memory (e.g., memory 504 and/or memory 506 ) of the node 500 in order to perform the various required functions of the node 500 .
- the processor 502 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the node 500 to operate in a wireless or wired environment.
- the processor 502 may run application-layer programs (e.g., browsers) and/or radio-access-layer (RAN) programs and/or other communications programs.
- the processor 502 may also perform security operations, such as authentication, security key agreement, and/or cryptographic operations. The security operations may be performed, for example, at the access layer and/or application layer.
- the processor 502 is coupled to its communication circuitry (e.g., transceiver 520 and transmit/receive element 522 ).
- the processor 502 may control the communication circuitry to cause the node 500 to communicate with other nodes via the network to which it is connected. While FIG. 5 depicts the processor 502 and the transceiver 520 as separate components, the processor 502 and the transceiver 520 may be integrated together in an electronic package or chip.
- the transmit/receive element 522 may be configured to transmit signals to, or receive signals from, other nodes, including servers, gateways, wireless devices, and the like.
- the transmit/receive element 522 may be an antenna configured to transmit and/or receive RF signals.
- the transmit/receive element 522 may support various networks and air interfaces, such as WLAN, WPAN, cellular, and the like.
- the transmit/receive element 522 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example.
- the transmit/receive element 522 may be configured to transmit and receive both RF and light signals.
- the transmit/receive element 522 may be configured to transmit and/or receive any combination of wireless or wired signals.
- the node 500 may include any number of transmit/receive elements 522 . More specifically, the node 500 may employ multiple-input and multiple-output (MIMO) technology. Thus, in an embodiment, the node 500 may include two or more transmit/receive elements 522 (e.g., multiple antennas) for transmitting and receiving wireless signals.
- MIMO multiple-input and multiple-output
- the transceiver 520 may be configured to modulate the signals to be transmitted by the transmit/receive element 522 and to demodulate the signals that are received by the transmit/receive element 522 .
- the node 500 may have multi-mode capabilities.
- the transceiver 520 may include multiple transceivers for enabling the node 500 to communicate via multiple RATs, such as Universal Terrestrial Radio Access (UTRA) and IEEE 802.11, for example.
- UTRA Universal Terrestrial Radio Access
- IEEE 802.11 for example.
- the processor 502 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 504 and/or the removable memory 506 .
- the processor 502 may store session context in its memory, as described above.
- the non-removable memory 504 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device.
- the removable memory 506 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like.
- SIM subscriber identity module
- SD secure digital
- the processor 502 may access information from, and store data in, memory that is not physically located on the node 500 , such as on a server or a home computer.
- the processor 502 may receive power from the power source 514 and may be configured to distribute and/or control the power to the other components in the node 500 .
- the power source 514 may be any suitable device for powering the node 500 .
- the power source 514 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.
- the processor 502 may also be coupled to the GPS chipset 516 , which is configured to provide location information (e.g., longitude and latitude) regarding the current location of the node 500 .
- location information e.g., longitude and latitude
- the node 500 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.
- the processor 502 may further be coupled to other peripherals 518 , which may include one or more software and/or hardware modules that provide additional features, functionality, and/or wired or wireless connectivity.
- the peripherals 518 may include various sensors such as an accelerometer, an e-compass, a satellite transceiver, a sensor, a digital camera (for photographs or video), a universal serial bus (USB) port or other interconnect interfaces, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, an Internet browser, and the like.
- the node 500 may be embodied in other apparatuses or devices, such as a sensor, consumer electronics, a wearable device such as a smart watch or smart clothing, a medical or eHealth device, a robot, industrial equipment, a drone, and a vehicle, such as a car, truck, train, or airplane.
- the node 500 may connect to other components, modules, or systems of such apparatuses or devices via one or more interconnect interfaces, such as an interconnect interface that may comprise one of the peripherals 518 .
- FIG. 6 is a block diagram of an exemplary computing system 600 that may be used to implement one or more nodes (e.g., clients, servers, or proxies) of a network, and which may operate as a server, gateway, device, or other node in a network.
- computing system 600 may include a network adapter 628 in communication with a communications network 630 .
- the computing system 600 may comprise a computer or server and may be controlled primarily by computer-readable instructions, which may be in the form of software, by whatever means such software is stored or accessed. Such computer-readable instructions may be executed within a processor, such as a central processing unit (CPU) 602 , to cause the computing system 600 to effectuate various operations.
- CPU central processing unit
- the CPU 602 is implemented by a single-chip CPU called a microprocessor. In other machines, the CPU 602 may comprise multiple processors.
- a co-processor 604 is an optional processor, distinct from the CPU 602 that performs additional functions or assists the CPU 602 .
- the CPU 602 fetches, decodes, executes instructions, and transfers information to and from other resources via the computer's main data-transfer path, a system bus 606 .
- a system bus 606 connects the components in the computing system 600 and defines the medium for data exchange.
- the system bus 606 typically includes data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus 606 .
- An example of such a system bus 606 is the PCI (Peripheral Component Interconnect) bus.
- Memories coupled to the system bus 606 include RAM 608 and ROM 610 . Such memories include circuitry that allows information to be stored and retrieved.
- the ROM 610 generally contains stored data that cannot easily be modified. Data stored in the RAM 608 may be read or changed by the CPU 602 or other hardware devices. Access to the RAM 608 and/or the ROM 610 may be controlled by a memory controller 612 .
- the memory controller 612 may provide an address translation function that translates virtual addresses into physical addresses as instructions are executed.
- the memory controller 612 may also provide a memory protection function that isolates processes within the system and isolates system processes from user processes. Thus, a program running in a first mode may access only memory mapped by its own process virtual address space. It cannot access memory within another process's virtual address space unless memory sharing between the processes has been set up.
- the computing system 600 may contain a peripherals controller 614 responsible for communicating instructions from the CPU 602 to peripherals, such as a printer 616 , a keyboard 618 , a mouse 620 , and a disk drive 622 .
- peripherals such as a printer 616 , a keyboard 618 , a mouse 620 , and a disk drive 622 .
- a display 624 which is controlled by a display controller 626 , is used to display visual output generated by the computing system 600 .
- Such visual output may include text, graphics, animated graphics, and video.
- the display 624 may be implemented with a CRT-based video display, an LCD-based flat-panel display, gas plasma-based flat-panel display, or a touch-panel.
- the display controller 626 includes electronic components required to generate a video signal that is sent to the display 624 .
Abstract
Description
-
- type: Entered
- #ROLEA Entered ROLEB
- participants:
- ROLEA: person
- ROLEB: vehicle
- ROLEA: object
- ROLEB: vehicle
- template: “MATCH (p:ROLEA)-[ap:Active]->(p)-[x:Proximity]->(v:ROLEB)-[av:Active]->(v)
- WHERE ap.end=x.end AND ay.start<x.start AND ay.end>x.end
- AND NOT EXISTS {MATCH (p)-[ap2:Active]->(p) WHERE 120>ap2.start-x.end>0}
- CREATE (p)-[r:Entered {start: x.end-90, end: x.end, bbox:x.bbox, duration: 90, desc:‘Entered:’+(x.end-90)}]->(v)
- RETURN p, v, r”
-
- type: Loaded
- #ROLEA Loaded ROLEB into ROLEC
- participants:
- —ROLEA: person
- ROLEB: object
- ROLEC: vehicle
- template: “MATCH (p:ROLEA)-[pr:Proximity]->(o:ROLEB)-[en:Entered]->(v:ROLEC)
- WHERE pr.start<en.start AND abs(pr.end-en.end)<25
- CREATE (p)-[r:Loaded {start: en.start, end: en.end, bbox: en.bbox, duration: en.end-en.start, with: v.id, linked_edge: id(en), desc:‘Loaded:’-+en.start}]->(o)
- RETURN p, o, r”
Claims (12)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/120,349 US11907293B2 (en) | 2020-12-14 | 2020-12-14 | Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/120,349 US11907293B2 (en) | 2020-12-14 | 2020-12-14 | Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220188356A1 US20220188356A1 (en) | 2022-06-16 |
US11907293B2 true US11907293B2 (en) | 2024-02-20 |
Family
ID=81942615
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/120,349 Active 2041-08-16 US11907293B2 (en) | 2020-12-14 | 2020-12-14 | Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs |
Country Status (1)
Country | Link |
---|---|
US (1) | US11907293B2 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11676399B1 (en) * | 2022-07-18 | 2023-06-13 | Motional Ad Llc. | Object tracking |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6441846B1 (en) * | 1998-06-22 | 2002-08-27 | Lucent Technologies Inc. | Method and apparatus for deriving novel sports statistics from real time tracking of sporting events |
US6711590B1 (en) * | 1998-07-10 | 2004-03-23 | Canon Kabushiki Kaisha | Linking metadata with a time-sequential digital signal |
US20050162423A1 (en) * | 2004-01-20 | 2005-07-28 | Goggin David E. | Method and apparatus for time series graph display |
GB2493580A (en) * | 2011-08-08 | 2013-02-13 | Vision Semantics Ltd | Method of searching for a target within video data |
US20150363644A1 (en) * | 2014-06-17 | 2015-12-17 | Nantworks, LLC | Activity recognition systems and methods |
US20170039765A1 (en) * | 2014-05-05 | 2017-02-09 | Avigilon Fortress Corporation | System and method for real-time overlay of map features onto a video feed |
-
2020
- 2020-12-14 US US17/120,349 patent/US11907293B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6441846B1 (en) * | 1998-06-22 | 2002-08-27 | Lucent Technologies Inc. | Method and apparatus for deriving novel sports statistics from real time tracking of sporting events |
US6711590B1 (en) * | 1998-07-10 | 2004-03-23 | Canon Kabushiki Kaisha | Linking metadata with a time-sequential digital signal |
US20050162423A1 (en) * | 2004-01-20 | 2005-07-28 | Goggin David E. | Method and apparatus for time series graph display |
GB2493580A (en) * | 2011-08-08 | 2013-02-13 | Vision Semantics Ltd | Method of searching for a target within video data |
US20170039765A1 (en) * | 2014-05-05 | 2017-02-09 | Avigilon Fortress Corporation | System and method for real-time overlay of map features onto a video feed |
US20150363644A1 (en) * | 2014-06-17 | 2015-12-17 | Nantworks, LLC | Activity recognition systems and methods |
Also Published As
Publication number | Publication date |
---|---|
US20220188356A1 (en) | 2022-06-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Chen et al. | An edge traffic flow detection scheme based on deep learning in an intelligent transportation system | |
US11004209B2 (en) | Methods and systems for applying complex object detection in a video analytics system | |
US11156630B2 (en) | Integrated tamper detection system and methods | |
Räty | Survey on contemporary remote surveillance systems for public safety | |
Xu et al. | Video structured description technology based intelligence analysis of surveillance videos for public security applications | |
Hakeem et al. | Video analytics for business intelligence | |
Yun et al. | Vision‐based garbage dumping action detection for real‐world surveillance platform | |
US10558862B2 (en) | Emotion heat mapping | |
CN104113784B (en) | Intelligent television system and its method | |
Avola et al. | Fusing depth and colour information for human action recognition | |
US9489582B2 (en) | Video anomaly detection based upon a sparsity model | |
Nikouei et al. | Kerman: A hybrid lightweight tracking algorithm to enable smart surveillance as an edge service | |
Alam et al. | Video big data analytics in the cloud: A reference architecture, survey, opportunities, and open research issues | |
Gaikwad et al. | Smart surveillance system for real-time multi-person multi-camera tracking at the edge | |
Himeur et al. | Deep visual social distancing monitoring to combat COVID-19: A comprehensive survey | |
Apostolakis et al. | DARLENE–Improving situational awareness of European law enforcement agents through a combination of augmented reality and artificial intelligence solutions | |
Martinez‐Alpiste et al. | Smartphone‐based object recognition with embedded machine learning intelligence for unmanned aerial vehicles | |
US11907293B2 (en) | Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs | |
Kwan-Loo et al. | Detection of violent behavior using neural networks and pose estimation | |
US10282672B1 (en) | Visual content analysis system with semantic framework | |
US20210264153A1 (en) | Machine learning method and apparatus for detection and continuous feature comparison | |
Nguyen et al. | Type-to-track: Retrieve any object via prompt-based tracking | |
CN117217839A (en) | Method, device, equipment and storage medium for issuing media resources | |
Kulbacki et al. | Intelligent video monitoring system with the functionality of online recognition of people’s behavior and interactions between people | |
Bakalos et al. | Fusing RGB and thermal imagery with channel state information for abnormal activity detection using multimodal bidirectional LSTM |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: CACI, INC. - FEDERAL, VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JORGENSEN, ZACHARY;STAUDINGER, TYLER;VISS, CHARLES;SIGNING DATES FROM 20201120 TO 20201207;REEL/FRAME:054766/0660 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |