Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
  • G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/48—Program initiating; Program switching, e.g. by interrupt
  • G06F9/4806—Task transfer initiation or dispatching
  • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
  • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
  • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
  • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
  • G06F9/5083—Techniques for rebalancing the load in a distributed system
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/96—Management of image or video recognition tasks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
  • G06V20/54—Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
  • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
  • G06V20/44—Event detection
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/178—Human faces, e.g. facial parts, sketches or expressions estimating age from face image; using age information for improving recognition

Definitions

• Video cameras are pervasively deployed in society; for instance, it is common for millions of cameras to be deployed in a city. Cameras can be deployed for any number of purposes. For example, some cameras can be installed in or on buildings for surveillance and business intelligence. Other cameras can be deployed to capture video of outdoor scenes (e.g., streets) for traffic control and crime prevention.
• video analytics can have significant resource demands. For instance, to track objects in video, some techniques can process 1 frame per second (or less) on an 8-core machine. Pursuant to another example, techniques that use Deep Neural Networks for object recognition can employ on the order of 30 GFlops to process a single frame. Due to the high processing costs and high data rates of video streams, quality of results returned and lag in time for returning the results for conventional video analytics approaches can be detrimentally impacted.
• Video streams can be captured by multiple cameras and continuously streamed to a video analytics computing system; thus, the video streams can be received at the video analytics computing system.
• Multiple video analytics queries can be executed on the video streams.
• the multiple video analytics queries can be concurrently executed by the video analytics computing system on the video streams as the video streams are continuously streamed to the video analytics computing system.
• the multiple video analytics queries can be executed utilizing resources (e.g., processor, memory) of the video analytics computing system allocated between the multiple video analytics queries. Execution of the multiple video analytics queries can return respective results for the multiple video analytics queries. Further, the results for the multiple video analytics queries can be outputted.
• the video analytics computing system can schedule the multiple video analytics queries. Scheduling of the multiple video analytics queries can be jointly based on resource-quality profiles of the multiple video analytics queries and lag tolerances of the multiple video analytics queries.
• a resource- quality profile of a video analytics query can specify resource demands and result qualities corresponding to candidate multi-dimensional configurations of the video analytics query.
• Scheduling of the multiple video analytics queries can include allocating the resources of the video analytics computing system between the multiple video analytics queries for the execution of each of the multiple video analytics queries. Further, scheduling of the multiple video analytics queries can include selecting multi-dimensional configurations for each of the multiple video analytics queries based on the resources as allocated. Scheduling of the multiple video analytics queries can also include placing the multiple video analytics queries on worker computing devices (e.g., particular machines) of the video analytics computing system for the execution of the multiple video analytics queries.
• worker computing devices e.g., particular machines
• resource-quality profiles of the multiple video analytics queries can be generated. For example, responsive to receiving a particular video analytics query (e.g., at the video analytics computing system), a resource-quality profile of the particular video analytics query can be generated.
• the resource-quality profile can specify resource demands and result qualities corresponding to candidate multi-dimensional configurations of the particular video analytics query.
• the resource-quality profile of the particular video analytics query can be generated by identifying the candidate multi-dimensional configurations utilizing a greedy local search in a resource-quality space, such that the candidate multi-dimensional configurations can be a subset of potential multi-dimensional configurations on a Pareto boundary.
• FIG. 1 illustrates a functional block diagram of an exemplary system that performs video analytics.
• Fig. 2 illustrates an exemplary comparison between fairness-based resource allocation and performance-based resource allocation for dividing resources among multiple video analytics queries.
• FIG. 3 illustrates a functional block diagram of another exemplary system that performs video analytics.
• Fig. 4 illustrates a functional block diagram of an exemplary system showing an exemplary scenario where two video analytics queries are performed.
• Fig. 5 illustrates an example of a profiler component and a scheduler component from Fig. 3 in greater detail
• Fig. 6 is a flow diagram that illustrates an exemplary methodology of performing video analytics.
• Fig. 7 is a flow diagram that illustrates another exemplary methodology of performing video analytics.
• Fig. 8 is a flow diagram that illustrates an exemplary methodology of scheduling multiple video analytics queries.
• FIG. 9 illustrates an exemplary computing device.
• Fig. 10 illustrates an exemplary computing system.
• the term "or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B.
• the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.
• Fig. 1 illustrates a system 100 that performs video analytics.
• the system 100 includes a video analytics computing system 102 that processes video analytics queries on video streams from a plurality of cameras.
• the video analytics computing system 102 can process a video analytics query 104, and a video analytics query 106 (collectively referred to herein as video analytics queries 104-106). It is contemplated that the video analytics computing system 102 can process substantially any number of video analytics queries 104-106.
• the video streams upon which the video analytics queries 104-106 are processed can be from a camera 108, ... , and a camera 110 (collectively referred to herein as cameras 108-110). Accordingly, the cameras 108- 110 are communicatively coupled (e.g., indirectly or directly, via wireless and/or wired connections) with the video analytics computing system 102, thereby enabling the video streams to be sent to the video analytics computing system 102.
• the video analytics computing system 102 supports live video analytics at scale with approximate and delay-tolerant processing.
• the video analytics computing system 102 can control result qualities of the video analytics queries 104-106; a result quality for a video analytics query can be an accuracy of a result returned responsive to the video analytics query relative to an actual value (e.g., a degree to which the actual value is approximated).
• the quality of a result generated responsive to the video analytics query can be based on how accurate the result is compared to the actual value (e.g., accuracy of the car count outputted responsive to the car counting query performed on the video stream relative to an actual car count in the video stream).
• the video analytics queries 104-106 can have various lag tolerances; for instance, the video analytics queries 104-106 can be processed such that production of results of at least some of the video analytics queries 104-106 by the video analytics computing system 102 are delayed in time.
• the video analytics computing system 102 can be or include one or more server computing devices (e.g., the video analytics computing system 102 can be or include a cluster of server computing devices).
• the video analytics computing system 102 can be or include one or more data centers, where a data center includes a plurality of server computing devices.
• One or more data centers can alternatively include the video analytics computing system 102.
• the video analytics computing system 102 can be a distributed computing system.
• the video analytics computing system 102 can include a plurality of computing devices. As depicted in Fig. 1, the video analytics computing system 102 includes a manager computing device 112 and a plurality of worker computing devices, namely, a worker computing device 114, and a worker computing device 116 (collectively referred to herein as worker computing devices 114-116). For example, the video analytics computing system 102 can include 100 worker computing devices 114-116; however, it is to be appreciated that fewer than 100 or more than 100 worker computing devices 114-116 can be included in the video analytics computing system 102.
• the manager computing device 112 can centrally manage the video analytics computing system 102; as described in greater detail herein, for instance, the manager computing device 112 can control the worker computing devices 114-116.
• the video analytics computing system 102 can include more than one manager computing device (e.g., a first manager computing device can generate profiles and a second manager computing device can perform scheduling as described herein).
• the worker computing devices 114-116 can execute the video analytics queries 104-106.
• Video streams can be captured by the multiple cameras 108-110 and continuously streamed to the video analytics computing system 102.
• the video analytics computing system 102 can receive the video streams from the cameras 108-110.
• the video analytics computing system 102 can execute the multiple video analytics queries 104-106 on the video streams.
• the multiple video analytics queries 104-106 can be concurrently executed by the video analytics computing system 102 on the video streams as the video streams are continuously streamed to the video analytics computing system 102.
• the multiple video analytics queries 104-106 can be executed utilizing resources of the video analytics computing system 102 allocated between the multiple video analytics queries 104-106.
• execution of the multiple video analytics queries 104-106 can return respective results for the multiple video analytics queries 104-106.
• the video analytics computing system 102 can further output the results for the multiple video analytics queries 104-106.
• the results can be retained in a log (e.g., in a data repository), transmitted from the video analytics computing system 102 to differing computing device(s) and/or differing computing system(s), or displayed on a display screen; however, it is contemplated that any other type of output is intended to fall within the scope of the hereto appended claims (e.g., audio, visual, or haptic feedback can be provided).
• a notification can be generated to output a result of a video analytics query (e.g., the notification can specify a location at which a license plate is found responsive to execution of a video analytics query that attempts to locate the license plate, the notification can be transmitted to one or more differing computing devices for presentation).
• a result of a video analytics query e.g., data specifying license plates on cars detected to pass through a toll booth
• a differing computing system e.g., an invoicing system
• the manager computing device 112 includes at least one processor 118 and memory 120.
• the processor 118 is configured to execute instructions loaded into the memory 120 (e.g., one or more systems loaded into the memory 120, one or more components loaded into the memory 120).
• the memory 120 includes a system manager component 122 that can be configured to manage resources of the worker computing devices 114-116 for processing the video analytics queries 104- 106.
• the worker computing devices 114-116 also each include at least one processor and memory.
• the worker computing device 114 includes at least one processor 124, ...
• the worker computing device 116 includes at least one processor 126 (collectively referred to as processors 124-126).
• the worker computing device 114 includes memory 128, and the worker computing device 116 includes memory 130 (collectively referred to as memories 128-130).
• the processors 124-126 of the worker computing devices 114-116 can each be configured to execute instructions loaded into the respective memories 128-130.
• worker components can be loaded into the memories 128-130, and executed by the processors 124-126; accordingly, the memory 128 of the worker computing device 114 includes a worker component 132, and the memory 130 of the worker computing device 116 includes the worker component 134 (collectively referred to as worker components 132-134).
• the worker components 132-134 execute the video analytics queries 104-106 utilizing resources (e.g., the processors 124- 126, the memories 128-130) of the worker computing devices 114-116.
• a video analytics query (e.g., each of the video analytics queries 104-106) can be specified as a graph of transforms applied to at least one of the video streams.
• each of the video analytics queries 104-106 can be a directed acyclic graph (DAG) of transforms on live video that is continuously streamed to the video analytics computing system 102.
• DAG directed acyclic graph
• Each of the transforms can process video frames (e.g., a time-ordered stream of messages) in the video stream and can pass corresponding outputs downstream.
• a video frame can be sequentially processed by the transforms such that corresponding outputs of the transforms are passed downstream.
• the worker components 132-134 of the worker computing devices 114-116 perform the transforms as controlled by the system manager component 122 of the manager computing device 112. Moreover, the worker component 132-134 provide feedback to the system manager component 122.
• the video analytics computing system 102 supports large scale analytics of live video, which allows the video analytics queries 104-106 to be submitted with arbitrary vision processors.
• the system manager component 122 of the manager computing device 112 can identify resource-quality profiles of the video analytics queries 104-106; for instance, the resource-quality profiles can be generated without exhaustively exploring a combinatorial space of knob configurations.
• the system manager component 122 can schedule the video analytics queries 104-106 jointly based on the resource-quality profiles and lag tolerances of the video analytics queries 104-106.
• the system manager component 122 can further schedule the video analytics queries 104-106 based on tradeoffs between resource usage, quality, and lag across the multiple video analytics queries 104- 106.
• the system manager component 122 can allocate resources of the worker computing devices 114-116 for processing a relatively large number the video analytics queries 104- 106, with such allocation being effectuated in a relatively short duration of time. According to an example, the system manager component 122 can allocate resources for thousands of video analytics queries 104-106 in hundreds of milliseconds; yet, the claimed subject matter is not so limited. Moreover, the techniques set forth herein can enable utilizing less CPU resources as compared to conventional approaches to generate profiles of queries. Further, the techniques described herein can enable providing results for the video analytics queries 104-106 with enhanced quality as compared to conventional techniques (e.g., traditional fairness-based scheduling approaches).
• the system manager component 122 can generate query resource-quality profiles and allocate resources to the video analytics queries 104-106 to maximize performance on quality and lag; in contrast, various conventional approaches use fair sharing of resources in clusters.
• the system manager component 122 can consider resource- quality tradeoffs with multi-dimensional configurations and variety in quality and lag goals for the video analytics queries 104-106 when managing resources of the video analytics computing system 102. Resource management provided by the system manager component 122 can be beneficial due to high costs of vision processing.
• the video analytics computing system 102 can support processing thousands of video analytics queries 104-106 on live video streams; however, the claimed subject matter is so limited, as more or less video analytics queries 104-106 can be processed. In view of the foregoing, the output of multiple cameras 108-110 can be efficiently analyzed in real-time.
• the system manager component 122 of the manager computing device 112 can manage resources of the video analytics computing system 102 for processing the video analytics queries 104-106 based on resource-quality tradeoffs with multi-dimensional configurations.
• Vision algorithms typically include various parameters, also referred to herein as knobs. Examples of knobs are video resolution, frame rate, and internal algorithmic parameters, such as a size of a sliding window to search for objects in object detectors.
• a combination of knob values is referred to herein as a configuration of a video analytics query. The configuration space grows exponentially with the number of knobs.
• Resource demand can be reduced by changing configurations (e.g., changing a video resolution and sliding window size), but such changes typically lower output quality.
• Some conventional approaches support changing a single knob value, namely, sampling rate; employing a single knob, however, allows these conventional approaches to use established analytical models to calculate quality and resource demand for different sampling rates.
• the video analytics queries 104-106 described herein are more complex black-boxes with more than one knob; thus, the video analytics queries 104-106 do not have known models to calculate the quality and resource demand. Rather, the system manager component 122 can generate the resource-quality profiles of the video analytics queries 104-106.
• the system manager component 122 of the manager computing device 112 can manage resources of the video analytics computing system 102 based on variety in quality and lag goals for the video analytics queries 104-106. While some of the video analytics queries 104-106 may require producing results in real-time, other video analytics queries 104-106 can tolerate lag of various durations (e.g., some queries can allow for lags of many minutes). Accordingly, some resources can be temporarily reallocated from lag tolerant video analytics queries during an interim shortage of resources. The shortage of resources can be due to a burst of new video analytics queries or spikes in resource usage of existing video analytics queries (e.g., due to an increase in a number of cars to track on a road).
• the video analytics queries 104-106 can have a wide variety of quality and lag goals.
• a particular video analytics query for counting cars to control traffic lights can work with moderate quality (e.g., approximate car counts) while providing results with relatively low lag.
• License plate readers at toll routes in contrast, can require relatively high quality (e.g., accuracy to properly recognize characters on the license plates) while tolerating higher lag (e.g., lag of many minutes can be permitted because billing can be delayed).
• license plate readers used for Amber Alerts can require high quality results without lag.
• system manager component 122 can jointly maximize the quality and minimize the lag of the streaming video analytics queries 104-106. In doing so, the system manager component 122 can use the generated resource-quality profiles as well as lag and quality goals.
• the system manager component 122 can allocate resources to each of the video analytics queries 104-106 and can select configurations for such queries based on the allocation (e.g., knob values for the video analytics queries 104-106 can be chosen).
• Scheduling for performance can lead to queries achieving better quality and lag.
• the system manager component 122 of the manager computing device 112 can schedule the video analytics queries 104-106, such that more resources can be allocated to queries whose qualities can have more improvement.
• the system manager component 122 can also schedule the video analytics queries 104-106 to allow queries with built-up lag in the processing to catch up.
• the system manager component 122 can adjust query configuration based on the resources allocated.
• the video analytics component system 102 can address the following challenges. There may not be analytical models for resource demand and quality for a query configuration, and the number of configurations for a query can make it expensive in terms of computing resources, time, and so forth to compute or estimate a resource-quality profile. Moreover, expressing quality and lag goals of individual video analytics queries 104-106 and across the set of video analytics queries 104-106 performed in the video analytics computing system 102 is non-trivial. Further, selecting allocations and configurations can be computationally difficult (e.g., exponential in the number of video analytics queries 104- 106 and knobs).
• the system manager component 122 can separately perform profiling and scheduling.
• the profiling can be performed in an offline phase and the scheduling can be performed in an online phase.
• the system manager component 122 can generate the resource-quality profiles of the video analytics queries 104-106 without exploring the entire combinatorial space of configurations. Rather, greedy search and domain-specific sampling can be used to identify candidate multi-dimensional configurations on the Pareto boundary of a resource-quality profile for a video analytics query.
• the system manager component 122 can consider the candidate multi-dimensional configurations on the Pareto boundary (as opposed to a larger set that includes other multi-dimensional configurations).
• the Pareto boundary can lack multi-dimensional configurations that are worse in both quality and resource demand compared to other multi-dimensional configurations.
• quality and lag goals of a video analytics query can be encoded in a utility function.
• Utility of a video analytics query can be based on achieved result quality and lag; thus, utility can be a weighted combination of the achieved quality and lag, with penalties for violating the goals. Penalties can allow for expressing priorities between the video analytics queries 104-106.
• the system manager component 122 Given utilities of multiple video analytics queries 104- 106, the system manager component 122 can schedule the video analytics queries 104-106 based on a scheduling objective for the video analytics computing system 102.
• Example scheduling objectives include maximizing a minimum utility (which can promote fairness) and maximizing a total utility (which can promote performance); however, other scheduling objectives are intended to fall within the scope of the hereto appended claims.
• Scheduling can be modeled by the system manager component 122 using
• Model-Predictive Control (MPC) to predict a future query lag over a time horizon; the predicted lag can be used in the utility function.
• Scheduling can be performed based on the resource-quality profiles of the video analytics queries 104-106 during allocation, and can allow for lagging queries to catch up. Further, inaccuracies in resource usages in the resource-quality profiles of the video analytics queries 104-106 can be handled.
• Query A is shown in table 200
• query B is shown in table 202
• Table 200 and table 202 include resource demand D and quality Q for three configurations C of each of the queries (Ax and Bx, respectively).
• table 204 shows time, capacity R, and, for each query, chosen configuration C, demand D, allocation A, achieved quality Q, and lag L for the fairness-based resource allocation.
• table 206 shows time, capacity R, and for each query, chosen configuration C, demand/ ) , allocation ⁇ , achieved quality Q, and lag J for the performance- based resource allocation.
• query B achieves a higher quality between times 10 and 22 using the performance-based resource allocation as compared to the fairness-based resource allocation shown in table 204, while a lag does not go beyond 8 seconds (e.g., a permissible lag for the query B).
• an improvement in quality Q for query A is less pronounced than an improvement in quality Q for query B for the same increase in resource demand D (e.g., a 1 unit increase in demand D from 1 to 2 results in a 0.1 unit improvement in quality Q for query A and a 0.2 unit improvement in quality Q for query B).
• resource demand D is a resource level to keep up with an incoming data rate.
• query A cannot tolerate any lag
• query B can tolerate up to 8 seconds of lag.
• Lag can be defined as a difference between a time of a last-arrived frame and a time of a last-processed frame (e.g., how much time's worth of frames are queued and unprocessed).
• table 204 shows the fairness-based resource allocation for the queries A and B. It is noted that a configuration that provides a highest quality Q to keep up with the live stream can be selected (e.g., demand D can be kept at or below allocation A).
• both queries can be allocated A of 2 units initially (at time 0); thus, configurations A2 and B2 can be respectively chosen. Thereafter, between times 10 and 22 when the capacity R drops to 2 units, the queries get an allocation ⁇ of 1 unit each, which leads to configurations Al and B 1 being selected. With the fair allocation approach, no lag is incurred for either of the queries.
• query A can be allocated A of 1 unit and query B can be allocated A of 3 units initially (at time 0); accordingly, configuration B3 can be selected for query B at time 0, thereby achieving higher quality compared to the fairness-based resource allocation (e.g., 0.3 unit increase in quality for query B, while a quality drop for query A is 0.1 unit due to use of configuration Al as opposed to configuration A2).
• Such allocation can be based on the system manager component 122 identifying the value in providing more resources to query B given the resource-quality profile.
• the system manager component 122 can allocate A of 1 unit to each of the queries; however, query configuration B3 can remain for query B. Since the resource demand D of B3 is 3 units, but query B has been allocated only 1 unit at time 10, query B can start to lag. Specifically, for every second, the lag in processing can increase by 2/3 of a second. However, query B can still produce results at quality Q of 0.9 unit, albeit delayed. At time 22, the capacity R recovers and query B has built up a lag of 8 seconds. The system manager component 122 can allocate A of 3 units to query B, but can switch query B to configuration B2 with a demand D of 2 units. This can enable query B to catch up, where for every second it can process 1.5 seconds of video. At time 38, the lag has been eliminated and the system manager component 122 can switch query B back to configuration B3 with quality Q of 0.9 unit.
• the performance-based resource allocation can enable achieving higher total quality by allocating resources based on the resource-quality profiles of the queries.
• the query configurations can be adjusted based on the allocations and the queries can be allowed to catch up to built-in lag.
• FIG. 3 illustrated is another system 300 that performs video analytics.
• the system 300 again includes the video analytics computing system 102, which further includes the manager computing device 112 and the worker computing devices 114- 116.
• the system manager component 122 of the manager computing device 112 can further include a profiler component 302 and a scheduler component 304.
• system manager component 122 of the manager computing device 112 is depicted to include both the profiler component 302 and the scheduler component 304, it is contemplated that the system manager component 122 can be distributed over more than one manager computing device (e.g., a first manager computing device can include the profiler component 302 and a second manager computing device can include the scheduler component 304).
• the profiler component 302 can be configured to generate resource-quality profiles for the video analytics queries 104-106.
• the resource-quality profiles can specify resource demands and result qualities corresponding to candidate multi-dimensional configurations of the video analytics queries 104-106.
• the scheduler component 304 can be configured to schedule the multiple video analytics queries 104-106 based on the resource-quality profiles of the multiple video analytics queries 104-106 and lag tolerances of the multiple video analytics queries 104-106.
• the scheduler component 304 can schedule the multiple video analytics queries 104-106 by allocating the resources of the video analytics computing system 102 (e.g., the resources of the worker computing devices 114-116) between the multiple video analytics queries 104-106 for the execution of each of the multiple video analytics queries 104-106.
• the scheduler component 304 can further be configured to select multi-dimensional configurations for each of the multiple video analytics queries 104-106 based on the resources as allocated.
• the scheduler component 304 can further place the multiple video analytics queries 104-106 on the worker computing devices 114-116 of the video analytics computing system 102 for the execution of the multiple video analytics queries 104-106.
• the scheduler component 304 can push the video analytics queries 104-106 as well as the multi-dimensional configurations to the worker computing devices 114-116.
• the worker components 132-134 of the worker computing device 114-116 can include device manager components and transform process components. More particularly, the worker component 132 of the worker computing device 114 can include a device manager component 306, and the worker component 134 of the worker computing device 116 can include a device manager component 308 (collectively referred to herein as device manager components 306-308). Moreover, the worker component 132 of the worker computing device 114 can include a transform process component 310, and the worker component 134 of the worker computing device 1 16 can include a transform process component 312 (collectively referred to herein as transform process components 310-312).
• the device manager components 306-308 cause the transform process components 310-312 to process transforms that are part of the video analytics queries 104- 106.
• the device manager components 306-308 can autonomously handle short-term fluctuations in the video analytics queries 104-106 and resources. Moreover, the device manager components 306-308 can periodically report resource utilization as well as status of the running transforms back to the system manager component 122 of the manager computing device 112. Accordingly, the scheduler component 304 of the system manager component 122 can use the reported information to allocate resources to the video analytics queries 104-106.
• the system manager component 122 as well as the device manager components 306-308 can be separate from the query data path; thus, videos from the cameras 108-110 can be streamed to decoding transforms processed by the transform process components 310-312, while control data can flow between the system manager component 122 and the device manager components 306-308.
• the system 400 includes the system manager component 122, the worker component 132, and the worker component 134.
• the manager computing device 112 can include the system manager component 122
• the worker computing device 114 can include the worker component 132
• the worker computing device 116 can include the worker component 134.
• Transforms of a video analytics query can include at least a first transform and a second transform.
• the first transform and the second transform can be placed on a common worker computing device (of the video analytics computing system 102) for execution.
• the first transform can be placed on a first worker computing device (of the video analytics computing system 102) for execution
• the second transform can be placed on a second worker computing device (of the video analytics computing system 102) for execution.
• a time-ordered stream of messages can be inputted to each of the transforms, and the transforms can provide outputs to downstream transforms.
• a first video analytics query can be or include a pipeline of transforms including a decode transform 402, a background subtraction transform 404, a track object transform 406, and a classify object transform 408.
• a second video analytics query can be or include a pipeline of transforms including a decode transform 410 and a detect license plates transform 412.
• the first video analytics query can be run across two machines
• the first video analytics query can be executed by the worker component 132 of the worker computing device 114 and the worker component 134 of the worker computing device 116. More particularly, the decode transform 402 and the background subtraction transform 404 are placed on the worker computing device 114 for execution by the worker component 132, and the track object transform 406 and the classify object transform 408 are placed on the worker computing device 116 for execution by the worker component 134. Further, the second video analytics query can be executed by one machine (e.g., the worker component 134 of the worker computing device 116 can execute the second video analytics query). Thus, the decode transform 410 and the detect license plates transform 412 are placed on the worker computing device 116 for execution by the worker component 134.
• a video stream can be inputted from a camera 414 to the worker computing device 114, such that the video stream is provided to the decode transform 402.
• the video stream received from the camera 414 can be processed, such that the decode transform 402, the background subtraction transform 404, the track object transform 406, and the classify object transform 408 are performed on frames of the video stream (e.g., the transforms 402-408 can be sequentially performed on a frame of the video stream as represented by the arrows between the transforms 402-408 in Fig. 4).
• an output of the background subtraction transform 404 is transmitted from the worker component 132 (e.g., the worker computing device 114) to the worker component 134 (e.g., the worker computing device 116).
• a video stream can be inputted from a camera 416 to the worker computing device 116, such that the video stream is provided to the decode transform 410.
• the video stream received from the camera 416 can be processed, such that the decode transform 410 is performed on frames of the video stream and then the detect license plates transform 412 is performed on the output of the decode transform 410.
• Fig. 5 illustrated is an example of the profiler component 302 and the scheduler component 304 of the system manager component 122 in greater detail.
• the profiler component 302 can implement offline profiling, and the scheduler component 304 can implement online scheduling.
• the profiler component 302 can efficiently generate resource-quality profiles for queries.
• the profiler component 302 can generate a resource-quality profile 504 (e.g., a query profile) for a query 502 (e.g., one of the video analytics queries 104-106).
• the profiler component 302 can generate the resource-quality profile 504 of the video analytics query 502 by identifying candidate multi-dimensional configurations utilizing a greedy local search in a resource- quality space.
• the candidate multi-dimensional configurations can be a subset of potential multi-dimensional configurations. Further, the candidate multi-dimensional configurations can be on a Pareto boundary.
• the profiler component 302 can generate the resource- quality profile 504 with a small number of configurations on the Pareto curve, which can reduce the configurations to be considered by the scheduler component 304.
• the scheduler component 304 in the online phase, can allocate resources (e.g., of the video analytics computing system 102) between multiple video analytics queries (e.g., the video analytics queries 104-106) for execution of the video analytics queries.
• the scheduler component 304 can also place the multiple video analytics queries on the worker components 132-134 (e.g., the worker computing device 114-116) for execution of the multiple video analytics queries.
• the scheduling of the multiple video analytics queries can be periodically adjusted by the scheduler component 304 over time based on changes in resource demands across the multiple video analytics queries and resource capacity (e.g., of the video analytics computing system 102).
• resource capacity e.g., of the video analytics computing system 1012.
• the scheduler component 304 can consider the running video analytics queries and can adjust the resource allocation, machine placement, and configurations based on the profiles, changes in demand, and/or capacity.
• the quality and lag requirements of each individual query can be encoded into a utility function, as described in greater detail herein.
• the performance goal across video analytics queries can be specified as maximizing the minimum utility or the sum of utilities.
• the scheduler component 304 can start, stop, migrate queries and/or transforms that are effectuated utilizing the worker components 132-134. Moreover, the scheduler component 304 can periodically change configurations for the video analytics queries executed by the worker components 132-134. The scheduler component 304 can further effectuate periodic resource changes, such that resources utilized for executing the queries can be changed over time. The worker components 132-134 can further report machine and/or query statistics back to be scheduler component 304 (e.g., reported back to the system manager component 122). The reporting can be performed periodically.
• the system manager component 122 can start executing the newly submitted query utilizing a default profile (e.g., the default profile can be for a substantially similar query performed on a video stream from a different one of the cameras 108-110).
• a default profile e.g., the default profile can be for a substantially similar query performed on a video stream from a different one of the cameras 108-110.
• the scheduler component 304 can initially schedule the newly submitted query based on the default profile.
• the profiler component 302 can perform offline profiling of the newly submitted video analytics query.
• the profiler component 302 can select a subset of potential multi-dimensional configurations (e.g., on a Pareto boundary) from a resource-quality space. Moreover, the profiler component 302 can compute a resource-quality profile T k (e.g., resource demands and result qualities of the selected configurations) for the query k.
• the profile can be computed either against a labeled dataset or using an initial part of a streaming video relative to a predetermined query configuration known to produce high-quality results.
• An example of a video analytics query is a license plate reader query that reads license plates from a video stream.
• the query scans a video frame to detect potential plates; thereafter, text on the plates can be recognized using optical character recognition.
• each frame can be utilized to provide a higher quality detection of license plates. Reducing the resolution and processing only a subset of the frames (e.g., sampling rate of 0.25) can reduce resource demand, but can also reduce the quality of the output (e.g., resulting in missed or incorrectly read license plates).
• Knobs that can be employed for the license plate reader query can include frame resolution and sampling rate as well as MinSize, Step, and MaxSize. For instance, the query can scan an image for license plates of size MinSize, then multiplicatively increase the size by Step; the foregoing can be repeated until the size reaches MaxSize. The set of potential license plates can then be sent to an optical character recognizer.
• qualities of configurations e.g., combinations of the five knobs above
• the harmonic mean between precision and recall Accordingly, 0 and 1 represent the lowest and highest qualities. For example, increasing MinSize or decreasing MaxSize can reduce the resources needed, but can result in some plates being missed and quality being decreased.
• the profiler component 302 can implement a greedy local search to identify configurations with high quality Q and low demand D. Notation used herein for a video analytics query k (e.g., from the video analytics queries 104-106) is set forth below.
• One or more exemplary enhancements can increase the efficiency of the search.
• the profiler component 302 can cache intermediate results in a DAG of a video analytics query; the profiler component 302 can reuse the intermediate results in evaluating configurations with overlapping knob values.
• hyperparameter searches can be used to enhance efficiency. Yet, it is to be appreciated that the claimed subject matter is not limited to the foregoing examples.
• the profiler component 302 can identify a subset of configurations that are on the Pareto boundary T of the resource-quality space. Let Q(c) be the quality and D (c) be the resource demand under configuration c. If c x and c 2 are two configurations such that ( c i) > Q (c 2 ) and £> ( c i) ⁇ D (c 2 ), then c 2 may not be useful in practice; under the foregoing scenario, c x is better than c 2 in both quality and resource demand. Accordingly, the profiler component 302 can extract the Pareto boundary of the explored configurations, which is referred to herein as the resource-quality profile T of the video analytics query.
• the scheduler component 304 can consider utilities of individual video analytics queries 104-106 and cluster-wide scheduling objective.
• the scheduler component 304 can periodically perform resource allocation and query placement. As part of resource allocation, it can be assumed that the video analytics computing system 102 is an aggregate bin of resources; the scheduler component 304, for example, can use an efficient scheduling heuristic to maximize cluster-wide performance by adjusting query allocation and configuration.
• the scheduler component 304 can place new queries (e.g., video analytics queries newly received by the manager computing system 1 12) to the worker computing devices 1 14-1 16 in the video analytics computing system 102 and/or migrate existing queries between the worker computing devices 1 14-1 16.
• Preferences can be set for each of the video analytics queries 104-106.
• the preferences can specify desired quality and lag.
• the preferences can set a minimum quality goal (Q ) .
• the preferences can also specify how much a query benefits from higher quality than the goal.
• the preferences can set a maximum lag goal (L M ).
• the maximum lag goal can indicate a maximum amount of tolerated lag and how sensitive the query is to violations of this goal.
• the foregoing preferences can be encoded in utility functions.
• the utility function for a query can have the following form, where (x)+ is the positive part of x.
• Query index k can be omitted for clarity.
• the second term, U Q describes how the utility responds to achieved quality Q above Q M , the soft quality goal.
• a multiplier Q and the minimum quality goal Q M are query specific and can be set based on the application analyzing the video. Results with quality below the minimum quality goal Q M are typically not useful.
• the third term, U L represents a penalty for results arriving later than the maximum lag goal L M . Further, a multiplier L can be in units of (1/second), making U L dimensionless.
• lag is a difference between a current time and an arrival time of a last processed frame.
• the lag is 15 minutes.
• Scheduling objectives can be utilized to define utility or performance of the entire video analytics computing system 102.
• the scheduling objective can be based on maximizing the sum of utilities.
• the scheduling objective can be based on a maximum-minimum fairness over utilities.
• the scheduler component 304 can allocate resources a k to the queries and can select multi-dimensional configurations for such queries (c k E T k ).
• the scheduler component 304 can periodically run (e.g., every few seconds) and can react to arrival of new queries, changes in query demand and lag, and changes in resource capacity of the video analytics computing system 102 (e.g., due to types of jobs other than execution of the video analytics queries 104-106).
• the scheduler component 304 can perform scheduling utilizing MPC.
• the scheduler component 304 can aim to maximize the minimum or sum of query utilities, which can depend on quality and lag. It is noted that while query quality can be instantaneously controlled by adjusting the configuration for the query, query lag accumulates over time if less resources than query demand is allocated.
• the scheduler component 304 can be employed to predict lags of the multiple video analytics queries 104-106 over a time horizon.
• the multiple video analytics queries 104-106 can be scheduled by the scheduler component 304 based on the predicted lags. Accordingly, the scheduler component 304, rather than optimizing the current performance, can optimize performance in the near future.
• the scheduling problem can be formulated using the MPC framework, where cluster performance over a time horizon Jean be modeled as a function of query configuration and allocation. In each step, the scheduler component 304 can select the configuration and allocation to maximize performance over the near future.
• the scheduler component 304 can predict query lag as set forth below:
• L k t+T an expected lag after time T, is a current lag L k t plus the lag that accumulates due to arrival of new data minus the data processed in time T.
• Time horizon J is a parameter that determines how far into the future the scheduling decision is made by the scheduler component 304.
• the predicted lag L k t+T can be utilized in the utility function to obtain a predicted utility.
• the scheduler component 304 can employ a scheduling heuristic for maximizing the sum of utilities to allocate resources. It is noted that the sum of allocated resources a k cannot exceed resource capacity R of the video analytics computing system 102.
• Maximizing the sum of utilities can be a variant of the knapsack problem, where it is attempted to include the queries at different allocation and configuration to maximize the total utility. Maximization can result in a distribution of resources.
• queries can be preferred that have a highest value of u k /a k (e.g., queries that receive a largest increase in utility normalized by resource spent).
• the scheduler component 304 can utilize the following scheduling heuristic.
• the scheduler component 304 can consider increasing a t (for queries i) by ⁇ (e.g., a step increase such as 1% of a core) and can consider configurations of q E P t .
• the scheduler component 304 can select the query i (and the corresponding configuration q) with a largest increase in utility.
• the scheduler component 304 can repeat the foregoing until resources are exhausted or until a best configuration for each query is selected.
• the scheduler component 304 can employ a scheduling heuristic for maximizing the minimum utility to allocate resources.
• a scheduling heuristic for maximizing the minimum utility to allocate resources.
• the scheduler component 304 can employ the following scheduling heuristic.
• the query i argmin fc i/ fc ( ⁇ 2 fc (c fc ), L fcjt+7 .) with a lowest utility can be selected to increase its allocation by a ⁇ (e.g., a step increase such as 1% of a core).
• the scheduler component 304 can compute a configuration q as argmax ceJ >.i/ i ((2 i (c), L i t+T ). The foregoing can be repeated by the scheduler component 304 until resources are exhausted or until a best configuration for each query is selected.
• the scheduler component 304 can determine placement of new queries and migration of existing queries.
• the suitability of placing a query q on a machine m can be quantified by the scheduler component 304 based on scores for the following goals: high utilization, load balancing, and spreading low lag queries.
• High utilization in the video analytics computing system 102 can be achieved by packing queries into machines, thereby minimizing fragmentation and wastage of resources.
• An alignment of a query relative to a machine can be defined using a weighted dot product p (e.g., p G [0,1]) between a vector of the machines available resources and the resource demands of the query.
• the query can be placed by the scheduler component 304 based on lag spreading. It can be desired to not concentrate a significant number of low-lag queries on a machine; accordingly, slack can be provided to accumulate lag for other queries (other than the low-lag queries) when resources are scarce without having to resort to migration of queries or violations of lag goals L M . The foregoing can be achieved by maintaining high average L M on each machine. Thus, the scheduler component 304 can compute I E [0,1] as the average L M after placing the query q on the machine m.
• a final score s q m computed by the scheduler component 304 can be an average of the above three scores; however, the claimed subject matter is not so limited (e.g., other scores can additionally or alternatively be used, all three of the foregoing scores need not be used by the scheduler component 304).
• For each new query q it can be placed on a machine with a largest s q m .
• migration from the machine m 0 to a new machine m 1 can be supported when the score improves above a threshold ⁇ (e.g., s(q, - s(q, m 0 ) > ⁇ ).
• the system manager component 122 can track a misestimation of the resource demands specified in the resource-quality profile of a particular video analytics query during execution of the particular video analytics query over time. Accordingly, the particular video analytics query can be scheduled by the scheduler component 304 based on the misestimation.
• the system manager component 122 e.g., the scheduler component 304) can incorporate the misestimation ⁇ in the lag predictor described above; for example, the misestimation can be included as follows:
• L k,t+T (a k , c k ) L t + T - T - ⁇ - ⁇ ⁇ j.
• machine level scheduling can be implemented by the device manager components 306-308. For instance, if a query fits on a single worker computing device (e.g., one of the worker computing devices 1 14-1 16), then the device manager component of such worker computing device can respond to changes in demand or lag at the machine level, without waiting for a decision to be made by the system manager component 122. Accordingly, the device manager component can execute the allocation described above, which can make the scheduling logic more scalable. Following this example, cluster wide scheduling performed by the scheduler component 304 can continue to perform allocation for purposes of determining query placement and migration, while the machine level scheduling can be implemented by the device manager components 306-308.
• the video analytics queries 104-106 can include a DAG of transforms that can be placed across multiple worker computing devices 1 14-1 16.
• the query resource allocation a k can be distributed to individual transforms based on per-transform resource demands.
• the individual transforms can then be placed to machines by the scheduler component 304, while accounting for the expected data flow across machines and network link capacities.
• the claimed subject matter is not limited to the foregoing example.
• Figs. 6-8 illustrate exemplary methodologies relating to performing video analytics. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein.
• the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media.
• the computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like.
• results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like.
• Fig. 6 illustrates a methodology 600 of performing video analytics.
• video streams can be received at a video analytics computing system.
• the video streams can be captured by multiple cameras and continuously streamed to the video analytics computing system.
• multiple video analytics queries can be executed on the video streams.
• the multiple video analytics queries can be concurrently executed by the video analytics computing system on the video streams as the video streams are continuously streamed to the video analytics computing system.
• the multiple video analytics queries can be executed utilizing resources of the video analytics computing system allocated between the multiple video analytics queries. Execution of the multiple video analytics queries can return respective results for the multiple video analytics queries.
• the results for the multiple video analytics queries can be outputted. [00104] Turning to Fig.
• a methodology 700 of performing video analytics can be performed at 702 .
• a video analytics query can be received at a video analytics computing system.
• a resource-quality profile of the video analytics query can be generated.
• the resource-quality profile can specify resource demands and result qualities corresponding to candidate multi-dimensional configurations of the video analytics query.
• the video analytics query can be executed on a video stream utilizing resources of the video analytics computing system based on the resource-quality profile.
• resources of a video analytics computing system can be allocated between the multiple video analytics queries for execution of each of the multiple video analytics queries.
• the multiple video analytics queries can concurrently be executed by the video analytics computing system on video streams as the video streams are streamed to the video analytics computing system from multiple camera.
• multi-dimensional configurations for each of the multiple video analytics queries can be selected based on the resources as allocated.
• the multiple video analytics queries can be executed utilizing the multi-dimensional configurations.
• the multiple video analytics queries can be placed on worker computing devices of the video analytics computing system for the execution of the multiple video analytics queries.
• the video analytics computing system 102 can include the computing device 900 (e.g., the computing device 900 can be the manager computing device, the computing device 900 can be one of the worker computing devices 114-116).
• the computing device 900 can be communicatively coupled with the video analytics computing system 102 (e.g., the computing device 900 can send a video analytics query to the video analytics computing system 102, the computing device 900 can receive a notification generated by the video analytics computing system 102).
• the computing device 900 includes at least one processor 902 that executes instructions that are stored in a memory 904.
• the instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above.
• the processor 902 may access the memory 904 by way of a system bus 906.
• the memory 904 may also store video analytics queries, resource-quality profiles, video streams (or portions thereof), and so forth.
• the computing device 900 additionally includes a data store 908 that is accessible by the processor 902 by way of the system bus 906.
• the data store 908 may include executable instructions, video analytics queries, resource-quality profiles, video streams (or portions thereof), etc.
• the computing device 900 also includes an input interface 910 that allows external devices to communicate with the computing device 900. For instance, the input interface 910 may be used to receive instructions from an external computer device, from a user, etc.
• the computing device 900 also includes an output interface 912 that interfaces the computing device 900 with one or more external devices. For example, the computing device 900 may display text, images, etc. by way of the output interface 912.
• the external devices that communicate with the computing device 900 via the input interface 910 and the output interface 912 can be included in an environment that provides substantially any type of user interface with which a user can interact.
• user interface types include graphical user interfaces, natural user interfaces, and so forth.
• a graphical user interface may accept input from a user employing input device(s) such as a keyboard, mouse, remote control, or the like and provide output on an output device such as a display.
• a natural user interface may enable a user to interact with the computing device 900 in a manner free from constraints imposed by input device such as keyboards, mice, remote controls, and the like. Rather, a natural user interface can rely on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and so forth.
• the computing device 900 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device 900.
• the computing system 1000 can be or include the video analytics computing system 102. Additionally or alternatively, the video analytics computing system 102 can be or include the computing system 1000.
• the computing system 1000 includes a plurality of server computing devices, namely, a server computing device 1002, and a server computing device 1004 (collectively referred to as server computing devices 1002-1004).
• the server computing devices 1002-1004 can be or include the manager computing device 112 and the worker computing devices 114-116.
• the server computing device 1002 includes at least one processor and a memory; the at least one processor executes instructions that are stored in the memory.
• the instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above.
• the server computing device 1002 Similar to the server computing device 1002, at least a subset of the server computing devices 1002-1004 other than the server computing device 1002 each respectively include at least one processor and a memory. Moreover, at least a subset of the server computing devices 1002-1004 include respective data stores.
• Processor(s) of one or more of the server computing devices 1002-1004 can be or include the processor 118 and the processors 124-126. Further, a memory (or memories) of one or more of the server computing devices 1002-1004 can be or include the memory 120 and the memories 128-130.
• the computing system 1000 further includes various network nodes 1006 that transport data between the server computing devices 1002-1004. Moreover, the network nodes 1002 transport data from the server computing devices 1002-1004 to external nodes (e.g., external to the computing system 1000) by way of a network 1008. The network nodes 1002 also transport data to the server computing devices 1002-1004 from the external nodes by way of the network 1008.
• the network 1008, for example, can be the Internet, a cellular network, or the like.
• the network nodes 1006 include switches, routers, load balancers, and so forth.
• a fabric controller 1010 of the computing system 1000 manages hardware resources of the server computing devices 1002-1004 (e.g., processors, memories, data stores, etc. of the server computing devices 1002-1004).
• the fabric controller 1010 further manages the network nodes 1006.
• the fabric controller 1010 manages creation, provisioning, de-provisioning, and supervising of virtual machines instantiated upon the server computing devices 1002-1004.
• Example 1 A method of performing video analytics, comprising: receiving video streams at a video analytics computing system, the video streams being captured by multiple cameras and continuously streamed to the video analytics computing system; executing multiple video analytics queries on the video streams, the multiple video analytics queries concurrently being executed by the video analytics computing system on the video streams as the video streams are continuously streamed to the video analytics computing system, the multiple video analytics queries being executed utilizing resources of the video analytics computing system allocated between the multiple video analytics queries, and execution of the multiple video analytics queries returns respective results for the multiple video analytics queries; and outputting the results for the multiple video analytics queries.
• Example 2 The method according to Example 1, further comprising: scheduling the multiple video analytics queries, wherein scheduling the multiple video analytics queries further comprises: allocating the resources of the video analytics computing system between the multiple video analytics queries for the execution of each of the multiple video analytics queries; and selecting multi-dimensional configurations for each of the multiple video analytics queries based on the resources as allocated.
• Example 3 The method according to Example 2, wherein scheduling the multiple video analytics queries further comprises: placing the multiple video analytics queries on worker computing devices of the video analytics computing system for the execution of the multiple video analytics queries.
• Example 4 The method according to any of Examples 1-3, further comprising: scheduling the multiple video analytics queries jointly based on resource- quality profiles of the multiple video analytics queries and lag tolerances of the multiple video analytics queries.
• Example 5 The method according to Example 4, further comprising: generating a resource-quality profile of a particular video analytics query, the resource- quality profile specifies resource demands and result qualities corresponding to candidate multi-dimensional configurations of the particular video analytics query.
• Example 6 The method according to Example 5, wherein generating the resource-quality profile of the particular video analytics query further comprises: identifying the candidate multi-dimensional configurations utilizing a greedy local search in a resource- quality space, the candidate multi-dimensional configurations being a subset of potential multi-dimensional configurations, and the candidate multi-dimensional configurations being on a Pareto boundary.
• Example 7 The method according to any of Examples 5-6, further comprising: tracking a misestimation of the resource demands specified in the resource- quality profile of the particular video analytics query during execution of the particular video analytics query over time; wherein the particular video analytics query is further scheduled based on the misestimation.
• Example 8 The method according to any of Examples 4-7, further comprising periodically adjusting the scheduling of the multiple video analytics queries over time based on changes in resource demands across the multiple video analytics queries and resource capacity of the video analytics computing system.
• Example 9 The method according to any of Examples 4-8, further comprising: predicting lags of the multiple video analytics queries over a time horizon; wherein the multiple video analytics queries are further scheduled based on the predicted lags.
• Example 10 The method according to any of Examples 1-9, wherein a video analytics query is specified as a graph of transforms applied to at least one of the video streams, and wherein a video frame in the at least one of the video streams is sequentially processed by the transforms such that corresponding outputs of the transforms are passed downstream.
• Example 11 The method according to Example 10, wherein: the transforms comprise at least a first transform and a second transform; the first transform is placed on a first worker computing device of the video analytics computing system for execution; and the second transform is placed on a second worker computing device of the video analytics computing system for execution.
• Example 12 The method according to any of Examples 1-11, further comprising: allocating the resources of the video analytics computing system between the multiple video analytics queries for the execution of each of the multiple video analytics queries based on: utilities of the multiple video analytics queries, wherein the utilities are based on achieved result quality and lag of the multiple video analytics queries; and a scheduling objective for the video analytics computing system.
• Example 13 The method according to Example 12, wherein the resources are allocated utilizing a scheduling heuristic for one of maximizing a minimum utility or maximizing a total utility.
• Example 14 The method according to any of Examples 1-13, further comprising: migrating a particular video analytics query between worker computing devices of the video analytics computing system for execution of the particular video analytics query.
• Example 15 A video analytics computing system, comprising: at least one processor; and memory that comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including: receiving a video analytics query at the video analytics computing system; responsive to receipt of the video analytics query, generating a resource-quality profile of the video analytics query, the resource-quality profile specifies resource demands and result qualities corresponding to candidate multi-dimensional configurations of the video analytics query; and executing the video analytics query on a video stream utilizing resources of the video analytics computing system based on the resource-quality profile.
• Example 16 The video analytics computing system according to Example
• the memory further comprising computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including: identifying the candidate multi-dimensional configurations utilizing a greedy local search in a resource-quality space, the candidate multi-dimensional configurations being a subset of potential multi-dimensional configurations, and the candidate multi-dimensional configurations being on a Pareto boundary.
• Example 17 The video analytics computing system according to any of
• Examples 15-16 the memory further comprising computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including: scheduling multiple video analytics queries based on resource-quality profiles and lag tolerances of the multiple video analytics query, wherein the multiple video analytics queries comprise the video analytics query, and the resource-quality profiles comprise the resource-quality profile.
• Example 18 A video analytics computing system, comprising: at least one processor; and memory that comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including: allocating resources of the video analytics computing system between multiple video analytics queries for execution of each of the multiple video analytics queries, the multiple video analytics queries concurrently being executed by the video analytics computing system on video streams as the video streams are streamed to the video analytics computing system from multiple camera; and selecting multi-dimensional configurations for each of the multiple video analytics queries based on the resources as allocated, the multiple video analytics queries being executed utilizing the multi-dimensional configurations.
• Example 19 The video analytics computing system according to Example
• the memory further comprising computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including: placing the multiple video analytics queries on worker computing devices of the video analytics computing system for the execution of the multiple video analytics queries.
• Example 20 The video analytics computing system according to any of
• Examples 18-19 wherein the resources of the video analytics computing system are allocated based on resource-quality profiles of the multiple video analytics queries and lag tolerances of the multiple video analytics queries.
• the terms "component” and "system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor.
• the computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.
• Computer-readable media includes computer-readable storage media.
• a computer-readable storage media can be any available storage media that can be accessed by a computer.
• such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.
• Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media.
• Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium.
• the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave
• coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave
• the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave
• the functionality described herein can be performed, at least in part, by one or more hardware logic components.
• illustrative types of hardware logic components include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Software Systems (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• General Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=62064709

Family Applications (1)

Country Status (4)

Families Citing this family (20)

Family Cites Families (23)

• 2016
  • 2016-12-08 US US15/373,301 patent/US10223604B2/en active Active
• 2017
  • 2017-10-31 CN CN201780066717.8A patent/CN110192393B/zh active Active
  • 2017-10-31 WO PCT/US2017/059135 patent/WO2018085210A1/en unknown
  • 2017-10-31 EP EP17798058.8A patent/EP3535976A1/de not_active Ceased

Also Published As

Similar Documents

Legal Events