KR20220151224A - Method and system for adaptive virtual broadcasting of digital content - Google Patents

Abstract

The virtual broadcasting system of the present invention optimizes the routing of digital content between nodes along overlay networks that are dynamically reconfigured based on predictions of frequently changing congestion levels of component interconnections within the underlying network. Concerning the delivery of streaming video to a large number of concurrent users over the Internet, the present invention predicts the limited capacity of congested ASN peering points, congestion levels across those ASN peering points, and based on the predictions , by using deep learning techniques to optimize the routing of video content along dynamically reconstructed overlay networks. The virtual broadcasting system handles scheduled as well as unscheduled events, streams live as well as recorded events, and streams those events in a highly scalable manner that maintains consistent QoE across large numbers of concurrent viewers with minimal delay. stream in real time.

Description

METHOD AND SYSTEM FOR ADAPTIVE VIRTUAL BROADCASTING OF DIGITAL CONTENT

CROSS REFERENCES OF RELATED APPLICATIONS

This application claims priority to US Provisional Patent Application Serial No. 62/096,938, filed on December 26, 2014, the disclosure of which is hereby incorporated by reference as if set forth herein in its entirety.

The present invention relates generally to an overlay network architecture for conveying digital content between nodes of an underlying network, and more particularly, based on forecasts of frequently changing congestion levels at component interconnects within the underlying network. A virtual broadcasting system that optimizes the routing of digital content between nodes along dynamically reconfigured overlay networks.

network congestion

As wired and wireless network traffic continues to grow exponentially, the finite capacity of shared links or interconnections between components within a network is becoming an increasingly important and vexing problem. Moreover, because the level of congestion on these shared links is dynamic and subject to great variability as network traffic changes, such congestion is difficult to measure at any given time and particularly difficult to predict, even in the short term.

This problem is somewhat similar to that of traffic congestion at intersections of shared roads and highways in increasingly populated areas. Whereas existing GPS navigation and traffic control systems measure the current congestion at these intersections and calculate optimal routes to reroute individual drivers around this congestion, the optimal route for any particular driver Their ability to predict routes in advance is hampered by the fluctuating nature of traffic congestion.

When a single company like Netflix accounts for more than a third of peak Internet traffic, companies that simultaneously deliver digital information (especially large amounts of linear data) over the Internet have to deal with some of the increasingly fluctuating nature of Internet congestion. . Similarly, as mobile voice and data usage proliferates, the limited availability of regulated RF spectrum is particularly important for companies developing high-bandwidth mobile applications.

Although a specific application of the present invention is described herein in connection with the delivery of streaming video over the Internet to a large number of concurrent users, the principles of the present invention are not limited to the limited capacity of shared links between network components in a digital format. The same applies in many other situations that constrain the routing of any type of information (eg audio, video, 3D model, etc.) that can be converted to . Other potential applications of the present invention (with respect to the congestion level of the shared links within the underlying network at any given point in time) include, for example, VoIP, corporate videoconferencing, virtual reality, multiplayer gaming, and a variety of other Includes bandwidth intensive applications.

As will be discussed in more detail below, the present invention does not “solve” the problem of limited capacity or “network congestion” at component links within an underlying network, such as the Internet, but instead at those links. It uses its limited capacity efficiently by monitoring and analyzing network traffic across its links to optimize the routing of digital content between nodes in overlay networks that are dynamically reconfigured based on predictions of congestion levels in the network.

video streaming event

Since the advent of the Internet and IP-based routing, many approaches to streaming video over the Internet have emerged. Before discussing their relative strengths and weaknesses, it is helpful to take a step back and look at the problem to be solved. In order to distribute video content over the Internet, the video content must first be captured and digitized. You can think of video content as an "event" that is captured "live" (or digitally generated) and distributed over the Internet. Reference herein to a video event includes the capture or generation of both video and audio, as well as any associated metadata.

Video events can be scheduled or unscheduled. For example, the "Super Bowl" is a scheduled event in that the time of its occurrence is known in advance, while other events (eg, natural disasters, toddler's first steps, or even video on demand (VOD)) have been forewarned. is unscheduled in that it may occur with little or no

Video content can be captured as a whole to generate a digitized video file, which is then distributed over the Internet when any other type of file is transferred (e.g., via file transfer protocol (FTP)). do. However, this "file transfer" approach imposes a delay on the receiver viewing (playing) the video content - that is, the receiver must wait until the entire file is transferred before viewing the video content. If the file sizes of the digitized video are relatively large, this delay can be significant.

Video content is thus often “streamed” to users so that they can continue to receive and view the content while it is still being transmitted. Essentially, video content is an ordered linear stream of small files or “video segments” (eg, 1 to 10 seconds in length) that are delivered to users, which users can begin viewing when received. divided into streams). In order to view a continuous stream of video content without delay or jitter, each video segment must be played at regular intervals - eg, at 30 frames per second (fps). However, it should be noted that the video segments need not be received at regular intervals if each segment is received before playback of the previous segment ends.

Whether an event is scheduled or unscheduled, the event may be streamed "live" (ie, when the event occurs) or may be "recorded" for streaming at any time after the event has occurred. For example, the Super Bowl could be caught and streamed live as the event occurs, or recorded for later streaming.

Finally, whether the event is scheduled or unscheduled, and whether the event is recorded or streamed live as it occurs, the event is "real time" (i.e., from the transmitter to the receiver with largely imperceptible delay). may be streamed) or delayed in transmission for seconds or even minutes. For example, viewers of a television program that is over the Internet but not streamed in real time (e.g., a baseball game) will experience the streamed event at a different time than viewers watching the same program broadcast over cable or satellite. can These delays (especially if greater than a few seconds) are called "quality of QoS", a measure of performance based on network-centric metrics (e.g., packet delay, packet loss, or jitter caused by routers or other network resources). service", it can degrade the user's "quality of experience" (ie, a user-centric or application-level quality perspective).

For example, social interaction between viewers may be constrained by the fact that they experience the same event at different times (in addition to jitter or other video artifacts). This means that today so many (scheduled and unscheduled) events are accessible in so many different ways - from broadcast radio or television accessible via mobile phones and desktop and laptop computers, as well as an ever-evolving field of consumer electronics devices to social media. Even to media and other Internet services - especially when delivered in real time.

Therefore, it is desirable for a video streaming system to handle scheduled as well as unscheduled events, stream live as well as recorded events, and stream those events with minimal latency to provide consistent QoE to viewers. do. Moreover, as the number of concurrent viewers of a streaming video event increases, maintaining a consistent QoE becomes a formidable problem. As such, scalability is a key design goal of any such system.

Despite recent advances in video streaming technology, historic "ad hoc" advances in the Internet's infrastructure still pose significant obstacles to Internet-based video streaming - chief among them, unpredictable occasional and across the Internet. Inconsistent QoS leading to network congestion in locations where there is Although a primary objective of the present invention is to maintain a consistent QoE for viewers of streaming video events, this objective is ultimately limited by network congestion across the Internet that cannot be eliminated.

Underlying Internet Architecture

Beginning with ARPANET (the first packet-switched network implementing the Internet protocol suite, or TCP/IP) and later NSFNET, the Internet "backbone" distributed control and allowed information to reach its desired destination. It is designed to be a redundant "network of networks" (ie, the Internet) that provides reliability or "resiliency" by providing alternative communication (routing) paths for However, maintaining consistent QoS or QoE over the Internet for packets following different paths between routers and other shared network resources remains an extremely difficult problem.

As the Internet backbone evolved and privatized, overlap and overlap emerged between traditional backbone networks and backbone networks owned by long-distance carriers. For purposes of this specification, we refer to large "public" networks that transmit data directly to customers or over other small "internet service provider (ISP) networks, and data only between themselves, or large ISPs." In either case, these large public and private networks are typically fiber-optic trunk line ) (e.g., multiple fiber optic cables bundled together to increase network capacity).

For routing purposes, the largest network providers carrying the most network traffic (e.g., large ISPs and private backbone networks) are called "autonomous systems" - each assigned an "autonomous system number" (ASN). blocks of IP routing prefixes, known as We refer to each of the large fiber optic rings owned by these companies as an ASN. The number of ASNs has increased dramatically in recent years, from approximately 5000 ASNs 15 years ago to over 50,000 ASNs worldwide today. As mentioned earlier, many large network providers also have backbone fiber optic ring networks (i.e., private ASNs) that do not service customers but can be connected to their own "public ASNs" or to public ASNs owned by other providers. ) owns.

Because different companies own ASNs, they contract with each other to facilitate routing of Internet traffic across ASNs and throughout the worldwide Internet. Each ASN uses a bank of routers, often referred to as “peering points,” to control access to other ASNs, using a routing protocol known as “border gateway protocol (BGP).” Any given ASN may use multiple peering points to connect to multiple different ASNs. Interconnected ASNs may be geographically contiguous, or may be remote, and may be connected via long fiber optic trunks that span great distances (eg, spanning countries or even oceans). Public ASNs may also be interconnected via “private ASNs” or backbone networks.

It is extremely difficult to monitor QoS within and across ASNs. Large network providers keep a large portion of the routing and performance information (including dynamic congestion metrics) within their ASNs as proprietary. Although the "Open Message Format" (currently in BPG version 4) provides a "data dump" of certain information when a TCP/IP connection to a BGP router is established, this mechanism is a practical problem. not very useful as Many BGP routers do not support the open message format, while others just turn it off. Moreover, the information is typically 5 minutes late, which is a relatively long time considering how often congestion levels change across the Internet.

Because this large amount of Internet traffic flows through the relatively high-bandwidth peering points that interconnect ASNs, these peering points often address "last mile" issues within ASNs (i.e., end-users and their " Other than gateways (congestion across the relatively lower bandwidth wired and wireless connections between ISPs), they are the major "bottlenecks" or sources of a significant portion of congestion across the Internet at any given time.

For example, when the traffic load across an ASN peering point increases, the routers at the ASNs on each side of the peering point become congested. In other words, these routers experience high utilization of RAM, CPU and other limited capacity shared resources. Increased demand for these resources can degrade performance (eg, bit rate) across these peering points and ultimately result in lost data packets. Because network traffic across the Internet is not centrally controlled, it is difficult to predict the frequently changing "peering point congestion" levels across the Internet at any given time.

If consistent QoS cannot be guaranteed within and across ASNs, it becomes very difficult to maintain consistent QoE for viewers of streaming video events. Any system streaming video over the Internet suffers from ever-changing congestion levels and unreliability of shared routers, especially at ASN peering points where a lot of Internet traffic flows. This problem is exacerbated when streaming video over the Internet and especially over these ASN peering points to a large number of concurrent viewers.

Traditional video streaming approaches

Various approaches to streaming video over the Internet have evolved over the past decades, characterizing and distinguishing different techniques for generating overlay network topologies (on the Internet) and delivering video content between network nodes along these overlay networks. A broad term is used to When comparing different approaches, returning briefly to the GPS navigation analogy, the factors that affect the time required to travel between any two points or nodes - namely distance, speed and congestion (typically along different routes) resolved by rerouting) - it is helpful to look at

When it comes to routing packets on the Internet, distance (or geographic proximity) has no direct relevance, since packets travel at nearly the speed of light. However, speed is affected by the number of stops or obstacles encountered along the route, or in this regard the number of "hops" encountered in intermediate routers between two nodes. As such, two nodes, regardless of their geographic proximity, may be said to be "nearby" (within "network proximity") to each other if they are only relatively few hops apart. Congestion at intermediate nodes along the path between two nodes affects the overall travel time, and dynamically reroutes traffic - i.e. overlay networks that determine the path between two nodes. Reconstructing with - can be solved by As will be discussed below, these factors serve to account for key differences between different approaches to streaming video over the Internet.

The most common method of delivering video other than the Internet is “broadcasting” a video stream (eg, a television program) from a “point of origin” to all destination viewers simultaneously—eg, over a dedicated cable or satellite infrastructure. is to do Although network hubs can be used to broadcast information to all network nodes in a LAN, broadcasting packets of video segments across switches and routers across the Internet is very impractical and inefficient. Most network users will not be interested in watching any given "channel" of video content, and significant congestion will occur near the point of origin as routers broadcasting video segments to other routers will easily be overwhelmed. Broadcast solutions are not feasible at all to deliver a channel of video content over the Internet from a single point of origin to a large number of concurrent viewers who may join the channel at any time.

An alternative "multicast" approach involves simultaneously streaming each video segment from a point of origin over the Internet to predefined groups of nodes. This approach is also impractical for large-scale video distribution over the Internet. Moreover, special infrastructure is required, such as dedicated routers with multicasting capability, which is also impractical and prohibitively expensive for large-scale commercial use.

Unlike broadcast and multicast techniques, a "unicast" technique for video streaming is a video segment from an originating point to a single destination node (eg, by establishing a TCP/IP connection with a defined destination node IP address). including sending them. However, delivering a large number of unicast packets to each viewing node will also quickly overwhelm the routers at or near the point of origin, with sufficient bandwidth to handle such a large number of simultaneous transmissions. You will not achieve consistent QoS for many of the reasons discussed above, not to mention the prohibitive cost of providing QoS.

Some VOD companies (such as Netflix and YouTube) are using variations of the unicast approach that typically rely on expensive "edge-server" infrastructure. This approach (sometimes referred to as a content delivery network (CDN)) involves setting up many physical servers across the Internet and distributing copies of each video content channel to each server. As a result, viewing nodes can receive the desired video content from a nearby server (within network proximity - only relatively few hops away from the viewing node).

Each edge server typically has significant bandwidth and computational power, and is essentially a separate video content source - from which nearby viewing-side nodes can obtain any channel of video content at any point in time ("on-demand"). Has - constitutes This approach of adding physical infrastructure adds additional value to allow more people to reach popular destinations more quickly (by fewer turns and less time spent on slower roads). It is somewhat similar to building highways and off-roads.

Although different users typically want to view different video channels at any given time, VOD systems use “peak” periods of demand (e.g., the final episode of a popular television series) when a particular video event must be streamed to a large number of concurrent viewers. , which can sometimes overwhelm even the largest streaming video company's infrastructure - or at least, the inefficient "worst case" of expensive infrastructure that is frequently underutilized (i.e., during more common periods of non-peak demand). results in a "worst-case" placement of -. Alternative VOD solutions have attempted to obviate the need for expensive edge server infrastructure by replicating and distributing video content among network nodes themselves (as discussed, for example, in US Patent Publication No. 2008/0059631). .

With or without an expensive edge server infrastructure, none of these VOD solutions solve the QoS problem for unscheduled video events, because they all - to ensure a nearby video content source. For - because it relies on "pre-seeding" edge servers or viewing nodes all over the internet with known content. Streaming a live, unscheduled event will require real-time simultaneous delivery of video content to all of these edge servers or viewing nodes, a problem not addressed by either of these VOD systems.

More recently, several unicast-based video streaming standards (eg, "WebRTC") have evolved to facilitate "point-to-point" streaming of video between desktop and mobile web browsers without requiring any plug-ins. Many existing smartphones, as well as desktop and laptop computers, have WebRTC libraries that support browser-to-browser video streaming, as well as a lower level for the viewing node to detect its bandwidth and CPU power in real time and adapt to changes in those metrics. Or "adaptive streaming" libraries that allow you to automatically request a higher "bit rate".

Adaptive streaming implementations include, among others, Apple's "HTTP Live Streaming" (HLS), Microsoft's "Smooth Streaming" and the "MPEG-Dash" ISO standard. In a typical point-to-point video streaming scenario, the receiving node sends "manifest files" to the HTTP server, containing the locations of the available bit rate version of each of the upcoming (e.g., next 8) video segments. periodically request For example, each video segment has a different "video resolution" requiring different streaming bit rates (bandwidth) so that each video segment is delivered in essentially the same amount of time, regardless of its resolution. may be available in 1080p, 720p and 480p versions, reflecting the

Standard HTML5 video players (in web browsers that support WebRTC) typically buffer three video segments before starting to play the video content. They use the current manifest file to send an HTTP request to the HTTP server for each video segment. The sending node then “pushes” each video segment (in small “chunks”) to the receiving node according to WebRTC standards for playback in the receiving node's web browser. If the receiving node supports adaptive streaming implementations and determines that the time required to receive recent video segments is increasing or decreasing significantly, then the receiving node may select a lower or higher value from among the selections in the manifest file. Automatically start requesting video segments of the bit rate. In other words, the receiving node “adapts” to its actual bandwidth over time by changing the resolution of the video segments it requests.

The "resolution" of a video frame is a measure of its width x height (e.g., 1920 x 1080 or 1080p) or the number of pixels in the frame, while its "bit rate" refers to the number of bits per second (bps) transmitted from the transmitter to the receiver do. For example, if 30 1080p-resolution video frames are delivered every second (30 frames per second (fps)) and each color pixel contains 24 bits (24 bits per pixel (bpp)), the bit rate is approximately 1.5 Tbps (1,492,992,000 bps - i.e. 1,492,992,000 = (1920 x 1080 ppf (pixels per frame)) x (24 bpp) x (30 fps) - would be

Standard video codecs use compression (eg, MPEG2 compression) and other video encoding techniques to produce lower effective bit rates (eg, 3 Mbps). In view of the foregoing, "bit rate" and "resolution" are highly correlated in that the effective bit rate can be increased or decreased by providing video frames of higher or lower resolution. Accordingly, these terms may be used somewhat interchangeably herein in this regard.

WebRTC and adaptive streaming standards enable virtually any smartphone user to capture and stream live video events, and also allow such users - from other individual smartphone users to large companies hosting large numbers of video content. to join a streaming video channel originating over the Internet from another originating point. However, these standards are designed for point-to-point video streaming and do not solve the "video delivery" problem of streaming a video channel over the Internet to a large number of simultaneous viewers.

To address this problem, some video streaming companies (e.g., StreamRoot) typically use a "peer-to-peer" or mesh network topology in which video content is relayed from one viewing node to another. (sometimes referred to as “peercasting”). In the context of video streaming, these terms can be used interchangeably to refer to overlay networks constructed over the Internet that enable viewing-side nodes to relay streaming video content to each other in a distributed manner. However, delivery of streaming video to a large number of concurrent viewers should be distinguished from non-streaming uses of a peer-to-peer or mesh network topology, such as for file transfer or file sharing applications.

Peer-to-peer video streaming systems deliver channels of video content from a single point of origin over the Internet to a large number of concurrent users. These systems work because their distributed nature facilitates recovery from individual points of failure (eg, by rerouting content through other nodes), and as more nodes are added to the network (ie, more They tend to be both flexible and scalable, in that their reliability and performance actually improve (as more and better routing "relay" options become available).

When new nodes join a video channel or existing nodes leave (stop watching) the channel, peer-to-peer video streaming systems must, to some extent, dynamically reconfigure the topology of the overlay network - that is, accommodate the new nodes. To do this, at least some of the routing paths between network nodes must be modified -. For example, when a new node is added, its geographic location may be taken into account to select existing nodes in the vicinity from which the new node will receive (and relay) video content.

However, if "peer" nodes are selected based solely on their geographic proximity, they may still be relatively "distant" from each other (and not within network proximity) - eg, if they are in different ASNs. As a result, traffic between them may pass through one or more possibly congested peering points. For example, the actual latency between two nodes within close geographic proximity may exceed the sum of the latency between each of those nodes and a node that is geographically distant. This phenomenon is sometimes referred to as "triangle inequality violation" (TIV), and represents the drawbacks of relying on BGP routing to carry digital content between nodes in an overlay network over ASN peering points.

One reason for this problem in existing peer-to-peer video streaming systems is that they are not configured to be "compatible" with the underlying architecture of the Internet. Any overlay network topology built on top of the Internet is still subject to many blocking or points of failure (other than new or fading nodes), such as the myriad QoS issues discussed above. By failing to address the fundamental QoS variability of the Internet, especially at ASN peering points, these systems face significant obstacles in providing consistent QoE to their users.

As such, existing peer-to-peer video streaming systems (such as GPS navigation systems) rely on geographic proximity (rather than network proximity) to select peer relay nodes, and traffic only “after the fact” when congestion occurs. reroute the Moreover, real-time streaming of linear data to concurrent users imposes an additional constraint that is not present in GPS navigation systems - the content must arrive at each node "at the same time." Edge servers and other physical infrastructure approaches (similar to building highways and ramps to provide higher speed routes) are costly and also suffer from unscheduled events and high concurrency rates of any particular event. Doesn't solve problems properly.

Thus, in order to address the disadvantages discussed above, and to provide consistent QoE to client nodes, creating and dynamically reconfiguring overlay networks (especially at ASN peering points through which a great deal of Internet traffic flows) is of the importance of the Internet. A digital content delivery system that considers the underlying architecture is needed.

In accordance with the present invention, various embodiments of novel methods and architectures provide (1) a map of shared links interconnecting components of an underlying network (e.g., ASNs and peering points interconnecting them) - maintaining - including the location of each node within one of the components (eg, within an ASN); (2) generating metrics by monitoring network traffic between nodes traversing their shared links (ASN peering points) along one or more overlay networks built on top of the underlying network (Internet); (3) analyzing metrics and maps over time to predict congestion levels that reflect the capacity of shared links (ASN peering points) that change over time; and (4) the underlying network by dynamically reconfiguring the topology of the overlay networks, based on predicted congestion levels, to generate optimal paths of digital content between client nodes along the overlay networks ( A digital content delivery system that provides consistent QoE to users of nodes on, eg, the Internet, is disclosed.

Certain embodiments of the "virtual broadcasting" system of the present invention (without requiring edge servers or other costly physical infrastructure) can provide one or more channels of video content - each video content channel from a single point of origin. It is described herein with respect to providing consistent QoE to viewers - streamed in real time, possibly to a large number of concurrent viewers who may join the channel at different times. As will be explained in more detail below, we use the term "virtual broadcasting" in connection with unicast streaming of linear content to concurrent users. From the users' point of view, unicast streaming is used to route the content, but the content is "broadcast" to the users in that they receive the content "simultaneously". Other embodiments of the present invention may be useful in many other situations where the limited capacity of shared links between network components constrains the routing of any type of information that can be converted to digital format, for those skilled in the art. will be clear to

For purposes of this specification, a single node receiving multiple different channels simultaneously may be considered a separate node on separate overlay networks, each defined by a single channel. In the context of VOD, each individual "showing" of a particular program can be considered a separate channel with its own network of viewing-side nodes.

The system of the present invention handles scheduled as well as unscheduled events, streams live events as well as recorded events, and is implemented in overlay networks built over the Internet that are subject to the QoS variability of the Internet. Despite this - it is possible to stream those events in real time with minimal latency in a highly scalable manner that maintains consistent QoE across large numbers of concurrent users. As the number of concurrent users (especially within any given ASN) increases, the performance of the system and the QoE of the users actually improves.

A client-server architecture is used to centralize server-side routing decisions. Distributed streaming delivery of video content is achieved through dynamically reconfigurable peer-to-peer overlay networks that allow video content to be relayed (ie, “pushed” to) viewing nodes of each video channel. Client nodes can use standard HTML5 video players (built into most desktop and mobile web browsers) to watch or play video, manage the receipt of video segments and send them to other nodes. It relies on custom embedded code (such as Javascript) to implement additional functionality, such as monitoring various performance metrics, as well as relaying to . In other embodiments, some or all of these functions may be incorporated into a custom application or mobile app.

The system of the present invention facilitates "point-to-point" video streaming between desktop and mobile web browsers and adapts to changes in node bandwidth by automatically requesting lower or higher bit rate video segments. In one embodiment, the virtual broadcasting system uses WebRTC and adaptation to facilitate video streaming without requiring web browser plug-ins and to enable nodes to detect their performance capabilities, such as bandwidth and CPU capacity. It uses unicast standards, including traditional streaming standards (such as HLS, MPEG-Dash or Smooth Streaming).

Each event is presented to a central "virtual broadcast server" for "point-of-origin" delivery of each channel to concurrent users across multiple dynamically reconfigurable overlay networks. do. Events can be acquired from almost any source (including CDNs), whether sent as complete files to a virtual broadcast server or streamed live. In embodiments using WebRTC, any user with a smartphone implementing WebRTC can upload recorded video events or capture events live to a virtual broadcast server for later delivery to users over overlay networks. You can upload (and watch other channels streamed from the virtual broadcast server).

The virtual broadcast server, in one embodiment, includes a "POI content server" that serves as the point of origin for each channel, from which video segments are delivered via dynamically reconfigurable overlay networks built over the Internet. Video segments are typically of a fixed size (eg, 1 to 10 seconds), as determined by the originating publisher of the video event. Video segments are viewed by client nodes and "pushed" from node to node along paths defined by overlay networks (ie, relayed as individual fixed-size "chunks" according to the WebRTC standard). In one embodiment, each video segment is divided into 64 KB chunks to match the size of a UDP datagram "packet" for maximum efficiency when streamed over the MPEG2 transport protocol.

In most cases video segments are effectively pushed to each client node, but in one embodiment, a client node can detect that not all of the chunks of a video segment have arrived in time (i.e., "fallback" feeding You can use the current manifest file to request video segments from the POI content server (which is the "fallback" feeding location).

As each node attempts to join a channel made available by the POI content server, the node (in another embodiment, with the help of the virtual broadcast server) determines the particular ASN on which that node resides. A virtual broadcast server, including the Internet (including ASNs and various peering point interconnections), to optimize routing of channel content between overlay networks that are dynamically reconfigured based on predictions of congestion levels at these ASN peering points. ), along with a dynamic "ASN interconnection map" and various monitored performance metrics, using this "ASN location" information. In another embodiment, the virtual broadcast server also uses each node's geographic location, in addition to each node's ASN location, to assist in this process.

In one embodiment, the topologies of overlay networks define routing paths of video segments between viewing-side nodes, and are dynamically reconstructed (in whole or in part) for each video segment of a channel. In another embodiment, they are dynamically reconstructed (in whole or in part) for each chunk of a video segment. In this way, the architecture of the Internet (as well as predicted congestion levels at ASN peering points) is taken into account in determining optimal routing paths for each video segment of a video channel. In another embodiment, if some or all of the paths along the overlay networks can deliver video segments in time (even if these paths are not optimal), then these paths meet a predefined congestion threshold or It is not reconstructed until other sufficiently significant problems are identified.

In one embodiment, the client nodes are capable of handling, for example, last mile issues and QoS issues (including congestion at ASN peering points) across the Internet as well as concurrent viewers of one or more channels of the virtual broadcasting system itself. Monitor performance issues related to congestion due to the number of They monitor the time required to relay video segments to other nodes, as well as the time required to contact designated sites across the Internet and across ASNs. Metrics monitored by the client are forwarded to the virtual broadcast server for use in making dynamic routing decisions. In one embodiment, the Virtual Broadcast Server includes a "Signaling Server" to communicate with client nodes via standard WebSocket protocols.

The client nodes optionally include an "Uploader", which allows users to capture video events and upload them to a virtual broadcast server in real time. Since the path from any client node to the virtual broadcast server may cross multiple ASNs, custom "showering" is used to facilitate streaming of video events and prevent packets from being delayed or blocked at intermediate routers. "The protocol is used. In one embodiment, the client nodes also search for "trending" events (referred to herein as "splashes") via the "Splash Extractor" search engine on the Virtual Broadcast Server and The splash extractor search engine identifies splashes and, based on user searches, allows users to stream and view trending events over the Internet that would otherwise not be available through the POI content server.

When requesting to join a channel, nodes monitor their relay capabilities - i.e., their connection type (e.g., 3G or 4G cellular, WiFi, LAN, etc.) as well as their CPU, operating system and memory configurations, and over time are classified by the virtual broadcast server based on their trusted "upstream" bandwidth - inferred from a variety of factors, including other fixed and variable performance metrics. In one embodiment, nodes are classified into three levels based on their relative relay capabilities. The lowest level nodes (“C” nodes) can watch video segments but cannot relay them to other nodes. Mid-level nodes ("B" nodes) may watch the video segments and may relay them within the ASN. Top level nodes (“A” nodes) can view video segments and relay them to other A nodes within the ASNs as well as across ASNs.

In another embodiment, node classifications may be dynamically changed, eg, based on monitored performance metrics and the system's current requirements for more or fewer relay nodes of a given classification. In addition, if there are enough A nodes within the ASN to relay the video segments, the A node will be treated as a B node, but promoted to an A node if necessary (e.g., when existing A nodes leave the channel). It can be specified as a "B:A" node, indicating that it can be. In one embodiment, if an individual node exhibits a significant change in performance (for better or for worse), the node may be reclassified (e.g., from a B node to a C node, or vice versa), and when the problem is resolved, It can be restored to its initial classification.

In another embodiment, client nodes may (e.g., based on their capabilities and client performance metrics) multiple "slots" to enable client nodes to relay and receive chunks of a video segment to and from multiple other nodes. are assigned ". In this embodiment, client nodes receive video segments from only one "feeding" node, but may "feed" or relay the video segments to multiple other client nodes. A nodes have up to 8 relay slots - 4 for relaying to A nodes in the same ASN and 4 for relaying to A nodes in different ASNs - i.e. via the ASN peering point - is assigned B:A and B nodes are assigned up to 8 slots for relaying to other client nodes in their ASN (ie other B:A, B and C nodes). In another embodiment, a client node may be fed by multiple other client nodes (eg, by alternating chunks in multiple destination slots). This technique can be used for high bit rate (eg, 4K) video streams where higher performance is required.

In another embodiment, certain nodes may receive multiple resolutions of a video segment from a single feeder node (eg, based on their capabilities and client performance metrics) (or, alternative implementations In an example, different resolutions may be received from different feeder nodes). If the upstream bandwidth of these nodes is sufficient, they can be considered "polycasting" nodes and, to the extent necessary, relay or feed those multiple resolutions of a video segment to one or more designated nodes.

To facilitate dynamic reconfiguration of overlay networks, the virtual broadcast server is used to predict the congestion level across the ASN peering points - that is, to predict the congestion level of the ASN peering point after a short period of time (eg, 1 minute). It uses a "Deep Mapper" deep learning engine that continuously analyzes performance metrics. In one embodiment, for each potential inter-ASN path between A nodes - e.g., from one A node in an ASN to an A node in another ASN - the predicted "congestion value" value)" is generated. In another embodiment, the congestion value reflects the predicted congestion level for the optimal path between each pair of A nodes.

In one embodiment, the virtual broadcast server is configured to generate and (in whole or in part) dynamically reconfigure both inter-ASN and intra-ASN overlay networks - e.g. within ASNs as well as To determine the best path for video segments to be pushed from one node to another across ASNs - use the "Overlay Network Creator". In this embodiment, the overlay network generator takes into account the number of available slots each node can use, as well as the number of resolutions each node can receive or relay to.

The overlay network generator generates (with the help of a deep mapper) and dynamically reconstructs an inter-ASN “Virtual Data Trunk” overlay network representing the topology of the A-nodes. In other words, the overlay network generator is the links or routing path that A nodes and video segments will follow between those A nodes within and (specifically) spanning ASNs—ie, possibly via congested ASN peering points. represent them

A virtual data trunk is a set of A nodes (that span ASNs as well as within ASNs) that will be instructed to request each video segment from a nearby POI content server (e.g., using the current manifest file), as well as their Each identifies a set of A nodes to push a video segment to, etc. As a result, the video segment will spread across all ASNs including the viewing node. To reach each viewing node, the segment may also travel over intermediate private backbone ASNs that do not have viewing nodes.

The overlay network generator also generates one or more "swarm" overlay networks within the ASN to relay video segments from A nodes within the ASN to B:A, B and C nodes within the ASN. These swarm overlay networks may be dynamically reconstructed (in whole or in part) for each video segment (or, in an alternative embodiment, for each chunk of a video segment). In one embodiment, each swarm overlay network within the ASN has nodes B:A, B, and C (within the ASN A) hierarchical topology.

Thus, the virtual broadcasting system and method of the present invention optimizes the routing of digital content between nodes in swarm overlay networks and a virtual data trunk that is dynamically reconfigured based on predictions of congestion levels at these key congestion points. efficiently use the limited capacity at ASN peering points and other major congestion points by monitoring and analyzing network traffic in order to maintain consistent QoE among system users.

1 is a graph representing one embodiment of the dynamically configured overlay networks of the present invention over the Internet;
2 is a block diagram illustrating one embodiment of the main client-side components of the client streaming video device of the present invention;
3 is a block diagram illustrating one embodiment of the main server-side components of the virtual broadcasting server of the present invention;
4 is a flowchart illustrating one embodiment of the dynamic video streaming process of the present invention.

Detailed embodiments of the systems and methods of the present invention are illustrated in the accompanying drawings and described below. It should be noted first that the present invention is not limited to the specific embodiments discussed below with reference to the drawings.

As noted above, although a specific application of the present invention is described herein in connection with the delivery of streaming video over the Internet to a large number of concurrent users, the principles of the present invention can be applied to shared links between network components. Limited capacity applies equally in many other situations that constrain the routing of any type of digital content.

Even in the context of delivering streaming video over the Internet, the allocation of functionality between client nodes and server components described herein is the result of design compromises, much of which is without departing from the spirit of the present invention. It can be reallocated between client-side components and server-side components. Similarly, client-side functionality can be assigned to a single modular component or distributed across multiple different components, as one or more standalone applications or mobile apps, or with standalone applications or apps and Javascript or other scripting or It can be implemented as a combination of programming languages. Moreover, server-side components may be implemented on a single hardware server or across multiple different servers. Such functionality may also be integrated into a single software module or allocated between different software modules distributed across one or more hardware servers.

Finally, in those embodiments where standard protocols and libraries (e.g., HTTP, WebSocket, WebRTC, STUN, and various adaptive streaming standards) are used, some or all of these standard protocols and libraries The functionality provided by may be replaced with other standard or proprietary implementations without departing from the spirit of the present invention.

overlay network

1 is a graph illustrating one embodiment of overlay networks 100 of the present invention mapped over the Internet. Although the Internet itself can be illustrated in a myriad of different ways, FIG. 1 illustrates the Internet as a set of ASN 110 fiber optic rings interconnected through peering points 120 . Individual client nodes viewing a particular video channel at any point in time are illustrated within each ASN 110 . Although not shown in FIG. 1 , multiple channels, and thus multiple sets of overlay networks 100 , may be active simultaneously (in one embodiment).

As noted above, the virtual data trunk overlay network is composed of A nodes 130 that are within ASN 110 (directly connected) as well as across ASNs 110 (i.e., via peering points 120). interconnections 175 between them. Backbone connector 195 is an A node between two ASNs 110, over a private ASN (not shown) that does not involve any commercial nodes and only interconnects the two public ASNs 110. represents the interconnection of them. For example, backbone connector 195 is shown connecting A node 130 in ASN 110-f with A node 130 in ASN 110-e. In this scenario, traffic between these two A-nodes 130 may travel over multiple “private” peering points 120 (or other proprietary connections with private ASNs).

As mentioned above, in one embodiment, such as in the case of connections 175 between A nodes 130 in two different public ASNs 110 (i.e., via peering point 120), The performance of these connections can be monitored only at the endpoints (ie the two A nodes 130). Traffic along a connection 175 between two A-nodes 130 within the same ASN 110 is relatively larger than traffic over ASNs 110, possibly because it does not traverse a congested peering point 120. It will likely be faster. Background connector 195 and connections 175 to/from A nodes 130 are illustrated with one-way arrows, despite the fact that bi-directional connections are supported between all client nodes illustrated in FIG. However, it reflects only the current one-way routing paths.

It should be noted that all traffic between any two client nodes of the present invention goes over the public Internet and thus passes through various intermediate routers (not shown) which affect QoS. The system monitors QoS effects within ASN 110 as well as across ASNs 110 (and thus one or more peering points 120). In one embodiment, this intra-ASN and inter-ASN traffic is monitored by each client node (according to the direction of the virtual broadcast server), and (virtual data trunk overlay network between A nodes 130 and ASN 110). A node represented by overlay networks 100 (including swarm overlay networks from each A node 130 in the ASN 110 to B (and B:A) nodes 140 and C nodes 150 in that ASN 110 It is passed to the virtual broadcast server for dynamic reconfiguration of fields and routing paths.

1 shows the various routing paths that a video segment follows between client nodes, given the “current state” of these overlay networks 100 . In other words, Figure 1 shows a diagram of these overlay networks 100 that can be dynamically reconfigured, in one embodiment, for each video segment (and, in an alternative embodiment, for each chunk of a video segment). Illustrates the current topology. For any particular video segment, the overlay networks 100 may or may not be reconstructed (either in whole or in part), as this determination depends at least in part on performance metrics collected over time. It should be noted that it is because it will depend.

ASN 110-c is such that a POI content server (not shown) resides at or near ASN 110-c (e.g., via one or two other ASNs 110), and overlay networks ( 100) to respond to an HTTP request to forward the current video segment to the A node 130-a to initiate streaming of the video segments over a channel. As will be discussed in more detail below, a POI content server will typically deliver each video segment to multiple requesting A nodes 130 within the same or nearby ASN 110, which A nodes ( 130) will in turn push the video segment to a number of different nodes along the overlay networks 100, resulting in multiple concurrent chunks of video segments delivered to and relayed from client nodes at any given point in time. There will be "redistribution" of copies.

In this scenario, A node 130-a sends a video segment to two other A nodes 130 - one within ASN 110-c and the other via peering point 120 to ASN 110-a. ) on the side - relays to . As noted above, the virtual data trunk overlay network represents the routing paths that a video segment follows as it is relayed within and across ASNs 110 between A nodes 130 . Thus, in this scenario, a video segment is distributed between multiple A nodes 130 within ASN 110-c, as well as multiple directly interconnected networks from ASN 110-a over various peering points 120. are relayed to ASNs (i.e., 110-a, 110-d, 110-f, and 110-g), from which the video segment is transmitted over multiple hops of the virtual data trunk overlay network to other ASNs 110. additionally relayed to

As will be explained in more detail below, the number of A nodes 130 required within an ASN 110 depends on the number of other client-viewing nodes within that ASN 110 (their class, open slots, etc.) number of open slots and their relative capabilities (as determined by performance metrics monitored over time). For example, although each of ASNs 110-b, 110-f, 110-i and 110-j are illustrated with only a single A node 130, they have a different number of other client nodes to feed (ASNs). Compare a single other node in (110-f) with multiple other nodes in ASN (110-i).

While a node's monitored upstream bandwidth is a major factor in determining how many nodes it will directly feed (i.e., how many outgoing slots will be used), how quickly these relays can be performed (typically It is important to note that the length of the "chain" of nodes within the ASN 110 (which relays a video segment from one node to the next, and so on) is largely irrelevant given . For example, two A nodes in external ASNs 110 (ASN 110-g and ASN 110-j) feed directly to two B nodes 130 in ASN 110-1 as well as , a single A node in ASN 110-i uses four outgoing slots (in this example, reflecting the relatively high monitored upstream bandwidth). However, the long chain of B nodes 140 and C nodes 150 that are indirectly fed from a single A node within ASN 110-i is not a reflection of its upstream bandwidth.

Within each ASN 110, video segments are routed from each A node within that ASN 110 (i.e., the "root" node of the swarm overlay network) to the various B (and B:A) nodes within that swarm overlay network. One or more swarm overlay networks are generated (in this embodiment, dynamically reconfigured for each video segment) to relay to 140 and C nodes 150. Although only one swarm overlay network is illustrated in ASN 110-c (compared to the two swarm overlay networks illustrated in ASN 110-h), a swarm overlay network generated within each ASN 110 The number of swarm overlay networks (and the internal topology of each swarm overlay network) will depend on various factors, such as the number of client viewing nodes within that ASN 110 as well as current and past performance metrics, number of open slots, etc. .

As noted above, a client node, such as A node 130-b in ASN 110-b, may receive requests from multiple other client nodes (in this case, in different ASNs 110-a and 110-d). may receive a video segment from two different A nodes 130). In one embodiment, these two different feeding nodes take turns sending chunks of a video segment to node A 130-b for performance reasons - e.g., because these chunks, as will be described in more detail, As such, congestion levels are continuously monitored through peering points 120 -. In other embodiments, this may be done for redundancy - e.g., because the reliability of the feeding nodes is questionable based on past performance metrics (apart from or in addition to congestion of the peering points 120). Because it can be embarrassing -.

The methods by which performance metrics are monitored, video segments are relayed, and overlay networks 100 are dynamically reconfigured are discussed after discussion of the main client-side (FIG. 2) and server-side (FIG. 3) functional components that implement these methods. , which is explored in more detail below with respect to FIG. 4 .

Client Streaming Video Device

Referring to Figure 2, a client device 200 represents one embodiment of the main components of the client streaming device of the present invention. Client device 200 may be implemented as a desktop or laptop computer, as well as a smartphone or other mobile device, or almost any other consumer electronic device capable of handling streaming content, such as streaming video. Client device 200 includes CPU 212, memory 214, operating system 216, network adapter 217, display 218 and camera 219, as are well known in the art. It includes any standard hardware and software computing components and associated peripherals 210 . The client device 200, together with certain standard libraries 220, to become a network node and to receive, display and relay streaming video content among other network nodes in the virtual broadcasting system of the present invention, this Components and peripherals 210 are used.

The present invention provides some standard libraries 220 (most smartphones as well as many other computing devices) that implement network protocols and other functionality that can be used to facilitate streaming video content between devices. also) are used. For example, video content can be streamed between two smartphone users and displayed on their mobile web browsers without requiring any plug-ins. Libraries 220 of the standard include the WebRTC 222 APIs (which facilitate inter-browser communication for streaming video content), the streaming bit rate (to “adapt” to real-time detection of changes in client bandwidth and CPU capacity), various adaptive streaming 224 implementations, such as HLS, MPEG-Dash, and Smooth Streaming, among others (high-speed bi-directional client-server communication over a single TCP/IP connection) WebSocket (226) protocol (which facilitates web browsers), and HTTP (228) (for a less frequent standard of communication between web servers and client web browsers).

The client device 200 also includes a standard player 232 (in one embodiment, a standard video player integrated into a standard HTML5 web browser 230) for viewing or playing streaming digital content. In other embodiments, the standard player 232 is integrated into a standalone desktop application or smartphone app. One advantage of using a standard HTML5 web browser 230 is that many of the standard libraries 220 are designed to work in web browsers and therefore do not require any plugins or plugins that would require a standalone desktop application or smartphone app. It does not require any other custom features.

Moreover, web browsers also support Javascript, which is often used to supplement standard web browser functionality (delivered from a standard web server as part of a web page, eg, without the need for any client browser plug-ins). Supports client-side scripting languages, such as In one embodiment, the non-standard main components of the client device 200 (Communicator 270, Performance Monitor 240, Receiver 250, Relayer 260 and Uploader ( 280) is implemented in JavaScript, and the content arrays 255 are generated and maintained by the JavaScript code. However, it should be noted that some or all of these components may be implemented in other programming languages and as standalone desktop applications or smartphone apps without departing from the spirit of the present invention.

Standard libraries 220 facilitate general point-to-point (unicast) streaming of content, including video content. The non-standard key components of client device 200 address the client-side aspects of the digital content delivery architecture implemented by the virtual broadcasting system of the present invention. In one embodiment, the streaming protocol is built on top of WebRTC 222, where the routing of content is centralized via a client-server architecture, and the content itself is streamed in a distributed fashion (node to node) over dynamically reconfigurable peer-to-peer overlay networks. push).

A user of client device 200 may first encounter one or more channels of content in a variety of different ways - eg, in an email or via links on a webpage, or even from within a standalone desktop application or smartphone app. In one embodiment, virtual broadcast server 300 (discussed in more detail below with respect to FIG. 3) delivers to HTML5 web browser 230 a standard HTML5 webpage with a collection of channels. This “channel webpage” communicates (e.g., using WebRTC 222 and adaptive streaming 224 libraries) with Signaling Server 330 as well as other client nodes, as well as sending chunks of video segments to these Contains proprietary Javascript code interpreted by HTML5 web browser 230 to implement functionality of non-standard components of client device 200, including receiving from nodes, processing and relaying to these nodes.

When clicking on a channel link within the channel webpage, the user issues a request to join the particular video content channel that is currently being streamed or, in another embodiment, streamed at a later, predefined point in time ("request to join"). will start doing The signaling server 330 of the virtual broadcasting server 300 responds to the join request by attempting to establish a WebSocket 226 connection with the client device 200 through the communicator 270 . As will be discussed in more detail below with respect to FIG. 3 , virtual broadcast server 300 establishes a WebSocket 226 connection with virtual broadcast server 300 for client devices 200 to receive and relay video content. and "STUN" 322 protocol to discover the public IP address of the client device 200 (e.g., inside a NAT firewall) so that it can establish WebRTC 222 connections with other client devices 200. Use

In the embodiments discussed herein, the client device 200 joins only one video channel at any given time. In other embodiments, the client device 200 can simultaneously join multiple channels without departing from the spirit of the present invention.

Client device 200 uses communicator 270 for two-way communication with signaling server 330 to facilitate rapid exchange of messages while maintaining a single TCP/IP connection open. As will be discussed in more detail below, this communication may involve (i) initial information about the client device 200 capabilities (e.g., OS, web browser, and connection type - 3G, 4G, WiFi, LAN, etc.) To provide to broadcast server 300, (ii) virtual broadcast server 300 to verify client node connectivity for subsequent WebRTC 222 node-to-node streaming of video segments over overlay networks 100; and (iii) exchanging real-time dynamic monitoring information (obtained via performance monitor 240, as discussed below) with virtual broadcast server 300.

In one embodiment, this Javascript code included in the channel webpage also includes client device 200 to determine if client device 200 is a C node (receiving video segments, but not relaying them to other client nodes). ) and provides this information to the signaling server 330. In other embodiments, certain capabilities of the client device 200 are transmitted to the virtual broadcast server 300, which determines whether the client device 200 is a C node.

This JavaScript code also facilitates communication with POI content server 380 to manage the receipt of video segments by receiver 250 for playback by standard player 232 . This process is, in effect, an extension of the standard point-to-point video streaming scenario, using the standard WebRTC 222 and adaptive streaming 224 capabilities.

In one embodiment, the standard web browser 230 interprets proprietary Javascript code from the channel webpage to periodically request manifest files as described above. These standard HTTP requests are sent to the POI content server 380, which serves the manifest files. The standard web browser 230 also adds the video segments themselves to the manifest file, including higher or lower bit rate versions of these video segments, as discussed above (e.g., when a change in bandwidth is detected). Uses standard adaptive streaming 224 libraries to request specified locations.

These requests for video segments are intercepted by proprietary Javascript code from the channel webpage - that is, because each video segment is sent from a different (feeder) node of the overlay networks 100 to a client device. Because it is pushed to 200 (so that the client device 200 does not have to initiate an HTTP "pull" request). In one embodiment (discussed in more detail below), the virtual broadcast server 300 directs one or more initial video segments to the client device 200 so that the client device 200 can begin playing the video content as quickly as possible. ), adds the client device 200 to the overlay networks 100 (and thus to the channel) immediately after the join request is received.

As receiver 250 receives the chunks of each video segment, receiver 250 receives and plays back the video segments, as well as (if client device 200 is not designated a C node) other clients. Generate content arrays 255 to facilitate relaying to nodes. Receiver 250 generates receive array 256 to compile the chunks into complete video segments, which are provided to a 3-segment buffer maintained by standard player 232. When intercepting an HTTP request for a video segment, if receiver 250 determines that a complete video segment is not yet in receive array 256, the video segment is sent to an alternate (or "fallback") location specified in the manifest file (i.e., POI content server 380). From the point of view of the standard player 232, the standard player 232 receives video segments in response to standard HTTP requests, and the video segments are actually sent to the client device 200 via the overlay networks 100. ) is being pushed to.

Moreover, in one embodiment, receiver 250 also determines the bit rate that client device 200 can handle (regardless of whether standard player 232 makes such a request in the usual way via a manifest file). Use the adaptive streaming 224 libraries to communicate to the signaling server 330 (via communicator 270). For example, if client device 200 experiences a temporary significant drop in its bandwidth (so that video segments do not arrive at receive array 256 before they are needed), client device 200 may of a (fallback) video segment may be requested from the POI content server 380 , and subsequent lower resolution video segments may then be pushed through the overlay networks 100 . When its bit rate returns to normal, the client device 200 can be pushed higher resolution video segments just like before the problem occurred.

As noted above, in one embodiment, the virtual broadcast server 300 connects virtual data trunk overlay networks (between A nodes within and across ASNs) and from each A node within the ASN to its ASN. Dynamically reconfigure overlay networks 100 for each video segment, including swarm overlay networks (to other nodes in the video segment). Unless client device 200 is classified as a C node (which receives video segments but does not relay them to other client nodes), repeater 260 has information about the node or nodes to which it will relay the video segments ( will receive instructions from the virtual broadcast server 300) relating to each video segment of the video channel it has joined. As discussed above with reference to FIG. 1 , regardless of whether client device 200 is A, B:A or B node, client device 200 may be required to relay video segments to a number of other client nodes. can

The length of the video segments (eg, 1 to 10 seconds) is defined by the creator of the video content according to the adaptive streaming 224 standards. Relay 260 will relay the video segment to each designated destination client node by pushing chunks according to the "RTCDataChannel" component of the WebRTC 222 standard (which does not require a signaling protocol).

In one embodiment, each video segment is divided into 64 KB chunks to match the size of a UDP datagram ("packet") for maximum efficiency when streamed over the MPEG2 transport protocol. The client device 200 sends and receives UDP “packets” one chunk at a time (falling back to TCP when necessary per the WebRTC 222 standard). For example, a 1 second video segment contains about 625 chunks (assuming a 1080p H.264 encoder producing about 5000 Kbps).

As receiver 250 receives the chunks of each video segment, receiver 250 generates receive array 256 to edit the chunks to construct complete video segments. Relay 260 generates relay array 257 to edit the chunks for the purpose of transmitting (relaying) the chunks to designated destination client nodes. In this way, relay array 257 acts as a buffer for incoming and outgoing chunks of a video segment. As will be discussed below, performance monitor 240 tracks the time required to stream an entire video segment to each designated destination client node, and (at a later stage in dynamically reconfiguring overlay networks 100) for use) and reports the metric back to the virtual broadcasting server 300.

In one embodiment, a receiving client node receives video segments from a single feeding node, such as client device 200 . In another embodiment, a number of potential feeding nodes are selected by virtual broadcast server 300, and those to negotiate the "top two" candidates (eg, based on current bandwidth or other monitored performance metrics). They take turns communicating among themselves and then sending chunks to designated receiving client nodes.

In another embodiment, multiple different resolutions (e.g., 1080p, 720p, and 480p) of each video segment are pushed between the A nodes, and virtual broadcast server 300 (e.g., as discussed in detail below) Instructs the A node at the root of each swarm overlay network which of its resolutions to push to other nodes in the swarm overlay network (based on the capabilities of those other nodes).

While receiver 250 is receiving chunks of a video segment for playback and repeater 260 is streaming the chunks to other designated client nodes, performance monitor 240 is directed by virtual broadcast server 300. As described above, various static and real-time dynamic performance metrics are collected, and these metrics are continuously provided to the virtual broadcasting server 300 through the signaling server 330.

As noted above, these metrics are used by virtual broadcast server 300 to dynamically reconfigure overlay networks 100 to optimize routing of the next video segment. Specifically, performance metrics are used to classify and reclassify client nodes, allocate and de-allocate slots for relaying video segments to other client nodes, and which resolutions of video segments are received and transferred to other client nodes. It is used to determine if a relay can be relayed, and ultimately to modify a subset of routing paths between client nodes when the overlay networks 100 are dynamically reconfigured. The exact manner in which these performance metrics are used by virtual broadcast server 300 will be discussed in more detail below with respect to FIG. 3 .

Static performance metrics, such as operating system, browser, and type of connection (e.g., 3G or 4G cellular, WiFi, LAN, etc.) are unlikely to change frequently and typically only signal server upon initial join request by client device 200. 330 (however, they will be reported if there is a change - e.g., change in cellular connection from 3G to 4G).

While the dynamic information may be aggregated and reported on a continuous basis (i.e., as it is collected), in one embodiment the “overhead” (how often these dynamic metrics are monitored and reported to signaling server 330) is the delivery of the video itself (i.e. , streaming chunks to and from the client device 200), various trade-offs are considered to ensure that performance is not impacted or the “payload”. In one embodiment, these metrics are used only for the next video segment, while in other embodiments, changes to the next chunk (or multiple chunks) may be made during delivery of the current video segment.

In one embodiment, two types of dynamic performance monitoring are performed. The first type of dynamic performance monitoring involves "pinging (e.g., to a Yahoo web server, a virtual broadcast server, etc.) ping)" times (or other similar measurements). Individually, these metrics provide insight into the performance of the client device 200, while providing additional insight into QoS within the ASN in which the client device 200 resides, as well as across ASNs over specific peering points. do. While virtual data trunk overlay networks (between A nodes) are of comparatively greater interest (due to congestion at peering points), congestion within an ASN is also of relevance (because congestion within an ASN, e.g. For example, it may require dynamic reconfiguration of at least a portion of one or more of the swarm overlay networks in the ASN).

Another type of dynamic performance monitoring involves the total time required to relay a video segment from one client node to another. In one embodiment, each node (except node C) has a "start" time when it transmits the first chunk of a video segment to a designated destination client node, as well as an "end" time after the last chunk of that video segment has been received. Record the time (eg, because the WebRTC 222 standard provides for verification of each packet). Performance monitor 240 sends this total time (for each video segment that the node transmits) to signaling server 330 . This metric also depends on the individual performance of the client device 200 as well as the congestion level across ASNs as well as within its ASN (e.g., if client device 200 is an A-node that feeds other A-nodes via ASN peering points). can provide insights on

In one embodiment, a user of client device 200 may also be a creator of video content. In most cases, this scenario results in continuously increasing quality of smartphone cameras (such as camera 219), enabling users to capture video events “anytime, anywhere”. However, it is also possible for users of desktop or laptop computers as well as smartphones to obtain the recorded video event from other sources.

The problem is that the client device 200 must somehow stream its video content over the Internet, over multiple ASNs, to the virtual broadcast server 300, which can be many hops away. Uploader 280 solves this problem through a proprietary "showering" protocol designed to prevent UDP packets from being delayed or blocked at intermediate routers. In one embodiment, uploader 280 is implemented via a dedicated smartphone app on client device 200, rather than relying on more limited client-side Javascript functionality.

To implement this showering protocol, the uploader 280 establishes a TCP/IP connection with the virtual broadcasting server 300, and forwards the maximum available IP packet sizes (“maximum transmission unit” (“MTU”)). UDP "bursts" are used for However, continuous UDP streams (whether sent through a single router port or distributed across multiple router ports) are often detected by intermediate routers as a "denial of service" attack and are therefore blocked. , these UDP streams can overflow a router's allocated memory (e.g., a FIFO queue), because routers typically (unlike the more typical TCP packets) This is because it allocates memory only for UDP packets.

To address these hurdles, uploader 280 not only distributes UDP packets among multiple ports (e.g., 6 ports in one embodiment), but also distributes any individual packets to prevent detection as a DOS attack. Delay packets sent through the port. In one embodiment, the delay at each port is long enough to prevent detection as a DOS attack, long enough to allow routers to allocate enough memory, but sufficient bandwidth to pass a video segment across multiple ASNs. short enough to provide, and short enough to prevent it from being recognized as the end of the UDP stream (this would stop the router from allocating memory for UDP packets and essentially "drop the UDP packets").

Because the uploader 280 delivers each video segment to the virtual broadcast server 300 in this way, the virtual broadcast server 300 can present the video content as if it were received from a more traditional CDN. Generates a channel for redistribution along overlay networks (100). In another embodiment, the virtual broadcast server 300 provides a fallback originating point source for a video segment to a client node whose current video segment did not arrive in time along the overlay networks 100. In the relatively rare scenario of , virtual broadcast server 300 uses this proprietary showering protocol.

virtual broadcasting server

Figure 3 shows one embodiment of the main server-side components of the virtual broadcasting server 300 of the present invention. As noted above, although the components of the virtual broadcasting server 300 are illustrated on a single physical hardware server, the functionality of these components may be interlinked between a number of different physical hardware devices and different software modules without departing from the spirit of the present invention. may be reallocated.

The virtual broadcast server 300 includes standard hardware/software 310 found on most hardware servers - e.g., CPU 312, memory 314, operating system 316, network adapter 317 and display. (318) - includes functions of certain standards, such as In certain embodiments, the virtual broadcast server 300 may also, for example, (i) allow client nodes to transmit and receive video to and from other client nodes, as well as establish connections with the virtual broadcast server 300. STUN 322 protocol ("Session Traversal Utilities for NAT"), which facilitates discovering the public IP addresses of client devices 200 inside a NAT firewall, so that it can be established; (ii) the WebSocket 326 protocol, which facilitates high-speed bi-directional client-server communication over a single TCP/IP connection; and (iii) standard libraries 320, which may include HTTP 328, which is used for less frequent standard communication with client web browsers, such as a standard HTML5 web browser 230. .

Virtual broadcast server 300 continuously analyzes performance metrics obtained from client nodes and dynamically reconfigures routing paths for video content channels distributed between those client nodes along overlay networks 100. Even so, there is no need to support the WebRTC 222 and adaptive streaming 224 standards, since the Virtual Broadcast Server 300 is not a client node on the overlay networks 100.

The virtual broadcast server 300 serves as a "channel originator" origination point for the overlay networks 100, specifically the virtual data trunk overlay network. In one embodiment, POI content server 380 designates one or more nearby A nodes (if possible, preferably in its ASN) to issue HTTP requests for video segments. These A-nodes effectively act as the root of a virtual data trunk overlay network, directing each video segment to other A-nodes within ASNs and via ASNs to other A-nodes and ultimately swarm overlay networks within each ASN. Push to other nodes via

As will be described in more detail below with reference to the POI content server 380, this "channel origination" function is a standard WebRTC 222 targeting cross-browser video streaming and adaptive streaming ( 224) does not require the use of libraries. As noted above, POI content server 380 also serves as an occasional fallback source of video segments for client nodes along overlay networks 100 that do not receive a current video segment in time. These client nodes issue HTTP requests to which the POI content server 380 responds by sending them the requested video segment.

Also, as noted above, the POI content server 380 determines whether the video content is obtained from the client device 200 via the uploader 280 or from a more traditional CDN (and the video content is transferred to the virtual broadcast server 300). ), whether streamed in real time or pre-provided for later streaming), serves (in one embodiment) as the originating point for all video channels.

Channel Admin 385 is responsible for setting up and maintaining each channel, while POI Content Server 380 prepares the video content itself for streaming as a channel to client nodes. In one embodiment, channel manager 385 is a signaling server when responding to join requests from client devices 200 wishing to join a particular channel and for delivery by POI content server 380 over the Internet. Generates and maintains a channel webpage for use by (330).

For support purposes, client devices 200 provide live monitoring of all video channels as well as individual viewers experiencing problems so that channel-specific and region-specific issues can be resolved (eg, when there are support requests in a particular region). A “viewer support console” is built and maintained by the channel manager 385 to support a “playout center” for In addition to these support functions, real-time monitoring of “channel analytics” is also maintained by the channel manager 385 to provide useful data to senders of video content (eg, at a CDN). For example, analysis can be performed on each video channel and current state of network nodes along overlay networks 100, as well as last mile and other issues related to video bit rates, congestion points, node latency, etc. Contains real-time metrics.

Finally, the signaling server 330 (e.g., joining the channel, providing performance metrics monitored by the client, routing to relay targets and obtaining resolution or bit rate changes A “channel management” function is provided to manage the video channels and interface with the signaling server 330 so that it has the current information needed to facilitate communication between it and the client devices 200 (regarding etc.).

Except for splash extractor 390, the remaining server-side functionality illustrated in FIG. 3 will, for simplicity, be described in terms of a single video content channel. However, it should be noted that this function is performed simultaneously, in one embodiment, for multiple channels and for various digital content at any given time.

Before client nodes access a video channel, the video content is transcoded to create multiple streams of lower resolution video segments. In one embodiment, POI content server 380 is implemented as an HTTP 228 server capable of communicating with standard HTML5 web browsers 230 in client devices 200 . Unlike signaling server 330, which establishes WebSocket 225 connections with client devices 200 for frequent two-way communication (eg, exchange of routing changes, performance data, etc.), POI content server 380 maintains a manifest file Responds to relatively infrequent client HTTP 228 requests from standard HTML5 web browsers 230 for fields, occasional video segments that do not arrive on time over overlay networks 100, and the like.

As noted above, the POI content server 380 also - i.e., nearby A nodes (at the root of the virtual data trunk overlay network, typically at the same ASN as the POI content server 380, or one or two hops away). It relies on the HTTP 228 protocol to implement its higher bandwidth channel creation functionality - by responding to HTTP requests for video segments from In other embodiments, these video segments are streamed according to WebRTC 222 and adaptive streaming 224 standards, or other video streaming techniques (showering protocol used by uploader 280 as discussed above). is pushed to the A nodes through (including ).

In one embodiment, POI content server 380 transcodes video content into three different resolutions (1080p, 720p, and 480p), while in other embodiments, a single fixed resolution for all video content, including , various other higher and lower resolutions (eg 4K, 360VR, 180VR, 240p, etc.) are supported. If the original source video is presented in a lower resolution (eg, 720p), only 720p and 480p resolutions may be supported for that video channel. This functionality facilitates adaptive bit rate streaming, whether initiated by the virtual broadcast server 300 or by client nodes (as discussed above) based on analysis of client performance metrics.

In one embodiment, POI content server 380 sends all available versions (e.g., three different resolutions) of each video segment to one or more nearby nodes (typically A nodes) in response to an HTTP request. initiates a channel by providing a , which initiates pushing each video segment along the overlay networks 100 . In another embodiment, these nodes relay all versions to B nodes (and B:A nodes) and ultimately to C nodes, so that all client nodes can use the adaptive streaming 224 capability. As described in more detail below, nodes relaying multiple resolutions to other nodes “polycast” these multiple versions of a video segment via overlay networks 100 to other client nodes.

Although POI content server 380 initiates a channel by providing video segments (in response to HTTP requests) to one or more nearby nodes, all client viewing nodes effectively receive and view each video segment simultaneously. ie, if each video segment passes through the overlay networks 100 before playback of the previous video segment ends, then all of the client viewing-side nodes are synchronized. Because client devices 200 buffer at least three video segments in this embodiment, this buffer provides some "margin for error" in case a video segment is occasionally delayed. Moreover, in another embodiment, when the POI content server 380 first starts “broadcasting” a channel, the start of the channel may be delayed to provide additional buffering. When client device 200 issues a request for a video segment directly to fallback POI content server 380 (eg, because the video segment did not arrive in time over overlay networks 100), for example For example, this buffer may be needed if the video segment traverses one or more ASNs.

As noted above, POI content server 380 also provides periodic manifest files in response to requests from client device 200 . Although these manifest files are delivered via standard HTTP 328 protocols, manifest files are relatively small and much less time critical than video segments. In one embodiment, each manifest file identifies the location of the next 8 video segments of various available bit rates. In this embodiment, the locations are fallback locations on the POI content server 380 as video segments are pushed to each client device 200 via the overlay networks 100 .

When a video content channel is ready for streaming (starting at POI content server 380 ), signaling server 330 awaits join requests from client devices 200 . Upon receiving a request to join the channel from the client device 200, the signaling server 330 may establish a WebSocket 326 connection through any NAT firewall that may exist on the client device 200. In order to do so, it relies on the STUN 322 protocol. Moreover, by identifying the public IP address of the client device 200, the signaling server 330 can pass that public IP address to other client nodes (eg, to relay a video segment to the client device 200). can be provided to

Once the WebSocket 326 connection is established, the client device 200 determines its capabilities, including, in one embodiment, whether the client device 200 is a C node (e.g., assumed for cellular connections in this embodiment). (eg, OS, web browser, and connection type - 3G, 4G, WiFi, LAN, etc.) is provided to the signaling server 330 . The client device 200 also provides its ASN location to the signaling server 330, which will be used later to add the client device 200 to the overlay networks 100.

In one embodiment, signaling server 330 directs one or more messages to client device 200 (via overlay networks 100) so that client device 200 can begin playing the video content of a channel as soon as possible. Give priority to delivery of early video segments. To initiate this process, signaling server 330 passes control to overlay network generator 350, which adds client device 200 to the swarm overlay network within its ASN (e.g., its by instructing the B node in the ASN to relay the video segments to the client device 200). It should be noted that the client device 200 has not yet been classified and will not relay any video segments to other client nodes yet. However, by being part of the overlay networks 100, the client device 200 can begin receiving video segments and playing the video content of a channel, as well as collect client performance metrics, which Its classification can be facilitated.

The signaling server 330 then obtains the upstream and downstream bandwidth of the client device 200 (via its WebSocket 326 connection). It should be noted that this metric is not very useful since a connection may cross multiple ASNs (even though signaling server 330 knows the ASN location of client device 200). A more relevant metric would relate to communication between the client device 200 and other client nodes within its own ASN.

Upon receiving client performance information (collected by the performance monitor 240 on the client device 200) from the client device 200 (and from other client nodes), the signaling server 330 performs an initial analysis and, For subsequent use by overlay network generator 350 and deep mapper 360 in dynamically reclassifying client nodes and reconstructing overlay networks 100 for the next video segment, as described below: It forwards that information to the performance tracker 340. Performance tracker 340 monitors the performance of each client node and determines if the client node is still “alive”. For example, if client device 200 closes the connection and leaves the channel, or does not respond to the “ping” within a threshold amount of time, then client device 200 (either intentionally or as a result of a hardware or software failure) ) will be considered as having left the channel. Performance tracker 340 also converts the client performance metrics into a format suitable for storage in Historical Performance DB 345 and for use by Overlay Network Generator 350 and Deep Mapper 360 .

In one embodiment, overlay network generator 350 also evaluates current and historical client performance metrics (maintained in historical performance DB 345), with the help of deep mapper 360, and for each video segment , dynamically (i) reclassify client nodes and (ii) a virtual data trunk overlay network (to relay video segments between A nodes, within and across ASNs) and (video segments to ASNs). A continuation of optimizing routing paths by generating and reconfiguring overlay networks 100, including a swarm overlay network (to relay from each A node in the ASN to any other B:A, B, and C nodes in that ASN) is in charge of the process. The topology of overlay networks 100 is maintained in overlay network DB 375 for use by overlay network generator 350 and deep mapper 360 .

For the performance metrics received from the newly added client device 200, the overlay network generator 350 uses the metrics to first classify the client device 200. In one embodiment, this process is also used to potentially reclassify client nodes for every video segment (not just when client nodes join a channel). Although client nodes typically do not reclassify very often, a client may experience temporary bandwidth degradation (eg, due to a household microwave or other interference). Also, when more A nodes are needed (eg, for redundancy or due to client nodes leaving the channel), the B:A nodes can be upgraded to A nodes. Other problems detected within or across ASNs may also require certain nodes to be reclassified.

Overlay network generator 350 may receive chunks of video segments pushed from other client nodes (via incoming slots) and may relay the chunks of video segments to other client nodes (via outgoing slots). Incoming and outgoing slots (i.e., network ports) are assigned to the client device 200. The WebRTC 224 standard supports 256 incoming and outgoing ports (slots), but in one embodiment only a single incoming slot (to maximize the quality of the video content that can be played on the client device 200). is allocated (to maximize throughput over overlay networks 100 and to support a wide range of client devices 200 and limited bandwidth connections) and up to eight outgoing slots are allocated. As seen above, A nodes have four outgoing slots for relaying video segments to other A nodes over ASN peering points, and four outgoing slots for relaying video segments to other A nodes within their ASN. be assigned As will be explained below, not all of the assigned slots are necessarily in use at any given point in time.

Overlay network generator 350 analyzes the downstream and upstream bandwidth of client device 200 to facilitate the classification process. As noted above, when client device 200 joins via a cellular connection (3G, 4G or even LTE), client device 200 is automatically deemed too unreliable to relay video segments, and thus C node classified as In other embodiments, this automatic classification may be limited to certain cellular connections (eg, 3G), or eliminated entirely.

In one embodiment, overlay network generator 350 may be configured to facilitate further classification, (1) LAN connections (eg, 100/100), (2) fiber optic connections (100/50), (3) ADSL. It uses typical downstream/upstream bandwidth categories (in Mbps), including connections (100/20), cable connections (100/10) and WiFi connections (which can vary greatly). In this embodiment, if client device 200 is not yet considered a C node and has an upstream bandwidth of at least 50 Mbps, then client device 200 initially acts as an A node (or whatever deep mapper 360 has to its ASN). If it indicates that no additional A nodes are needed, it is classified as a B:A node). Otherwise, the client device 200 would be classified as a B node.

As will be discussed below, the overlay network generator 350 calculates the number of available outgoing slots it can use before determining the extent to which it must dynamically reconfigure the overlay networks 100 (if any). To do so, further analyze (in one embodiment) the upstream bandwidth of the client device 200 . Overlay network generator 350 also determines the extent to which client device 200 can receive and/or polycast multiple resolutions.

In one embodiment, all of a client node's downstream bandwidth is used for its single incoming slot, while only one-third of its upstream bandwidth is used to relay video segments between its outgoing slots. All of its upstream bandwidth is not used because relaying the video segments may disrupt TCP/IP and other connections that the client device 200 is using for other applications.

Overlay network generator 350 analyzes the downstream bandwidth of client device 200 (even if classified as a C node) to determine the number of resolutions that client device 200 can support through its single ingress slot. do. For example, if 1080p requires a bit rate of 3 Mbps, 720p requires a bit rate of 1.5 Mbps, and 480p requires a bit rate of 500 Kbps, the client device 200 can use all three resolutions At least 5 Mbps to support, at least 4.5 Mbps to support 1080p and 720p, at least 3 Mbps to support only 1080p, at least 2 Mbps to support 720p and 480p, at least 1.5 Mbps to support only 720p, and to support only 480p This will require a downstream bandwidth of at least 500 Kbps. In one embodiment, bit rates below 500 Kbps will not be supported. In other embodiments, lower resolutions may be supported, and other techniques may be used to reduce bandwidth requirements (eg, greater compression, different video formats, etc.).

As noted above, in one embodiment, A, B: A and B nodes can also be considered polycasting nodes that can relay multiple resolutions to other nodes via one or more of their outgoing slots. . In this regard, overlay network generator 350 analyzes the upstream bandwidth of client device 200 to determine the number of resolutions that client device 200 can relay to other client nodes.

Since the client node can only use 1/3 of its upstream bandwidth in this embodiment, the client device 200 must polycast at least 15 Mbps, 1080p and 720p (per outgoing slot) to polycast all three resolutions. At least 13.5 Mbps (per outgoing slot) to cast, at least 9 Mbps (per outgoing slot) to polycast only 1080p, at least 6 Mbps (per outgoing slot) to polycast 720p and 480p, to relay only 720p (outgoing At least 4.5Mbps per slot), and at least 1.5Mbps upstream bandwidth (per outgoing slot) to relay only 480p.

A client device 200 cannot relay resolutions it has not received. Moreover, the polycasting capability of the client device 200 is taken into account in terms of which other client nodes can receive multiple resolutions, as described below. However, as noted above, client device 200 uses adaptive streaming 224 implementations to request lower or higher resolution versions of video segments when experiencing significant changes in its bandwidth. When the client device 200 receives multiple different resolutions of a video segment, it will simply play back the highest resolution it received.

Assuming that the client device 200 is not a C node, the overlay network generator 350 analyzes the upstream bandwidth of the client device 200 as well as the ability of the client device 200 to polycast multiple resolutions. By considering the degree, the number of available outgoing slots that the client device 200 can use is calculated. For example, if client device 200 is classified as a node A with a LAN connection with an upstream bandwidth of 100 Mbps, client device 200 can only use the ASN to polycast video segments within and across ASNs. About 6 outgoing slots are available. In this embodiment, overlay network generator 350 will allocate 4 slots for polycasting across ASNs to other A nodes (giving priority to these inter-ASN slots), and another A in its ASN. Leave the remaining 2 slots for polycasting to nodes. In other embodiments, these assignments may of course vary without departing from the spirit of the present invention.

Similarly, if the client device 200 is classified as a B:A or B node with a cable connection with an upstream bandwidth of 10 Mbps, the client device 200 can only polycast 720p and 480p resolutions or only 1080p. Only one outgoing slot can be used to transmit. In one embodiment, higher quality resolutions are given priority (to the extent that nodes can receive that resolution), so one slot will be allocated for 1080p only. Here too, these assignments may vary without departing from the spirit of the invention.

Having classified the client device 200 and determined the number of slots available (including polycasting multiple resolutions), the overlay network generator 350 uses the overlay networks 100 to optimize routing paths. Determines the degree to dynamically reconfigure , and if the client device 200 is node A, the overlay network generator 350 first obtains congestion levels for each inter-ASN path between A nodes from the deep mapper 360. (discussed in more detail below), and then dynamically reconfigure at least a portion of the virtual data trunk overlay network to include the client device 200 .

For example, given a set of weighted routes (each route having a “congestion level” weight), overlay network generator 350 (e.g., similar to GPS navigation route establishment) can divide a video segment into node A It uses standard path finding techniques to determine the optimal path to distribute between them. However, this process requires multiple relaying slots - e.g., 4 outgoing slots for A nodes relaying to A nodes within the ASN, and 4 outgoing slots for A nodes relaying to A nodes via the ASN peering point. It should be noted that it is slightly complicated by the use of outgoing slots). However, this is just a slight variation on the simplest case where node A has only one outgoing slot. In other words, the overlay network generator 350 keeps track of the number of open (unused) slots during creation or reconfiguration of the virtual data trunk overlay network, and if it no longer has any unused open slots, a particular A node Stop assigning as a relay source.

If the client device 200 is a B:A or B node, the overlay network generator 350 dynamically reconfigures some or all of the inter-ASN swarm overlay networks in the ASN in which the client device 200 resides. It should be noted that if there are multiple A nodes within that ASN, their routes to each other will be determined as part of the virtual data trunk overlay network. In one embodiment, (if enough slots are available) While only one A node may be used to create a swarm overlay network, in other embodiments other nodes may be allocated equally among multiple A nodes or distributed based on relative upstream bandwidth or other metrics. have.

Regarding any particular A nodes and the rest of the B, B:A and C nodes in the ASN, these nodes are first based on their classification (i.e., B:A, then B, then C) and then their relative bandwidth. (ie, the number of available slots that can be used, as described above). Note that the swarm overlay network is hierarchical in this embodiment, as each node has only a single feeder node. Similar techniques may be used for non-hierarchical “mesh” swarms in other embodiments.

In this hierarchical swarm embodiment, the process starts with a root A node, which will have a certain number of outgoing slots available (eg, two outgoing slots). Those slots will be routed to the next level in the hierarchy - eg the two B:A nodes with the highest number of available slots that can be used. Once these routes are determined, the available outgoing slots of those nodes will be routed to the remaining B:A nodes with the largest number of available slots. This process continues down the hierarchy (via B nodes and finally C nodes) until all paths have been determined.

Given the relatively high speed (less than 1 ms) of relaying between nodes in an ASN, the length of the chain under any client node (e.g., 100 client nodes, each with a single outgoing slot) is relatively small. Note that it doesn't matter. Given a one-second video segment, a chain of hundreds of nodes can still be accommodated (which is rare since many nodes in an ASN are likely to support multiple outgoing slots). If not all of the nodes can be included in a swarm (e.g. C nodes and B nodes with 0 available slots are not considered), they have open slots in that ASN to be allocated when they become available. Additional nodes will be required. In the meantime, these nodes will be instructed to request video segments from POI content server 380 .

Limitations of BGP routing protocols to understand the significance of ASN peering point congestion before turning to Deep Mapper 360, which predicts and quantifies congestion levels across ASN peering points (eg, for the next minute). It is helpful to understand them. BGP routers do not have the ability to determine and predict congestion “at routing”. BGP routers are only aware of their own routers, and the latency is "one hop away" through the ASN peering point. BGP routers do not know the hop number or latency to any final destination, which may be many hops away over multiple ASN peering points. Given a choice of multiple ASN peering points, the BGP routers essentially choose the one with the current maximum available bandwidth (i.e. open slot and lowest latency one hop away).

In contrast, deep mapper 360 uses its knowledge of the Internet's underlying architecture. In one embodiment, deep mapper 360 maintains an ASN interconnection map (including ASNs and their various peering point interconnections) of the Internet, as schematically illustrated in FIG. 1 . This map does not change often, but in one embodiment is monitored every 5 minutes to catch these infrequent changes.

However, the overlay networks 100 built on top of these ASNs are frequently analyzed by the virtual broadcast server 300 (e.g., via client-side monitoring as discussed above) and possibly every video segment (e.g., one in the embodiment every second). However, in practice, the overlay networks 100 only operate when it is actually warranted - eg, not only when new nodes join or leave the channel (with current and historical information maintained in the historical performance DB 345). based) - even when enough problems are detected - to be corrected.

For example, in one embodiment multiple internal "congestion thresholds" are used. Upon initial detection of a relatively low congestion threshold within an ASN or specific to a particular client device 200, the overlay network generator 350 merely "indicates" to the client device 200 or ASN (e.g., the next video segment) to see if the problem repeats. In such a case, the overlay network generator 350 may lower the resolution (and thus bit rate) of the next video segment relayed to that client node (or all client nodes in the "offending" ASN). Ultimately, if the problem gets worse (eg, a higher congestion threshold is exceeded), a portion of the overlay networks 100 (eg, a subset IP range within an ASN) may be dynamically reconfigured. Finally, the entire ASN, or perhaps the virtual data trunk overlay network itself, may require dynamic reconfiguration.

In either case, the goal of these congestion thresholds is to prioritize deteriorating to more serious problems that cause video segments to be lost or even cause client nodes to rely on obtaining video segments from the POI content server 380's fallback location. , to proactively identify and correct problems.

By maintaining an awareness of the ASN interconnection map of the Internet and the ASN location of nodes on overlay networks 100 and monitoring the current and historical performance of these nodes in real time, the Deep Mapper 360 is able to use any client node minimizes the possibility of unnecessarily relaying a video segment to a distant client node (eg, many hops away over multiple ASN peering points). For example, as an initial matter in one embodiment, the virtual data trunk overlay network can transfer video segments from one A-node to another A-node within the same ASN or via a single ASN peering point to another A-node within a nearby ASN (whenever possible). ) tend to route.

However, not all single hops are created equal. For example, deep mapper 360 may detect that the peering point between “ASN 1” and “ASN 2” is becoming congested over time (based on client performance metrics maintained in historical performance DB 345). can "know" that the 2-hop route from "ASN 1" via "ASN 3" to "ASN 2" is actually faster than the current 1-hop route (or based on recent and past trends). It can be "predicted" that it will be faster in the very near future. By quantifying peering point congestion based on the actual current and past performance of the A nodes across the peering points, deep mapper 360 - maybe every video segment or at least peering point congestion (based on an internal threshold) to facilitate dynamic reconfiguration of the topology of the virtual data trunk overlay network - when such changes are required.

In one embodiment, deep mapper 360 uses a scale of 1 to 10, where 1 is the lowest level of predicted short-term congestion and 10 is the highest, using each pair of A nodes (to the same ASN or to different ASNs). ) quantifies congestion for As noted above, the overlay network generator 350 uses this congestion level "score" to compare the different potential routes between A nodes and determine the most efficient route (i.e., the lowest "weighted hop" route). " use As a result, the most "furthest" A-nodes (measured in weighted hops) from the POI content server 380 are the time required for a video segment to traverse the virtual data trunk overlay network from the POI content server 380 to these A-nodes. will minimize the amount of

In one embodiment, for each pair of A nodes, deep mapper 360 generates a predicted congestion level score for each path from one A node to another A node, followed by that pair of A nodes. Selects the lowest congestion level score to be applied to, and returns it to the overlay network 350. In other embodiments, deep mapper 360 generates a different function of the predicted congestion level scores (such as mean, median, etc., for each path from one A node to another A node).

Deep mapper 360, in one embodiment, is a deep learning engine that continuously analyzes performance metrics maintained in historical performance DB 345 and predicts congestion levels across ASN peering points (e.g., 1 minute in the future). to be. It should be noted that, like any deep learning engine, deep mapper 360 uses a number of non-linear transformations to model the behavior of ASN peering points with respect to traffic between A nodes across those peering points.

As discussed above, the deep mapper 360 cannot monitor most of the Internet traffic that actually passes through the peering points, and over time, this traffic passes through the peering points at inter-ASN hops between A nodes. Only the effect of the branch is monitored. As more performance metrics are obtained, deep mapper 360 calculates the time required for these inter-ASN hops (e.g., compared to typically much less congested intra-ASN hops, which are also monitored in this embodiment, however) - which is then Quantified as the relative congestion level - can better predict

Because the congestion level of peering points is highly dynamic, these predictions may only be accurate for a short period of time. However, if this analysis is performed continuously and can change for the next one-second video segment, it is not critical that the prediction be accurate over a long period of time.

In one embodiment, deep mapper 360 initially quantifies ASN peering points based on very coarse information (ie, before many client performance metrics have been obtained). For example, if an ASN has 1000 peering points, it can be assumed that that ASN is a backbone that is likely to be much faster than another ASN with 6 peering points. As more client performance metrics are obtained, these ASN peering point congestion levels will become more accurate. In another embodiment, multiple “learning nodes” are deployed to “jump start” a new channel. These running nodes are transmit-only nodes that do not watch the video, and are only deployed to quickly provide client performance information so that deep mapper 360 can start making more accurate predictions sooner than it would otherwise.

Moreover, in one embodiment, deep mapper 360 also takes into account congestion within the ASN, since this implies the need for creation of, for example, additional A nodes within the ASN and thus additional swarm overlay networks. Because you can. For example, if many client nodes in the ASN are taking progressively longer time to acquire video segments over time, deep mapper 360 may indicate to the ASN to indicate that additional A nodes are needed. and overlay network generator 350 may “promote” one or more B:A nodes to A nodes, resulting in partial reconfiguration of the virtual data trunk overlay network, and ultimately new swarm overlay networks within the ASN. in need. In another embodiment, deep mapper 360 applies deep learning techniques within each ASN and helps overlay network generator 350 generate swarm overlay networks within the ASN.

Thus, the overlay network generator 350 and the deep mapper 360, in order to minimize relaying of video segments over unnecessarily distant paths (i.e., across multiple ASN peering points), the Internet's underlying architecture ( ASN interconnection map) and the ASN location of the client nodes overlaid on the architecture. Moreover, overlay network generator 350 and deep mapper 360 also continuously analyze real-time client performance metrics obtained by client devices 200 and these metrics (often congestion at ASN peering points). to dynamically reconfigure the overlay networks 100 in the case of presenting significant problems (due to . As a result, the QoS variability of the Internet can be monitored, and "before it happens" (based on predicted congestion levels generated by the deep mapper 360) by dynamically rerouting around these problems (in particular ASN The impact on client nodes of congestion (at peering points) can be minimized.

In one embodiment, virtual broadcast server 300 identifies trending video events (“splashes”) and allows users to search the domain of these events to obtain desired splash results from POI content server 380 as a video channel. includes a splash extractor 390 search engine for immediate streaming (where these channels would otherwise not be available from the virtual broadcast server 300).

In one embodiment, splash extractor 390 continuously collects data from multiple news sources—eg, via APIs for Twitter, RSS Feeds, Reddit, and tens of thousands of online magazines. On average, thousands of unique "current events" appear from these sources every hour. Splash extractor 390 uses a novel automated method to identify these trending events (splashes) and find and extract related videos that can be acquired and streamed through POI content server 380 .

Splash extractor 390 identifies “deviations from the norm” to detect splashes. For example, in the domain of news sources, a baseline is created (without the need for normalized data) using, for example, the standard Levenshtein comparison algorithm. On average, no more than a few sources will discuss the same "topic" (i.e., collection of keywords) in a short period of time, unless a particular topic actually represents a trend. At that point (eg, when 15 or more sources discuss the same topic within a short period of time), the topic is identified as a deviation and thus a splash.

Splash extractor 390 then - in one embodiment, by using standard neural network techniques to learn and predict unique keywords from "splash related" articles - selects the "most important" keywords from those sources (e.g. , 40 keywords in one embodiment). These keywords are then categorized (eg, news, sports, etc.) and ranked by frequency.

Splash extractor 390 then uses those keywords to search social media for videos related to each splash, and indexes the relevant text associated with the potential splash video channels. Users can then search that index, or simply browse the categories of splash video events. Upon selecting a result (whether searched or browsed), users can immediately stream the video they want. In one embodiment, the user is simply linked to the video's current source, while in another embodiment, the video is acquired through the virtual broadcast server 300 and streamed from the POI content server 380 (e.g., many useful when many concurrent users request the same splash video channel).

Dynamic video streaming process

Having discussed the main client-side and server-side components of the virtual broadcasting system of the present invention, flowchart 400 of FIG. 4 shows how these components interact dynamically. In other words, flowchart 400 represents one embodiment of the dynamic streaming process of the present invention implemented by these components. It should be noted that flow chart 400 presents steps in terms of interactions between client-side and server-side functions, as much of this process is event-based and not linear.

Step 401 is performed by uploader 280, where the video event is either captured by a client node (e.g., smartphone camera 219 on client device 200), generated digitally, or obtained from an external source. represents the process being (and previously described). In either case, the client (eg, client device 200) then streams the video segments (whether captured live or recorded) of that video event to the virtual broadcast server 300.

Whether the video events are obtained from clients or from a more traditional CDN (and whether they are recorded or streamed play-by-play), the virtual broadcast server 300, as discussed above, at step 410, is a POI content server ( 380) prepares each video channel for live streaming. At this point, in one embodiment, a channel webpage is generated and ultimately encounters a potential client node. When the user of the client device 200 clicks on the desired channel, a join request is sent to the signaling server 330 along with the client capabilities (operating system, browser, type of connection, etc.). Alternatively, the user of client device 200 encounters a trending splash video event (as discussed above) and selects that video event for streaming as a video channel from POI content server 380 (at step 410). do.

At step 412, the signaling server 330 verifies the client connection to the channel (eg, by using the STUN 322 protocol to identify the client's public IP address), and then It establishes a WebSocket 326 connection through an arbitrary NAT firewall and later provides its public IP address to other client nodes for relaying video segments to that client. The Signaling Server 330 then passes control to the Overlay Network Creator 350, which adds the (yet unclassified) client as a node on the Overlay Networks 100, and allows the user to immediately select the video channel in step 415. Initial video segments will be pushed from the node on the overlay networks 100 to the client so that they can start viewing (step 414).

The signaling server 330 then classifies the client device 200 as an A, B:A, B or C node, at step 416, and at step 430 includes the client device 200 into the network topology. It uses both the overlay network generator 350 and the deep mapper 360 to dynamically reconfigure the inter-ASN (virtual data trunk) and intra-ASN (swarm) overlay networks 100 to Signaling server 330 then provides relevant route information to other client nodes to begin relaying the video segments to client device 200 .

The POI content server 380, in step 435, then sends the video segments along the current (reconfigured) overlay networks 100 to those nodes that are the originating points of the video channel to nearby nodes ( Typically in response to HTTP requests from A nodes), each video segment is relayed from node to node until relayed to and viewed by the client device 200 .

While the client device 200 receives the chunks for viewing each video segment of the channel and, at step 450, edits them (and possibly also at step 440), the chunks are transferred to other designated client nodes. While relaying), the client device 200 also monitors its performance at step 425 and provides the client performance metrics to the signaling server 330, as discussed above with respect to the performance monitor 240. . Additionally, as each video segment is requested, these requests are intercepted (in one embodiment, by client Javascript code at receiver 250) at step 455, for reasons discussed above. As such, video segments are pushed along the overlay networks 100 to the client device 200 . The arrow from step 455 to step 425 simply indicates that the monitoring process in step 425 is a continuous process, simultaneous with the receiving, viewing and relaying of chunks of video segments.

Also, as noted above, the client device 200, even if video segments are pushed to the client device 200 from other client nodes, sends the POI content server 380 the manifest files (e.g., the location of the next 8 video segments). ) periodically initiates HTTP requests for Sometimes, if a video segment does not arrive in time, the client device 200 will request the video segment directly from the POI content server 380, which is a fallback location. Moreover, sometimes, in accordance with the adaptive streaming 224 standards, the client device 200 may also (eg, when detecting a change in its performance levels) to request a modified bit rate for subsequent video segments. , can contact the POI content server 380. However, as noted above, the receiver 250 may well detect this request earlier and contact the virtual broadcast server 300 to make these changes via the overlay networks 100, giving the feeding client node a more It may be instructed to automatically push lower or higher resolution video segments to the client device 200 (ie, not in response to its request).

At step 452 , POI content server 380 responds to these HTTP requests and delivers the requested manifest files and fallback video segments to client device 200 . As noted above, the change in bit rates is addressed via the overlay networks 100 (and in step 430), resulting in lower or higher resolution video segments being pushed to the client device 200.

Step 454 follows the continuous process performed by performance tracker 340, overlay network generator 350, and deep mapper 360 (in one embodiment, performed for each video segment and described in detail above). include In this step 454, the client capability information is continuously updated and, if necessary, (as indicated by the arrow from step 454 to step 430), the overlay networks 100 in step 430. It is dynamically reconfigured, and new routing information is provided to relevant relay nodes through the signaling server 330.

Finally, at step 460, splash extractor 390 continuously identifies trending splash video events that users of client devices 200 can browse or search for, then for immediate viewing as discussed above. Stream.

The invention has been described herein with reference to specific embodiments as illustrated in the accompanying drawings. In view of this disclosure, it will be appreciated that additional embodiments of the concepts disclosed herein may be devised and implemented within the scope of the invention by those skilled in the art.

Claims (1)

in the way.

Images