US20210084425A1 - Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape - Google Patents
Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape Download PDFInfo
- Publication number
- US20210084425A1 US20210084425A1 US17/094,589 US202017094589A US2021084425A1 US 20210084425 A1 US20210084425 A1 US 20210084425A1 US 202017094589 A US202017094589 A US 202017094589A US 2021084425 A1 US2021084425 A1 US 2021084425A1
- Authority
- US
- United States
- Prior art keywords
- audio
- client
- stream
- remote users
- streams
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S1/00—Two-channel systems
- H04S1/002—Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
- H04L65/403—Arrangements for multi-party communication, e.g. for conferences
-
- H04L65/604—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field
Definitions
- This application relates generally to network-based conferencing wherein participants “listen” to and participate in the conference using a web browser and, in particular, to techniques to enhance delivery of audio to such conference participants
- CDN content delivery network
- overlay network that is operated and managed by a service provider.
- the service provider typically provides the content delivery service on behalf of third parties (customers) who use the service provider's shared infrastructure.
- a distributed system of this type typically refers to a collection of autonomous computers linked by a network or networks, together with the software, systems, protocols and techniques designed to facilitate various services, such as content delivery, web application acceleration, or other support of outsourced origin site infrastructure.
- a CDN service provider typically provides service delivery through digital properties (such as a website), which are provisioned in a customer portal and then deployed to the network.
- Real-time communications e.g., videoconferencing, shared document editing, screen sharing, and the like
- WebRTC an Internet standard
- WebRTC seeks to take the most critical elements of video chat and move them to one of the most commonly used tools for accessing the Internet, namely, a web browser.
- WebRTC is supported with plugins by both Google Chrome and Mozilla Firefox. It allows the browser to access the client machine's camera and microphone, provides a method for establishing a direct connection between two users' browser and to use that connection to send audio and video, and it provides a method for sending arbitrary data streams across a connection.
- An overlay network platform facilitates a multi-party conference. End users participate in the conference using client-based web browser software, and using a protocol such as WebRTC. According to this disclosure, an enhanced “audio” experience for a user participating in the conference is provided. To this end, a method is implemented at a client machine to render different participants' audio in the multi-party conference at different positions in a three-dimensional (3D) space. In operation, and given a set of audio streams that comprise the conference and that are being received, a three-dimensional (3D) position for each audio stream in a 3D soundscape is computed.
- the 3D position for each audio stream is computed by identifying a target area region of space (e.g., a 45° cone positioned a given distance in front of the listener), and then determining the 3D position for each stream in the target area region of space.
- the relative 3D positions may be determined based on contextual information related to the content of an audio stream, as well as a computed “relevance” of one stream versus another.
- the relative 3D positions of the audio streams may be deterministic (e.g., equally spaced).
- Given streams may be positioned relative to one another, e.g., up or down, left or right, forward or backward.
- Each audio stream is then rendered (output or played) to the listener at the 3D position computed, e.g., using the HTML5 Web Audio API.
- FIG. 1 is a block diagram illustrating a known distributed computer system configured as a content delivery network (CDN);
- CDN content delivery network
- FIG. 2 is a representative CDN edge machine configuration
- FIG. 3 depicts the various components of a web-based collaboration solution according to this disclosure
- FIG. 4 illustrates a multi-party videoconference setup that is enabled by associating the web-based solution of this disclosure with an overlay network
- FIG. 5 is a process flow depicting a method of this disclosure.
- a distributed computer system 100 is configured as a content delivery network (CDN) and is assumed to have a set of machines 102 a - n distributed around the Internet.
- CDN content delivery network
- machines 102 a - n distributed around the Internet.
- machines typically, most of the machines are servers located near the edge of the Internet, i.e., at or adjacent end user access networks.
- a network operations command center (NOCC) 104 manages operations of the various machines in the system.
- Third party sites such as web site 106 , offload delivery of content (e.g., HTML, embedded page objects, streaming media, software downloads, and the like) to the distributed computer system 100 and, in particular, to “edge” servers.
- content e.g., HTML, embedded page objects, streaming media, software downloads, and the like
- content providers offload their content delivery by aliasing (e.g., by a DNS CNAME) given content provider domains or sub-domains to domains that are managed by the service provider's authoritative domain name service. End users that desire the content are directed to the distributed computer system to obtain that content more reliably and efficiently.
- the distributed computer system may also include other infrastructure, such as a distributed data collection system 108 that collects usage and other data from the edge servers, aggregates that data across a region or set of regions, and passes that data to other back-end systems 110 , 112 , 114 and 116 to facilitate monitoring, logging, alerts, billing, management and other operational and administrative functions.
- Distributed network agents 118 monitor the network as well as the server loads and provide network, traffic and load data to a DNS query handling mechanism 115 , which is authoritative for content domains being managed by the CDN.
- a distributed data transport mechanism 120 may be used to distribute control information (e.g., metadata to manage content, to facilitate load balancing, and the like) to the edge servers.
- a given machine 200 in the content delivery network comprises commodity hardware (e.g., an Intel Pentium processor) 202 running an operating system kernel (such as Linux or variant) 204 that supports one or more applications 206 a - n .
- given machines typically run a set of applications, such as an HTTP proxy 207 (sometimes referred to as a “global host” or “ghost” process), a name server 208 , a local monitoring process 210 , a distributed data collection process 212 , and the like.
- the machine may include one or more media servers, such as a Windows Media Server (WMS) or Flash server, as required by the supported media formats, or it may utilize HTTP-based delivery of chunked content fragments that constitute a stream.
- WMS Windows Media Server
- a CDN edge server is configured to provide one or more extended content delivery features, preferably on a domain-specific, customer-specific basis, preferably using configuration files that are distributed to the edge servers using a configuration system.
- a given configuration file preferably is XML-based and includes a set of content handling rules and directives that facilitate one or more advanced content handling features.
- the configuration file may be delivered to the CDN edge server via the data transport mechanism.
- U.S. Pat. No. 7,111,057 illustrates a useful infrastructure for delivering and managing edge server content control information, and this and other edge server control information can be provisioned by the CDN service provider itself, or (via an extranet or the like) the content provider customer who operates the origin server.
- the CDN may include a storage subsystem, such as described in U.S. Pat. No. 7,472,178, the disclosure of which is incorporated herein by reference.
- the CDN may operate a server cache hierarchy to provide intermediate caching of customer content; one such cache hierarchy subsystem is described in U.S. Pat. No. 7,376,716, the disclosure of which is incorporated herein by reference.
- the CDN may provide secure content delivery among a client browser, edge server and customer origin server in the manner described in U.S. Publication No. 20040093419.
- Secure content delivery enforces SSL-based links between the client and the edge server process, on the one hand, and between the edge server process and an origin server process, on the other hand. This enables an SSL-protected web page and/or components thereof to be delivered via the edge server.
- a content provider identifies a content provider domain or sub-domain that it desires to have served by the CDN.
- the CDN service provider associates (e.g., via a canonical name, or CNAME) the content provider domain with an edge network (CDN) hostname, and the CDN provider then provides that edge network hostname to the content provider.
- CDN edge network
- those servers respond by returning the edge network hostname.
- the edge network hostname points to the CDN, and that edge network hostname is then resolved through the CDN name service. To that end, the CDN name service returns one or more IP addresses.
- the requesting client browser then makes a content request (e.g., via HTTP or HTTPS) to an edge server associated with the IP address.
- the request includes a host header that includes the original content provider domain or sub-domain.
- the edge server Upon receipt of the request with the host header, the edge server checks its configuration file to determine whether the content domain or sub-domain requested is actually being handled by the CDN. If so, the edge server applies its content handling rules and directives for that domain or sub-domain as specified in the configuration. These content handling rules and directives may be located within an XML-based “metadata” configuration file.
- WebRTC Traditional point-to-point communication and collaboration tools, such as LifeSize, WebEx, and Skype, require specialized applications and infrastructure.
- the WebRTC APIs have the potential to make direct point-to-point communication ubiquitous by bringing it to the browser.
- WebRTC enables video communication and collaboration without any additional plugins or software. That said, however, there are still a number of challenges remaining for an entity that wants to deploy a WebRTC-based application.
- a relay infrastructure using TURN
- multi-user video conferencing over WebRTC requires full mesh connectivity between all users; that is, a separate connection must be established between each pair of users.
- an overlay network fabric is used.
- a solution that facilitates multi-user collaboration such as video conferencing, chat, document sharing, and desktop sharing, is provided.
- a primary use case as described below is for high-quality video conferencing that is scalable to large numbers of users, this is not a limitation, as the cloud-supported multiplexing and relay techniques herein may be used to provide other multi-user collaboration, such as chat, document sharing, and desktop sharing, all in a seamless and scalable manner.
- the overlay network can also provide additional functions and features to support a collaboration session; these may include, without limitation, persistent storage and recording of sessions and documents, integration with existing videoconferencing and telecommunications infrastructure (LifeSize rooms, PSTN, etc.), and others.
- FIG. 3 depicts a representative architecture 300 for an enterprise collaboration platform using an overlay network according to an aspect of this disclosure.
- the front-end application (app) 300 preferably is built on a number of components (described below) that are preferably accessed through the one or more
- the platform components 304 include signaling 306 , connectivity 308 , multiplexing 310 , storage 312 , and PSTN integration 314 .
- the platform 304 comprises part of an overlay network (or leverages elements thereof), but this is not a requirement, as the solution herein may be provided as a standalone architecture.
- the notion of a “component” herein may involve multiple machines, whether co-located or distributed, as well as the processes and programs executing thereon.
- the signaling component 306 preferably is a distributed signaling system that keeps track of users' state (e.g., “Online”, “Away”, “Busy”, etc. . . . ), and it is used to transmit the information (i.e., SDP) necessary to initiate an RTCPeerConnection.
- the signaling component 306 preferably integrates with various user authentication and identity management solutions, although this is not a requirement.
- the connectivity component 308 manages video, voice and data connections routed though the overlay network platform to handle Network Access Translation (NAT) traversal, as well as to provide enhanced performance and security.
- NAT Network Access Translation
- the multiplexing component 310 comprises multiplexing machines to allow for scalable, multi-peer sessions. This component makes it so that each peer only needs to upload its media stream once. Other peers are then able to access peers' media streams through overlay network edge machines (rather than by direct connections to peers).
- the multiplexing component provides for multiplexing in the cloud to significantly reduce edge bandwidth requirements that would otherwise be required to support WebRTC (which otherwise dictates a new connection be setup for pair of peers in a multi-user collaboration). With this approach herein of using the overlay network, there is no requirement to setup a new connection for each pair of peers in a multi-peer collaboration (conference, chat, etc.) session.
- the multiplexing component 310 intelligently adjusts the quality of different users' streams to enhance performance—e.g., only deliver HD streams for people who are currently speaking, deliver lower-quality streams to mobile devices, etc.
- the storage component 312 allows overlay network customers to (optionally) store data from a collaboration session (e.g., record a meeting, save work on a collaborative document, etc.).
- the PTSN integration component 314 allows users to join sessions from the PSTN and legacy telecommunications equipment, and it allows users to call out over the PSTN.
- the platform may include a transcoding component that allows for communications between browsers that do not have the same video codecs implemented, and for one-way broadcasting to browsers that do not support WebRTC.
- the front-end components 300 interact with the back-end platform 304 using an application programming interface, such as RESTful APIs 302 .
- RESTful APIs 302 provide methods for exchanging SDPs to set up calls, provide information on which chat rooms are available, which media streams are available in each chat room, which user media streams in a given chat room are most “relevant” at any given moment, and so forth.
- the APIs preferably also provide methods for interacting with other parts of the back-end, e.g., verifying users' identities, accessing storage (saving data, retrieving data, searching), and the like.
- the APIs also preferably include a JavaScript (JS) API 303 , referred to herein as “iris.js,” which is a thin layer on top of the base WebRTC API and other HTML5 components.
- JS JavaScript
- the iris.js API 303 preferably uses the other RESTful APIs to integrate with the overlay network fabric.
- the iris.js API allows applications to establish and use video, voice, and data channels.
- the front-end web app is built on the JavaScript API, and third party applications may use this API to build apps that seamlessly integrate with the platform.
- the front-end components 300 comprise a web application (or web app) 316 , which is a unified communication tool built on iris.js.
- the web app 316 routes video, voice, and data through the overlay network fabric.
- the web app also provides (or interfaces to) one or more collaboration functions or technologies, such as video chat, collaborative document editing, desktop sharing, and the like. Because the web app 316 preferably is built in an API (such as iris.js 303 , which can support several data channels), it is easily extensible.
- the web app 316 is skinnable so it can be rebranded and used by enterprise customers.
- iris.js is built on top of the WebRTC API's, third parties are able to easily adapt existing WebRTC applications to use the solution described herein.
- the third party applications 318 are depicted here as part of the front-end, but they may be separate and distinct.
- the RESTful API 302 also makes integration with other collaboration tools possible.
- the front end may include or have associated therewith legacy on-premises equipment 320 , such as LifeSize rooms. Further, the front-end may include or have associated therewith native mobile apps 322 , such as devices and tablets that run native iOS and Android apps (as opposed to HTML5 apps in mobile browsers, which are also supported).
- native mobile apps 322 such as devices and tablets that run native iOS and Android apps (as opposed to HTML5 apps in mobile browsers, which are also supported).
- the API layer 302 enables a service provider or third parties to easily build native mobile applications for the solution.
- FIG. 4 depicts further implementation details of a multi-party solution implemented within an overlay network 400 , such as the Akamai content delivery network (CDN).
- CDN Akamai content delivery network
- there are two end user peers 402 and 404 and each peer is associated (e.g., using conventional CDN DNS mapping operations) to respective edge servers 406 and 408 .
- Each peer also establishes a WebRTC connection to a media server 410 that hosts the videoconference (in this example scenario).
- a signaling back-end is powered by a distributed data store 412 .
- the platform is implemented using a combination of Node.js, PHP, Apache, Cassandra, and Kurento Media server running on Ubuntu Linux machines. Cassandra data is accessed via the
- RESTful API which is powered by Node.js running behind an Apache proxy 414 .
- signaling information is exchanged via HTTPS interactions using the RESTful API. Multiplexing is accomplished using the Kurento Media Server running on cloud Ubuntu VMs running in geographically-distributed locations.
- the Node.js signaling application performs a DNS lookup to the CDN mapping to determine an optimal (in terms of one or more factors such as latency, loss, load, availability, reachability, etc.) media server to which as client should connect.
- Clients upload their live media stream via WebRTC to the chosen media server.
- the connection is set up by the signaling layer through the RESTful API.
- Other clients who wish to subscribe to that media stream connect to the same media server (via the signaling layer) and receive the stream.
- the underlying network environment may allow for direct connectivity between peers. This requirement is met among users, for example, as long as peers are connected to an enterprise VPN.
- Another approach is to use STUN and TURN servers that run, for example, on cloud virtual machine (VM) infrastructure.
- VM virtual machine
- a TURN-compliant version of a relay network for peer-to-peer connectivity may be used. In the multi-party case described above, STUN and TURN are not needed because it is assumed that clients can connect directly to multiplexing servers.
- Still another approach to connectivity may involve a multicast overlay network to distribute streams.
- the API is powered by a Node.js web application.
- the Node.js application interacts with Kurento Media Server and Cassandra to orchestrate calls.
- the “iris.js” JavaScript API is a client-side ECMAScript 6 library that allows web applications to interact with the system via the Iris RESTful API. It contains functionality that allows for easy WebRTC connection management, call orchestration, and automatic, dynamic quality switching, e.g., as the relevancy of different participants in a room changes.
- the web application is an HTMLS Web App written on top of iris.js. The views are powered by a PHP application.
- the HTMLS Web Audio API is used to have a client browser render different participants' audio at particular positions in 3D space.
- the 3D position of a speaker might depend on one or more factors, such as a position of that person's video window in the screen, whether that person is speaking, the location of the person, when the person joined the call, or simply by hashing the speaker's ID to a position on a ring.
- Different speakers' relative positions may remain stable to communicate contextual information. Users move “forward” and “backward,” or “up” and “down” in space, as their relevancy (as described below) changes.
- the implementation is in client-side JavaScript running in a web browser, preferably that functions as follows:
- the client Given a set of audio streams (via WebRTC or HTTPS), the client computes a 3D position for each audio stream by carrying out a set of operations.
- a “target area” region of space is identified. Typically, this region is in front of the listener (e.g., within a 45° cone in front of the viewer, and between, say, 30 cm and 3 m away from the listener.
- each audio stream is assigned a 3D position within that target region as follows.
- “left,” “right,” “up,” “down,” “forward” and “backward” are from the listener's perspective. If applicable (i.e., if video corresponding to the audio stream is displayed in the browser window), the 3D position corresponds to the point on the screen where the video is displayed; otherwise, the stream is given a position based on contextual information related to the content of the stream.
- a hash function takes a unique identifier for each stream (e.g., each participant's name), and maps that to a left/right position within the target area.
- each individual speaker will maintain a stable position relative to other speakers, even if other speakers join or leave the conference.
- a hash function maps each stream's unique identifier to a number in the range [0,1], and then each stream is assigned a position within the left/right target region such that the positions are evenly (or otherwise deterministically) spaced within the target region.
- the up/down and forward/backward position are then based on “relevancy” and other contextual information for each stream. For example, if one person is giving a presentation, their position may be “up” relative to other participants (who may be asking questions). Or, someone who just joins the room may be “back” compared to someone who has recently spoken.
- the browser renders each audio stream at the 3D position associated with it in step (2).
- the audio is played over the browser's stereo (or surround sound) speakers.
- the effect to the end user is that it will sound like each stream originates at its associated 3D position.
- steps (2) and (3) are performed continuously, so, for example, as an individual stream's associated 3D position changes, the end-user experiences this as that stream “moving” in the stereophonic soundscape.
- the following describes a method comprising server- and client-based logic for intelligently and dynamically determining which streams are most important—i.e., which streams correspond to users who are currently speaking, have spoken recently, or are about to speak—and delivering those streams at higher quality. Remaining users' streams are delivered at lower video quality (or audio-only).
- This approach saves bandwidth and enables scalable, real-time multi-party videoconferencing. Stated another way, while each client consumes a fixed total bandwidth, the way in which the bandwidth is apportioned is dynamically adjusted to enhance user experience.
- the platform maintains a set of variables, continuously updated for each participant, which are then used to determine an importance or relevancy of each user's audiovisual (a/v) stream.
- these variables are derived from multiple sources, including audio filters, video filters, user input, and other measures of individual and group behavior.
- Example variables/filters include, without limitation: speaking (audio), crowd noise (audio), face detection (video), group detection (video), microphone mute (user input), raise hand (user input), and so forth.
- the variables are used to construct a probability density function (PDF) across several possible stream “attributes,” such as whether a stream represents an individual speaking, a small group, an individual who is participating in a conversation, audience members who are reacting to a main presenter, and so forth.
- PDF probability density function
- heuristics are built by leveraging statistical modeling and/or machine learning (ML) techniques (e.g., using a training set of users in a multi-party context) to construct PDFs for each attribute, e.g., from (raw or smoothed) measurements of the variables.
- ML machine learning
- these PDFs are combined using relative weighting techniques to drive both client and server behavior.
- the nature of the relative weighting techniques may vary. There may be a predefined set of techniques, or a set of best practices, a default set, or some combination.
- the PDFs are used to drive the end-user experience.
- the video images of different speakers on the screen and the loudness of different speakers' audio streams may be given different prominence based on the relevance of each speaker, and on the capabilities of the client.
- a client with a high-bandwidth connectivity displays the two most “relevant” speakers in HD, N other speakers in small video windows, and only audio for everyone else, whereas a mobile client with low connectivity displays only one low-quality video, and audio for everyone else.
- PDFs preferably are used to guide routing and multiplexing of individual user streams. Streams that are more relevant may be afforded increased bandwidth and resource allocation. The PDFs may also be used to guide assignments of streams to different media servers.
- one or more additional transforms such as decay functions, are used to dampen oscillations in relevancy changes. For example, if two people are speaking, the decay function prevents constant switching between each person's video.
- the clients are browsers
- the servers are cloud machines.
- Logic is implemented in client-side JavaScript, and in the server-side in Node.js JavaScript.
- client-server API calls are via RESTful HTTPS requests
- media flows are via the HTMLS WebRTC API.
- a client makes a RESTful API call via HTTPS to the server indicating that it wants to join a conference.
- the server adds client to conference. Typically, this involves adding information to a distributed database.
- the client makes another API request indicating what type of media it has to offer (audio, video, etc. . . . ), what capabilities it has (e.g., total bandwidth does it have, type of device, number of video streams it can handle, number of HD video streams it can handle, how many lower quality streams, etc.).
- the information on client capabilities typically comes from several sources, such as information that the client explicitly sends in the API request, information (e.g., on the client's hardware capabilities, on the client's connectivity performance, etc.) that is collected by a client-side JavaScript code (using HTML5 APIs such as the Navigation Timing API, the Web Performance API, and the Network Information API, and WebRTC), and information that is inferred from server-side code, such as the client's User-Agent, the network performance of ISP/network that the client is in, and the actual measured bandwidth, throughput, latency, etc., to the client.
- the server allocates resources for the media stream that the client will upload.
- this step involves making a DNS request to a load balancer to find a free media server, communicating the offer/answer SDP information needed to create a WebRTC connection between the client and the media server, and recording information about the connection in the database.
- the server adds the client to list of participants in room, and assigns an initial relevancy.
- relevancy is preferably a multi-dimensional data structure that comprises one or more variables derived from multiple sources, including audio filters, video filters, user input, and other measures of individual and group behavior.
- Example variables/filters include: speaking (audio), crowd noise (audio), face detection (video), group detection (video), mic mute (user input), raise hand (user input), and the like.
- the server compiles a list of all participants in room along with each client's full multidimensional relevancy, as well as a summary score.
- clients can determine an ordered ranking of which participant in the conference is “most relevant” based on the summary score.
- the “summary” can comprise a single variable (e.g., time since hand last raised), a weighted average of variables, the PDFs as described above, or some combination.
- clients periodically poll the server for changes to the participant list and the relevancy for each participant. In lieu of polling, clients may also be notified via a push notifications or via a publisher/subscriber system).
- clients also periodically send information back to the server that is used to build their relevancy object and summary scores. The server combines this information with information that it collects (for example, information from server-side a/v filters, information on how long a client has been in a room, etc.) and continuously updates the relevancy object and summary score for each participant.
- clients change which WebRTC media streams they subscribe to, preferably via the following mechanism: (i) first, given the client capabilities (step (3)), the client knows what kind of stream it can handle (e.g., N high-quality streams, M low-quality streams, S audio streams, and so forth); (ii) the client JavaScript (iris.js) sorts the other participants in order of “summary” relevancy score; (iii) the client then associates the most relevant participants with the highest quality streams, preferably in order; and (iv) updates the associations as needed, e.g., by examining if a client is associated with a different quality stream than the client is currently subscribed, unsubscribing from the old quality stream and subscribes to the quality of stream that the client is currently associated with. In other words, the client JavaScript seamlessly swaps the different qualities so that an interruption is not visible to the end user. Subscription preferably is via API calls, as described in step (4).
- a client can handle 1 HD stream 4 low-quality streams, and 100 audio streams, then the participant with the highest summary relevancy will get the HD stream, the next 4 highest summary scores will get low-quality streams, and the remaining participants will be associated with audio-only streams.
- a third client may be subscribed to an HD stream for Alice and an SD stream for Bob. If Bob starts speaking, then the client will drop the HD connection to Alice and switch to an SD connection. The client will then subscribe to an HD connection for Bob and swap that in for the old SD connection.
- the client makes a series of API calls to the server to allocate media server resources and to broker a new connection in a way that is analogous to step (4) above.
- An alternative method in step (9) involves having each client subscribe to only one WebRTC media stream per other participant.
- the server performs the above steps of sorting participants based on summary relevancy score, and the server takes care of adjusting the quality of each stream when summary relevancy score changes.
- the client does not have to subscribe/unsubscribe to a different quality WebRTC stream at any time; clients only need to subscribe/unsubscribe to WebRTC streams when other participants join/leave the conference.
- the server keeps track of a maximum total bandwidth that a given client can handle, and it makes sure that the aggregate quality of all WebRTC streams delivered to that client is below the maximum total bandwidth threshold. In this case, the client only needs to consume the relevancy to handle the following step.
- JavaScript on the client preferably uses the full multi-dimensional relevancy object to determine how to display other participants' video in the browser window.
- the full multi-dimensional relevancy object determines how to display other participants' video in the browser window.
- current speakers are given a position of prominence, people with hands raised are highlighted, etc.
- each client always consumes an amount of bandwidth that is below a fixed maximum, but within this constraint, the bandwidth is dynamically apportioned such that the most-relevant participants are delivered at the highest quality, resulting in an enhanced experience for the end-user.
- dynamic quality adjustment is driven (e.g., by the client) using relevancy.
- Relevancy is a property of each media stream; preferably, it is a multi-dimensional object that contains information such as whether the user is speaking, whether the user has muted his or her microphone, how recently the user spoke, if the user is raising his or her hand, that is used to determine which streams are most important.
- server-side switching is used so that only the most relevant streams are delivered, say, at high quality.
- switching is done by the client; in particular, the server lets the client know the relevancy of all participants' streams, and the client (based on its capabilities), dynamically subscribes to the high-quality version of the most relevant streams and low-quality versions of other streams.
- the multi-party collaboration solution has significant advantages. It enables seamless video conferencing directly within the user's browser. No plugins, downloads or special equipment (beyond a webcam) are needed.
- the approach enables web-based multi-party video conferencing.
- the technique enables direct point-to-point and multi-party communications (video conferencing, and real time screen sharing, document collaboration, etc.) between and among end users through the overlay network fabric.
- the overlay network fabric provides support for multipath packet transport, as described in U.S. Pat. Nos. 6,665,726 and 6,751,673, assigned to Akamai Technologies, Inc.
- the various connections used in the overlay network and as described typically are secured via SSL or other transport layer security (TLS) techniques.
- TLS transport layer security
- the use of the overlay network to provide a peer-to-peer relay as described in U.S. Pat. No. 9,137,196 may also be utilized. That approach facilitates implementation of the TURN protocol (“Traversal Using Relay NAT,” as specified in Internet Request for Comment 5766), as well as STUN protocol, as described in RFC 3489.
- each peer executes an application (e.g., a video calling application), and the peers communicate with one another, preferably using a relay service established by the overlay network service provider.
- the overlay network comprises sets of servers, typically organized as edge regions.
- Each region typically comprises multiple edge servers, each of which is a machine such as has been described above.
- An edge region may comprise a single relay server.
- a directory service which may be part of the overlay network or a service associated with a third party, is used to facilitate communications between the peers even if each peer is behind a NAT device.
- each client a peer
- the directory service performs DNS lookups to determine which overlay network relay server (a virtual IP, or VIP) to which each of the clients should connect.
- the directory service tells each client the VIP it should use and the VIP the other machine will use.
- the directory service also assigns a unique connection ID for the end-to-end connection.
- Each client then sends a session allocation request to the assigned overlay relay server.
- the allocation request includes the connection ID and the remote peer's relay server VIP.
- the overlay relay server responds with the NAT IP/port that was provisioned for the connection. The information is relayed to the remote peer via the directory service.
- Each client then sends a channel binding request to its assigned overlay relay server, indicating the NAT IP/port of the remote peer.
- each overlay relay server is aware of all necessary identifying information for the connection, namely: connection ID, each client's NAT IP/port, and each client's relay server VIP (and service port, if it is necessary for the port to be different).
- Each above-described process preferably is implemented in computer software as a set. of program instructions executable in one or more processors, as a special-purpose machine.
- Representative machines on which the subject matter herein is provided may be Intel Pentium-based computers running a Linux or Linux-variant operating system and one or more applications to carry out the described functionality.
- One or more of the processes described above are implemented as computer programs, namely, as a set of computer instructions, for performing the functionality described.
- This apparatus may be a particular machine that is specially constructed for the required purposes, or it may comprise a computer otherwise selectively activated or reconfigured by a computer program stored in the computer.
- a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including an optical disk, a CD-ROM, and a magnetic-optical disk, a read-only memory (ROM), a random access memory (RAM), a magnetic or optical card, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
- a given implementation of the present invention is software written in a given programming language that runs in conjunction with a DNS-compliant name server (e.g., BIND) on a standard Intel hardware platform running an operating system such as Linux.
- the functionality may be built into the name server code, or it may be executed as an adjunct to that code.
- a machine implementing the techniques herein comprises a processor, computer memory holding instructions that are executed by the processor to perform the above-described methods.
- the techniques herein generally provide for the above-described improvements to a technology or technical field, as well as the specific technological improvements to various fields including collaboration technologies including videoconferencing, chat, document sharing and the like, distributed networking, Internet-based overlays, WAN-based networking, efficient utilization of Internet links, and the like, all as described above.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Acoustics & Sound (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
- This application relates generally to network-based conferencing wherein participants “listen” to and participate in the conference using a web browser and, in particular, to techniques to enhance delivery of audio to such conference participants
- Distributed computer systems are well-known in the prior art. One such distributed computer system is a “content delivery network” (CDN) or “overlay network” that is operated and managed by a service provider. The service provider typically provides the content delivery service on behalf of third parties (customers) who use the service provider's shared infrastructure. A distributed system of this type typically refers to a collection of autonomous computers linked by a network or networks, together with the software, systems, protocols and techniques designed to facilitate various services, such as content delivery, web application acceleration, or other support of outsourced origin site infrastructure. A CDN service provider typically provides service delivery through digital properties (such as a website), which are provisioned in a customer portal and then deployed to the network.
- Real-time communications (e.g., videoconferencing, shared document editing, screen sharing, and the like) over the Internet have been a part of our daily lives at work and at home. WebRTC, an Internet standard, was created to make videoconferencing and point-to-point data transfer easier to implement. In particular, WebRTC (which stands for Web Real Time Communications) seeks to take the most critical elements of video chat and move them to one of the most commonly used tools for accessing the Internet, namely, a web browser. WebRTC is supported with plugins by both Google Chrome and Mozilla Firefox. It allows the browser to access the client machine's camera and microphone, provides a method for establishing a direct connection between two users' browser and to use that connection to send audio and video, and it provides a method for sending arbitrary data streams across a connection.
- An overlay network platform facilitates a multi-party conference. End users participate in the conference using client-based web browser software, and using a protocol such as WebRTC. According to this disclosure, an enhanced “audio” experience for a user participating in the conference is provided. To this end, a method is implemented at a client machine to render different participants' audio in the multi-party conference at different positions in a three-dimensional (3D) space. In operation, and given a set of audio streams that comprise the conference and that are being received, a three-dimensional (3D) position for each audio stream in a 3D soundscape is computed. In one embodiment, the 3D position for each audio stream is computed by identifying a target area region of space (e.g., a 45° cone positioned a given distance in front of the listener), and then determining the 3D position for each stream in the target area region of space. The relative 3D positions may be determined based on contextual information related to the content of an audio stream, as well as a computed “relevance” of one stream versus another. Alternatively, the relative 3D positions of the audio streams may be deterministic (e.g., equally spaced). Given streams may be positioned relative to one another, e.g., up or down, left or right, forward or backward. Each audio stream is then rendered (output or played) to the listener at the 3D position computed, e.g., using the HTML5 Web Audio API.
- The foregoing has outlined some of the more pertinent features of the disclosed subject matter. These features should be construed to be merely illustrative. Many other beneficial results can be attained by applying the disclosed subject matter in a different manner or by modifying the subject matter as will be described.
- For a more complete understanding of the subject matter and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a block diagram illustrating a known distributed computer system configured as a content delivery network (CDN); -
FIG. 2 is a representative CDN edge machine configuration; -
FIG. 3 depicts the various components of a web-based collaboration solution according to this disclosure; -
FIG. 4 illustrates a multi-party videoconference setup that is enabled by associating the web-based solution of this disclosure with an overlay network; and -
FIG. 5 is a process flow depicting a method of this disclosure. - In a known system, such as shown in
FIG. 1 , adistributed computer system 100 is configured as a content delivery network (CDN) and is assumed to have a set ofmachines 102 a-n distributed around the Internet. Typically, most of the machines are servers located near the edge of the Internet, i.e., at or adjacent end user access networks. A network operations command center (NOCC) 104 manages operations of the various machines in the system. Third party sites, such asweb site 106, offload delivery of content (e.g., HTML, embedded page objects, streaming media, software downloads, and the like) to thedistributed computer system 100 and, in particular, to “edge” servers. Typically, content providers offload their content delivery by aliasing (e.g., by a DNS CNAME) given content provider domains or sub-domains to domains that are managed by the service provider's authoritative domain name service. End users that desire the content are directed to the distributed computer system to obtain that content more reliably and efficiently. Although not shown in detail, the distributed computer system may also include other infrastructure, such as a distributeddata collection system 108 that collects usage and other data from the edge servers, aggregates that data across a region or set of regions, and passes that data to other back-end systems network agents 118 monitor the network as well as the server loads and provide network, traffic and load data to a DNSquery handling mechanism 115, which is authoritative for content domains being managed by the CDN. A distributeddata transport mechanism 120 may be used to distribute control information (e.g., metadata to manage content, to facilitate load balancing, and the like) to the edge servers. - As illustrated in
FIG. 2 , a givenmachine 200 in the content delivery network comprises commodity hardware (e.g., an Intel Pentium processor) 202 running an operating system kernel (such as Linux or variant) 204 that supports one ormore applications 206 a-n. To facilitate content delivery services, for example, given machines typically run a set of applications, such as an HTTP proxy 207 (sometimes referred to as a “global host” or “ghost” process), aname server 208, alocal monitoring process 210, a distributeddata collection process 212, and the like. For streaming media, the machine may include one or more media servers, such as a Windows Media Server (WMS) or Flash server, as required by the supported media formats, or it may utilize HTTP-based delivery of chunked content fragments that constitute a stream. - A CDN edge server is configured to provide one or more extended content delivery features, preferably on a domain-specific, customer-specific basis, preferably using configuration files that are distributed to the edge servers using a configuration system. A given configuration file preferably is XML-based and includes a set of content handling rules and directives that facilitate one or more advanced content handling features. The configuration file may be delivered to the CDN edge server via the data transport mechanism. U.S. Pat. No. 7,111,057 illustrates a useful infrastructure for delivering and managing edge server content control information, and this and other edge server control information can be provisioned by the CDN service provider itself, or (via an extranet or the like) the content provider customer who operates the origin server.
- The CDN may include a storage subsystem, such as described in U.S. Pat. No. 7,472,178, the disclosure of which is incorporated herein by reference.
- The CDN may operate a server cache hierarchy to provide intermediate caching of customer content; one such cache hierarchy subsystem is described in U.S. Pat. No. 7,376,716, the disclosure of which is incorporated herein by reference.
- The CDN may provide secure content delivery among a client browser, edge server and customer origin server in the manner described in U.S. Publication No. 20040093419.
- Secure content delivery as described therein enforces SSL-based links between the client and the edge server process, on the one hand, and between the edge server process and an origin server process, on the other hand. This enables an SSL-protected web page and/or components thereof to be delivered via the edge server.
- In a typical operation, a content provider identifies a content provider domain or sub-domain that it desires to have served by the CDN. The CDN service provider associates (e.g., via a canonical name, or CNAME) the content provider domain with an edge network (CDN) hostname, and the CDN provider then provides that edge network hostname to the content provider. When a DNS query to the content provider domain or sub-domain is received at the content provider's domain name servers, those servers respond by returning the edge network hostname. The edge network hostname points to the CDN, and that edge network hostname is then resolved through the CDN name service. To that end, the CDN name service returns one or more IP addresses. The requesting client browser then makes a content request (e.g., via HTTP or HTTPS) to an edge server associated with the IP address. The request includes a host header that includes the original content provider domain or sub-domain. Upon receipt of the request with the host header, the edge server checks its configuration file to determine whether the content domain or sub-domain requested is actually being handled by the CDN. If so, the edge server applies its content handling rules and directives for that domain or sub-domain as specified in the configuration. These content handling rules and directives may be located within an XML-based “metadata” configuration file.
- Unified Browser-Based Enterprise Collaboration Platform using an Overlay Network
- The following assumes familiarity with WebRTC.
- Traditional point-to-point communication and collaboration tools, such as LifeSize, WebEx, and Skype, require specialized applications and infrastructure. The WebRTC APIs have the potential to make direct point-to-point communication ubiquitous by bringing it to the browser. As described above, WebRTC enables video communication and collaboration without any additional plugins or software. That said, however, there are still a number of challenges remaining for an entity that wants to deploy a WebRTC-based application. Thus, for example, a relay infrastructure (using TURN) is needed to establish connections between two peers behind NATs, and building a robust and scalable relay infrastructure is challenging. Additionally, multi-user video conferencing over WebRTC requires full mesh connectivity between all users; that is, a separate connection must be established between each pair of users. Each user needs to upload their video (and other data) multiple times—once for each peer—and the resources required grow in a way proportional to the square of the number of users, which does not scale. These issues are not limited to WebRTC; indeed, existing, dedicated video conferencing solutions struggle with the same problems. For example, Microsoft's Skype relays are often overloaded, significantly impacting the quality of Skype calls that cannot use a direct peer-to-peer connection. LifeSize needs the same full-mesh connectivity described above, which severely limits the number of different remote sites that can participate in one meeting.
- To address these issues, and according to one aspect of this disclosure, an overlay network fabric is used. In particular, by distributing multiplexing and the relay infrastructure over a platform, such as a CDN (as described above), a solution that facilitates multi-user collaboration, such as video conferencing, chat, document sharing, and desktop sharing, is provided. While a primary use case as described below is for high-quality video conferencing that is scalable to large numbers of users, this is not a limitation, as the cloud-supported multiplexing and relay techniques herein may be used to provide other multi-user collaboration, such as chat, document sharing, and desktop sharing, all in a seamless and scalable manner. The overlay network can also provide additional functions and features to support a collaboration session; these may include, without limitation, persistent storage and recording of sessions and documents, integration with existing videoconferencing and telecommunications infrastructure (LifeSize rooms, PSTN, etc.), and others.
-
FIG. 3 depicts arepresentative architecture 300 for an enterprise collaboration platform using an overlay network according to an aspect of this disclosure. There are three primary elements: a front-end application 300,RESTful APIs 302, and a back-end orplatform 304. The front-end application (app) 300 preferably is built on a number of components (described below) that are preferably accessed through the one or more -
RESTful APIs 302. As will be seen, this architecture allows on-premises infrastructure to be integrated with the solution, and it could also be used to allow third parties to create applications powered by the platform. Theplatform components 304 include signaling 306,connectivity 308, multiplexing 310,storage 312, andPSTN integration 314. In one embodiment, theplatform 304 comprises part of an overlay network (or leverages elements thereof), but this is not a requirement, as the solution herein may be provided as a standalone architecture. Further, the notion of a “component” herein may involve multiple machines, whether co-located or distributed, as well as the processes and programs executing thereon. - Turning first to the platform, the
signaling component 306 preferably is a distributed signaling system that keeps track of users' state (e.g., “Online”, “Away”, “Busy”, etc. . . . ), and it is used to transmit the information (i.e., SDP) necessary to initiate an RTCPeerConnection. Thesignaling component 306 preferably integrates with various user authentication and identity management solutions, although this is not a requirement. - The
connectivity component 308 manages video, voice and data connections routed though the overlay network platform to handle Network Access Translation (NAT) traversal, as well as to provide enhanced performance and security. - The
multiplexing component 310 comprises multiplexing machines to allow for scalable, multi-peer sessions. This component makes it so that each peer only needs to upload its media stream once. Other peers are then able to access peers' media streams through overlay network edge machines (rather than by direct connections to peers). The multiplexing component provides for multiplexing in the cloud to significantly reduce edge bandwidth requirements that would otherwise be required to support WebRTC (which otherwise dictates a new connection be setup for pair of peers in a multi-user collaboration). With this approach herein of using the overlay network, there is no requirement to setup a new connection for each pair of peers in a multi-peer collaboration (conference, chat, etc.) session. - As will be described, and as a further feature, preferably the
multiplexing component 310 intelligently adjusts the quality of different users' streams to enhance performance—e.g., only deliver HD streams for people who are currently speaking, deliver lower-quality streams to mobile devices, etc. - The
storage component 312 allows overlay network customers to (optionally) store data from a collaboration session (e.g., record a meeting, save work on a collaborative document, etc.). ThePTSN integration component 314 allows users to join sessions from the PSTN and legacy telecommunications equipment, and it allows users to call out over the PSTN. Although not depicted, the platform may include a transcoding component that allows for communications between browsers that do not have the same video codecs implemented, and for one-way broadcasting to browsers that do not support WebRTC. - As noted, the front-
end components 300 interact with the back-end platform 304 using an application programming interface, such asRESTful APIs 302. TheseAPIs 302 provide methods for exchanging SDPs to set up calls, provide information on which chat rooms are available, which media streams are available in each chat room, which user media streams in a given chat room are most “relevant” at any given moment, and so forth. - The APIs preferably also provide methods for interacting with other parts of the back-end, e.g., verifying users' identities, accessing storage (saving data, retrieving data, searching), and the like. As also depicted, the APIs also preferably include a JavaScript (JS)
API 303, referred to herein as “iris.js,” which is a thin layer on top of the base WebRTC API and other HTML5 components. The iris.jsAPI 303 preferably uses the other RESTful APIs to integrate with the overlay network fabric. In particular, the iris.js API allows applications to establish and use video, voice, and data channels. Preferably, the front-end web app is built on the JavaScript API, and third party applications may use this API to build apps that seamlessly integrate with the platform. - The front-
end components 300 comprise a web application (or web app) 316, which is a unified communication tool built on iris.js. Theweb app 316 routes video, voice, and data through the overlay network fabric. The web app also provides (or interfaces to) one or more collaboration functions or technologies, such as video chat, collaborative document editing, desktop sharing, and the like. Because theweb app 316 preferably is built in an API (such asiris.js 303, which can support several data channels), it is easily extensible. Thus, users are able to choose which voice, video, and data channels to connect to for a given session—for example, several users in a video conference room could use the room's camera and mic for videoconferencing with a remote site, but each individual user might use his or her personal laptop to edit a shared document. Preferably, theweb app 316 is skinnable so it can be rebranded and used by enterprise customers. As noted, because iris.js is built on top of the WebRTC API's, third parties are able to easily adapt existing WebRTC applications to use the solution described herein. Thethird party applications 318 are depicted here as part of the front-end, but they may be separate and distinct. As noted above, theRESTful API 302 also makes integration with other collaboration tools possible. As also depicted, the front end may include or have associated therewith legacy on-premises equipment 320, such as LifeSize rooms. Further, the front-end may include or have associated therewith nativemobile apps 322, such as devices and tablets that run native iOS and Android apps (as opposed to HTML5 apps in mobile browsers, which are also supported). TheAPI layer 302 enables a service provider or third parties to easily build native mobile applications for the solution. - In one embodiment, the above-described solution provides a multi-party voice and video chat system.
FIG. 4 depicts further implementation details of a multi-party solution implemented within anoverlay network 400, such as the Akamai content delivery network (CDN). As depicted, there are two end user peers 402 and 404, and each peer is associated (e.g., using conventional CDN DNS mapping operations) torespective edge servers media server 410 that hosts the videoconference (in this example scenario). A signaling back-end is powered by a distributeddata store 412. In an example implementation, the platform is implemented using a combination of Node.js, PHP, Apache, Cassandra, and Kurento Media server running on Ubuntu Linux machines. Cassandra data is accessed via the - RESTful API, which is powered by Node.js running behind an
Apache proxy 414. In this approach, signaling information is exchanged via HTTPS interactions using the RESTful API. Multiplexing is accomplished using the Kurento Media Server running on cloud Ubuntu VMs running in geographically-distributed locations. In operation, the Node.js signaling application performs a DNS lookup to the CDN mapping to determine an optimal (in terms of one or more factors such as latency, loss, load, availability, reachability, etc.) media server to which as client should connect. Clients upload their live media stream via WebRTC to the chosen media server. The connection is set up by the signaling layer through the RESTful API. Other clients who wish to subscribe to that media stream connect to the same media server (via the signaling layer) and receive the stream. - While the approach shown in
FIG. 4 is typical, it is not a limitation. In an alternative embodiment, the underlying network environment may allow for direct connectivity between peers. This requirement is met among users, for example, as long as peers are connected to an enterprise VPN. Another approach is to use STUN and TURN servers that run, for example, on cloud virtual machine (VM) infrastructure. In still another embodiment, a TURN-compliant version of a relay network for peer-to-peer connectivity may be used. In the multi-party case described above, STUN and TURN are not needed because it is assumed that clients can connect directly to multiplexing servers. Still another approach to connectivity may involve a multicast overlay network to distribute streams. - In this example implementation, the API is powered by a Node.js web application. The Node.js application interacts with Kurento Media Server and Cassandra to orchestrate calls. The “iris.js” JavaScript API is a client-side ECMAScript 6 library that allows web applications to interact with the system via the Iris RESTful API. It contains functionality that allows for easy WebRTC connection management, call orchestration, and automatic, dynamic quality switching, e.g., as the relevancy of different participants in a room changes. The web application is an HTMLS Web App written on top of iris.js. The views are powered by a PHP application.
- According to this aspect, it is assumed that there are multiple participants to a conference, such as a videoconference (or even just an audio conference), that is being facilitated using the above-described platform in association with the overlay network. In this aspect, different participants' audio is then projected from different positions in a 3D stereophonic soundscape in the following manner.
- In particular, preferably the HTMLS Web Audio API is used to have a client browser render different participants' audio at particular positions in 3D space. The 3D position of a speaker might depend on one or more factors, such as a position of that person's video window in the screen, whether that person is speaking, the location of the person, when the person joined the call, or simply by hashing the speaker's ID to a position on a ring. Different speakers' relative positions may remain stable to communicate contextual information. Users move “forward” and “backward,” or “up” and “down” in space, as their relevancy (as described below) changes.
- The implementation is in client-side JavaScript running in a web browser, preferably that functions as follows:
- Given a set of audio streams (via WebRTC or HTTPS), the client computes a 3D position for each audio stream by carrying out a set of operations.
- A method according to this disclosure is depicted in
FIG. 5 . At step (1), a “target area” region of space is identified. Typically, this region is in front of the listener (e.g., within a 45° cone in front of the viewer, and between, say, 30 cm and 3 m away from the listener. - At step (2), each audio stream is assigned a 3D position within that target region as follows. In this discussion, “left,” “right,” “up,” “down,” “forward” and “backward” are from the listener's perspective. If applicable (i.e., if video corresponding to the audio stream is displayed in the browser window), the 3D position corresponds to the point on the screen where the video is displayed; otherwise, the stream is given a position based on contextual information related to the content of the stream. For a multiparty real-time video conversation, preferably this is done as follows: a hash function takes a unique identifier for each stream (e.g., each participant's name), and maps that to a left/right position within the target area. Thus, each individual speaker will maintain a stable position relative to other speakers, even if other speakers join or leave the conference. Alternately, a hash function maps each stream's unique identifier to a number in the range [0,1], and then each stream is assigned a position within the left/right target region such that the positions are evenly (or otherwise deterministically) spaced within the target region. Preferably, the up/down and forward/backward position are then based on “relevancy” and other contextual information for each stream. For example, if one person is giving a presentation, their position may be “up” relative to other participants (who may be asking questions). Or, someone who just joins the room may be “back” compared to someone who has recently spoken.
- At step (3), and using the HTMLS Web Audio API, the browser renders each audio stream at the 3D position associated with it in step (2). The audio is played over the browser's stereo (or surround sound) speakers. The effect to the end user is that it will sound like each stream originates at its associated 3D position.
- Preferably, steps (2) and (3) are performed continuously, so, for example, as an individual stream's associated 3D position changes, the end-user experiences this as that stream “moving” in the stereophonic soundscape.
- If relevancy support is included in the solution (and this is not required), the following details may be implemented.
- As noted above, traditional multiparty videoconferencing methods involve gathering an audio/video stream from each individual end user's client, and distributing each user's stream to all other end users. The delivery of these streams is usually facilitated by either a full-mesh topology or a centralized multiplexing server. That approach does not scale to large numbers of users.
- As an alternative, the following describes a method comprising server- and client-based logic for intelligently and dynamically determining which streams are most important—i.e., which streams correspond to users who are currently speaking, have spoken recently, or are about to speak—and delivering those streams at higher quality. Remaining users' streams are delivered at lower video quality (or audio-only). This approach saves bandwidth and enables scalable, real-time multi-party videoconferencing. Stated another way, while each client consumes a fixed total bandwidth, the way in which the bandwidth is apportioned is dynamically adjusted to enhance user experience.
- According to this approach, the platform maintains a set of variables, continuously updated for each participant, which are then used to determine an importance or relevancy of each user's audiovisual (a/v) stream. Preferably, these variables are derived from multiple sources, including audio filters, video filters, user input, and other measures of individual and group behavior. Example variables/filters include, without limitation: speaking (audio), crowd noise (audio), face detection (video), group detection (video), microphone mute (user input), raise hand (user input), and so forth. Preferably, the variables are used to construct a probability density function (PDF) across several possible stream “attributes,” such as whether a stream represents an individual speaking, a small group, an individual who is participating in a conversation, audience members who are reacting to a main presenter, and so forth. In particular, preferably heuristics are built by leveraging statistical modeling and/or machine learning (ML) techniques (e.g., using a training set of users in a multi-party context) to construct PDFs for each attribute, e.g., from (raw or smoothed) measurements of the variables. Preferably, these PDFs are combined using relative weighting techniques to drive both client and server behavior.
- The nature of the relative weighting techniques may vary. There may be a predefined set of techniques, or a set of best practices, a default set, or some combination.
- Thus, in the client case, preferably the PDFs, along with information on the client device's capabilities and network connectivity, are used to drive the end-user experience. For example, the video images of different speakers on the screen and the loudness of different speakers' audio streams may be given different prominence based on the relevance of each speaker, and on the capabilities of the client. In this manner, a client with a high-bandwidth connectivity displays the two most “relevant” speakers in HD, N other speakers in small video windows, and only audio for everyone else, whereas a mobile client with low connectivity displays only one low-quality video, and audio for everyone else. These are just representative examples of course. In the server case, the
- PDFs preferably are used to guide routing and multiplexing of individual user streams. Streams that are more relevant may be afforded increased bandwidth and resource allocation. The PDFs may also be used to guide assignments of streams to different media servers. In both the client and server cases, preferably one or more additional transforms, such as decay functions, are used to dampen oscillations in relevancy changes. For example, if two people are speaking, the decay function prevents constant switching between each person's video.
- The following describes one implementation using the above-described collaboration infrastructure. In this example, the clients are browsers, and the servers are cloud machines. Logic is implemented in client-side JavaScript, and in the server-side in Node.js JavaScript. As noted above, preferably client-server API calls are via RESTful HTTPS requests, and the media flows are via the HTMLS WebRTC API.
- At step (1), a client makes a RESTful API call via HTTPS to the server indicating that it wants to join a conference. At step (2), the server adds client to conference. Typically, this involves adding information to a distributed database. At step (3), the client makes another API request indicating what type of media it has to offer (audio, video, etc. . . . ), what capabilities it has (e.g., total bandwidth does it have, type of device, number of video streams it can handle, number of HD video streams it can handle, how many lower quality streams, etc.). The information on client capabilities typically comes from several sources, such as information that the client explicitly sends in the API request, information (e.g., on the client's hardware capabilities, on the client's connectivity performance, etc.) that is collected by a client-side JavaScript code (using HTML5 APIs such as the Navigation Timing API, the Web Performance API, and the Network Information API, and WebRTC), and information that is inferred from server-side code, such as the client's User-Agent, the network performance of ISP/network that the client is in, and the actual measured bandwidth, throughput, latency, etc., to the client. At step (4), the server allocates resources for the media stream that the client will upload. Typically, this step involves making a DNS request to a load balancer to find a free media server, communicating the offer/answer SDP information needed to create a WebRTC connection between the client and the media server, and recording information about the connection in the database. At step (5), the server adds the client to list of participants in room, and assigns an initial relevancy. As noted above, relevancy is preferably a multi-dimensional data structure that comprises one or more variables derived from multiple sources, including audio filters, video filters, user input, and other measures of individual and group behavior. Example variables/filters include: speaking (audio), crowd noise (audio), face detection (video), group detection (video), mic mute (user input), raise hand (user input), and the like. At step (6), the server compiles a list of all participants in room along with each client's full multidimensional relevancy, as well as a summary score.
- Preferably, clients can determine an ordered ranking of which participant in the conference is “most relevant” based on the summary score. In one exemplary but non-limiting embodiment, the “summary” can comprise a single variable (e.g., time since hand last raised), a weighted average of variables, the PDFs as described above, or some combination.
- At step (7), clients periodically poll the server for changes to the participant list and the relevancy for each participant. In lieu of polling, clients may also be notified via a push notifications or via a publisher/subscriber system). At step (8), clients also periodically send information back to the server that is used to build their relevancy object and summary scores. The server combines this information with information that it collects (for example, information from server-side a/v filters, information on how long a client has been in a room, etc.) and continuously updates the relevancy object and summary score for each participant.
- At step (9), clients change which WebRTC media streams they subscribe to, preferably via the following mechanism: (i) first, given the client capabilities (step (3)), the client knows what kind of stream it can handle (e.g., N high-quality streams, M low-quality streams, S audio streams, and so forth); (ii) the client JavaScript (iris.js) sorts the other participants in order of “summary” relevancy score; (iii) the client then associates the most relevant participants with the highest quality streams, preferably in order; and (iv) updates the associations as needed, e.g., by examining if a client is associated with a different quality stream than the client is currently subscribed, unsubscribing from the old quality stream and subscribes to the quality of stream that the client is currently associated with. In other words, the client JavaScript seamlessly swaps the different qualities so that an interruption is not visible to the end user. Subscription preferably is via API calls, as described in step (4).
- Thus, for example, if a client can handle 1 HD stream 4 low-quality streams, and 100 audio streams, then the participant with the highest summary relevancy will get the HD stream, the next 4 highest summary scores will get low-quality streams, and the remaining participants will be associated with audio-only streams. As a more concrete example, if Alice is speaking and Bob is quiet, a third client may be subscribed to an HD stream for Alice and an SD stream for Bob. If Bob starts speaking, then the client will drop the HD connection to Alice and switch to an SD connection. The client will then subscribe to an HD connection for Bob and swap that in for the old SD connection.
- Although not intending to be limiting, for each new WebRTC media stream description, preferably the client makes a series of API calls to the server to allocate media server resources and to broker a new connection in a way that is analogous to step (4) above.
- An alternative method in step (9) involves having each client subscribe to only one WebRTC media stream per other participant. Here, the server performs the above steps of sorting participants based on summary relevancy score, and the server takes care of adjusting the quality of each stream when summary relevancy score changes. In this case, however, the client does not have to subscribe/unsubscribe to a different quality WebRTC stream at any time; clients only need to subscribe/unsubscribe to WebRTC streams when other participants join/leave the conference. The server keeps track of a maximum total bandwidth that a given client can handle, and it makes sure that the aggregate quality of all WebRTC streams delivered to that client is below the maximum total bandwidth threshold. In this case, the client only needs to consume the relevancy to handle the following step.
- At step (10), JavaScript on the client preferably uses the full multi-dimensional relevancy object to determine how to display other participants' video in the browser window. Thus, for example, current speakers are given a position of prominence, people with hands raised are highlighted, etc.
- In summary, each client always consumes an amount of bandwidth that is below a fixed maximum, but within this constraint, the bandwidth is dynamically apportioned such that the most-relevant participants are delivered at the highest quality, resulting in an enhanced experience for the end-user.
- Thus, and according to this aspect, dynamic quality adjustment is driven (e.g., by the client) using relevancy. Relevancy is a property of each media stream; preferably, it is a multi-dimensional object that contains information such as whether the user is speaking, whether the user has muted his or her microphone, how recently the user spoke, if the user is raising his or her hand, that is used to determine which streams are most important. Preferably, server-side switching is used so that only the most relevant streams are delivered, say, at high quality. In another embodiment, switching is done by the client; in particular, the server lets the client know the relevancy of all participants' streams, and the client (based on its capabilities), dynamically subscribes to the high-quality version of the most relevant streams and low-quality versions of other streams.
- The multi-party collaboration solution has significant advantages. It enables seamless video conferencing directly within the user's browser. No plugins, downloads or special equipment (beyond a webcam) are needed. The approach enables web-based multi-party video conferencing. The technique enables direct point-to-point and multi-party communications (video conferencing, and real time screen sharing, document collaboration, etc.) between and among end users through the overlay network fabric. To support the techniques herein, and depending on the implementation of the underlying distribution, the overlay network fabric provides support for multipath packet transport, as described in U.S. Pat. Nos. 6,665,726 and 6,751,673, assigned to Akamai Technologies, Inc. The various connections used in the overlay network and as described typically are secured via SSL or other transport layer security (TLS) techniques. Further, the use of the overlay network to provide a peer-to-peer relay as described in U.S. Pat. No. 9,137,196 may also be utilized. That approach facilitates implementation of the TURN protocol (“Traversal Using Relay NAT,” as specified in Internet Request for Comment 5766), as well as STUN protocol, as described in RFC 3489. In the relay approach, each peer executes an application (e.g., a video calling application), and the peers communicate with one another, preferably using a relay service established by the overlay network service provider. The overlay network comprises sets of servers, typically organized as edge regions. Each region typically comprises multiple edge servers, each of which is a machine such as has been described above. An edge region may comprise a single relay server. A directory service, which may be part of the overlay network or a service associated with a third party, is used to facilitate communications between the peers even if each peer is behind a NAT device. In the relay approach, each client (a peer) has a connection to a directory service. The directory service performs DNS lookups to determine which overlay network relay server (a virtual IP, or VIP) to which each of the clients should connect. The directory service then tells each client the VIP it should use and the VIP the other machine will use. The directory service also assigns a unique connection ID for the end-to-end connection. Each client then sends a session allocation request to the assigned overlay relay server. The allocation request includes the connection ID and the remote peer's relay server VIP. The overlay relay server responds with the NAT IP/port that was provisioned for the connection. The information is relayed to the remote peer via the directory service. Each client then sends a channel binding request to its assigned overlay relay server, indicating the NAT IP/port of the remote peer. At this point, each overlay relay server is aware of all necessary identifying information for the connection, namely: connection ID, each client's NAT IP/port, and each client's relay server VIP (and service port, if it is necessary for the port to be different).
- Each above-described process preferably is implemented in computer software as a set. of program instructions executable in one or more processors, as a special-purpose machine.
- Representative machines on which the subject matter herein is provided may be Intel Pentium-based computers running a Linux or Linux-variant operating system and one or more applications to carry out the described functionality. One or more of the processes described above are implemented as computer programs, namely, as a set of computer instructions, for performing the functionality described.
- While the above describes a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary, as alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, or the like. References in the specification to a given embodiment indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic.
- While the disclosed subject matter has been described in the context of a method or process, the subject matter also relates to apparatus for performing the operations herein. This apparatus may be a particular machine that is specially constructed for the required purposes, or it may comprise a computer otherwise selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including an optical disk, a CD-ROM, and a magnetic-optical disk, a read-only memory (ROM), a random access memory (RAM), a magnetic or optical card, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. A given implementation of the present invention is software written in a given programming language that runs in conjunction with a DNS-compliant name server (e.g., BIND) on a standard Intel hardware platform running an operating system such as Linux. The functionality may be built into the name server code, or it may be executed as an adjunct to that code. A machine implementing the techniques herein comprises a processor, computer memory holding instructions that are executed by the processor to perform the above-described methods.
- While given components of the system have been described separately, one of ordinary skill will appreciate that some of the functions may be combined or shared in given instructions, program sequences, code portions, and the like.
- While given components of the system have been described separately, one of ordinary skill will appreciate that some of the functions may be combined or shared in given instructions, program sequences, code portions, and the like. Any application or functionality described herein may be implemented as native code, by providing hooks into another application, by facilitating use of the mechanism as a plug-in, by linking to the mechanism, and the like.
- The techniques herein generally provide for the above-described improvements to a technology or technical field, as well as the specific technological improvements to various fields including collaboration technologies including videoconferencing, chat, document sharing and the like, distributed networking, Internet-based overlays, WAN-based networking, efficient utilization of Internet links, and the like, all as described above.
Claims (9)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/094,589 US20210084425A1 (en) | 2016-12-30 | 2020-11-10 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662440473P | 2016-12-30 | 2016-12-30 | |
US15/857,694 US10834514B2 (en) | 2016-12-30 | 2017-12-29 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
US17/094,589 US20210084425A1 (en) | 2016-12-30 | 2020-11-10 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/857,694 Continuation US10834514B2 (en) | 2016-12-30 | 2017-12-29 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210084425A1 true US20210084425A1 (en) | 2021-03-18 |
Family
ID=62712214
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/857,694 Active US10834514B2 (en) | 2016-12-30 | 2017-12-29 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
US17/094,589 Abandoned US20210084425A1 (en) | 2016-12-30 | 2020-11-10 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/857,694 Active US10834514B2 (en) | 2016-12-30 | 2017-12-29 | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape |
Country Status (1)
Country | Link |
---|---|
US (2) | US10834514B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11910131B1 (en) * | 2022-09-28 | 2024-02-20 | Amazon Technologies, Inc. | Publisher-subscriber architecture for video conferencing |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10708320B2 (en) * | 2017-06-27 | 2020-07-07 | Atlassian Pty Ltd | Selective internal forwarding in conferences with distributed media servers |
US20190327140A1 (en) * | 2018-04-24 | 2019-10-24 | Level 3 Communications, Llc | Subscriber configuration ingestion in a content delivery network |
US20230007067A1 (en) * | 2021-06-30 | 2023-01-05 | Tencent America LLC | Bidirectional presentation datastream |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130058470A1 (en) * | 2011-09-07 | 2013-03-07 | Elwha LLC, a limited liability company of the State of Delaware | Computational systems and methods for identifying a communications partner |
US20150052455A1 (en) * | 2012-03-23 | 2015-02-19 | Dolby Laboratories Licensing Corporation | Schemes for Emphasizing Talkers in a 2D or 3D Conference Scene |
US20170010853A1 (en) * | 2015-07-12 | 2017-01-12 | Jeffrey Gelles | System for remote control and use of a radio receiver |
US9800990B1 (en) * | 2016-06-10 | 2017-10-24 | C Matter Limited | Selecting a location to localize binaural sound |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8351589B2 (en) * | 2009-06-16 | 2013-01-08 | Microsoft Corporation | Spatial audio for audio conferencing |
US9143880B2 (en) * | 2013-08-23 | 2015-09-22 | Tobii Ab | Systems and methods for providing audio to a user based on gaze input |
-
2017
- 2017-12-29 US US15/857,694 patent/US10834514B2/en active Active
-
2020
- 2020-11-10 US US17/094,589 patent/US20210084425A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130058470A1 (en) * | 2011-09-07 | 2013-03-07 | Elwha LLC, a limited liability company of the State of Delaware | Computational systems and methods for identifying a communications partner |
US20150052455A1 (en) * | 2012-03-23 | 2015-02-19 | Dolby Laboratories Licensing Corporation | Schemes for Emphasizing Talkers in a 2D or 3D Conference Scene |
US20170010853A1 (en) * | 2015-07-12 | 2017-01-12 | Jeffrey Gelles | System for remote control and use of a radio receiver |
US9800990B1 (en) * | 2016-06-10 | 2017-10-24 | C Matter Limited | Selecting a location to localize binaural sound |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11910131B1 (en) * | 2022-09-28 | 2024-02-20 | Amazon Technologies, Inc. | Publisher-subscriber architecture for video conferencing |
Also Published As
Publication number | Publication date |
---|---|
US10834514B2 (en) | 2020-11-10 |
US20180192219A1 (en) | 2018-07-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10623700B2 (en) | Dynamic speaker selection and live stream delivery for multi-party conferencing | |
US10587756B2 (en) | Collecting and correlating microphone data from multiple co-located clients, and constructing 3D sound profile of a room | |
US11956317B2 (en) | Unified, browser-based enterprise collaboration platform | |
US20210084425A1 (en) | Representation of contextual information by projecting different participants' audio from different positions in a 3D soundscape | |
US10869001B2 (en) | Provision of video conferencing services using a micro pop to extend media processing into enterprise networks | |
US9794201B2 (en) | Messaging based signaling for communications sessions | |
US11716368B2 (en) | Multicast overlay network for delivery of real-time video | |
US9402054B2 (en) | Provision of video conference services | |
US9525849B2 (en) | Provision of video conferencing with load balancing | |
EP2241067B1 (en) | Multiway peer-to-peer media streaming | |
US20160285945A1 (en) | Method and Systems for Optimizing Bandwidth Utilization in a Multi-Participant Full Mesh Peer-to-Peer Video Session | |
US11159586B2 (en) | Dynamically controlling relay communication links during a communication session | |
WO2018126134A1 (en) | Unified, browser-based enterprise collaboration platform | |
da Silva et al. | Cross-layer multiuser session control for optimized communications on SDN-based cloud platforms | |
EP3563248B1 (en) | Unified, browser-based enterprise collaboration platform | |
US11863592B2 (en) | Active speaker tracking using a global naming scheme | |
Sakomaa | Analysis of a web conferencing system: development and customisation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: AKAMAI TECHNOLOGIES, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MEHTA, ABHIJIT C.;GERO, CHARLES E.;HOUMAN, THOMAS;SIGNING DATES FROM 20180108 TO 20180131;REEL/FRAME:058829/0578 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |