KR20100021435A - Active speaker identification - Google Patents

Abstract

Procedures for identifying clients in an audio event are described. In an example, a media server may order clients providing audio based on the input level. An identifier may be associated with the client for identifying the client providing input within the event. The ordered clients may be included in a list which may be inserted into a packet header carrying the audio content.

Description

Active speaker identification {ACTIVE SPEAKER IDENTIFICATION}

Media conference participants may have difficulty identifying other participants in the conference. Participants may not be familiar with the speaker's voice or the speaker's face or the audio exchange may confuse the listener. In the latter case, the listener may be confused if multiple participants speak at the same time or there is a quick exchange between the participants, whether or not to speak. In some cases, the speaker may include his / her name, such as "I am Bob ..." or the listener may ask for the identity of the previous speaker. The complexity of these issues can increase as the number of participants speaking or contributing to the audio input increases. Although the listener may guess the speaker's identity from the "contextual clue" in the conversation, in some cases the participants may not understand which participants provide the audio input.

It may also be desirable to minimize bandwidth consumption, or data throughput, for information delivery. For example, a physical connection to send data may have additional throughput, but consuming communication link resources may reduce the throughput available for other data transfers, or if the user has limited network bandwidth Can affect audio data delivery.

Acceptance of media conferencing enhancements may be limited if these improvements are not "backwards compatible". For example, if a modification is inconsistent with existing protocols and versions, the user will have to obtain an updated version and / or seek institutional approval to communicate with the participant implementing the modified version. The above described situation can hinder the acceptance of the modified technology.

Procedures for identifying clients in an audio or audio / video event are described. In one example, the media server may order clients providing audio based on the input level. An identifier may be associated with the client to identify the client providing the input within the event. The ordered clients can be included in a list that can be inserted into a packet header that carries audio content.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

Detailed description will be made with reference to the accompanying drawings. In these figures, the leftmost digit (s) of a reference number identifies the figure in which the reference number first appeared. In other instances of the figures and description the use of the same reference numbers may indicate similar or identical items.

1 illustrates an environment in an example implementation that may use techniques for allowing aggressive speaker identification.

FIG. 2 illustrates a real time protocol data packet including an ordered / reordered active client list in a list of contributing sources (CSRC) fields.

3 is a flow diagram illustrating a procedure in an example implementation for identifying aggressive clients.

4 is a flow diagram illustrating a procedure in an example implementation for identifying active clients in a real time protocol conference.

survey

Techniques for identifying active audio contributors in a media event are described. In implementations, a list of contributing or participating audio clients can be constructed based on the client contribution to the session. The ID can be associated with participating clients so that clients can identify which client (s) are actively contributing to the event. The configured list can be inserted in data stream packet headers for delivery to conference clients. In implementations, the identification information can be included in control packets used in connection with data transmission. The techniques described herein can provide speaker information without consuming minimal network resources and causing synchronization problems.

In other implementations, a media server can be configured to switch / mix audio streams such that an ordered list of active clients is inserted into the data packet headers. For example, the media server may include a list of active speakers that may be ordered based on the current active speaker so that information related to which clients are actively speaking is provided to the clients. The list can be provided without increasing the media transfer overhead for the network.

Example environment

1 illustrates an environment 100 of example implementations that may operate to use aggressive speaker identification. For example, media server 102 may mix and switch between audio streams provided by the client in a media event while identifying active audio clients. Although audio data processing is described, media server 102 may process other types of media data, including video, and the like, based on the functions and conferencing of the client devices. For example, media server 102 may manipulate audio / video data for some clients when delivering audio data to clients without video capabilities or the like.

For example, media server processor 104 may determine which client or clients actively contribute to audio content when mixing / switching audio streams for the clients. The media server processor 104 may determine which clients are actively entering audio data based on the mixing / switching algorithm / technique employed by the processor to generate send media streams. This determination can be used to order a list of clients contributing to outbound media streams from media server 102, or clients contributing to media server output.

Clients "A" 106 and "E" 114 (such as clients "A" 106 and "E" 114 carrying on a conversation) contribute to the audio input. In an audio event including "A" 106, "B" 108, "C" 110, "D" 112, and "E" 114, an inactive client "B" 108 ), "C" 110, "D" 112 can be provided with an "A + E" transmission stream from media server 102, or a combination of these two speakers, while client "A" ( 106 and " E " 114 respectively receive the other party's transmission stream from media server 102 (e.g., client A 106 receives the client E transmission stream while client E 114 receives the client). A receives a transmission stream). Suitable client devices include, but are not limited to, voice over internet protocol (VoIP) phones, computing devices with audio capabilities, publicly switched network telephones (PSTN) phones connected to digital audio sessions through gateways, and the like. Do not.

In some implementations, active speakers may not be provided with a signal that includes the speaker's own transmission stream to avoid feedback or echo (eg, client A 106 has an audio stream that includes client A audio). May not be sent). Some general identification scenarios can be considered, for example, when the participant "E" (relative to client A 106) speaks loudly while the participant "E" (relative to client E 114) communicates in a relatively normal voice. Likewise, if Client A 106 can talk over to Client E, then Participants "A" and "E" are involved in a quick exchange in which the current speaker changes between these two participants, Or participant "A" leads the conversation while participant "E" provides relatively less input. An example of the latter situation could include a participant adding minor acknowledgments to the overwhelming monologue of the main speaker.

In implementations, media server 102 may determine the dominant client (ie, speaker) based on the number of packets received from the client, when the audio content is received, packet size, energy audio level, and the like. Thus, although two or more clients contribute to the content at the same time, one active client can be designated as the dominant client (ie, speaker) based on the factors mentioned above. For example, media server 102 may be based on a current active client (and based on current data packets containing audio content received from an active client with respect to mixing and / or switching between inputs received from different clients. Related speaker). For example, if client E is not contributing to data packets at this time, media server 102 can designate client A 106 as the current "active" client. In another example, if both Client A 106 and Client E 114 are active, but Client A 106 is contributing to the audio content at a higher energy level than Client E 114 (ie, Participant A speaks loudly) And E is in low tone), client A 106 may be designated as the dominant and active speaker. Clients may be provided with an active client list beginning with Client A 106. This type of decision can be made when mixing / switching client input audio streams for one or more ongoing conferences. For example, when employing a mixing algorithm, the media server 102 processor may distinguish between active clients, while the identification module 116 may be used to insert this information into applicable data packets. Can be.

Referring generally to FIG. 2, in implementations, when implementing a real-time transport protocol (RTP) and its associated real-time control protocol (RTP), media server 102 includes data transmission and signaling streams. By examining the data in the streams sent from the clients, it is possible to identify active clients, ie active speakers. For client A 106, media server 102 checks the audio source (RSRC) field in the RTP packet or from the client name (CNAME) and client SSRC (client's ID in session) included in the RTCP report. It can be identified that the transmission stream originated from client A 106. Other information can also be checked. SSRC can also be obtained from the RTP packet header. For example, the SSRC may be mapped to a CNAME in the RTCP report.

RTCP signaling may be used to identify lost packets, to ensure data transmission quality, and RTCP reporting may be obtained from RTCP out-of-band signals. For example, the RTCP report may include an arbitrarily generated client SSRC that maps to the client CNAME. In general, CNAMEs are IDs / records associated with aliases used on client devices. In some examples, CNAME is a string of numbers and the like. In implementations, media server 102 may be assigned an SSRC in a session. In some examples, the SSRC may change for clients involved in the session. For example, the client SSRC changes when the client cuts-off (for example, rejoins after a long pause), when the client SSRCs collide (an SSRC common to more than one client is issued), and so on. Can be. In this way, the incoming data stream can be identified from the SSRC or from the RTCP signaling in the data stream. Media server 102 may also obtain a CNAME from RTCP signaling for use in client identification.

When generating a transmission stream (including audio output), media server 102 may identify from the CNAME and SSRC obtained from the active client which clients are contributing to the audio input for the session. For example, media server 102 may associate the CRC inserted in the SSRC, RTCP packets with the audio content transmission stream (ie, media server output stream (s) carrying audio data). Returning to the previous exemplary session between the clients "A" 106, "B" 108, "C" 110, "D" 112, and "E" 114, the mixed signal "A" In the case of + E ", media server 102 may now order clients" A "and" E "depending on which client is active, who is active and session dominant, and so forth. This sort order can be changed based on the client providing the audio input. In this case, if Client A is currently providing input, or if Client A is dominant in the conversation, the list may include Client E 114 starting with the ID of Client A 106. In situations where audio is exchanged between client A 106 and client E 114, the sort order may be changed based on the currently speaking participant, as indicated on a per packet basis.

Referring to FIG. 2, in an RTP configuration, the media server identification module 116 can insert this ordered list of SSRCs in the output stream RTP packet header. For example, ordered IDs are inserted into the list of contributing sources (CSRC) field 204 of the packet header sent in the data stream. If Client A and Client E are currently exchanging active roles, the configuration of SSRC is from "Client A, Client E ..." 204 (a) to "Client E, Client A ..." (204 (b). Can be changed to)). In the former way, clients receiving a data stream (listening clients or participants in a session), while avoiding additional signaling related synchronization issues and network overhead, have no bearing on what clients are providing input, relative contributions, etc. Receive notification. For example, the CSRC field may be allowed to contain up to 15 IDs of 32 bits per one, and the rest may be related to the specification. Clients that are not performed in connection with the techniques discussed herein may participate without the benefits discussed herein. Thus, building systems and technologies can be backwards compatible.

Although SSRC can identify active clients, using SSRC can be randomly assigned by SSRC and changed by collisions with other clients with similar SSRCs, which drop out of the session. This may be a problem, for example, after the SSRC is reassigned after rejoining this session.

Media server 102 may insert aggressive client CNAME (s) in RTCP packets delivered to clients (eg, to allow other “listening” clients to recognize the aggressive client CNAME and SSRC). . For example, media server identification module 116 may "fan-out" aggressive client IDs sent to "listening clients" into media server RTCP packets. For example, if several active clients are contributing to the conference, the media server may insert the obtained client IDs into the RTCP packets sent with respect to the media server data stream at specified intervals. RTCP packets may include a CNAME in each packet. The CNAME (s) may be distributed into RTCP packets that are delivered to listening clients to minimize transmission overhead. Clients receiving media server RTCP data, including aggressive client IDs, can store the data in local memory so that a CNAME can be associated with the data packets when receiving audio content. For example, the CNAME, mapped SSRC, and other related information may be stored in a look-up table or the like. For example, audio content included in a data stream may be transmitted in a generally continuous manner, while RTCP signaling may only occur intermittently at specified intervals (e.g., every 5 or 10 seconds). Can be. Thus, a client receiving a data packet can associate the SSRC in the CSRC with a previously received CNAME. In implementations, a globally routable user agent universal resource indicator (GRUU) may be used to identify a particular client.

In implementations, an active client may be informed that this client is an active person in a meeting. For example, a participant (associated with an active client) may want to know that he / she is not "talking" to another participant. Returning to the session between the clients "A" 106, "B" 108, "C" 110, "D" 112 and "E" 114, for example, client A 106 If aggressive, but clients "B" 108, "C" 110, "D" 112, and "E" 114 are not aggressive, this may be identified through an RTCP signal sent to Client A. have. Accordingly, media server 102 can generate a transmission media stream for clients "B", "C", "D" and "E" by delivering the transmission stream through client A, while client A 106 May identify that no other client is active as a member of the client or session "listening" based on the CSRC / RTCP packets.

In other implementations, human understandable information is associated with the SSRC and CNAME. For example, when the participant is speaking, the user may want the picture of the speaking participant to be displayed on the associated monitor. In implementations, human understandable client information may be exchanged between clients. For example, data can generally be exchanged when initiating an event or session.

The Internet (WWW) may be used to connect the client and other components, but other networks and various links are also suitable. For example, a network connecting media server 102 to a client may include a wide area network (WAN), a local area network (LAN), a wireless network, a public telephone network, an intranet, and the like. The network may be configured to include a plurality of sub-networks.

The following discussion describes techniques that can be implemented using the systems and apparatuses previously described. Aspects of the respective procedures may be implemented in hardware, firmware, software, or a combination thereof. Procedures are shown as a set of blocks that specify operations to be performed by one or more devices, but are not necessarily limited to the illustrated order for performing the operations by each block.

Example Procedures

The following discussion describes techniques that can be implemented using the systems and apparatuses previously described. Aspects of the respective procedures may be implemented in hardware, firmware, software, or a combination thereof. Procedures are shown as a set of blocks that specify operations to be performed by one or more devices, but are not necessarily limited to the illustrated order for performing the operations by each block. Various other examples are also contemplated.

3 illustrates example procedures for identifying active audio input clients in a media session. For example, these techniques may be used in media conferencing or teleconferences where some clients do not have video capabilities.

In implementations, the media server acting as a host or central point may determine 302 audio input clients related to the input provided by each active client. For example, the determination may be made as part of mixing and / or switching audio client input. Thus, client A may be designated as the primary active client until another client provides audio input. In another example, if client A and client E are contributing but client A's audio has a higher energy level, client A may be selected. Client audio may have a higher energy level if the participant associated with the client is speaking loudly or if the client is speaking more consistently, as in the case where the client is leading the audio input.

An audio input client may be identified as a "top" client if the client is currently active, conversational, and the like. In RTP / RTCP systems functioning in accordance with the present techniques, the media server may obtain client input streams and associated RTCP packets (eg, RTCP packets sent from the client), 304 These RTCP packets contain an SSRC mapped to the CNAME for the particular client creating the stream containing the audio content. For example, the media server can obtain the SSRC and CNAME for the client. The CNAME identifies the client with respect to the SSRC. The media server may order the input client SSRCs according to which clients are currently providing audio input, dominating the conversation, and so on (306). For example, the media server may order active client SSRC IDs from the current active "speaker", eg, descending from an active client providing input. In an example, RTP may allow the identification of 15 active speakers using a 32-bit ID per active client included in the CRSC.

The media server can associate the ID with the audio input client. For example, the media server may obtain the SSRC and CNAME from the audio input client RTCP packet. SSRC in the CRSC field included in the media server output stream may be used to identify the audio input client.

Clients can associate other data with a receive / audio input client. For example, the receiving client (the listening client or the client of the media event) may have information that a person associated with the CNAME can understand. For example, the client may have a picture of the participant (associated with the client CNAME / SSRC), the participant's name, and other information.

The ordered audio input client IDs may be inserted 308 into the packet header. For example, if clients "A" and "E" are providing audio input (currently client A is an active client), the CSRC field of the RTP header contains SSRCs that start the list with SSRC of client "A". can do. In this way, the identity of the speaker can be notified to the listening client (which can include an audio input client receiving audio input from another active client) within the content stream. In another example, the sorting order of the audio input clients in the list may be based at least in part on which audio input client is dominant in the media session. Considerations for dominance may include the energy level of the audio input, the duration of the input, the duration of the silence period, the packet size, and the like. For example, the list may begin with client A because client A is currently active and the client A transmission stream is indicating a higher energy level than one or more other audio input clients.

The media server may send 310 the SSRC and CNAME to the listening client (session client) in the media server transmission stream (s) (such as RTCP packets sent in connection with the content transfer). SSRC for audio input clients may also be located in the CSRC field of the data stream packet header of RTP packets. For example, in five client media events, if three participants are speaking, the media (sent to the listening clients) associated with the RTP packets carrying the audio content are associated with the client SSRC and CNAME associated with the audio input clients. May be included in server RTCP packets. In this way, the media server can send SSRC and CNAME identifying the active audio clients to the clients. Thus, the listening client can identify the originating source of the audio content with respect to the SSRC (s) and CNAME (s) in the RTP packet. The client SSRC may be updated if this client SSRC conflicts with an SSRC issued to another client, or if the client changes the source transport address for some other reason. The SSRC (s) and CNAME (s) can be stored in local memory (312) so that the listening client can access information throughout the media event.

4 illustrates exemplary techniques for identifying active clients in a media conference. For example, the techniques can be used during media conferencing when some of the clients do not have video, or can be used in audio teleconferencing.

In the present implementations, the media server may receive active client input (audio content) as well as active client's ID (402). For example, a client contributing to an audio conference can send an SSRC and a CNAME identifying this client. For example, the SSRC may be included in the RTP packet header and in the RTCP packet along with the CNAME in the data stream.

An ordered 404 list of one or more active clients in the conference is generated. For example, a media server mixing / switching an audio input stream can configure an active client list (SSRC IDs in RTP / RTCP), or clients providing input to a meeting or session. For example, the media server is an audio / video mixing server (AVMCU) that obtains an SSRC identifier from an active client transmission stream, which may include a data area and an associated signaling area. The AVMCU may then determine the relative configuration of the active client SSRCs or other ID of the client in the session. The SSRC may be identified from an RTCP report that may be mapped to a client CNAME. For example, the ranking may be based on which client is currently active. In other implementations, factors such as energy level, number of data packets provided, duration of silence period, packet size, and the like can be considered. For example, an ordered list can start with an active client that can be in session with a given number of packets, while a second concurrent and active client is assigned a lower relative status. .

The ordered list may be inserted 406 into the CSRC list field included in the packet headers in the media server transmission data stream flows. For example, the media server output includes audio provided by an active client, with the CSRC field containing an ordered list of SSRC IDs of these active clients. As a result, it is possible to inform the listening client, i.e., the client receiving the audio content stream, about which clients are active and the relative relationship of these active clients. In addition, SSRC and CNAME may be included in the media server transmission RTCP packets.

SSRCs may be associated with a CNAME of an active audio client (408). For example, the media server may transmit an RTCP that includes a client CNAME related to the SSRC included in the CRSC field of the RTP packet header. This CNAME can be obtained from RTCP packets.

Human understandable information may also be associated with the CNAME and / or audio input client SSRC. For example, a picture or name can be associated with the client CNAME so that the participant's picture or name appears when the relevant client is providing audio content. This information can be delivered within the meeting or the client can enter information that this person can understand.

In other implementations, the GRUU may be associated with the SSRC of the active client. In some cases where one client is active but not others, the media server provides an indication to the active client (410) to notify the active client that no other client is active, but the active client transmission stream. Is not returned to the active client. In this way, the active client can be made aware that the participant is not "talking" to the other participant.

Although RTP and RTCP have been described, the techniques and implementations of the present disclosure can be applied to other protocol data transfer mechanisms.

conclusion

Although the invention has been described in language specific to structural features and / or methodological acts, it is understood that the invention as defined by the appended claims is not necessarily limited to the specific features or acts described. Should be. Rather, the specific features and acts are disclosed as example forms of implementing the invention claimed in the claims.

Claims (20)

Ordering one or more audio input clients according to input provided by respective audio input clients included in the one or more audio input clients (306); Associating an identifier (ID) with the respective audio input client (304); Inserting an ordered list of one or more audio input client IDs into a packet header (308) How to include. The method of claim 1, And the list is inserted into a real-time transport protocol (RTP) contributing source list (CSRC) in the packet header. The method of claim 1, Ordering is determined by the host mixing audio streams such that the list is descended from the currently active audio input client. The method of claim 1, Sending (310) for each client a canonical name (CNAME) and a synchronization source (SSRC) identifier mapped to the CNAME. The method of claim 4, wherein Wherein the CNAME and associated SSRC are obtained from a real-time control protocol (RTCP) record for the respective client. The method of claim 5, The CNAME and associated SSRC are sent in an RTCP packet to a listening client. The method of claim 1, Storing (312) the CNAME and SSRC in a listening client local memory. The method of claim 1, The dominant client is determined based on at least one of energy level, duration of silence period, duration or packet size. The method of claim 1, Updating the SSRC identifier by the CNAME when the client changes the source transport address in the session. Ordering a list of one or more active audio clients in a meeting based on individual active audio client participants in the meeting (404), wherein the respective active audio clients are associated with a CNAME and SSRC identifier; Inserting the ordered list into an RTP CSRC list field in one or more audio streams (408). How to include. The method of claim 10, The ordered list starts with a dominant and active client. The method of claim 11, Determining a dominant client based on at least one of an energy level, a duration of a silent period, a duration, or a packet size. The method of claim 11, The SSRC identifier is mapped to a CNAME obtained from a control packet for one or more received audio streams. The method of claim 13, The control packet is compatible with RTCP. The method of claim 13, The SSRC and the CNAME are included in an RTCP packet. The method of claim 10, And (410) providing an indication that the predominant and active audio client included in the one or more active clients, i.e., the other audio client, is not active. A media server (102) for generating one or more outgoing media streams from received aggressive clients, wherein the media server inserts an ordered list of one or more aggressive clients into the one or more media streams. The method of claim 17, The media server inserts the ordered list into an RTP CSRC in a packet header. The method of claim 17, And the ordered list is representative of a client predominant based on at least one of an energy level, a duration of a silent period, a duration, or a packet size associated with a media stream received from the client predominant. The method of claim 17, And the media server transmits aggressive client IDs in RTCP packets to clients receiving transmission media streams, including a CNAME for the aggressive client and an SSRC identifier mapped to the CNAME.

Images