CN111367493A

CN111367493A - Method and device for controlling audio in multimedia conference

Info

Publication number: CN111367493A
Application number: CN202010152948.4A
Authority: CN
Inventors: 吴佳宝; 田建超
Original assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd; Guangzhou Shizhen Information Technology Co Ltd
Current assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd; Guangzhou Shizhen Information Technology Co Ltd
Priority date: 2020-03-06
Filing date: 2020-03-06
Publication date: 2020-07-03
Also published as: WO2021174982A1

Abstract

The application discloses a method and a device for controlling audio in a multimedia conference. Wherein, the method comprises the following steps: receiving a first operation in a multimedia conference window, wherein the multimedia conference window displays a plurality of clients accessed to a multimedia conference; selecting at least one client from the plurality of clients according to a first operation; collecting first audio information sent to the selected at least one client; sending the first audio information to a server, so that the server sends the first audio information to the selected at least one client; receiving second audio information sent by the server; the second audio information is audio information generated in the multimedia conference process; and playing the second audio information. The method and the device solve the technical problem that the multimedia conference in the prior art cannot meet the requirement of participants for independent communication, and are simple and convenient to operate.

Description

Method and device for controlling audio in multimedia conference

Technical Field

The application relates to the field of intelligent equipment, in particular to a method and a device for controlling audio in a multimedia conference.

Background

The multimedia conference is used for realizing the remote conference holding of a plurality of people through software. At present, when a plurality of people are online simultaneously, the voice of a speaker is transmitted to access parties of all other multimedia conferences, so that in the multimedia conference, if two parties need to communicate independently, the current ongoing multimedia conference needs to be quitted, and the mode is complex to operate and is easy to miss important information in the conference.

Aiming at the problem that the multimedia conference in the prior art cannot meet the requirement of independent communication of participants, an effective solution is not provided at present.

Disclosure of Invention

The embodiment of the application provides a method and a device for controlling audio in a multimedia conference, which can solve the problem that the multimedia conference in the prior art cannot meet the requirement of independent communication of participants. The technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a method for controlling audio in a multimedia conference, including: receiving a first operation in a multimedia conference window, wherein the multimedia conference window displays a plurality of clients accessed to a multimedia conference; selecting at least one client from the plurality of clients according to a first operation; collecting first audio information sent to the selected at least one client; sending the first audio information to a server, so that the server sends the first audio information to the selected at least one client; receiving second audio information sent by the server, wherein the second audio information is audio information generated in the multimedia conference process; and playing the second audio information.

Optionally, a second operation in the multimedia conference window is received, and the selected at least one client is deselected.

Optionally, the first operation is a long-press operation, and after the first audio information is sent to the server, so that the server sends the first audio information to the selected at least one client, when the long-press operation is finished, the selected at least one client is deselected.

Optionally, receiving third audio information sent by the server; the third audio information is sent by a third client terminal accessed to the multimedia conference, and the current client terminal is selected by the third client terminal according to the first operation of the third client terminal; adjusting the volume parameter of the second audio information and/or the volume parameter of the received third audio information; and playing the received third audio information and the second audio information together.

Optionally, the volume parameter of the second audio information is decreased, and/or the third volume parameter of the third audio information is increased, so that the volume parameter of the second audio information is lower than the volume parameter of the third audio information.

In a second aspect, an embodiment of the present application provides a method for controlling audio in a multimedia conference, including: receiving first audio information sent by a first client; determining that the first client has the selected second client, and sending the first audio information to the second client; and determining that the first client does not have the selected second client, and sending the first audio information to all clients except the first client in the multimedia conference.

Optionally, before determining that the first client has the selected second client, receiving selected information sent by the first client for the second client; the selected information is determined by the first client according to a first operation in the multimedia conference window; the selection of the second client by the first client is recorded.

Optionally, deselection information of the second client sent by the first client is received; the deselection information is determined by the first client according to a second operation in the multimedia conference window; and clearing the selected record of the first client to the second client.

Optionally, the selected information of the second client sent by the first client is received while the first audio information sent by the first client is received.

Optionally, determining that there is second audio information sent to the second client; the second audio information and the first audio information are sent by different clients accessing the multimedia conference; adjusting a first volume parameter of the first audio information and/or a second volume parameter of the second audio information; and sending the second audio information to the second client.

Optionally, the second volume parameter of the second audio information is decreased, and/or the first volume parameter of the first audio information is increased, so that the second volume parameter is lower than the first volume parameter.

In a third aspect, an embodiment of the present application provides a device for controlling audio in a multimedia conference, including: the first receiving module is used for receiving a first operation in a multimedia conference window, and the multimedia conference window displays a plurality of clients accessed to a multimedia conference; the selection module is used for selecting at least one client from the plurality of clients according to the first operation; the acquisition module is used for acquiring first audio information sent to the selected at least one client; the sending module is used for sending the first audio information to the server, so that the server sends the first audio information to the selected at least one client; the second receiving module is used for receiving second audio information sent by the server; the second audio information is audio information in the multimedia conference process; and the playing module is used for playing the second audio information.

In a fourth aspect, embodiments of the present application provide a computer storage medium having stored thereon a plurality of instructions adapted to be loaded by a processor and to perform the above-mentioned method steps.

In a fifth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the above-mentioned method steps.

In the embodiment of the application, a first operation in a multimedia conference window is received, and a plurality of clients accessing a multimedia conference are displayed in the multimedia conference window; selecting at least one client from the plurality of clients according to a first operation; collecting first audio information sent to the selected at least one client; sending the first audio information to a server, so that the server sends the first audio information to the selected at least one client; receiving second audio information sent by the server, wherein the second audio information is audio information generated in the multimedia conference process; and playing the second audio information. In the scheme, the first operation is carried out on at least one client side in the multimedia conference, and the audio information is sent to the server, so that the server only sends the audio information to at least one client side selected by the first operation, the independent communication among part of the client sides in the multimedia conference is realized, the operation is simple and convenient, and the technical problem that the multimedia conference in the prior art cannot meet the requirement of independent communication among participants is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a flowchart of a method for controlling audio in a multimedia conference according to an embodiment of the present application;

FIG. 2 is a schematic diagram of selecting a client by a first operation according to an embodiment of the present application;

FIG. 3 is a schematic diagram of another selection of a client by a first operation according to an embodiment of the present application;

fig. 4 is a flowchart of another audio control method in a multimedia conference according to an embodiment of the present application;

FIG. 5 is a schematic diagram of processing audio information in a multimedia conference according to an embodiment of the present application;

FIG. 6 is an information interaction diagram of an audio processing method according to an embodiment of the present application;

fig. 7 is a schematic diagram of an audio control apparatus in a multimedia conference according to an embodiment of the present application; and

fig. 8 is a schematic diagram of another audio control device in a multimedia conference according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

It should be understood that the embodiments described are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.

In the description of the present application, it is to be understood that the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not necessarily used to describe a particular order or sequence, nor are they to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate. Further, in the description of the present application, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Example 1

According to an embodiment of the present application, there is provided an embodiment of a method for controlling audio in a multimedia conference, it should be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that here.

Fig. 1 is a flowchart of a method for controlling audio in a multimedia conference, in which a plurality of clients access the multimedia conference, as shown in fig. 1, the method includes the following steps:

step S102, receiving a first operation in a multimedia conference window, wherein the multimedia conference window displays a plurality of clients accessing a multimedia conference.

First, it should be noted that the multimedia conference in the present application may be a video conference, which transmits video and audio simultaneously, may include transmission of video and/or audio of multiple parties participating in the video conference, and sharing of multimedia files, and may also be an audio conference, which transmits audio only, and the specific form of the present application is not limited specifically. The interactive interface provided by the multimedia conference can display the video/head portrait/client identifier of each participant, can also display the multimedia file shared by the speakers, and can also switch between the two or be displayed in a split-screen manner, and the specific display manner is not specifically limited in the present application.

In the above scheme, the multimedia conference window displays a client screen corresponding to a multimedia conference participant, the client screen may include an image obtained by a camera of the participant or a default image set by the client, and the first operation may be an operation such as a check, a long press, a click and the like performed on the screen of the client, or an operation such as a check, a long press, a click and the like performed on the client in a member list of the multimedia conference.

The first operation in the multimedia conference window is to select a specific client among a plurality of clients accessing the multimedia conference. Fig. 2 is a schematic diagram of selecting a client through a first operation according to an embodiment of the present application, and with reference to fig. 2, a multimedia conference window is displayed on a display interface, where the multimedia conference window includes pictures of clients participating in a multimedia conference, and when a user performs the above-mentioned click, long-press or other operations on the picture of at least one client, at least one of the clients is selected.

Fig. 3 is a schematic diagram of another client selection through a first operation according to an embodiment of the present application. The current interface displays the PPT when the speaker speaks, on the basis, a conference staff list at the lower right corner can be called out, the conference staff list comprises clients participating in the multimedia conference, and a user can perform selection operation on the identification of the client needing to be communicated independently, so that at least one client can be selected.

And step S104, selecting at least one client from the plurality of clients according to the first operation.

As before, after receiving the first operation, the client indicated by the first operation is taken as the selected client.

Still referring to FIG. 2, if the user presses the screen of the client with ID132465 for a long time, then the client is selected, and referring again to FIG. 3, if the user clicks on ID132465, then the client is also caused to be selected.

After at least one client is selected, the interface of the user performing the first operation may send a prompt message to determine whether the user needs to perform the individual communication, and after the user determines that the user needs to perform the individual communication, the step S106 is performed to collect the first audio information, so as to avoid transmission of the audio information due to the misoperation. For example, the user interface performing the first operation may prompt "whether to communicate with the user 132465 alone", and when the user selects "yes", the process proceeds to step S106.

And step S106, collecting the first audio information sent to the selected at least one client.

The first audio information is acquired by the client performing the first operation, and the first audio information is the audio information which is independently sent to the selected at least one client.

Step S108, the first audio information is sent to the server, so that the server sends the first audio information to the selected at least one client.

In an alternative embodiment, when at least one client is selected, the identity of the selected at least one client is sent to the server, and the server records the identity of the selected at least one client first. The server thus sends the first audio information to the selected at least one client upon receiving the first audio information, based on the previous recording of the selected at least one client. Here, the identification of the client may be an ID, an IP address, or a nickname of the client in the multimedia conference, etc.

In an optional embodiment, the identifier of the selected at least one client is also sent to the server while the first audio information is sent to the server, and after the server receives the first audio information, the server forwards the first audio information to the selected at least one client according to the identifier of the selected at least one client.

Specifically, in an optional embodiment, after the client collects the first audio information through a microphone and performs encoding processing, the client actively transmits the first audio information to the server through the voice communication link between the client and the server, and the server transmits the first audio information to the selected at least one client. In another optional embodiment, after the first audio information is collected, the client sends a notification message to the server, the server stores the received notification message in a preset message queue, and the server sequentially acquires the audio information according to the notification message in the message queue. When the notification message sent by the current client is in turn, the server sends response information corresponding to the notification message to the current client so as to allow the client to send the first audio information to the server.

Step S1010, receiving second audio information sent by the server, where the second audio information is audio information generated in the multimedia conference process.

Specifically, the audio information in the multimedia conference process refers to audio information facing all clients in the normal multimedia conference process. Here, all clients are directed to that the audio information of the speaking client is transmitted to all other clients except the speaking client. And after receiving the audio information of the multimedia conference process, the server forwards the audio information to all the other clients except the speaking client.

In an alternative embodiment, during the speaking of the speaker of the multimedia conference, the speaking of the speaker is transmitted to all other clients participating in the multimedia conference except the client of the speaker through the server. When a client communicates with at least one selected client independently, the audio information of the main speaker sent by the server is not influenced, namely, the audio information of the main speaker sent by the server is still received. For example, when the client a communicates with the selected at least one client b independently, all client audio information of the client c facing the multimedia conference can be received at the same time.

Step S1012, playing the second audio information.

In the scheme, when one client side sends audio information to other client sides independently, the client side does not influence the receiving and playing of the audio information in the multimedia conference process, so that even if two or more client sides in the multimedia conference are communicated independently, the client sides are not influenced to receive the audio information in the multimedia conference process, and therefore the client sides in the multimedia conference can communicate independently and cannot miss important information of the conference.

Therefore, in the embodiment of the present application, a first operation in a multimedia conference window is received, and a plurality of clients accessing a multimedia conference are displayed in the multimedia conference window; selecting at least one client from the plurality of clients according to a first operation; collecting first audio information sent to the selected at least one client; sending the first audio information to a server, so that the server sends the first audio information to at least one selected client, and receives second audio information sent by the server, wherein the second audio information is audio information generated in the multimedia conference process; and playing the second audio information. In the scheme, the first operation is carried out on at least one client in the multimedia conference, the identification of at least one client selected by the first operation and the audio information are sent to the server, so that the server only sends the audio information to the selected at least one client, the independent communication among part of the clients in the multimedia conference is realized, the operation is simple and convenient, and the technical problem that the multimedia conference in the prior art cannot meet the requirement of independent communication among participants is solved.

As an alternative embodiment, the method further includes: and receiving a second operation in the multimedia conference window, and canceling the selection of the selected at least one client.

The steps are executed after the first operation in the multimedia conference window is received, and the selected client is cancelled after the second operation in the multimedia conference window is received. After the selected client is cancelled, if the audio information is collected again, the audio information is sent to all the clients participating in the multimedia conference except the client collecting the audio information by the server, namely the audio information faces all other clients, the independent communication is finished, and the switching from the independent communication to the communication of the whole staff in the multimedia conference is realized.

In an alternative embodiment, as shown in fig. 3, the first selection operation of the user is to check ID132465, so that the client with ID132465 is selected, and when the user cancels the check of ID132465, the client receives the second operation, so that the client with ID132465 is deselected.

As an optional embodiment, after sending the first audio information to the server, so that the server sends the first audio information to the selected at least one client, the method further includes: and when the long-press operation is finished, canceling the selection of the selected at least one client.

In the above scheme, the first operation is a long-press operation, and when the long-press operation is finished, the selection of the client selected by the long-press operation is cancelled.

In an alternative embodiment, in conjunction with the interactive interface shown in FIG. 2, to communicate with the 132465 client individually, the user may start holding the window of 132465 client to select it, at which time the identity of 132465 client may be sent to the server so that the server knows the at least one client that was selected. Then the user starts speaking, after the speaking is finished, the user lifts the finger to finish the long-time pressing operation, at the moment, the audio information of the speaking is sent to the server, the server sends the audio information to at least one selected client according to the received identification, and the client with the ID of 132465 is deselected.

The scheme realizes the selection and the deselection of the client by long-press operation, so that a user can speak after long-press the client needing to be communicated independently and stop long-press after speaking is finished, and the aim of sending voice information to the specified client in the multimedia conference can be fulfilled, and the method is simple and convenient.

As an alternative embodiment, the method further includes: receiving third audio information sent by the server; the third audio information is sent by a third client terminal accessed to the multimedia conference, and the current client terminal is selected by the third client terminal according to the first operation of the third client terminal; adjusting the volume parameter of the second audio information and/or the volume parameter of the received third audio information; and playing the received third audio information and the second audio information together.

Specifically, the current client is used for representing the client that collects the first audio information and receives the second audio information sent by the server. The third audio information refers to audio information forwarded to the current client by the server after the current client is selected by the other clients except the current client through the first operation in the video conference, and the third audio information is only receivable by the current client.

In the scheme, the current client not only can actively select other clients and send the audio information to the selected other clients to actively realize the independent communication with the other clients, but also can be selected by the other clients, receive the audio information which is appointed by the other clients and is sent by the other clients, and passively realize the independent communication with the other clients. The operation mode of the selected client is the same for different clients, and therefore, the first operation representation is also adopted in the above scheme, and the specific mode is the same as the foregoing embodiment.

When the current client receives the second audio information in the multimedia conference process, if any other client selects the current client and appoints to send the third audio information to the current client, the current client receives the second audio information and the third audio information at the same time, so that audio conflict occurs. When the audio conflict occurs at the current client, the current client can process the audio conflict by adjusting the volume parameter of the second audio information and/or the volume parameter of the third audio information.

As an optional embodiment, adjusting the volume parameter of the second audio information and/or the volume parameter of the third audio information includes: and reducing the volume parameter of the second audio information, and/or increasing the third volume parameter of the third audio information so that the volume parameter of the second audio information is lower than the volume parameter of the third audio information.

In the above scheme, when an audio conflict occurs at the current client, the volume of the second audio information in the multimedia conference process is reduced to be smaller than that of the third audio information, and even the volume of the second audio information can be reduced to zero, so that the second client can hear the third audio information more clearly; the third volume parameter of the third audio information can be increased to be larger than the second volume parameter, so that the second client can hear the third audio information more clearly.

Example 2

Fig. 4 is a flowchart of a method for controlling audio in a multimedia conference, in which a plurality of clients access the multimedia conference, as shown in fig. 4, the method includes the following steps:

step S402, receiving first audio information sent by the first client.

The steps are executed by a server of the multimedia conference, the server of the multimedia conference is provided with multimedia conference software and can be arranged at a cloud end, clients participating in the multimedia conference access the multimedia conference through accessing the server at the cloud end and can also be arranged at the local of one party of the multimedia conference, such as the local of a speaker of the conference, and other clients participating in the multimedia conference access the local server of the speaker through a network to join the multimedia conference.

The multimedia conference in the present application may be a video conference, which transmits video and audio simultaneously, may include transmission of video and/or audio of multiple parties participating in the video conference, and sharing of multimedia files, and may also be an audio conference, which transmits only audio, and the specific form of the present application is not specifically limited. The interactive interface provided by the multimedia conference can display the video/head portrait/client identifier of each participant, can also display the multimedia file shared by the speakers, and can also switch between the two or be displayed in a split-screen manner, and the specific display manner is not specifically limited in the present application.

The first client may be any client participating in the multimedia conference, and the first audio information represents audio information sent by the first client to the server. After receiving the audio information sent by the user, the first client sends the audio information to the server. In an alternative embodiment, the client collects the sound through a microphone, encodes the sound, and transmits the sound to the receiving end of the server.

Step S404, determining that the first client has the selected second client, and sending the first audio information to the second client.

The second client may also be any client participating in the multimedia conference. The first client has a selected second client to represent that the first client performed the first operation of embodiment one on the second client. The operations such as checking, long-pressing, and clicking may be performed on the screen of the client, or the operations such as checking, long-pressing, and clicking may be performed on the client in the member list of the multimedia conference.

It should be noted that, the selection of the second client by the first client may be performed before the first client sends the first audio information, for example, before the first client sends the first audio information, the second client may be subjected to operations such as checking, long pressing, clicking and the like to inform the server of the second client selected by the first client, and then the first audio information is collected and sent to the server, or before the first client sends the first audio information, the second client may be subjected to operations such as checking, long pressing, clicking and the like to select the second client, and then the first audio information is collected, and simultaneously the first audio information and the information of the selected second client are sent to the server; the second client can also be selected while the first client sends the first audio information, for example, the first client collects the first audio information while performing long-time press on the second client to select the second client, so that the first client sends the first audio information to the second client and carries the identifier of the second client, so that the server receives the first audio information and the information of the selected second client at the same time.

In an alternative embodiment, the first operation for client selection may result in the first client generating a forwarding label, where the forwarding label may be an identification of the second client selected by the first client, for example, an ID of the second client, a nickname of the second client in the multimedia conference, and the like. If the first client needs to forward the first audio information to the designated second client, the second client needs to be selected. In this embodiment, after receiving the first audio information, the server first determines whether the first client has a forwarding label. Specifically, the server stores a forwarding marking bit corresponding to the transmission channel of each client, the forwarding marking bit is used for storing a forwarding marking of the client, and the server determines whether the first client has a selected second client by querying the forwarding marking bit of the first client.

If the first client has the selected second client, it indicates that the first client wishes to forward the first audio information to the designated second client, and therefore the server forwards the first audio information to the second client designated in the forwarding label.

It should be noted that the number of the second clients may be one or more, that is, the first client may have one selected second client or may have a plurality of selected second clients.

Step S406, determining that the first client does not have the selected second client, and sending the first audio information to all clients except the first client in the multimedia conference.

And if the first client does not have the selected second client, determining that the first audio information is not used for independent communication with a certain client in the multimedia conference, and directly forwarding the first audio information to all other clients except the first client, which are accessed into the multimedia conference.

Whether the first audio information is forwarded to all other clients or to the designated client is determined by judging whether the first client has the selected second client, so that in the multimedia conference, one client can speak to all other clients and communicate with only the designated client.

In the above scheme, if the current server receives the audio information sent by one client without the selected second client, and receives the audio information sent by another client without the selected second client in the process of transmitting and forwarding the audio information to all other clients, the current server may send prompt information to the client which sends the audio information backward to prompt whether the client needs to add a forwarding label before speaking, so as to avoid a situation that the audio information is sent to all other clients due to forgetting to add the forwarding label.

For example, in the process of speaking by the speaker, the client a needs to communicate with the client B separately, but forgets to select the client B, so the server receives the audio information sent by the client a while receiving the audio information sent by the client of the speaker. At this time, the server sends prompt information to the client A: and if the user selects no on the client A, the server transmits the audio to all other clients in the multimedia conference, and after waiting for the client A to select other clients, the audio is forwarded according to at least one client selected by the client A.

Fig. 5 is a schematic diagram of processing audio information in a multimedia conference according to an embodiment of the present application. As shown in connection with fig. 5, there are four clients shown participating in a multimedia conference in this example: each client has a sending end (Audio Send Stream) of Audio information and a receiving end (Audio Rev Stream) of the Audio information, wherein the sending end is used for collecting the Audio information through a microphone, carrying out coding processing on the Audio information and then transmitting the Audio information to the receiving end of the server, and the receiving end is used for receiving the Audio information transmitted by the server and then carrying out decoding playing.

The server creates a multimedia conference Room0 for the above four clients, Room0 includes a processing module corresponding to each client, EP1 corresponding to client1, EP2 corresponding to client2, EP3 corresponding to client3, and EP4 corresponding to client 4. Each processing module comprises a sending end, a processor receiving end and a unique client identification, wherein the sending end of the processing module is used for sending audio stream data to the client, the receiving end of the processing module is used for receiving the audio stream data sent by the client, the unique client identification can be a user ID or an equipment physical address, and the server determines a data forwarding link through the identification.

The Room0 also includes a filter module for filtering audio stream data to be forwarded by the server, which can forward the audio stream data to all clients except the sender to avoid the sound producer hearing own echo. If only one person in the multimedia conference is to be restricted from speaking, it is possible to deal with this by making a policy, such as: when a plurality of audio stream data need to be forwarded, only the audio stream data with the largest volume is forwarded, or the audio stream data which arrives recently is forwarded.

Also included in Room0 is a selective Filter module (Audio Select Filter) that forwards Audio stream data to the client of instructions when it is determined at the time of forwarding that the client has marked a client designated for forwarding.

In the following, two examples are described, that the client1 does not select other clients to speak, and that the client4 selects other clients to speak.

When the client1 has no other selected client, that is, the client1 has no forwarding label, when the client1 speaks, the sending end in the client1 encodes the audio information to obtain audio stream data, and sends the audio stream data to the receiving end in the EP 1. At this time, the processor determines whether the client1 has other selected clients, and the determination result is that the client1 does not have other selected clients, so that the receiving end in EP1 sends the received audio stream data carrying the identifier of the client1 to the channel elimination filter module, and the channel elimination filter module forwards the audio stream of the client1 to all other clients except the client1 in the rom 0, that is, the client2, the client3, and the client4, thereby realizing the speaking of the client 1.

The client4 selects other clients and makes a speech, when the client4 makes a speech, the sending end in the client4 encodes audio information to obtain audio stream data, and sends the audio stream data to the receiving end in the EP 4. At this time, the processor determines whether the client4 has other selected clients, and the determination result is that the client4 has other selected clients, and the forwarding flag in the client4 indicates that the client3 is, so that the receiving end in EP1 sends the received audio stream data carrying the identifier of the client4 to the selective filtering module, the selective filtering module sends the audio stream data of the client4 to the sending end of EP3, and the sending end of EP3 sends the audio stream data of the client4 to the receiving end of the client3, so that only the client3 in rom 0 can receive the audio stream data of the client 4.

The above-mentioned scheme can be applied to a multimedia conference, and in the case that a current speaker is not affected, any one client needs to communicate with at least one other client, which is exemplified below.

In an optional embodiment, when the speaker speaks, the client a needs to communicate with the client B separately, the client a may add the tagging information as the ID of the client B, and the server may send only the audio information of the client a to the client B when processing the audio information sent by the client a. Similarly, when the client B replies to the client a separately, the ID of the client a needs to be added with the tagging information, and the server only sends the audio information of the client B to the client a when processing the audio information sent by the client B, thereby completing the separate communication between the two clients in the multimedia conference.

In another optional embodiment, when the speaker speaks, the client a needs to send a prompt voice to the speaker, and the prompt voice does not need to be heard by other clients in the multimedia conference, so the client a may add the annotation information as the ID of the speaker, and the server may only send the audio information of the client a to the client of the speaker when processing the audio information sent by the client a. Under the condition, in the multimedia conference, other clients can receive the audio information of the speaker except the speaker, and only the speaker can receive the prompt voice of the client A sending end, so that the remote prompt of the speaker is realized.

Therefore, the embodiment of the present application receives the first audio information sent by the first client; determining that the first client has a selected second client, and sending the first audio information to the second client; and determining that the first client does not have the selected second client, and sending the first audio information to all clients except the first client in the multimedia conference. In the above scheme, the server determines the forwarding mode of the audio information by judging whether the client sending the audio information has the selected second client, so that only the audio information is forwarded to the specified client determined by the forwarding label under the condition that the client sending the audio information has the selected second client, thereby realizing the independent communication among part of the clients in the multimedia conference, being simple and convenient to operate, and solving the technical problem that the multimedia conference in the prior art cannot meet the independent communication among participants.

As an alternative embodiment, before determining that the first client has the selected second client, the control method further includes: receiving selected information of a second client sent by a first client; the selected information is determined by the first client according to a first operation in the multimedia conference window; the selection of the second client by the first client is recorded.

In the above scheme, the multimedia conference window displays a client screen corresponding to a multimedia conference participant, the client screen may include an image obtained by a camera of the participant or a default image set by the client, and the first operation may be an operation such as a check, a long press, a click and the like performed on the screen of the client, or an operation such as a check, a long press, a click and the like performed on the client in a member list of the multimedia conference. And after receiving the first operation, taking the client indicated by the first operation as the selected client.

As an optional embodiment, the control method further includes: receiving deselection information of a second client sent by a first client; the deselection information is determined by the first client according to a second operation in the multimedia conference window; and clearing the selected record of the first client to the second client.

The steps are executed after the first operation in the multimedia conference window is received, and the selected record of the first client to the second client is cleared after the second operation of the multimedia conference window is received. After the selected record of the first client to the second client is cleared, if the audio information sent by the first client is collected again, the audio information is sent to all other clients participating in the multimedia conference except the first client by the server, namely the audio information faces all other clients, so that the independent communication between the first client and the second client is finished, and the switching from the independent communication to the whole communication in the multimedia conference is realized.

As an optional embodiment, the receiving of the first audio information sent by the first client specifically includes: and receiving selected information of a second client sent by the first client while receiving the first audio information sent by the first client.

In the above scheme, the selected information of the second client from the first client is sent out at the same time when the first client sends the first audio information, but not before the first client sends the first audio information. Specifically, before the first client sends the first audio information, the second client may be subjected to operations such as checking, long-time pressing, clicking and the like, the second client is selected, then the first audio information is collected, and the first audio information and the information of the selected second client are sent to the server; the second client may also be selected while the first client sends the first audio information, for example, the first client collects the first audio information while performing long-time pressing on the second client to select the second client, so that the first client sends the first audio information to the second client and carries the identifier of the second client, so that the server receives the first audio information and the information of the selected second client at the same time.

In an optional embodiment, after receiving the first audio information, the first client encodes the first audio information to obtain a code stream corresponding to the first audio information, and then packs the identification information of the second client to be selected and the code stream corresponding to the first audio information and sends the packed information to the server at the same time, so that the server receives the selected information of the second client from the first client while receiving the first audio information.

As an optional embodiment, after determining that the first client has the selected second client and sending the first audio information to the second client, the control method further includes: determining that there is second audio information sent to the second client; the second audio information and the first audio information are sent by different clients accessing the multimedia conference; adjusting a first volume parameter of the first audio information and/or a second volume parameter of the second audio information; and sending the second audio information to the second client.

The second audio information may be from the first audio information, which may be the current audio information of the speaker, and the audio information is forwarded by the processor to all other clients except the speaker, and the second audio information may be from another client needing to communicate with the second client separately, except the first audio information. When the server forwards the first audio information to the second client according to the forwarding label of the first client, if the second client also receives the second audio information forwarded by other clients at the moment, an audio conflict occurs at the second client. In the above scheme, when an audio collision occurs at the second client, the processor processes the audio collision by adjusting the first volume parameter of the first audio information and/or the first volume parameter of the second audio information.

It should be noted that, when an audio conflict occurs at the second client, the volume parameters of the two audio information that have a conflict may be adjusted based on the processor, and the volume parameters of the two audio information that have a conflict may also be adjusted by the user on the second client.

As an optional embodiment, adjusting a first volume parameter of the first audio information and/or a second volume parameter of the second audio information includes: the second volume parameter of the second audio information is decreased, and/or the first volume parameter of the first audio information is increased, so that the second volume parameter is lower than the first volume parameter.

In the above scheme, when an audio conflict occurs at the second client, the volume of the second audio information in the multimedia conference process is reduced to be smaller than the volume of the first audio information, and even the volume of the second audio information can be reduced to zero, so that the second client can listen to the first audio information more clearly; the first volume parameter of the first audio information can be increased to be larger than the second volume parameter, so that the effect that the second client can listen to the first audio information more clearly can be achieved.

Fig. 6 is an information interaction diagram of an audio processing method according to an embodiment of the present application, and referring to fig. 6, in this embodiment, when performing audio processing, an interaction step between a first client a and a server includes:

and S61, pressing and holding the picture of the appointed client on the multimedia conference interface of the client A, and determining to select the client B.

In the above steps, when the user carries out long press on the appointed client side in the interface of the multimedia conference, the client side B is selected. In this example, the operation that triggers the client a to send the forwarding annotation request is a long press operation.

S62, the client a sends a request to the server to add the identifier of the specified client B to the forwarding label.

S63, the server stores the forwarding label of the client A, and the forwarding label includes the ID of the client B.

S64, the user at the client a keeps the long press operation in step S61 and speaks.

S65, the client a sends the audio stream data to the server.

S66, the server receives the audio stream data sent by the client A and judges whether the client A has a forwarding label.

Since the client a sends the forwarding label request to the server in step S62, the determination result indicates that the client a has the forwarding label.

S67, the user at client a cancels the long press on the screen of client B.

S68, the client a sends a request to cancel the audio forwarding annotation to the server.

S69, the server nulls the annotation request of the client A.

Example 3

According to an embodiment of the present application, there is also provided a device for controlling audio in a multimedia conference, fig. 7 is a schematic diagram of a device for controlling audio in a multimedia conference according to an embodiment of the present application, and multiple clients access to a multimedia conference, as shown in fig. 7, the device includes:

the first receiving module 70 is configured to receive a first operation in a multimedia conference window, where a plurality of clients accessing a multimedia conference are displayed in the multimedia conference window.

A selecting module 72 is configured to select at least one client from the plurality of clients according to a first operation.

And a collecting module 74, configured to collect the first audio information sent to the selected at least one client.

A sending module 76 is configured to send the first audio information to the server, so that the server sends the first audio information to the selected at least one client.

A second receiving module 78, configured to receive second audio information sent by the server; the second audio information is audio information generated in the multimedia conference process.

And the playing module 710 is configured to play the second audio information.

As an alternative embodiment, the apparatus further comprises: and the third receiving module is used for receiving the second operation in the multimedia conference window and canceling the selection of the selected at least one client.

As an alternative embodiment, the first operation is a long press operation, and the apparatus further comprises: and the cancellation module is used for canceling the selection of the selected at least one client when the long-press operation is finished after the first audio information and the identification of the selected at least one client are sent to the server so that the server sends the first audio information to the selected at least one client.

As an alternative embodiment, the apparatus further comprises: the fourth receiving module is used for receiving third audio information sent by the server; the third audio information is sent by a third client terminal accessed to the multimedia conference, and the current client terminal is selected by the third client terminal according to the first operation of the third client terminal; the adjusting module is used for adjusting the volume parameter of the second audio information and/or the volume parameter of the received third audio information; and the second playing module is used for playing the received third audio information and the second audio information together.

As an alternative embodiment, the adjusting module comprises: and the adjusting submodule is used for reducing the volume parameter of the second audio information and/or increasing the third volume parameter of the third audio information so as to enable the volume parameter of the second audio information to be lower than the volume parameter of the third audio information.

The control device for audio in a multimedia conference of this embodiment can execute the control method for audio in a multimedia conference provided in the foregoing method embodiment 1, can execute any combination implementation steps not limited to this method, and has corresponding functions and advantageous effects of this method.

Example 4

According to an embodiment of the present application, there is also provided a device for controlling audio in a multimedia conference, fig. 8 is a schematic diagram of a device for controlling audio in a multimedia conference according to an embodiment of the present application, and a plurality of clients access to the multimedia conference, as shown in fig. 8, the device includes:

the first receiving module 80 is configured to receive first audio information sent by the first client.

A first sending module 82, configured to determine that the first client has the selected second client, send the first audio information to the second client.

And a second sending module 84, configured to determine that the first client does not have the selected second client, and send the first audio information to all clients except the first client in the multimedia conference.

As an alternative embodiment, the apparatus further comprises: the second receiving module is used for receiving the selected information of the second client sent by the first client before the first client is determined to have the selected second client; the selected information is determined by the first client according to a first operation in the multimedia conference window; and the recording module is used for recording the selection of the first client to the second client.

As an alternative embodiment, the apparatus further comprises: the third receiving module is used for receiving the deselection information of the second client side, which is sent by the first client side; the deselection information is determined by the first client according to a second operation in the multimedia conference window; and the clearing module is used for clearing the selected record of the first client to the second client.

As an alternative embodiment, the first receiving module includes: and the receiving submodule is used for receiving the selected information of the second client side, which is sent by the first client side, while receiving the first audio information sent by the first client side.

As an alternative embodiment, the apparatus further comprises: a determining module for determining that there is second audio information sent to the second client; the second audio information and the first audio information are sent by different clients accessing the multimedia conference; the adjusting module is used for adjusting a first volume parameter of the first audio information and/or a second volume parameter of the second audio information; and the third sending module is used for sending the second audio information to the second client.

As an alternative embodiment, the adjusting module comprises: and the adjusting submodule is used for reducing the second volume parameter of the second audio information and/or increasing the first volume parameter of the first audio information so as to enable the second volume parameter to be lower than the first volume parameter.

It should be noted that, when the control device for audio in a multimedia conference provided in the foregoing embodiment executes control of audio in a multimedia conference, the division of each functional module is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the control device for audio in a multimedia conference and the control method embodiment for audio in a multimedia conference provided in the above embodiments belong to the same concept, and details of implementation processes are found in the method embodiments, which are not described herein again.

Example 5

An embodiment of the present application further provides a computer storage medium, where the computer storage medium may store a plurality of instructions, where the instructions are suitable for being loaded by a processor and executing the method steps in the embodiments shown in fig. 1 to 7, and a specific execution process may refer to specific descriptions of the embodiments shown in fig. 1 to 7, which are not described herein again.

The device on which the storage medium is located may be a smart interactive tablet.

Example 6

According to an embodiment of the present application, there is also provided an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps of embodiment 1.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for controlling audio in a multimedia conference, comprising:

receiving a first operation in a multimedia conference window, wherein the multimedia conference window displays a plurality of clients accessing a multimedia conference;

selecting at least one client from the plurality of clients according to the first operation;

collecting first audio information sent to the selected at least one client;

sending the first audio information to a server, so that the server sends the first audio information to the selected at least one client;

receiving second audio information sent by the server, wherein the second audio information is audio information generated in the multimedia conference process;

and playing the second audio information.

2. The method of claim 1, further comprising:

and receiving a second operation in the multimedia conference window, and canceling the selection of the at least one selected client.

3. The method of claim 1, wherein the first operation is a long press operation, and wherein after sending the first audio information to a server such that the server sends the first audio information to the selected at least one client, the method further comprises:

and when the long-press operation is finished, canceling the selection of the at least one selected client.

4. The method of claim 1, further comprising:

receiving third audio information sent by the server;

the third audio information is sent by a third client terminal accessed to the multimedia conference, and the current client terminal is selected by the third client terminal according to the first operation of the third client terminal;

adjusting a volume parameter of the second audio information and/or a volume parameter of the received third audio information;

and playing the received third audio information and the second audio information together.

5. The method of claim 4, wherein the adjusting the volume parameter of the second audio information and/or the volume parameter of the third audio information comprises:

the volume parameter of the second audio information is decreased,

and/or, a third volume parameter of the third audio information is adjusted to be high,

so that the volume parameter of the second audio information is lower than the volume parameter of the third audio information.

6. A method for controlling audio in a multimedia conference, the method comprising:

receiving first audio information sent by a first client;

determining that the first client has a selected second client, and sending the first audio information to the second client;

and determining that the first client does not have the selected second client, and sending the first audio information to all clients except the first client in the multimedia conference.

7. The method of claim 6, wherein prior to the determining that the first client has the selected second client, the controlling method further comprises:

receiving selected information of a second client sent by the first client; the selected information is determined by the first client according to a first operation in a multimedia conference window;

recording the selection of the second client by the first client.

8. The method of claim 7, wherein the control method further comprises:

receiving deselection information of a second client side, which is sent by the first client side; the deselection information is determined by the first client according to a second operation in the multimedia conference window;

and clearing the selected record of the first client to the second client.

9. The method according to claim 6, wherein the receiving the first audio information sent by the first client is specifically:

and receiving selected information of a second client side sent by a first client side while receiving first audio information sent by the first client side.

10. The method of claim 6, wherein after determining that the first client has the selected second client and sending the first audio information to the second client, the controlling method further comprises:

determining that there is second audio information sent to the second client; the second audio information and the first audio information are sent by different clients accessing the multimedia conference;

adjusting a first volume parameter of the first audio information and/or a second volume parameter of the second audio information;

and sending the second audio information to the second client.

11. The method of claim 10, wherein the adjusting the first volume parameter of the first audio information and/or the second volume parameter of the second audio information comprises:

lowering a second volume parameter of the second audio information,

and/or, turning up a first volume parameter of the first audio information,

such that the second volume parameter is lower than the first volume parameter.

12. An apparatus for controlling audio in a multimedia conference, comprising:

the system comprises a first receiving module, a second receiving module and a third receiving module, wherein the first receiving module is used for receiving a first operation in a multimedia conference window, and the multimedia conference window displays a plurality of clients accessed to a multimedia conference;

a selection module for selecting at least one client from the plurality of clients according to the first operation;

the acquisition module is used for acquiring first audio information sent to the selected at least one client;

the sending module is used for sending the first audio information to a server, so that the server sends the first audio information to the selected at least one client;

the second receiving module is used for receiving second audio information sent by the server; the second audio information is audio information of the multimedia conference process;

and the playing module is used for playing the second audio information.

13. A computer storage medium, characterized in that it stores a plurality of instructions adapted to be loaded by a processor and to perform the method steps of any of claims 1 to 11.

14. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps of any of claims 1 to 11.