CN117749947A - Multi-terminal protocol-based multi-party call processing method and system - Google Patents

Abstract

The invention discloses a multi-party call processing method and a multi-party call processing system based on a multi-terminal protocol, wherein the multi-party call processing method comprises the following steps: the server is connected with each first terminal through a relay, receives first audio data sent by the first terminals, performs unified protocol header adaptation and identifier generation on each first audio data, performs pretreatment on each first audio data, periodically acquires each third audio data obtained after pretreatment, mixes each third audio data to obtain first mixed audio data, performs audio deletion on the first mixed audio data according to each third audio data, determines a terminal corresponding to each second mixed audio data obtained after audio deletion according to the audio identifier of each third audio data, packages each second mixed audio data according to the protocol header of the terminal, and sends the packaged second mixed audio data to the corresponding terminal, thereby improving the effect of multiparty call.

Description

Multi-terminal protocol-based multi-party call processing method and system

Technical Field

The invention relates to the technical field of communication processing, in particular to a multi-party call processing method and system based on a multi-terminal protocol.

Background

In the present internet era, with the continuous upgrading of network conditions and the continuous popularization of intelligent terminals, the business form of performing audio and video session based on multiparty calls is more and more paid attention to and developed.

In the prior art, a set of commodity service system is often required to be used in audio transmission of a multiparty call in different terminals, the single commodity service system only supports a single set of signaling protocol, all terminal devices accessed by a third party need to develop and be compatible around the signaling protocol, the defect of poor flexibility exists, and because all terminals need to call around the signaling protocol, the terminals are required to meet the requirement of the signaling protocol to access the commodity service system, terminal devices meeting the requirement must be purchased when the multiparty call is performed, the cost of the multiparty call is increased, and in the later period, along with the increase of conference participants, the development and compatible workload around the single signaling protocol also generates an increase in magnitude, so that the data volume required to be processed in audio transmission of the single commodity service system is increased, the audio uploaded by the multiparty terminal cannot be timely processed, and the effect of the multiparty call is poor.

Disclosure of Invention

In order to solve the technical problems, the invention discloses a multi-party call processing method and system based on a multi-terminal protocol, which improves the flexibility and effect of multi-party call.

In order to achieve the above object, the present invention discloses a multi-party call processing method based on a multi-port protocol, comprising:

the method comprises the steps that a server receives a signaling sent by each of a plurality of first terminals through a preset relay, and is connected with each first terminal according to the signaling;

the server receives the first audio data sent by each first terminal, and performs protocol header unified adaptation and identifier generation on each first audio data according to a preset unified protocol header and identifier generation method to obtain a plurality of second audio data; the second audio data comprises an audio identification;

the server performs preprocessing on each second audio data respectively and stores third audio data obtained after preprocessing each second audio data;

the server side periodically acquires each third audio data, and gathers each third audio data through a preset audio mixing algorithm to acquire first audio mixing data;

The server performs audio deletion on the first audio data according to each third audio data to obtain a plurality of second audio data, and determines a second terminal corresponding to each second audio data according to an audio identifier corresponding to each third audio data;

the server acquires the protocol header corresponding to the second terminal, packages each second audio data according to the protocol header, and sends the packaged second audio data to the corresponding second terminal.

The invention discloses a multiparty call processing method based on multiport protocol, firstly, setting relay in service terminal to make the service terminal compatible with signaling protocol of third party terminal, processing and transmitting audio frequency in service terminal, realizing multiparty call of various terminals, improving flexibility of multiparty call, then after the service terminal is compatible with the multiparty terminal, obtaining audio frequency data sent by each party terminal, the service terminal is used for uniformly processing the audio frequency, thereby using preset protocol head to process protocol head adaptation for the second audio frequency data, so as to uniformly process the audio frequency data in later period, generating identification corresponding to each audio frequency data, facilitating matching of audio frequency and platform in later period, after generating the second audio frequency data, respectively preprocessing and saving the second audio frequency data, so as to eliminate noise and the like, then mixing audio frequency with improved audio frequency effect, so that multiparty terminal receives audio frequency sent by other terminals, further, for improving audio frequency mixing effect, the service terminal is used for uniformly processing audio frequency according to different audio frequency, and sending audio frequency data to different audio frequency terminals, and deleting the same audio frequency, and sending audio frequency data to the terminal is further processed by the invention, and the invention is used for mixing terminal is further used for determining the audio frequency data to be mixed by the same as the terminal, thereby improving flexibility of the invention, and the server performs unified forwarding, so that the tone quality of the mixed sound is improved, and the conversation effect is further improved.

As a preferred example, the receiving, at the server, the signaling sent by each of the plurality of terminals through the preset relay includes:

the server side binds different ports through the relay and receives a connection instruction sent by each terminal in the plurality of terminals respectively;

according to the connection instruction, the server side respectively acquires the signaling protocol document provided by each terminal through the port, and is compatible with the signaling protocol of each terminal according to the signaling protocol document, so as to be connected with each terminal in the plurality of terminals.

The invention utilizes the relays to bind different ports, can cooperate with each type of terminal to realize the compatibility processing of the terminal by a single relay, and further completes the compatibility of the signaling protocol of the third party terminal by the server, but not the compatibility of the signaling protocol of the third party terminal in the prior art, thereby reducing the compatible workload, avoiding the equipment requirement on the third party terminal and reducing the cost of multiparty call.

As a preferred example, the performing, according to the preset unified protocol header and identifier generating method, the unified protocol header adaptation and identifier generation on each first audio data includes:

The server receives the audio protocol header rules respectively sent by each terminal, and converts the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules;

the server side combines the terminal address, the terminal port and the terminal identifier corresponding to each piece of first audio data to generate an audio identifier corresponding to each piece of first audio data.

The invention is based on the difference of the audio data into the protocol header, in order to uniformly process the audio data in the later stage, the audio effect is improved, and the protocol header of the audio data is adapted, so that the protocol header of all the audio data is uniformly changed into the protocol header of the server, the server can conveniently analyze the protocol header, and meanwhile, the corresponding identification of each audio data is generated, so that the terminal which is required to be transmitted by the current video is determined according to the identification in the later stage.

As a preferred example, the preprocessing is performed on each second audio data at the server, and third audio data obtained by preprocessing each second audio data is stored, including:

the server decodes and resamples each second audio data to obtain a plurality of third audio data;

The server writes each third audio data in the plurality of third audio data into a corresponding annular memory respectively, and stores the third audio data.

The invention decodes and resamples the data, so that all the audio data are in the same format, the later data mixing is convenient, the mixing effect is improved, then different audio data are respectively stored in different annular memories, and the memory copying times are greatly reduced according to the annular memories, thereby improving the software execution efficiency and the timeliness of conversation.

As a preferred example, the step of periodically obtaining each third audio data at the server and summarizing each third audio data by a preset audio mixing algorithm includes:

the server side periodically acquires each third audio data, and adds and gathers all the third audio data according to bit depth to acquire initial audio mixing data;

and carrying out maximum audio limitation and minimum audio limitation on the initial mixed audio data according to a preset limiting threshold value to obtain the first mixed audio data.

The invention periodically gathers the audio of all the third audios, so that all terminals participating in the call can timely receive audio data sent by other terminals participating in the call, and the effect of the call is improved.

As a preferred example, the server performs audio deletion on the first audio data according to each third audio data to obtain a plurality of second audio data, and determines a second terminal corresponding to each second audio data according to an audio identifier corresponding to each third audio data, where the method includes:

the server side respectively performs subtraction processing on the first audio data according to the bit depth of each third audio data, and determines a sending terminal of second audio data obtained after the current subtraction processing according to the audio identification of the current third audio data; wherein the sending terminal is the second terminal.

The invention deletes the sound of the terminal from the mixed sound, so that the terminal can hear the audio data of other terminals participating in the call except the terminal, and the audio playing effect is improved.

As a preferred example, the obtaining, at the server, a protocol header corresponding to the second terminal, and packaging each second audio mixing data according to the protocol header, includes:

and deleting the unified protocol header of the second audio mixing data by the server, and adding the acquired protocol header corresponding to the second terminal to the second audio mixing data so as to package the second audio mixing data.

The invention removes the unified protocol header in the audio data, adds the protocol header needed by the corresponding terminal, so that the terminal receiving the audio data can identify and play the audio data according to the added protocol header, and the multiparty call is completed.

On the other hand, the invention discloses a multiparty call processing system based on a multiport protocol, which comprises a terminal compatible module, an audio identification module, an audio processing module, an audio mixing module, a mixing classification module and an audio packaging module;

the terminal compatible module is used for receiving a signaling sent by each terminal in a plurality of first terminals through a preset relay by the service terminal, and connecting the signaling with each first terminal according to the signaling;

the audio identification module is used for receiving the first audio data sent by each first terminal by the server, and carrying out protocol header unified adaptation and identification generation on each first audio data according to a preset unified protocol header and identification generation method to obtain a plurality of second audio data; the second audio data comprises an audio identification;

the audio processing module is used for respectively preprocessing each second audio data by the server and storing third audio data obtained after preprocessing each second audio data;

The audio mixing module is used for periodically acquiring each third audio data by the server and summarizing each third audio data through a preset mixing algorithm to acquire first mixed audio data;

the audio mixing classification module is used for the server to perform audio deletion on the first audio mixing data according to each third audio data to obtain a plurality of second audio mixing data, and determine a second terminal corresponding to each second audio mixing data to send according to the audio identifier corresponding to each third audio data;

the audio packaging module is used for acquiring a protocol header corresponding to the second terminal by the server, packaging each piece of second audio mixing data according to the protocol header, and sending the packaged second audio mixing data to the corresponding second terminal.

The invention discloses a multiparty call processing system based on multiport protocol, firstly, setting relay in service terminal to make the service terminal compatible with signaling protocol of third party terminal, processing and transmitting audio frequency in service terminal, realizing multiparty call of various terminals, improving flexibility of multiparty call, then after the service terminal is compatible with the multiparty terminal, obtaining audio frequency data sent by each party terminal, the service terminal is used for uniformly processing the audio frequency, thereby using preset protocol head to process protocol head adaptation of the second audio frequency data, so as to uniformly process the audio frequency data in later period, generating identification corresponding to each audio frequency data, facilitating matching of audio frequency and platform in later period, after generating the second audio frequency data, respectively preprocessing and saving the second audio frequency data, so as to eliminate noise and the like, then mixing audio frequency with improved audio frequency effect, so that multiparty terminal receives audio frequency sent by other terminals, further, in order to improve audio frequency mixing effect, the service terminal is used for uniformly processing audio frequency according to different audio frequency, and sending audio frequency data to different audio frequency terminals, and deleting the audio frequency data from the terminal is further processed by mixing terminal, thereby making the invention, further comprises the invention, determining the audio frequency data is mixed by the terminal is used for mixing audio frequency data by the same as the terminal, and sending audio frequency data by the terminal is further processed by the invention, and the invention is better than the terminal is used for mixing audio frequency data by the user terminal is mixed by itself, and the server performs unified forwarding, so that the tone quality of the mixed sound is improved, and the conversation effect is further improved.

As a preferable example, the terminal compatible module includes a port connection unit and a compatible unit;

the port connection unit is used for the server to bind different ports through the relay and respectively receive the connection instruction sent by each terminal in the plurality of terminals;

the compatible unit is used for respectively acquiring the signaling protocol document provided by each terminal through the port according to the connection instruction, and is compatible with the signaling protocol of each terminal according to the signaling protocol document to connect with each terminal in the plurality of terminals.

The invention utilizes the relays to bind different ports, can cooperate with each type of terminal to realize the compatibility processing of the terminal by a single relay, and further completes the compatibility of the signaling protocol of the third party terminal by the server, but not the compatibility of the signaling protocol of the third party terminal in the prior art, thereby reducing the compatible workload, avoiding the equipment requirement on the third party terminal and reducing the cost of multiparty call.

As a preferred example, the audio identification module includes an adaptation unit and an identification unit;

the adaptation unit is used for receiving the audio protocol header rules respectively sent by each terminal by the server, and converting the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules;

The identification unit is used for the server to combine the terminal address, the terminal port and the terminal identification corresponding to each first audio data to generate the audio identification corresponding to each first audio data

The invention is based on the difference of the audio data into the protocol header, in order to uniformly process the audio data in the later stage, the audio effect is improved, and the protocol header of the audio data is adapted, so that the protocol header of all the audio data is uniformly changed into the protocol header of the server, the server can conveniently analyze the protocol header, and meanwhile, the corresponding identification of each audio data is generated, so that the terminal which is required to be transmitted by the current video is determined according to the identification in the later stage.

Drawings

Fig. 1: the embodiment of the invention discloses a multi-terminal protocol-based multi-party call processing method;

fig. 2: the embodiment of the invention discloses a multi-terminal protocol-based multiparty call processing system structure schematic diagram;

fig. 3: the embodiment of the invention discloses a micro-service architecture structure schematic diagram;

fig. 4: a flow diagram of a multi-party call processing method based on a multi-port protocol is disclosed for a further embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

The embodiment discloses a multi-party call processing method based on a multi-port protocol, and the specific implementation flow of the processing method can refer to fig. 1, and mainly comprises steps 101 to 106, wherein the steps are as follows:

step 101: the method comprises the steps that a server receives a signaling sent by each of a plurality of first terminals through a preset relay, and the server is connected with each first terminal according to the signaling.

In this embodiment, the steps mainly include: the server side binds different ports through the relay and receives a connection instruction sent by each terminal in the plurality of terminals respectively; according to the connection instruction, the server side respectively acquires the signaling protocol document provided by each terminal through the port, and is compatible with the signaling protocol of each terminal according to the signaling protocol document, so as to be connected with each terminal in the plurality of terminals.

In this embodiment, the step uses the relays to bind different ports, so that a single relay can be matched with each type of terminal to perform compatibility processing on the terminal, and further, the service end performs compatibility of a signaling protocol of the third party terminal, instead of the third party terminal in the prior art, so that the compatible workload is reduced, the equipment requirement on the third party terminal is avoided, and the cost of multiparty call is reduced.

Step 102: the server receives the first audio data sent by each first terminal, and performs protocol header unified adaptation and identifier generation on each first audio data according to a preset unified protocol header and identifier generation method to obtain a plurality of second audio data; the second audio data includes an audio identification.

In this embodiment, the steps mainly include: the server receives the audio protocol header rules respectively sent by each terminal, and converts the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules; the server side combines the terminal address, the terminal port and the terminal identifier corresponding to each piece of first audio data to generate an audio identifier corresponding to each piece of first audio data.

In this embodiment, the step is based on the difference of the audio data as the protocol header, so as to perform unified processing on the audio data in the later stage, improve the audio effect, and adapt the protocol header of the audio data, so that the protocol header of all the audio data is unified to be changed into the protocol header of the server, which is convenient for the server to analyze and process, and meanwhile generates the identifier corresponding to each audio data, so that the terminal to which the current video should be sent is determined according to the identifier in the later stage.

Step 103: the server side respectively preprocesses each second audio data and stores third audio data obtained after preprocessing each second audio data.

In this embodiment, the steps mainly include: the server decodes and resamples each second audio data to obtain a plurality of third audio data; the server writes each third audio data in the plurality of third audio data into a corresponding annular memory respectively, and stores the third audio data.

In this embodiment, the decoding and resampling are performed on the data in this step, so that all the audio data are in the same format, so that the later data mixing is facilitated, the mixing effect is improved, then different audio data are respectively stored in different annular memories, so that the memory copying times are greatly reduced according to the annular memories, the software execution efficiency is improved, and the timeliness of the call is further improved.

Step 104: and the server side periodically acquires each third audio data, and gathers each third audio data through a preset audio mixing algorithm to acquire first audio mixing data.

In this embodiment, the steps mainly include: the server side periodically acquires each third audio data, and adds and gathers all the third audio data according to bit depth to acquire initial audio mixing data; and carrying out maximum audio limitation and minimum audio limitation on the initial mixed audio data according to a preset limiting threshold value to obtain the first mixed audio data.

In this embodiment, the step periodically gathers the audio of all the third audio, so that all the terminals participating in the call can timely receive the audio data sent by other terminals participating in the call, and the effect of the call is improved.

Step 105: the server performs audio deletion on the first audio data according to each third audio data to obtain a plurality of second audio data, and determines a second terminal corresponding to and sending each second audio data according to the audio identifier corresponding to each third audio data.

In this embodiment, the steps mainly include: the server side respectively performs subtraction processing on the first audio data according to the bit depth of each third audio data, and determines a sending terminal of second audio data obtained after the current subtraction processing according to the audio identification of the current third audio data; wherein the sending terminal is the second terminal.

In this embodiment, the step deletes the sound of the terminal from the audio mix, so that the terminal can hear the audio data of other terminals participating in the call except the terminal itself, and the audio playing effect is improved.

Step 106: the server acquires the protocol header corresponding to the second terminal, packages each second audio data according to the protocol header, and sends the packaged second audio data to the corresponding second terminal.

In this embodiment, the steps mainly include: and deleting the unified protocol header of the second audio mixing data by the server, and adding the acquired protocol header corresponding to the second terminal to the second audio mixing data so as to package the second audio mixing data.

In this embodiment, the step removes the unified protocol header from the audio data, and adds the protocol header required by the corresponding terminal, so that the terminal that receives the audio data performs identification playing on the audio data according to the added protocol header, thereby completing the multiparty call.

On the other hand, the embodiment also discloses a multiparty call processing system based on a multiport protocol, and the specific structure of the system is shown in fig. 2, and mainly includes a terminal compatibility module 201, an audio identification module 202, an audio processing module 203, an audio mixing module 204, a mixing classification module 205 and an audio packaging module 206.

The terminal compatible module 201 is configured to receive, by a server, a signaling sent by each of a plurality of first terminals through a preset relay, and connect to each of the first terminals according to the signaling.

The audio identification module 202 is configured to receive, by the server, first audio data sent by each first terminal, and perform protocol header unified adaptation and identification generation on each first audio data according to a preset unified protocol header and identification generation method, so as to obtain a plurality of second audio data; the second audio data includes an audio identification.

The audio processing module 203 is configured to perform preprocessing on each second audio data by using the server, and store third audio data obtained by preprocessing each second audio data.

The audio mixing module 204 is configured to periodically obtain each third audio data by using the server, and collect each third audio data by using a preset mixing algorithm to obtain first mixed audio data.

The mixing classification module 205 is configured to perform audio deletion on the first mixed audio data according to each third audio data by using the server to obtain a plurality of second mixed audio data, and determine a second terminal corresponding to and sending each second mixed audio data according to an audio identifier corresponding to each third audio data.

The audio packaging module 206 is configured to obtain a protocol header corresponding to the second terminal from the server, package each second audio data according to the protocol header, and send the packaged second audio data to the corresponding second terminal.

In this embodiment, the terminal compatible module 201 includes a port connection unit and a compatible unit.

The port connection unit is used for the server to bind different ports through the relay and respectively receive the connection instruction sent by each terminal in the plurality of terminals.

The compatible unit is used for respectively acquiring the signaling protocol document provided by each terminal through the port according to the connection instruction, and is compatible with the signaling protocol of each terminal according to the signaling protocol document to connect with each terminal in the plurality of terminals.

In this embodiment, the audio identification module 202 includes an adaptation unit and an identification unit.

The adaptation unit is used for receiving the audio protocol header rules sent by each terminal respectively by the server, and converting the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules.

The identification unit is used for the server to combine the terminal address, the terminal port and the terminal identification corresponding to each first audio data to generate the audio identification corresponding to each first audio data.

Example two

In this embodiment, a multi-party call processing method based on multi-port protocol under a micro-service architecture is provided, where the basic structure of the micro-service architecture can refer to fig. 3, and as can be seen from fig. 3, the micro-service architecture divides a module into sub-service modules according to a functional division, and performs integrated communication on the sub-service modules through a specific means, and specifically, the micro-service architecture includes a Redis, a logic preemption service, a relay service, a central scheduling service, a media scheduling service and a media algorithm processing service.

Referring to fig. 3, the dis is used for storing a hot data account book after logic preemption, recovering data to a state before abnormality at any time in case of abnormality of a logic preemption service or a media scheduling service, and simultaneously is used as a communication medium between the logic preemption service and the media scheduling service, and after the logic preemption of a task is performed, the task is written into the dis and is quickly read by the media scheduling service and immediately processes the media data for scheduling.

The logical preemption service performs logical preemption on tasks, and whether the tasks are preempted successfully is determined by the task type priority, the task priority and the audio mixing mode according to the service design.

The relay service is responsible for carrying out signaling communication on all terminals, analyzing the received terminal signaling request, carrying out request event on the central scheduling service, converting event data of the central scheduling service into corresponding type terminal signaling data and transmitting the corresponding type terminal signaling data to a specific terminal.

The central scheduling service is responsible for scheduling the resources of the whole system, sending the task created by triggering the front end of the upper layer to the logic preemption service for business preemption, receiving the success result of preemption, sending the event data of the initialization terminal needed according to the task data to the relay service (starting sound card acquisition and starting power amplifier play), and simultaneously, parallelly sending a media algorithm processing service request to start the conference task.

The media scheduling service is used for processing and analyzing private audio data protocols of various terminals, integrating the various private audio data protocols into a unified audio data protocol and sending the unified audio data protocol to the media algorithm processing service for carrying out specific algorithm processing, wherein if the terminals exist in the task in other media scheduling services, data streams are needed to be sent to the corresponding media scheduling services, and the purpose of balancing the data streams of the terminals is achieved.

The media algorithm processing service is used for processing audio frequency, consuming resources and consuming time, and comprises a conference mixing, voice control operation and task recording module.

Referring to fig. 4, it can be seen from fig. 4 that the specific implementation flow of the multi-party call processing method based on the multi-port protocol according to fig. 3 mainly includes steps 401 to 405, where the steps are as follows:

step 401: the server side is compatible with a signaling protocol of each terminal in a plurality of different terminals according to a preset relay service, and receives first audio data sent by each terminal respectively according to the signaling protocol.

In this embodiment, this step is mainly: the server side respectively receives the connection instruction sent by each terminal in the plurality of terminals through the ports with different relay binding, specifically, the server side respectively receives the signaling of the third party terminal through the ports with different relay service binding respectively, namely, the server side uses the socket port as a distinguishing processing third party terminal, and further, if the terminal is used as the server side, the web side can carry out configuration of terminal IP, port and platform type and then carry out active connection through the relay.

According to the port, the server receives the signaling protocol document docking provided by the third party, and translates the data issued by the central scheduling service into the third party signaling protocol according to the relay of the server.

In a specific embodiment, if the server uses a udp+protobuf data structure and the third party system terminal uses a udp+binary private protocol, a network protocol may be used as UDP, and a single socket is defined to connect all the terminals, and if the network protocol between the server and the terminals is tcp, a thread cycle is opened to manage socket connections of all the terminals.

According to the relay service, the server can receive the audio data respectively sent by different terminals.

Step 402: the server side respectively carries out protocol header adaptation on the first audio data sent by each terminal according to a preset media scheduling service to generate a plurality of second audio data, and generates a unique audio identifier corresponding to each second audio data through a preset media algorithm service.

In this embodiment, this step is mainly: the server receives the audio protocol header rule provided by the terminal, replaces each protocol header of the first audio data with a unified protocol header of the server according to the audio protocol header rule, specifically, provides the audio protocol header rule through a third party, then uses a terminal audio connection port as a terminal platform distinguishing mark, sets the audio communication of the terminal through a connection 8800 port, and uses the audio communication of the terminal through the connection 6600 port, so that the third party device can be queried through binding two network sockets, and the audio stream protocol header reported by the third party can be converted into the unified protocol header of the third party and sent to other service modules for processing.

After the protocol header of each first audio data is converted, a preset identifier generating method is utilized to generate a unique identifier corresponding to each first audio data, wherein the identifier generating method is that a server side takes a unique identifier set by a terminal MAC address+port or an IP address+port or a third party as an identifier, 8 bytes are taken as the unique identifier of each first audio data in total, and a media algorithm service obtains the unique identifier of each terminal or audio data sent by each terminal according to an interface at the beginning of call task creation.

Step 403: the server performs preprocessing on each second audio data, and stores each third audio data obtained after the preprocessing into a corresponding annular memory.

In this embodiment, this step is: each received second audio data stream is decoded, such as MP3, SBC, PCM and the like, resampled into uniform sampling rate, sampling precision and sound channel, finally, a ring memory is respectively allocated for each third audio data stream, the third audio data stream is written into the ring memory, and the socket address, namely the socket address, is recorded, wherein the ring memory is a fixed large memory for initializing a piece of fixed memory, the frame number of corresponding data can be stored, the memory writing is normally carried out at the last frame end address, the data is written from the first position of the memory when the data is written to the end of the memory, so that the purpose of writing the ring data is achieved, the memory mode can greatly reduce the memory copy times, improve the software execution efficiency, and effectively control the total size of the memory so as to avoid software abnormality caused by memory leakage.

Step 404: the server side periodically acquires each third audio data from the annular memory, performs audio mixing processing on the third audio data, and sends the audio data after audio mixing to each terminal connected with the server side.

In this embodiment, this step is mainly: the server side periodically acquires each third audio data, adds and gathers all third audio data according to bit depth to acquire initial audio mixing data, specifically, after acquiring a meeting room creation request of a set meeting task room from a central scheduling service, the media algorithm processing service module traverses a ring memory corresponding to each terminal participating in a call in the meeting room, acquires a frame of audio data through the ring memory to synthesize, and when audio mixing is performed, the media algorithm processing service receives each frame of audio data reported by the terminal from the media scheduling service, writes all corresponding audio data into the corresponding ring memory, and then performs audio mixing according to 48000 sampling rate/100- >480; obtaining sampling points of every 10 ms; 480 samples 1 mono 2 bits deep; calculating to obtain 960Byte; and (3) regularly carrying out audio mixing once every 10ms by another thread, wherein the audio mixing process is to add and collect 10ms one frame of data of all members in a conference according to bit depth, obtain a total conference task audio mixing data frame according to the minimum audio frequency range of the limiting threshold, then subtract the total conference task audio mixing data frame according to bit depth according to the audio frequency frame before audio mixing of each conference member to obtain individual audio frequency frames of the members, and meanwhile, label and assign the audio frequency obtained after the individual audio frequency frames are removed by utilizing the audio frequency identification corresponding to the audio frequency frames of each member.

After the mixed audio data of the plurality of removed personal audio frames are obtained, the identification is utilized to find the terminal corresponding to each mixed audio data, then the media algorithm service only needs to send a unified protocol header to the media scheduling service, the media scheduling service analyzes the platform unique audio protocol header, and after the private protocol header corresponding to each terminal is replaced by the platform unique audio protocol header, the mixed audio data after the protocol header is replaced is sent to the corresponding terminal.

The audio data finally received by the terminal is audio data which is obtained by mixing and synthesizing the audio data by the media algorithm processing module and removing the audio frames of the terminal, namely the received data can be played without additional processing, so that the terminal which supports playing and collecting the PCM audio can be added into the multiparty call of the server side no matter the platform type or the version of the terminal.

The multi-party call processing method based on the multi-terminal protocol provided by the embodiment is compatible with the third party terminal signaling protocol and the audio data stream by the server side, audio broadcasting is realized to the third party terminal and the system terminal, all kinds of system terminals are compatible with audio or visual functions such as intercom, multi-party intercom and the like, meanwhile, the method is based on a micro-service type architecture, so that the multi-class system terminal signaling mode processing can be easily and clearly realized, and only a single relay system is matched with each class of system terminal to realize the compatible processing of the terminals, and the flexibility of multi-party call can be improved.

The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present invention, and are not to be construed as limiting the scope of the invention. It should be noted that any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art without departing from the spirit and principles of the present invention are intended to be included in the scope of the present invention.

Claims (10)

1. The multi-party call processing method based on the multi-terminal protocol is characterized by comprising the following steps of:

the method comprises the steps that a server receives a signaling sent by each of a plurality of first terminals through a preset relay, and is connected with each first terminal according to the signaling;

the server receives the first audio data sent by each first terminal, and performs protocol header unified adaptation and identifier generation on each first audio data according to a preset unified protocol header and identifier generation method to obtain a plurality of second audio data; the second audio data comprises an audio identification;

the server performs preprocessing on each second audio data respectively and stores third audio data obtained after preprocessing each second audio data;

The server side periodically acquires each third audio data, and gathers each third audio data through a preset audio mixing algorithm to acquire first audio mixing data;

the server performs audio deletion on the first audio data according to each third audio data to obtain a plurality of second audio data, and determines a second terminal corresponding to each second audio data according to an audio identifier corresponding to each third audio data;

the server acquires the protocol header corresponding to the second terminal, packages each second audio data according to the protocol header, and sends the packaged second audio data to the corresponding second terminal.

2. The multi-party call processing method based on multi-terminal protocol as claimed in claim 1, wherein the server receives signaling sent by each of a plurality of terminals through a preset relay, comprising:

the server side binds different ports through the relay and receives a connection instruction sent by each terminal in the plurality of terminals respectively;

according to the connection instruction, the server side respectively acquires the signaling protocol document provided by each terminal through the port, and is compatible with the signaling protocol of each terminal according to the signaling protocol document, so as to be connected with each terminal in the plurality of terminals.

3. The multi-party call processing method according to claim 1, wherein the performing protocol header unified adaptation and identifier generation on each first audio data according to a preset unified protocol header and identifier generation method includes:

the server receives the audio protocol header rules respectively sent by each terminal, and converts the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules;

the server side combines the terminal address, the terminal port and the terminal identifier corresponding to each piece of first audio data to generate an audio identifier corresponding to each piece of first audio data.

4. The multi-party call processing method according to claim 1, wherein the server performs preprocessing on each second audio data, and stores third audio data obtained by preprocessing each second audio data, respectively, and the method comprises:

the server decodes and resamples each second audio data to obtain a plurality of third audio data;

the server writes each third audio data in the plurality of third audio data into a corresponding annular memory respectively, and stores the third audio data.

5. The multi-party call processing method according to claim 1, wherein the server periodically obtains each third audio data and gathers each third audio data by a preset audio mixing algorithm, and the method comprises:

the server side periodically acquires each third audio data, and adds and gathers all the third audio data according to bit depth to acquire initial audio mixing data;

and carrying out maximum audio limitation and minimum audio limitation on the initial mixed audio data according to a preset limiting threshold value to obtain the first mixed audio data.

6. The multi-party call processing method according to claim 1, wherein the server performs audio deletion on the first audio data according to each third audio data to obtain a plurality of second audio data, and determines a second terminal corresponding to each second audio data according to an audio identifier corresponding to each third audio data, including:

the server side respectively performs subtraction processing on the first audio data according to the bit depth of each third audio data, and determines a sending terminal of second audio data obtained after the current subtraction processing according to the audio identification of the current third audio data; wherein the sending terminal is the second terminal.

7. The multi-party call processing method according to claim 1, wherein the server obtains a protocol header corresponding to the second terminal, and packages each second audio data according to the protocol header, and the method comprises:

and deleting the unified protocol header of the second audio mixing data by the server, and adding the acquired protocol header corresponding to the second terminal to the second audio mixing data so as to package the second audio mixing data.

8. The multi-party call processing system based on the multi-terminal protocol is characterized by comprising a terminal compatible module, an audio identification module, an audio processing module, an audio mixing module, a mixing classification module and an audio packaging module;

the terminal compatible module is used for receiving a signaling sent by each terminal in a plurality of first terminals through a preset relay by the service terminal, and connecting the signaling with each first terminal according to the signaling;

the audio identification module is used for receiving the first audio data sent by each first terminal by the server, and carrying out protocol header unified adaptation and identification generation on each first audio data according to a preset unified protocol header and identification generation method to obtain a plurality of second audio data; the second audio data comprises an audio identification;

The audio processing module is used for respectively preprocessing each second audio data by the server and storing third audio data obtained after preprocessing each second audio data;

the audio mixing module is used for periodically acquiring each third audio data by the server and summarizing each third audio data through a preset mixing algorithm to acquire first mixed audio data;

the audio mixing classification module is used for the server to perform audio deletion on the first audio mixing data according to each third audio data to obtain a plurality of second audio mixing data, and determine a second terminal corresponding to each second audio mixing data to send according to the audio identifier corresponding to each third audio data;

the audio packaging module is used for acquiring a protocol header corresponding to the second terminal by the server, packaging each piece of second audio mixing data according to the protocol header, and sending the packaged second audio mixing data to the corresponding second terminal.

9. The multi-party call processing system according to claim 8, wherein the terminal compatible module comprises a port connection unit and a compatible unit;

The port connection unit is used for the server to bind different ports through the relay and respectively receive the connection instruction sent by each terminal in the plurality of terminals;

the compatible unit is used for respectively acquiring the signaling protocol document provided by each terminal through the port according to the connection instruction, and is compatible with the signaling protocol of each terminal according to the signaling protocol document to connect with each terminal in the plurality of terminals.

10. The multi-party call processing system according to claim 8, wherein the audio identification module comprises an adaptation unit and an identification unit;

the adaptation unit is used for receiving the audio protocol header rules respectively sent by each terminal by the server, and converting the protocol header of each first audio data into a unified protocol header preset in the server according to the audio protocol header rules;

the identification unit is used for the server to combine the terminal address, the terminal port and the terminal identification corresponding to each first audio data to generate the audio identification corresponding to each first audio data.