CN114499747B

CN114499747B - Audio and video data processing method and device, electronic equipment and storage medium

Info

Publication number: CN114499747B
Application number: CN202011236732.2A
Authority: CN
Inventors: 丁长文
Original assignee: Chengdu TD Tech Ltd
Current assignee: Chengdu TD Tech Ltd
Priority date: 2020-11-09
Filing date: 2020-11-09
Publication date: 2023-06-20
Anticipated expiration: 2040-11-09
Also published as: CN114499747A

Abstract

The embodiment of the application provides a processing method, a processing device, electronic equipment and a storage medium for audio and video data, wherein the processing method comprises the following steps: transmitting an audio and video establishment request, wherein the audio and video establishment request comprises Forward Error Correction (FEC) coding parameters of a transmitting end; acquiring a confirmation message of an audio and video establishment request; when the confirmation message comprises the FEC coding negotiation parameters, carrying out anti-packet processing on the audio and video data of the sending end based on the FEC coding negotiation parameters, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end. The transmitting end and the receiving end negotiate the FEC coding parameters, the FEC coding negotiation parameters of the audio and video data between the transmitting end and the receiving end are determined, and the data is subjected to anti-packet processing based on the FEC coding negotiation parameters, so that the packet loss phenomenon of the audio and video data is reduced, the accuracy of audio and video data transmission is improved, and the user experience is improved.

Description

Audio and video data processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of mobile communications technologies, and in particular, to a method and apparatus for processing audio and video data, an electronic device, and a storage medium.

Background

B-trunk (Broadband Trunking Communication ) is a private network broadband trunking system standard established by the broadband trunking industry alliance organization. The B-trunk protocol defines the end-to-end flow of the video service in the broadband cluster network, and the media parameter format initiated by the service can be determined through the negotiation flow of the media parameters at the beginning of the service.

In real-time communication, audio and video data are generally transmitted by using UDP (User Datagram Protocol, user data protocol), and since UDP is a connectionless transmission protocol, the quality of data transmission cannot be guaranteed, when the network transmission environment is poor, packet loss will inevitably occur, so that phenomena such as audio discontinuity, video screen display or pause are caused, and the communication experience of a user is affected.

The existing anti-packet-loss strategy is usually carried out based on a packet-loss retransmission mechanism, and the packet-loss retransmission mechanism requires a receiving end to send a message to a sending end after detecting data packet loss, so that the sending end determines a corresponding data packet according to the message and sends the corresponding data packet to the receiving end to finish correction of packet loss.

However, the packet loss retransmission can be completed only by relying on feedback from the receiving end, and is not suitable for communication services without feedback from the receiving end, and the application range is limited.

Disclosure of Invention

The embodiment of the application provides a processing method, a device, electronic equipment and a storage medium for audio and video data, wherein the processing method, the device, the electronic equipment and the storage medium are used for carrying out anti-packet loss processing on the audio and video data based on FEC coding, and can automatically carry out packet loss detection and recovery without feedback of a receiving end, so that the accuracy of data transmission is improved.

In a first aspect, an embodiment of the present application provides a method for processing audio and video data, where the method includes:

transmitting an audio and video establishment request, wherein the audio and video establishment request comprises Forward Error Correction (FEC) coding parameters of a transmitting end;

acquiring a confirmation message of the audio and video establishment request;

and when the confirmation message comprises an FEC coding negotiation parameter, carrying out anti-lost packet processing on the audio and video data of the sending end based on the FEC coding negotiation parameter, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end.

Optionally, the FEC encoding parameters include: FEC switching parameters, FEC packet parameters, FEC redundancy and FEC anti-packet loss mode.

Optionally, when the FEC switch parameter of the sending end and the FEC switch parameter of the receiving end are both on FEC codes, the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the FEC code negotiation parameters are determined respectively according to the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the sending end and the receiving end.

Optionally, the FEC encoding parameters of the transmitting end include: the method comprises the steps of sending end FEC switch parameters, FEC packet alternative parameters, FEC redundancy alternative parameters and FEC anti-lost mode alternative parameters.

Optionally, the FEC encoding negotiation parameter is an FEC packet parameter, an FEC redundancy and an FEC anti-packet loss mode of the receiving end, which are determined according to the FEC encoding capability of the receiving end, the FEC packet alternative parameter, the FEC redundancy alternative parameter and the FEC anti-packet loss mode alternative parameter of the sending end.

Optionally, before sending the audio-video establishment request, the method further includes:

adding FEC coding capability into terminal cluster capability of broadband cluster communication protocol;

the sending end sends a registration request message to a cluster core network, wherein the registration request message comprises FEC coding capability of the sending end;

and acquiring a registration passing message of the cluster core network, wherein the registration passing message is a message determined by the cluster core network according to the FEC coding capacity in the registration request message and the subscription information of the sending end.

Optionally, the performing anti-packet processing on the audio and video data of the sending end based on the FEC encoding negotiation parameter includes:

Determining FEC redundant data of the audio and video data of the transmitting end according to the FEC coding negotiation parameters;

and adding FEC redundant data into the audio and video data of the transmitting end so as to perform anti-packet loss processing on the audio and video data of the transmitting end according to the FEC redundant data.

In a second aspect, the present application further provides a method for processing audio and video data, where the method includes:

acquiring an audio and video establishment request of a transmitting end and FEC coding parameters of a receiving end, wherein the audio and video establishment request comprises the FEC coding parameters of the transmitting end;

determining a confirmation message of the audio and video establishment request according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end, wherein the confirmation message is a response message of the audio and video establishment request;

and when the confirmation message comprises an FEC coding negotiation parameter, an FEC starting message is sent to the sending end and the receiving end, so that the sending end and the receiving end perform anti-packet processing on the audio and video data based on the FEC coding negotiation parameter and the FEC starting message, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end.

In a third aspect, the present application further provides a method for processing audio and video data, where the method includes:

and when the confirmation message comprises an FEC coding negotiation parameter, carrying out anti-lost packet processing on the audio and video data of the receiving end based on the FEC coding negotiation parameter, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end.

In a fourth aspect, the present application further provides an apparatus for processing audio and video data, where the apparatus includes:

the system comprises an establishment request sending module, a receiving module and a sending module, wherein the establishment request sending module is used for sending an audio and video establishment request, and the audio and video establishment request comprises Forward Error Correction (FEC) coding parameters of a sending end;

the confirmation message acquisition module is used for acquiring the confirmation message of the audio and video establishment request;

And the first data processing module is used for carrying out anti-packet loss processing on the audio and video data of the sending end based on the FEC coding negotiation parameters when the confirmation message comprises the FEC coding negotiation parameters, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end.

In a fifth aspect, the present application further provides an apparatus for processing audio and video data, where the apparatus includes:

the first request acquisition module is used for acquiring an audio and video establishment request of the transmitting end and FEC coding parameters of the receiving end, wherein the audio and video establishment request comprises the FEC coding parameters of the transmitting end;

the first confirmation message determining module is used for determining a confirmation message of the audio and video establishment request according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end, wherein the confirmation message is a response message of the audio and video establishment request;

and the FEC starting module is used for sending an FEC starting message to the sending end and the receiving end when the confirmation message comprises the FEC coding negotiation parameter so that the sending end and the receiving end can carry out anti-packet processing on the audio and video data based on the FEC coding negotiation parameter and the FEC starting message, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end.

In a sixth aspect, the present application further provides an apparatus for processing audio and video data, where the apparatus includes:

the second request acquisition module is used for acquiring an audio and video establishment request of the transmitting end and FEC coding parameters of the receiving end, wherein the audio and video establishment request comprises the FEC coding parameters of the transmitting end;

the second confirmation message determining module is used for determining a confirmation message of the audio and video establishment request according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end, wherein the confirmation message is a response message of the audio and video establishment request;

and the second data processing module is used for carrying out anti-packet loss processing on the audio and video data of the receiving end based on the FEC coding negotiation parameters when the confirmation message comprises the FEC coding negotiation parameters, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the transmitting end and the FEC coding parameters of the receiving end.

In a seventh aspect, the present application further provides an electronic device, including: a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes the computer-executed instructions stored in the memory, so that the at least one processor executes the processing method of the audio/video data provided in any embodiment of the present application.

In an eighth aspect, the present application further provides a computer readable storage medium, where computer executable instructions are stored, where the computer executable instructions are used to implement a method for processing audio and video data as provided in any embodiment of the present application when executed by a processor.

The embodiment of the application provides a processing method, a device, electronic equipment and a storage medium for audio and video data, wherein FEC coding parameters of a sending end are added in an audio and video establishment request, and acknowledgement information about the audio and video establishment request is obtained, wherein the acknowledgement information is response information determined by a negotiation process of the receiving end and the sending end about the FEC coding parameters, if the acknowledgement information comprises the FEC coding negotiation parameters, packet loss prevention processing is carried out on the audio and video data of the sending end and the receiving end based on the FEC coding negotiation parameters, the packet loss prevention processing of the audio and video data is realized, the FEC coding can automatically carry out packet loss detection and recovery without returning the packet loss data, the application range is wide, and the accuracy of data transmission is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Fig. 1 is an application scenario diagram of a processing method of audio and video data provided in an embodiment of the present application;

fig. 2 is a flowchart of a processing method of audio and video data according to an embodiment of the present application;

fig. 3 is a flowchart of a processing method of audio and video data according to another embodiment of the present application;

FIG. 4 is a flow chart of terminal cluster registration in the embodiment of FIG. 3 of the present application;

fig. 5 is a flowchart of a negotiation process of FEC encoding parameters in the embodiment shown in fig. 3 of the present application;

fig. 6 is a flowchart of a processing method of audio and video data according to another embodiment of the present application;

fig. 7 is a flowchart of a processing method of audio and video data according to another embodiment of the present application;

fig. 8 is a schematic structural diagram of an audio/video data processing device according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an audio/video data processing device according to another embodiment of the present application;

fig. 10 is a schematic structural diagram of an audio/video data processing device according to another embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

The following describes the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

The application scenario of the embodiment of the present application is explained below:

fig. 1 is an application scenario diagram of a processing method of audio and video data provided in this embodiment of the present application, as shown in fig. 1, a calling UE (User Equipment) 110 initiates an audio and video request to a called UE120, and sends the audio and video request to the called UE120 through a trunking core network 130, the called UE120 acknowledges the audio and video request, and generates a request acknowledgement message, and sends the request acknowledgement message to the calling UE110 through the trunking core network 130, so that the calling UE110 and the called UE120 establish an audio and video service, and the transmission of the audio and video data is performed through the trunking core network 130.

However, in the existing audio and video service based on the B-trunk protocol, in order to improve the real-time performance of the service, the UDP protocol is adopted to transmit RTP data, but the quality of the transmitted data cannot be guaranteed by the UDP protocol, so that when a network is congested, a packet loss problem often occurs, and the phenomena of audio discontinuity, video clip or screen display are caused, so that the user experience is poor.

In order to solve the above problems, an embodiment of the present application proposes an anti-packet-loss policy for audio and video data, and the main concept of the technical solution of the embodiment of the present application is as follows: the FEC coding parameters of the sending end are added in the audio and video establishment request, and the FEC coding negotiation parameters corresponding to the communication of the sending end and the receiving end are determined through the negotiation process of the FEC coding parameters of the sending end and the receiving end, so that the packet loss prevention treatment is carried out on the audio and video data of the sending end and the receiving end according to the negotiation parameters, the packet loss prevention strategy of the audio and video data is realized, the accuracy of data transmission is improved, and the user experience is improved.

Fig. 2 is a flowchart of a processing method of audio and video data according to an embodiment of the present application. The processing method of the audio and video data can be executed by the terminal equipment, and concretely can be executed by the transmitting end. As shown in fig. 2, the processing method of audio/video data provided in this embodiment includes the following steps:

Step S201, an audio/video establishment request is sent, where the audio/video establishment request includes a forward error correction FEC encoding parameter of the sending end.

The audio and video establishment REQUEST is a REQUEST initiated by the transmitting end to the receiving end to establish audio and video, and may also be called as a CALL REQUEST. FEC (Forward Error Correction) coding parameters include various parameters of FEC coding, such as whether FEC coding is enabled, redundancy of FEC coding, FEC coding algorithm employed, etc.

Specifically, FEC coding is an error control coding, which is processed in a certain format in advance before a signal or a data packet is transmitted, and decoded according to a specified algorithm at a receiving end, so as to achieve the purposes of detecting an error code and correcting errors. Redundant error correction codes are added into data codes through the FEC coding technology, so that the disordered data codes can be automatically corrected through decoding, the error rate of data transmission is reduced, and the accuracy of data is improved.

Wherein the FEC switching parameter includes two states, on and off, when it is on, it indicates that FEC encoding is employed or on, and when it is off, it indicates that FEC encoding is not employed or on. The FEC packet parameters are used to characterize how many packets or segments the data encoding is divided into. And FEC redundancy indicates the length of FEC encoding. The FEC anti-lost mode corresponds to the lost packet mode of the transmitted data, if the lost packet mode of the transmitted data is random lost packet, the FEC anti-lost mode is the FEC anti-random lost packet mode, and if the lost packet mode of the transmitted data is assault lost packet, the FEC anti-lost mode is the FEC anti-assault lost packet mode. Specifically, the packet loss mode may be determined according to the received data.

Illustratively, assuming that the data encoding is divided into m packets, each packet including k bits of data, and n-k bits of redundancy encoding, the FEC switching parameter corresponding to the data encoding is on, the FEC packet parameter is m, and the FEC redundancy is n-k.

Further, the FEC encoding parameters may also include an FEC encoding algorithm. The FEC coding algorithm may include any one or more of Reed Solomon (RS, reed-Solomon) codes, hamming codes, low-density parity check codes (Low Density Parity Check Code, LDPC) and Turbo product codes (Turbo Product Code, TPC), among others.

Specifically, the transmitting end transmits an audio and video establishment request to the receiving end, and the audio and video establishment request can be received and processed by the cluster core network and then transmitted to the receiving end, or can be directly transmitted to the receiving end through the cluster core network. The sending end may also be called a calling UE or a calling end, and the receiving end is called a called UE or a called end, respectively.

Further, the audio/video establishment request carries a Video Description (video description) message, and the Video Description message includes FEC encoding parameters of the transmitting end. The audio/video establishment request can also include or carry call type, call attribute, called number, media information, etc.

Illustratively, table 1 is a FEC field table of a Video Description message in the embodiment shown in fig. 1 of the present application, as shown in table 1: in this embodiment, a field of FEC encoding parameters is added in the Video Description message or Video Description cell, and the FEC field includes 3 parameters, which are a parameter type, optional or necessary, and a parameter length, respectively. Specifically, the FEC field or Video FEC Parameters is Video FEC Parameters, and the optional/mandatory parameter may be optional or mandatory, and its length is 1 Byte (Byte).

Table 1 FEC field table of Video Description messages

Cell unit	Type(s)	Optional/necessary optional	Length of
				Video FEC Parameters	Video FEC Parameters	Optionally, a	1

Exemplary, table 2 is a parameter table of Video FEC Parameters in the embodiment shown in fig. 1 of the present application, wherein Video FEC Parameters is the FEC encoding parameter described above, as shown in table 2, table 2 Video FEC Parameters

Wherein, FEC coding parameter 1, FEC coding parameter 2 and FEC coding parameter 3

Step S202, obtaining the confirmation message of the audio/video establishment request.

The confirmation message is a response message of the audio/video establishment request message generated according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end. The number of the receiving ends can be one or a plurality of. When the number of the receiving terminals is one, the corresponding scene is a single call scene, and when the number of the receiving terminals is a plurality of the receiving terminals, the corresponding scene is a group call scene. The acknowledgement message may also be referred to as CALL complete.

Specifically, when the FEC coding parameters of the transmitting end and the FEC switching parameters in the FEC coding parameters of the receiving end are both on FEC coding, the negotiation of the FEC coding negotiation parameters of the acknowledgement message may be further performed according to other FEC coding parameters of the transmitting end and the receiving end. When at least one FEC switch parameter of the receiving end is not to start FEC encoding, the acknowledgement message will not include FEC encoding negotiation parameters, i.e. the audio and video data service between the transmitting end and the receiving end cannot use FEC encoding to perform anti-packet loss processing.

Optionally, when the FEC switch parameter of the transmitting end and the FEC switch parameter of the receiving end are both on FEC codes, the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the FEC code negotiation parameter in the acknowledgement message are determined according to the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the transmitting end and the receiving end, respectively.

Specifically, the FEC encoding parameters of the receiving end are the same as the specific form of the FEC encoding parameters of the transmitting end, and the specific parameters may be the same or different. When specific parameters of the FEC coding parameters, such as the FEC packet parameter, FEC redundancy, and FEC anti-packet loss mode, are the same, the acknowledgement message of the audio/video establishment request can be determined according to the FEC coding parameters of the receiving end or the transmitting end. When the specific parameters of the two parameters are different, the FEC code negotiation parameters of the acknowledgement message may be determined according to a predetermined protocol, for example, the FEC code parameters of the transmitting end may be determined as the FEC code negotiation parameters.

Specifically, when the number of the receiving ends is one, the step of determining the FEC encoding negotiation parameters may be performed by the trunking core network, or may be performed by the receiving ends. And when the number of the receiving ends is at least two, such as 3, 5 or other numbers, the method needs to be executed by the cluster core network.

Further, when executed by the trunking core network, the negotiation flow of the FEC encoding negotiation parameters is:

the cluster core network determines whether the service can adopt FEC coding according to FEC parameters carried in an audio and video establishment request of the transmitting end and FEC coding capacity reported by the receiving end when the terminal cluster is registered, if so, the FEC coding negotiation parameters of the transmitting end and the receiving end are further determined according to the FEC coding parameters of the transmitting end and related subscription protocols, and the FEC coding negotiation parameters are added into a confirmation message and transmitted to the receiving end and the transmitting end. And when the FEC encoding capability of the receiving end is not provided with the FEC encoding capability, it means that the data communication between the transmitting end and the receiving end cannot use FEC encoding, and the acknowledgement message does not include FEC encoding negotiation parameters. Or when the transmitting end and the receiving end can not agree on specific parameters of the FEC codes, the confirmation message also does not comprise the FEC code negotiation parameters. Wherein, the FEC coding capability of the receiving end is added into the terminal cluster capability of the receiving end when the receiving end registers.

Specifically, the FEC encoding capability of the receiving end may be carried in UE Trunking Capability (terminal cluster capability) of the receiving end, and reported when the receiving end cluster terminal registers. The FEC encoding capability is used to characterize whether the receiving end has the capability of adopting FEC encoding.

Accordingly, the specific procedure performed by the receiving end is similar to that described above, and only the step of transmitting the acknowledgement message to the receiving end is omitted.

Step S203, when the acknowledgement message includes an FEC encoding negotiation parameter, performing anti-packet loss processing on the audio/video data of the transmitting end based on the FEC encoding negotiation parameter.

The FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the transmitting end and the FEC coding parameters of the receiving end.

Further, when the acknowledgement message does not include the FEC encoding negotiation parameter, the anti-packet processing cannot be performed on the audio and video data of the transmitting end based on the FEC encoding negotiation parameter, that is, the anti-packet processing is not performed on the audio and video data, but the transmission is performed according to the related protocol.

Specifically, the packet loss prevention processing is to perform FEC encoding on the audio and video data at the transmitting end, and decode the encoded audio and video data at the receiving end, so as to detect and correct packet loss.

In this embodiment, by adding the FEC encoding parameter of the transmitting end to the audio/video establishment request, and obtaining the acknowledgement message about the audio/video establishment request, where the acknowledgement message is a response message determined by the negotiation process of the receiving end and the transmitting end about the FEC encoding parameter, if the acknowledgement message includes the FEC encoding negotiation parameter, the anti-packet processing is performed on the audio/video data of the transmitting end and the receiving end based on the FEC encoding negotiation parameter, so that the anti-packet processing of the audio/video data is implemented, and the FEC encoding can automatically perform packet loss detection and recovery without returning the packet loss data, which has a wide application range and improves the accuracy of data transmission.

Fig. 3 is a flowchart of a processing method of audio and video data according to another embodiment of the present application, and as shown in fig. 3, the processing method of audio and video data according to this embodiment refines step S203 on the basis of the processing method of audio and video data according to the embodiment shown in fig. 2, and adds a step of cluster registration of a terminal before step S201, where the processing method of audio and video data according to this embodiment may include the following steps:

step S301, adding FEC encoding capability into terminal trunking capability of broadband trunking communication protocol.

Specifically, the broadband trunking communication protocol is the B-trunk protocol. In the B-trunk protocol, the terminal performs UE Trunking Capability reporting when registering, and adds reporting whether the terminal supports FEC coding capability.

Illustratively, table 3 is a table of parameters of UE Trunking Capability in the embodiment of fig. 3 of the present application, as shown in table 3:

table 3 UE Trunking Capability parameter table

As can be seen, UE Trunking Capability includes: the end-to-end encryption parameter is used for encrypting the audio and video data so as to prevent the data from being eavesdropped; PDCP (Packet Data Convergence Protocol ) header compression for defining header compression modes of packets of a trunking channel; the AMR Over PDCP is used for describing whether the AMR Over PDCP technology is adopted for encoding and decoding; half duplex call parameters for; NAS (Network Attached Storage ) video service parameters for; and the FEC coding capability parameter is used for describing whether the terminal equipment has FEC coding capability.

In step S302, the transmitting end sends a registration request message to the trunking core network, where the registration request message includes FEC encoding capability of the transmitting end.

Specifically, the transmitting end sends a registration request message to the trunking core network based on the trunking base station of the trunking core network, where the registration request message may also be called TRUNKING REGISTER REQUEST, and the registration request message carries FEC coding capability of the transmitting end, and its specific form may refer to table 3.

Step S303, obtaining the registration passing message of the cluster core network.

The registration passing message is a message determined by the trunking core network according to the FEC coding capability in the registration request message and the subscription information of the sending end.

Specifically, the trunking core network receives the registration request message from the sender, searches trunking subscription information of the sender, determines whether the registration request is passed according to the subscription signal and the FEC coding capability in the registration request message, and if so, generates a registration passing message and sends the registration passing message to the sender to complete registration.

Specifically, steps S301 to S303 constitute a cluster registration procedure of the transmitting end, where the procedure may occur after the transmitting end starts up Attach to the end of the network, or after the FEC encoding capability of the transmitting end changes.

When the cluster registration of the transmitting end is completed, the subsequent steps can be performed.

Correspondingly, the receiving end can also comprise a cluster registration process corresponding to the sending end, and the sending end is only required to be replaced by the receiving end.

Fig. 4 is a flowchart of terminal cluster registration in the embodiment shown in fig. 3 of the present application, where, as shown in fig. 4, when a terminal device UE sends a TRUNKING REGISTER REQUEST cluster registration message to a cluster core network, a terminal cluster registration process is initiated, and the cluster registration message carries FEC coding capability of the UE; after the TRUNKING core network receives TRUNKING REGISTER REQUEST, searching TRUNKING subscription information of the UE, and if the TRUNKING core network side processes normally, recovering trunk REGISTER ACCEPT to the UE; after receiving the trunk REGISTER ACCEPT, the UE replies TRUNKING REGISTER COMPLETE to the TRUNKING core network, completing registration. The terminal device UE may be the transmitting end or the receiving end.

Step S304, an audio and video establishment request is sent, wherein the audio and video establishment request comprises Forward Error Correction (FEC) coding parameters of a sending end.

Step S305, obtaining a confirmation message of the audio/video establishment request.

Wherein, the FEC encoding parameters of the transmitting end include: the method comprises the steps of sending end FEC switch parameters, FEC packet alternative parameters, FEC redundancy alternative parameters and FEC anti-lost mode alternative parameters.

Specifically, the FEC switching parameter of the transmitting end, that is, the FEC switching parameter of the transmitting end, is used to describe whether the transmitting end adopts or starts FEC coding; the FEC packet alternative parameters comprise a plurality of optional FEC packet parameters or a set of optional FEC packet parameters; the FEC redundancy alternative parameters comprise a plurality of optional FEC redundancy parameters or are a set of optional FEC redundancy parameters; the FEC anti-packet loss mode alternative parameters include a plurality of optional FEC anti-packet loss mode parameters or a set of optional FEC anti-packet loss mode parameters.

Specifically, the confirmation message may be generated by the trunking core network or the receiving end. The FEC coding negotiation parameters required for communication between the receiving end and the transmitting end can be determined from the FEC packet alternative parameters, the FEC redundancy alternative parameters and the FEC anti-packet loss mode alternative parameters of the transmitting end according to the FEC coding capability of the receiving end, and are added into the acknowledgement message. Namely, the sender provides the option of selectable FEC coding parameters, and the receiver selects proper specific parameters from the options as FEC coding parameters used in communication between the sender and the receiver, namely FEC coding negotiation parameters.

Wherein, the FEC encoding capability of the receiving end is added in UE Trunking Capability when the receiving end cluster is registered.

Specifically, fig. 5 is a flowchart of a negotiation process of FEC encoding parameters in the embodiment shown in fig. 3 of the present application, and as shown in fig. 5, taking a video CALL service as an example, a calling UE initiates a video individual establishment REQUEST CALL to a trunking core network, where the REQUEST carries Video Description, and includes the FEC encoding parameters of the calling UE. When the negotiation process of the FEC coding parameters occurs at the cluster core network side, the cluster core network determines whether the service adopts FEC coding and determines corresponding FEC coding negotiation parameters according to the FEC coding parameters carried when the calling UE initiates the request and the FEC coding capability reported when the called UE terminal performs cluster registration, and then transmits a confirmation message CALL CONFIRMED after negotiation to the called terminal. When the negotiation process of the FEC coding parameters occurs at the called UE side, the trunking core network needs to send the FEC coding parameters of the calling UE to the called UE, the called UE selects whether to adopt FEC coding according to its FEC coding capability and determines the corresponding FEC coding negotiation parameters, and then transmits the acknowledgement message CALL configured after negotiation to the trunking core network, and the trunking core network sends the acknowledgement message to the calling UE.

When the video group call service is the video group call service, the negotiation process can only occur at the cluster core network side, and the FEC coding capacity or FEC coding parameters of each called UE can be acquired by the monitoring UE and transmitted to the cluster core network, and the process is similar to that of the single call service and will not be repeated here.

Further, after the sender receives the confirmation message, a special bearer for the sender and a special bearer for the receiver can be established, so that audio and video services of the sender and the receiver can be performed.

Step S306, when the acknowledgement message includes the FEC encoding negotiation parameter, determining FEC redundancy data of the audio/video data of the transmitting end according to the FEC encoding negotiation parameter.

The FEC coding negotiation parameters are FEC packet parameters, FEC redundancy and FEC anti-packet loss mode of the receiving end, which are determined according to the FEC coding capability of the receiving end, the FEC packet alternative parameters, FEC redundancy alternative parameters and FEC anti-packet loss mode alternative parameters of the sending end.

Specifically, the number of packets of the audio and video data and the length of the FEC redundancy data can be determined according to the FEC packet parameters and the FEC redundancy in the FEC encoding negotiation parameters, and the FEC encoding mode is determined according to the FEC anti-packet loss mode, so that a corresponding FEC encoding algorithm is determined according to the FEC encoding mode, and the FEC redundancy encoding is added after the data encoding of the audio and video data.

Step S307 adds FEC redundancy data to the audio and video data of the transmitting end, so as to perform anti-packet loss processing on the audio and video data of the transmitting end according to the FEC redundancy data.

Specifically, before transmission, FEC redundancy data is added to the audio and video data at the transmitting end, that is, FEC encoding is added to the audio and video data based on the FEC encoding parameters, so that FEC encoded audio and video data is obtained, and at the receiving end, the FEC encoded audio and video data can be decoded according to a corresponding decoding algorithm, so that error correction of error codes is achieved.

In this embodiment, during terminal cluster registration, FEC related parameters are added in the B-trunk c protocol, and during terminal registration, the FEC parameters are added to the terminal cluster capability, so that when a transmitting end initiates an audio and video establishment request to a receiving end, a cluster core network can perform FEC negotiation flow according to the FEC parameters of the transmitting segment and the receiving end, so as to determine FEC coding negotiation parameters corresponding to the two communication, so that packet loss prevention processing is performed on audio and video data between the two based on the FEC coding negotiation parameters, thereby improving accuracy of data transmission and improving user experience.

Fig. 6 is a flowchart of a processing method of audio and video data according to another embodiment of the present application, where the processing method of audio and video data provided by the present embodiment is applicable to a related device of a cluster core network, and as shown in fig. 6, the processing method of audio and video data includes the following steps:

step S601, acquiring an audio/video establishment request of the transmitting end and FEC encoding parameters of the receiving end.

The audio and video establishment request comprises Forward Error Correction (FEC) coding parameters of a transmitting end.

Optionally, before acquiring the audio and video establishment request of the transmitting end and the FEC encoding parameters of the receiving end, the method further includes:

receiving a registration request message of a transmitting end or a receiving end, wherein the registration request message comprises FEC coding capability of the transmitting end, and the FEC coding capability is added in terminal trunking capability of a broadband trunking communication protocol; generating a registration passing message according to the FEC coding capability in the registration request message and the subscription information of the sending end or the receiving end, and sending the registration passing message to the sending end or the receiving end.

Step S602, determining the acknowledgement message of the audio/video establishment request according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end.

The confirmation message is a response message of the audio/video establishment request.

Optionally, determining the acknowledgement message of the audio/video establishment request according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end includes:

generating a confirmation message of the audio and video establishment request;

and determining an FEC coding negotiation parameter according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end, and adding the FEC coding negotiation parameter into the confirmation message of the audio/video establishment request.

Optionally, determining the FEC encoding negotiation parameter according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end includes:

when the FEC switch parameter of the transmitting end and the FEC switch parameter of the receiving end are both on FEC codes, respectively determining the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the FEC code negotiation parameter in the acknowledgement message according to the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the transmitting end and the receiving end.

Correspondingly, determining the FEC coding negotiation parameters according to the FEC coding parameters of the transmitting end and the FEC coding parameters of the receiving end includes: and the FEC packet parameters, the FEC redundancy and the FEC anti-lost mode of the receiving end are determined according to the FEC coding capability of the receiving end, the FEC packet alternative parameters, the FEC redundancy alternative parameters and the FEC anti-lost mode alternative parameters of the sending end, and are used as FEC coding negotiation parameters.

Step S603, when the acknowledgement message includes an FEC encoding negotiation parameter, sending an FEC start message to the sending end and the receiving end, so that the sending end and the receiving end perform anti-packet processing on the audio/video data based on the FEC encoding negotiation parameter and the FEC start message.

Optionally, performing anti-packet processing on the audio and video data based on the FEC encoding negotiation parameters and the FEC start message includes:

determining FEC redundant data of the audio and video data of the transmitting end according to the FEC coding negotiation parameters; and adding FEC redundant data into the audio and video data of the transmitting end so as to perform anti-packet loss processing on the audio and video data of the transmitting end according to the FEC redundant data.

Fig. 7 is a flowchart of a processing method of audio and video data according to another embodiment of the present application, where the processing method of audio and video data provided by the present embodiment is applicable to a called UE or a receiving end, and as shown in fig. 7, the processing method of audio and video data includes the following steps:

step S701, acquiring an audio and video establishment request of the transmitting end and FEC encoding parameters of the receiving end.

Adding FEC coding capability into terminal cluster capability of broadband cluster communication protocol; the method comprises the steps that a receiving end sends a registration request message to a cluster core network, wherein the registration request message comprises FEC coding capability of the receiving end; and acquiring a registration passing message of the cluster core network, wherein the registration passing message is a message determined by the cluster core network according to the FEC coding capacity in the registration request message and the subscription information of the receiving end.

Step S702 determines the acknowledgement message of the audio/video establishment request according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end.

The confirmation message is a response message of the audio/video establishment request. The FEC encoding parameters of the transmitting end may be sent to the receiving end by the trunking core network.

generating a confirmation message of the audio and video establishment request; and determining an FEC coding negotiation parameter according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end, and adding the FEC coding negotiation parameter into the confirmation message of the audio/video establishment request.

In step S703, when the acknowledgement message includes an FEC encoding negotiation parameter, anti-packet loss processing is performed on the audio/video data of the receiving end based on the FEC encoding negotiation parameter.

Optionally, performing anti-packet processing on the audio and video data of the receiving end based on the FEC encoding negotiation parameter includes:

determining an FEC decoding algorithm of the audio and video data of the receiving end according to the FEC coding negotiation parameters; and carrying out packet loss detection and error correction on the audio and video data of the transmitting end according to the FEC decoding algorithm.

Fig. 8 is a schematic structural diagram of an audio/video data processing device according to an embodiment of the present application, and as shown in fig. 8, the audio/video data processing device according to the embodiment includes: a setup request sending module 810, an acknowledgement message obtaining module 820 and a first data processing module 830.

The establishment request sending module 810 is configured to send an audio/video establishment request, where the audio/video establishment request includes a forward error correction FEC encoding parameter of a sending end; a confirmation message obtaining module 820, configured to obtain a confirmation message of the audio/video establishment request; and the first data processing module 830 is configured to perform anti-packet processing on the audio and video data of the transmitting end based on the FEC coding negotiation parameter when the acknowledgement message includes the FEC coding negotiation parameter, where the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end.

Optionally, the processing device of audio and video data further includes:

the transmitting end registration module is used for adding FEC coding capacity into terminal cluster capacity of the broadband cluster communication protocol before transmitting the audio and video establishment request; the sending end sends a registration request message to a cluster core network, wherein the registration request message comprises FEC coding capability of the sending end; and acquiring a registration passing message of the cluster core network, wherein the registration passing message is a message determined by the cluster core network according to the FEC coding capacity in the registration request message and the subscription information of the sending end.

Optionally, the first data processing module 830 is specifically configured to:

when the confirmation message comprises the FEC coding negotiation parameters, determining FEC redundant data of the audio and video data of the transmitting end according to the FEC coding negotiation parameters; and adding FEC redundant data into the audio and video data of the transmitting end so as to perform anti-packet loss processing on the audio and video data of the transmitting end according to the FEC redundant data.

Fig. 9 is a schematic structural diagram of an audio/video data processing apparatus according to another embodiment of the present application, and as shown in fig. 9, the audio/video data processing apparatus according to the present embodiment includes: a first request acquisition module 910, a first acknowledgement message determination module 920, and an FEC initiation module 930.

The first request obtaining module 910 is configured to obtain an audio and video establishment request of the transmitting end and FEC encoding parameters of the receiving end, where the audio and video establishment request includes forward error correction FEC encoding parameters of the transmitting end; a first acknowledgement message determining module 920, configured to determine an acknowledgement message of the audio/video establishment request according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end, where the acknowledgement message is a response message of the audio/video establishment request; and the FEC initiation module 930 is configured to send an FEC initiation message to the sending end and the receiving end when the acknowledgement message includes an FEC encoding negotiation parameter, so that the sending end and the receiving end perform anti-packet processing on the audio and video data based on the FEC encoding negotiation parameter and the FEC initiation message, where the FEC encoding negotiation parameter is an FEC encoding parameter determined according to the FEC encoding parameter of the sending end and the FEC encoding parameter of the receiving end.

Optionally, the processing device of audio and video data further includes:

the terminal registration module is used for receiving a registration request message of a sending terminal or a receiving terminal before acquiring an audio and video establishment request of the sending terminal and FEC coding parameters of the receiving terminal, wherein the registration request message comprises FEC coding capacity of the sending terminal, and the FEC coding capacity is added in terminal trunking capacity of a broadband trunking communication protocol; generating a registration passing message according to the FEC coding capability in the registration request message and the subscription information of the sending end or the receiving end, and sending the registration passing message to the sending end or the receiving end.

Optionally, the first acknowledgement message determining module 920 includes:

the first confirmation message generation unit is used for generating a confirmation message of the audio and video establishment request; the FEC negotiation parameter determining unit is used for determining FEC negotiation parameters according to the FEC encoding parameters of the sending end and the FEC encoding parameters of the receiving end; and the FEC negotiation parameter adding unit is used for adding the FEC coding negotiation parameters into the confirmation message of the audio/video establishment request.

Optionally, the FEC negotiation parameter determining unit is specifically configured to:

Correspondingly, the FEC negotiation parameter determining unit is specifically configured to:

and the FEC packet parameters, the FEC redundancy and the FEC anti-lost mode of the receiving end are determined according to the FEC coding capability of the receiving end, the FEC packet alternative parameters, the FEC redundancy alternative parameters and the FEC anti-lost mode alternative parameters of the sending end, and are used as FEC coding negotiation parameters.

Fig. 10 is a schematic structural diagram of an audio/video data processing device according to another embodiment of the present application, and as shown in fig. 10, the audio/video data processing device according to the present embodiment includes: a second request acquisition module 1010, a second acknowledgement message determination module 1020, and a second data processing module 1030.

The second request obtaining module 1010 is configured to obtain an audio and video establishment request of the transmitting end and FEC encoding parameters of the receiving end; a second acknowledgement message determining module 1020, configured to determine an acknowledgement message of the audio/video establishment request according to the FEC encoding parameter of the transmitting end and the FEC encoding parameter of the receiving end, where the acknowledgement message is a response message of the audio/video establishment request; and the second data processing module 1030 is configured to perform anti-packet processing on the audio and video data of the receiving end based on the FEC coding negotiation parameter when the acknowledgement message includes the FEC coding negotiation parameter, where the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end.

Optionally, the processing device of audio and video data further includes:

the receiving end registration module is used for adding the FEC coding capacity into the terminal cluster capacity of the broadband cluster communication protocol before acquiring the audio and video establishment request of the transmitting end and the FEC coding parameters of the receiving end; the method comprises the steps that a receiving end sends a registration request message to a cluster core network, wherein the registration request message comprises FEC coding capability of the receiving end; and acquiring a registration passing message of the cluster core network, wherein the registration passing message is a message determined by the cluster core network according to the FEC coding capacity in the registration request message and the subscription information of the receiving end.

Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application, as shown in fig. 11, where the electronic device includes: memory 1110, processor 1120, and computer programs.

The computer program is stored in the memory 1110 and is configured to be executed by the processor 1120 to implement a method for processing audio and video data provided in any of the embodiments corresponding to fig. 2-8 of the present application.

Wherein memory 1111 and processor 1120 are coupled via bus 1130.

The description may be understood correspondingly with reference to the description and effects corresponding to the steps of fig. 2 to fig. 7, which are not repeated here.

An embodiment of the present application provides a computer readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement a method for processing audio and video data provided in any one of the embodiments corresponding to fig. 2 to fig. 7 of the present application.

The computer readable storage medium may be, among other things, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An audio-video data processing method, characterized in that the method comprises:

acquiring a confirmation message of the audio and video establishment request;

when the confirmation message comprises an FEC coding negotiation parameter, carrying out anti-lost packet processing on the audio and video data of the sending end based on the FEC coding negotiation parameter, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end;

the performing anti-packet loss processing on the audio and video data of the transmitting end based on the FEC coding negotiation parameters includes:

2. The method of claim 1, wherein the FEC encoding parameters comprise: FEC switching parameters, FEC packet parameters, FEC redundancy and FEC anti-packet loss mode.

3. The method of claim 2, wherein when the FEC switch parameter of the transmitting end and the FEC switch parameter of the receiving end are both on FEC codes, the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the FEC code negotiation parameter in the acknowledgement message are determined according to the FEC packet parameter, the FEC redundancy and the FEC anti-packet loss mode of the transmitting end and the receiving end, respectively.

4. The method of claim 1, wherein the FEC encoding parameters of the transmitting end include: the method comprises the steps of sending end FEC switch parameters, FEC packet alternative parameters, FEC redundancy alternative parameters and FEC anti-lost mode alternative parameters.

5. The method of claim 4, wherein the FEC encoding negotiation parameters are FEC packet parameters, FEC redundancy and FEC anti-packet loss mode of the receiving end determined according to FEC encoding capability of the receiving end, FEC packet alternative parameters, FEC redundancy alternative parameters and FEC anti-packet loss mode alternative parameters of the transmitting end.

6. The method of claim 1, further comprising, prior to sending the audio video setup request:

7. The method according to any one of claims 1-6, wherein performing anti-packet loss processing on the audio/video data of the transmitting end based on the FEC encoding negotiation parameters includes:

8. A method for processing audio and video data, the method comprising:

when the confirmation message comprises an FEC coding negotiation parameter, an FEC starting message is sent to the sending end and the receiving end, so that the sending end and the receiving end perform anti-packet processing on the audio and video data based on the FEC coding negotiation parameter and the FEC starting message, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the sending end and the FEC coding parameter of the receiving end;

the step of performing anti-packet processing on the audio and video data based on the FEC encoding negotiation parameters and the FEC start message includes: determining an FEC decoding algorithm of the audio and video data of the receiving end according to the FEC coding negotiation parameters; and carrying out packet loss detection and error correction on the audio and video data of the transmitting end according to the FEC decoding algorithm.

9. A method for processing audio and video data, the method comprising:

when the confirmation message comprises an FEC coding negotiation parameter, carrying out anti-lost packet processing on the audio and video data of the receiving end based on the FEC coding negotiation parameter, wherein the FEC coding negotiation parameter is an FEC coding parameter determined according to the FEC coding parameter of the transmitting end and the FEC coding parameter of the receiving end;

the performing anti-packet loss processing on the audio and video data of the receiving end based on the FEC encoding negotiation parameters includes:

10. An apparatus for processing audio-visual data, said apparatus comprising:

the first data processing module is used for carrying out anti-packet loss processing on the audio and video data of the sending end based on the FEC coding negotiation parameters when the confirmation message comprises the FEC coding negotiation parameters, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end;

the first data processing module is specifically configured to,

11. An apparatus for processing audio-visual data, said apparatus comprising:

the FEC starting module is used for sending FEC starting information to the sending end and the receiving end when the confirmation information comprises FEC coding negotiation parameters so that the sending end and the receiving end can carry out anti-lost packet processing on the audio and video data based on the FEC coding negotiation parameters and the FEC starting information, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the sending end and the FEC coding parameters of the receiving end;

the method for performing anti-packet loss processing on the audio and video data based on the FEC coding negotiation parameters and the FEC starting message comprises the following steps: determining an FEC decoding algorithm of the audio and video data of the receiving end according to the FEC coding negotiation parameters; and carrying out packet loss detection and error correction on the audio and video data of the transmitting end according to the FEC decoding algorithm.

12. An apparatus for processing audio-visual data, said apparatus comprising:

The second request acquisition module is used for acquiring an audio and video establishment request of the transmitting end and FEC coding parameters of the receiving end;

the second data processing module is used for carrying out anti-packet loss processing on the audio and video data of the receiving end based on the FEC coding negotiation parameters when the confirmation message comprises the FEC coding negotiation parameters, wherein the FEC coding negotiation parameters are FEC coding parameters determined according to the FEC coding parameters of the transmitting end and the FEC coding parameters of the receiving end;

the second data processing module is specifically configured to,

13. An electronic device, comprising: a memory and at least one processor;

the memory stores computer-executable instructions;

The at least one processor executing computer-executable instructions stored in the memory causes the at least one processor to perform the method of processing audiovisual data as claimed in any one of claims 1 to 9.

14. A computer-readable storage medium, in which computer-executable instructions are stored, which, when executed by a processor, implement the method of processing audio-video data according to any one of claims 1 to 9.