CN114158089A

CN114158089A - Audio transmission method, terminal, electronic device and storage medium

Info

Publication number: CN114158089A
Application number: CN202111465686.8A
Authority: CN
Inventors: 邢文浩; 张晨; 韩润强; 井伟明
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-12-03
Filing date: 2021-12-03
Publication date: 2022-03-08

Abstract

The present disclosure relates to an audio transmission method, a terminal, an electronic device, and a storage medium, the audio transmission method including: the method comprises the steps that a first terminal sends a plurality of audio packets to a second terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment; the first terminal receives feedback of the second terminal for the received audio packets; the first terminal determines an audio packet received by the second terminal according to the feedback, and determines an audio packet to be retransmitted according to a redundant audio frame in the audio packet received by the second terminal; and the first terminal sends the audio packet to be retransmitted to the second terminal.

Description

Audio transmission method, terminal, electronic device and storage medium

Technical Field

The present disclosure relates to the field of signal processing, and in particular, to an audio transmission method, a terminal, an electronic device, and a storage medium.

Background

In the real-time communication process, the audio data loss is difficult to avoid under the influence of the actual conditions of the network. When network packet loss occurs, the sound quality heard by the receiving end is poor, and even the sound sent by the sending end cannot be heard clearly. Recovery of lost audio data during real-time communication is a necessary technique.

The recovery of lost audio data is mainly Forward Error Correction (FEC) technology and retransmission technology. The FEC technology is mainly implemented by adding a part of redundant information when a transmitting end transmits an audio packet, so that a receiving end can recover lost data through the redundant information when a network packet loss occurs. The retransmission technique is that after finding that a received audio packet is lost, a receiving end notifies the sending end to resend the lost audio packet. However, the FEC technique may have difficulty in efficiently recovering lost audio data in the case of a high packet loss rate, while the retransmission technique may increase the code rate due to excessive retransmission operations and cause audio delay in the case of a high packet loss rate. That is, both of these techniques have difficulty in efficiently recovering lost audio data at a high packet loss rate.

Disclosure of Invention

The present disclosure provides an audio transmission method, device electronic equipment, and storage medium to at least solve the problem in the related art that it is difficult to efficiently recover lost audio data under a high packet loss rate.

According to a first aspect of embodiments of the present disclosure, there is provided an audio transmission method, including: the method comprises the steps that a first terminal sends a plurality of audio packets to a second terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment; the first terminal receives feedback of the second terminal for the received audio packets; the first terminal determines an audio packet received by the second terminal according to the feedback, and determines an audio packet to be retransmitted according to a redundant audio frame in the audio packet received by the second terminal; and the first terminal sends the audio packet to be retransmitted to the second terminal.

Optionally, the determining, according to a redundant audio frame in an audio packet received by the second terminal, an audio packet to be retransmitted includes: the first terminal acquires a first number of audio frames which are not continuously received by the second terminal according to redundant audio frames in the audio packet received by the second terminal; obtaining a second number of redundant audio frames contained in each audio packet; and determining the audio packets to be retransmitted according to the first quantity and the second quantity.

Optionally, the second number is N; the determining the audio packets to be retransmitted according to the first number and the second number includes: judging the first quantity; when the first number is less than or equal to N +1, determining that the audio packets to be retransmitted are: taking the last audio frame in the first number of audio frames as an audio packet of the audio frame at the current moment; when the first number is larger than N, determining that the audio packets to be retransmitted are: taking the (N + 1) th audio frame in the first number of audio frames as an audio packet of an audio frame at the current moment, acquiring the number of the remaining continuous non-received audio frames starting from the next audio frame of the (N + 1) th audio frame, updating the first number to the number of the remaining continuous non-connected audio frames, and returning to the step of judging the first number until determining that the audio packet to be retransmitted is: and taking the last audio frame in the remaining continuous unreceived audio frames as an audio packet of the audio frame at the current moment.

Optionally, the obtaining, by the first terminal, a first number of consecutive audio frames that are not received by the second terminal according to the redundant audio frame in the audio packet received by the second terminal includes: and the first terminal starts counting when determining that the second terminal does not receive a certain audio frame according to the redundant audio frame in the audio packet received by the second terminal, and stops counting when determining that the second terminal receives the audio frame again, so as to obtain the first number of the audio frames which are not received continuously by the second terminal.

Optionally, the audio transmission method further includes: the first terminal obtains the packet loss rate of the audio packet sent to the second terminal; and adjusting the preset quantity according to the packet loss rate.

Optionally, the adjusting the preset number according to the packet loss ratio includes: when the packet loss rate is increased, increasing the preset number; and when the packet loss rate is reduced, reducing the preset number.

Optionally, the feedback includes a packet identifier of the received audio packet, where the packet identifier includes index information of the audio packet received by the second terminal.

According to a second aspect of the embodiments of the present disclosure, there is provided an audio transmission method including: receiving a plurality of audio packets from a first terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment; sending feedback for the received audio packets to the first terminal; and receiving an audio packet to be retransmitted sent by a first terminal, wherein the audio packet to be retransmitted is determined by the first terminal according to the feedback and is obtained according to a redundant audio frame in the audio packet received by the second terminal.

Optionally, the audio transmission method further includes: and the second terminal recovers the audio frames corresponding to the redundant audio frames in the audio packets which are not received by the second terminal according to the redundant audio frames in the received audio packets.

Optionally, the audio transmission method further includes: and the second terminal sends feedback aiming at the packet loss rate of the audio packet sent by the first terminal to the first terminal.

According to a third aspect of embodiments of the present disclosure, there is provided a first terminal for audio transmission, the first terminal comprising: the audio transmitting unit is configured to transmit a plurality of audio packets to a second terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment; a feedback receiving unit configured to receive feedback of the second terminal for the received audio packets; a retransmission determination unit configured to: determining an audio packet received by a second terminal according to the feedback, and determining an audio packet to be retransmitted according to a redundant audio frame in the audio packet received by the second terminal; wherein the audio transmitting unit is further configured to transmit the audio packet to be retransmitted to the second terminal.

Optionally, the determining, according to a redundant audio frame in an audio packet received by the second terminal, an audio packet to be retransmitted includes: acquiring a first number of audio frames which are not received continuously by the second terminal according to redundant audio frames in the audio packet received by the second terminal; obtaining a second number of redundant audio frames contained in each audio packet; and determining the audio packets to be retransmitted according to the first quantity and the second quantity.

Optionally, the second number is N; the determining the audio packets to be retransmitted according to the first number and the second number includes: judging the first quantity; when the first number is less than or equal to N +1, determining that the audio packets to be retransmitted are: taking the last audio frame in the first number of audio frames as an audio packet of the audio frame at the current moment; when the first number is larger than N, determining that the audio packets to be retransmitted are: taking the (N + 1) th audio frame in the first number of audio frames as an audio packet of an audio frame at the current moment, acquiring the number of the remaining continuous non-received audio frames starting from the next audio frame of the (N + 1) th audio frame, updating the first number to the number of the remaining continuous non-connected audio frames, and returning to the step of judging the first number until determining that the audio packet to be retransmitted is: and taking the last audio frame in the remaining continuous unreceived audio frames as an audio packet of the current audio frame.

Optionally, the obtaining, according to redundant audio frames in an audio packet received by the second terminal, a first number of consecutive audio frames that are not received by the second terminal includes: and according to redundant audio frames in the audio packet received by the second terminal, starting counting when the second terminal is determined not to receive a certain audio frame, stopping counting when the second terminal is determined to receive the audio frame again according to the redundant audio frames in the received audio packet, and acquiring the first number of the audio frames which are not received continuously by the second terminal.

Optionally, the first terminal further includes: an adjustment unit configured to: acquiring the packet loss rate of an audio packet sent to the second terminal; and adjusting the preset quantity according to the packet loss rate.

According to a fourth aspect of embodiments of the present disclosure, there is provided a second terminal for audio transmission, the second terminal comprising: an audio receiving unit configured to receive a plurality of audio packets from a first terminal, wherein each of the plurality of audio packets includes an audio frame at a current time and a preset number of redundant audio frames before the current time; a feedback transmitting unit configured to transmit feedback for the received audio packet to the first terminal; the audio receiving unit is further configured to receive an audio packet to be retransmitted sent by a first terminal, where the audio packet to be retransmitted is determined by the first terminal according to the feedback and is obtained according to a redundant audio frame in the audio packet received by the second terminal.

Optionally, the second terminal further includes: and the recovery unit is configured to recover the audio frame corresponding to the redundant audio frame in the audio packet which is not received by the second terminal according to the redundant audio frame included in the received audio packet.

Optionally, the feedback sending unit is further configured to: and sending feedback aiming at the packet loss rate of the audio packet sent by the first terminal to the first terminal.

According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic apparatus, including: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the audio transmission method as described above.

According to a sixth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium storing instructions, which when executed by at least one processor, cause the at least one processor to perform the audio transmission method as described above.

According to a seventh aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer instructions, characterized in that the computer instructions, when executed by a processor, implement the audio transmission method as described above.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects: according to the audio transmission method executed by the first terminal in the embodiment of the disclosure, each audio packet includes an audio frame at the current time and a preset number of redundant audio frames before the current time, and the audio packet received by the second terminal is determined according to the feedback of the second terminal for the received audio packet, and the audio packet to be retransmitted is determined according to the redundant audio frame in the audio packet received by the second terminal, so that compared with a conventional FEC scheme, under the condition of a high packet loss rate, the audio transmission method disclosed in the disclosure can further recover lost audio data through active retransmission, and compared with a conventional retransmission scheme, due to selective retransmission, a code rate can be effectively saved under the high packet loss rate. Therefore, the audio transmission method of the present disclosure, which combines the FEC technique and the retransmission technique, makes it possible to perform efficient recovery of lost audio data at a high packet loss rate.

According to the audio transmission method executed by the second terminal in the embodiment of the disclosure, since the audio packet received from the first terminal includes the preset number of redundant audio frames before the current time in addition to the audio frame at the current time, and can receive the audio packet to be retransmitted sent by the first terminal (the audio packet to be retransmitted is determined by the first terminal according to the feedback and is determined according to the redundant audio frames in the audio packet received by the second terminal), the audio packet to be retransmitted selectively can be received under the condition of high packet loss rate, and thus the lost audio data can be recovered efficiently.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.

FIG. 1 is an exemplary system architecture to which exemplary embodiments of the present disclosure may be applied;

fig. 2 is a flowchart of an audio transmission method performed by a first terminal according to an exemplary embodiment of the present disclosure;

fig. 3 is a format example of an audio packet carrying redundant audio frames according to an exemplary embodiment of the present disclosure.

Fig. 4 is a schematic diagram of an audio transmission method of an exemplary embodiment of the present disclosure determining that an audio packet needs to be retransmitted;

fig. 5 is a flowchart of an audio transmission method performed by a second terminal according to an exemplary embodiment of the present disclosure;

fig. 6 is a block diagram illustrating a first terminal for audio transmission according to an exemplary embodiment of the present disclosure;

fig. 7 is a block diagram illustrating a second terminal for audio transmission according to another exemplary embodiment of the present disclosure;

fig. 8 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

In this case, the expression "at least one of the items" in the present disclosure means a case where three types of parallel expressions "any one of the items", "a combination of any plural ones of the items", and "the entirety of the items" are included. For example, "include at least one of a and B" includes the following three cases in parallel: (1) comprises A; (2) comprises B; (3) including a and B. For another example, "at least one of the first step and the second step is performed", which means that the following three cases are juxtaposed: (1) executing the step one; (2) executing the step two; (3) and executing the step one and the step two.

Fig. 1 illustrates an exemplary system architecture 100 in which exemplary embodiments of the present disclosure may be applied.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. A user may use the

terminal devices

101, 102, 103 to interact with the server 105 over the network 104 to receive or send messages (e.g., video data upload requests, video data download requests), etc. Various communication client applications, such as audio and video communication software, audio and video recording software, instant messaging software, conference software, mailbox clients, social platform software, and the like, may be installed on the

terminal devices

101, 102, and 103. The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices having a display screen and capable of playing, recording, editing, etc. audio and video, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc. When the

terminal device

101, 102, 103 is software, it may be installed in the electronic devices listed above, it may be implemented as a plurality of software or software modules (for example, to provide distributed services), or it may be implemented as a single software or software module. And is not particularly limited herein.

The

terminal devices

101, 102, 103 may be equipped with an image capturing device (e.g., a camera) to capture video data. In practice, the smallest visual unit that makes up a video is a Frame (Frame). Each frame is a static image. Temporally successive sequences of frames are composited together to form a motion video. Further, the

terminal apparatuses

101, 102, 103 may also be mounted with a component (e.g., a speaker) for converting an electric signal into sound to play the sound, and may also be mounted with a device (e.g., a microphone) for converting an analog audio signal into a digital audio signal to pick up the sound. In addition, the

terminal apparatuses

101, 102, 103 can perform voice communication or video communication with each other, for example, perform real-time voice communication or video communication.

The server 105 may be a server providing various services, such as a background server providing support for multimedia applications installed on the

terminal devices

101, 102, 103. The background server can analyze, store and the like the received data such as the audio and video data uploading request, can also receive the audio and video data downloading request sent by the

terminal equipment

101, 102 and 103, and feeds back the audio and video data indicated by the audio and video data downloading request to the

terminal equipment

101, 102 and 103.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be noted that the audio transmission method provided by the embodiment of the present disclosure is generally executed by the terminal device, or may also be executed by the terminal device and the server in cooperation. Accordingly, the first terminal and the second terminal mentioned in the embodiments of the present disclosure may be provided in the terminal device, or in both the terminal device and the server.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation, and the disclosure is not limited thereto.

Fig. 2 is a flowchart of an audio transmission method performed by a first terminal according to an exemplary embodiment of the present disclosure. Referring to fig. 2, a first terminal transmits a plurality of audio packets to a second terminal at step S210. Here, according to an exemplary embodiment, each of the plurality of audio packets includes an audio frame at a current time and a preset number of redundant audio frames before the current time. For example, each audio packet may carry a redundant audio frame at the first n time instants in addition to the audio frame at the current time instant, for example, n may be 3, but is not limited thereto. As an example, a plurality of audio packets may be transmitted sequentially in time order, and each audio packet may have a packet sequence number. For example, the packet identification of the audio packet may be included in the header of the audio packet. For example, the packet identification may include index information of the audio packet. The index information may be, for example, a packet sequence number, but is not limited thereto. Since the audio packet includes the redundant audio frames of the preset number before the current time, if a packet loss occurs in the transmission process and the redundant audio frames corresponding to the lost packet exist in the audio packet received by the second terminal after the lost packet, the receiving end can recover the audio frames corresponding to the redundant audio frames in the lost packet by using the redundant audio frames, so that the lost audio data can be recovered faster than the lost packet is retransmitted, and the audio delay is reduced.

In step S220, the first terminal receives feedback of the second terminal for the received audio packets. By way of example, the feedback may include, but is not limited to, a packet identification of the received audio packet. For example, the packet identifier may include index information of an audio packet received by the second terminal, and the index information may be, for example, a packet sequence number. According to an exemplary embodiment, the second terminal may send acknowledgement feedback, e.g., an acknowledgement character ack (acknowledgement character), to the first terminal if the second terminal successfully receives the audio packet. Whereas if the second terminal fails to receive the audio packet successfully (i.e., there is a packet loss), the first terminal will not receive acknowledgement feedback for the audio packet. For example, the acknowledgement feedback may include the packet sequence number of the received audio packet. In this way, the first terminal can know which audio packets were received by the second terminal based on the acknowledgement feedback. In the present disclosure, if a first terminal receives feedback for a received audio packet from a second terminal, the first terminal is considered to have received feedback for all audio frames in the audio packet. For example, if one audio packet is an audio packet with ACK, all audio frames included in the audio packet are audio frames with ACK.

In step S230, the first terminal determines the audio packet received by the second terminal according to the feedback, and determines the audio packet to be retransmitted according to the redundant audio frame in the audio packet received by the second terminal. For example, as shown in fig. 3, if

audio packets

7, 8, and 9 are lost during transmission, but the first terminal receives feedback for packet number 10, the first terminal may determine that the second terminal received audio packet 10 according to the feedback. Since the redundant audio frames 7, 8, and 9 corresponding to the

audio packets

7, 8, and 9 are included in the packet No. 10, the second terminal can recover the audio frames corresponding to the redundant audio frames in the lost

audio packets

7, 8, and 9 (i.e., the audio frames at the current time in the

audio packets

7, 8, and 9), respectively, by using the redundant audio frames 7, 8, and 9 included in the packet No. 10. In this case, the first terminal may regard the audio frame corresponding to the redundant audio frame as an audio frame with acknowledgement feedback, that is, the first terminal may regard the audio frame corresponding to the redundant audio frame as being received by the second terminal, and therefore, the audio packet corresponding to the audio frame is not retransmitted. That is, when a redundant audio frame corresponding to a lost audio packet is included in an audio packet received by the second terminal following the lost audio packet, the first terminal does not need to retransmit the lost audio packet. In contrast, when the audio packet received by the second terminal after the lost audio packet does not include the redundant audio frame corresponding to the lost audio packet, the first terminal needs to determine the audio packet to be retransmitted according to the redundant audio frame in the audio packet received by the second terminal. However, since each audio packet also carries redundant audio frames corresponding to other audio packets, in the present invention, each audio packet that is not received is not retransmitted, but the retransmission of the audio packet is selectively performed.

Optionally, according to an exemplary embodiment, although not shown in fig. 2, the method shown in fig. 2 may further include: the first terminal obtains the packet loss rate of the audio packet sent to the second terminal; and adjusting the preset quantity according to the packet loss rate. For example, the first terminal may calculate the packet loss rate according to the feedback of the second terminal for the received audio packet, or the first terminal may also directly receive the feedback of the packet loss rate from the second terminal. For example, the first terminal may calculate how many audio packets are received by the second terminal with respect to the packet identifier (e.g., packet sequence number) included in the feedback of the received audio packets, thereby calculating the packet loss rate. Or, the second terminal may know how many audio packets are received and how many audio packets are lost according to packet identifiers (e.g., packet sequence numbers) in the received audio packets, so as to calculate a packet loss rate, and then feed back the packet loss rate to the first terminal. As an example, the first terminal may periodically receive feedback of the packet loss rate from the second terminal, e.g., the first terminal may receive feedback of the packet loss rate from the second terminal every time, or may receive feedback of the packet loss rate from the second terminal every several times. The first terminal may correspondingly adjust the number of redundant audio frames added in the audio packet according to the packet loss rate fed back by the second terminal. According to an exemplary embodiment, when the packet loss rate increases, the preset number is increased; and when the packet loss rate is reduced, reducing the preset number. For example, the greater the packet loss rate, the more redundant audio frames at the previous time point can be contained in each audio packet. Conversely, the smaller the packet loss rate, the fewer redundant audio frames at the previous time point can be contained in each audio packet.

Fig. 3 is a format example of an audio packet carrying redundant audio frames according to an exemplary embodiment of the present disclosure. In the example shown in fig. 3, each audio packet carries, in addition to the audio frame at this time, a redundant audio frame 3 times before the current time. However, as described above, the number of redundant audio frames may be adaptively adjusted, for example, the number of added redundant audio frames may be adaptively adjusted according to a packet loss rate, thereby facilitating more efficient packet loss recovery by the second terminal.

Fig. 4 is a schematic diagram of an audio transmission method for determining an audio packet to be retransmitted according to an exemplary embodiment of the present disclosure. The manner of determining the audio packets to be retransmitted is described below in conjunction with fig. 4.

According to an exemplary embodiment, specifically, to determine an audio packet to be retransmitted, first, the first terminal obtains, according to a redundant audio frame in an audio packet received by the second terminal, a first number of audio frames that are not continuously received by the second terminal; secondly, acquiring a second number of redundant audio frames contained in each audio packet; and finally, determining the audio packets to be retransmitted according to the first quantity and the second quantity. For example, the first number may be obtained by: and the first terminal starts counting when determining that the second terminal does not receive a certain audio frame according to the redundant audio frame in the audio packet received by the second terminal, and stops counting when determining that the second terminal receives the audio frame again, so as to obtain the first number of the audio frames which are not received continuously by the second terminal.

For example, in the example of fig. 4, it is assumed that each audio packet includes, in addition to the audio frame at the current time, a redundant audio frame 6 times before the current time, that is, each audio packet carries 6 redundant audio frames. In the example shown in (a) of fig. 4, if an audio frame (for example, the audio frame 7) preceding the audio frame 8 is an audio frame with confirmation feedback, it is considered that the audio frame preceding the audio frame 8 is received by the second terminal. The counting is started when it is determined that the second terminal has not received the audio frame 8, at which point the first number is equal to 1, continues if it is subsequently determined that the second terminal has not received the audio frame 9, at which point the first number is equal to 2, and subsequently continues if it is determined that the second terminal has not received the audio frame 10, at which point the first number is equal to 3, continues if it is determined that the second terminal has not received the audio frame 11, at which point the first number is equal to 4, until the counting is stopped when it is determined that the second terminal has received the audio frame again, for example, when it is determined that the second terminal has received the audio frame 12, at which point the first number may be obtained equal to 4.

Here, it should be noted that the second terminal does not receive the

audio frame

8, 9 or 10, which indicates that the second terminal does not receive the audio packet having the

audio frame

8, 9 or 10 as the audio frame at the current time, and does not include the redundant audio frame corresponding to the

audio frame

8, 9 or 10 in the other audio packets received by the second terminal, because, as mentioned above, if the redundant audio frame corresponding to the

audio frame

8, 9 or 10 is included in the other audio packets received by the second terminal, the

audio frame

8, 9 or 10 is considered to be received by the second terminal, that is, the

audio frame

8, 9 or 10 is also considered to be an audio frame with ACK.

As mentioned above, since each audio packet may carry redundant audio frames (even if it is a retransmitted audio packet, a preset number of redundant audio frames may be carried), the present invention determines an audio packet to be retransmitted based on the first number and the second number by considering a second number of redundant audio frames (assuming that the second number is N) included in each audio packet in addition to the first number when determining the audio packet to be retransmitted.

Specifically, for example, the audio packet to be retransmitted can be determined by:

step 1: and judging the first quantity.

Step 2: when the first number is less than or equal to N +1, determining that the audio packets to be retransmitted are: and taking the last audio frame in the first number of audio frames as an audio packet of the audio frame at the current moment. Here, the audio packet in which the last audio frame of the first number of audio frames is taken as the audio frame at the current time includes, in addition to the last audio frame of the first number of audio frames, a preset number of audio frames before the last audio frame, that is, a preset number of redundant audio frames before the current time (that is, the time corresponding to the last audio frame).

For example, as shown in fig. 4 (a), if the number of redundant audio frames included in each audio packet is 6 (i.e., N is 6), and the audio frames 8 to 11 are determined to be the number of consecutive unreceived audio frames (i.e., the first number is 4), since the first number is 4 and N is 6, it is satisfied that the first number is less than N +1, and therefore, it is determined that the audio packet to be retransmitted is an audio packet having the last audio frame (i.e., the audio frame 11) of the 4 consecutive unreceived audio frames as the audio frame of the current time, i.e., the audio packet to be retransmitted includes 6 redundant audio frames before the audio frame 11 in addition to the audio frame 11, i.e., includes redundant audio frames 5 to 10 6 times before the current time corresponding to the audio frame 11. Because the audio packet to be retransmitted contains the redundant audio frames 8 to 10, the second terminal can recover the audio frames corresponding to the redundant audio frames in the audio packet which is not received by the second terminal by using the received redundant audio frames in the retransmitted audio packet after receiving the retransmitted audio packet.

And step 3: when the first number is larger than N, determining that the audio packets to be retransmitted are: taking the (N + 1) th audio frame in the first number of audio frames as an audio packet of an audio frame at the current moment, acquiring the number of the remaining continuous non-received audio frames starting from the next audio frame of the (N + 1) th audio frame, updating the first number to the number of the remaining continuous non-connected audio frames, and returning to the step of judging the first number until determining that the audio packet to be retransmitted is: and taking the last audio frame in the remaining continuous unreceived audio frames as an audio packet of the audio frame at the current moment. Here, the N +1 th audio frame in the first number of audio frames is taken as the audio packet of the audio frame at the current time, and includes a preset number of audio frames before the N +1 th audio frame, that is, a preset number of redundant audio frames before the current time (that is, the time corresponding to the N +1 th audio frame). Similarly, the last audio frame of the remaining consecutive non-received audio frames is taken as the audio packet of the audio frame at the current time, and includes, in addition to the last audio frame of the remaining consecutive non-received audio frames, a preset number of audio frames before the last audio frame, that is, a preset number of redundant audio frames before the current time (that is, the time corresponding to the last audio frame).

For example, as shown in (b) of fig. 4, if the number of redundant audio frames included in each audio packet is 6 (i.e., N is 6), and the audio frames 8 to 16 are determined to be 9 consecutive audio frames that are not received (i.e., the first number is 9), since the first number is 9 and N is 6, it is satisfied that the first number is greater than N, and therefore, it is first determined that the audio packet to be retransmitted is an audio packet having the N +1 th audio frame (i.e., the 7 th audio frame, that is, the audio frame 14) as the audio frame at the current time, and the audio packet includes 6 audio frames before the audio frame 14 in addition to the audio frame 14, that is, includes the audio frames 8 to 13.

Next, the number of remaining consecutive non-received audio frames starting from the next audio frame of the 7 th audio frame, i.e., the number of consecutive non-received audio frames starting from the 8 th audio frame (i.e., audio frame 15) is obtained, and since only two consecutive non-received audio frames remain thereafter, the number of remaining consecutive non-received audio frames is 2. At this time, the first number is updated to 2, and then the procedure returns to step 1. Since the first number is 2 and N is 6, the first number is smaller than N +1, and therefore, the audio packet to be retransmitted is determined as: the last audio frame (i.e., audio frame 16) of the remaining consecutive unreceived audio frames is taken as the audio packet of the audio frame at the current time. This audio packet contains, in addition to the audio frame 16, a predetermined number of audio frames preceding the audio frame 16.

Finally, for the example shown in fig. 4 (b), the audio packets that need to be retransmitted are only the audio packet with the audio frame 14 as the audio frame at the current time and the audio packet with the audio frame 16 as the audio frame at the current time. After the second terminal receives the two retransmitted audio packets, since the first retransmitted audio packet includes redundant audio frames 8 to 13 in addition to the audio frame 14, and the second retransmitted audio packet includes at least redundant audio frame 15 in addition to the audio frame 16, the second terminal can recover the audio frames corresponding to the redundant audio frames in the unreceived audio packets by using the redundant audio frames in the received retransmitted audio packets after receiving the retransmitted audio packets, for example, recover the audio frames 8 to 13 and the audio frame 15.

Finally, in step S240, the first terminal sends the audio packet to be retransmitted to the second terminal.

Because the first terminal does not directly retransmit the audio packet but selectively retransmits the audio packet when the second terminal does not receive the audio packet, the code rate can be effectively saved under the condition of high packet loss rate, because each audio packet also has a packet header, and the packet header can consume a plurality of code rates, so that the code rate is saved when several audio frames are combined into one audio packet for retransmission compared with the case that each lost audio packet is retransmitted respectively.

Above, the audio transmission method performed by the first terminal according to the exemplary embodiment of the present disclosure has been described with reference to fig. 2 to 4, and since not only the audio frame at the current time and the preset number of redundant audio frames before the current time are included in each audio packet, but also the audio packet received by the second terminal is determined according to the feedback of the second terminal for the received audio packet, and the audio packet to be retransmitted is determined according to the redundant audio frame in the audio packet received by the second terminal, compared to the conventional FEC scheme, in the case of a high packet loss rate, the above-mentioned audio transmission method of the present disclosure may further perform recovery of lost audio data through active retransmission. Compared with the traditional retransmission scheme, the selective retransmission is carried out, so that the code rate can be effectively saved under the condition of high packet loss rate. Therefore, the audio transmission method of the present disclosure, which combines the FEC technique and the retransmission technique, makes it possible to perform efficient recovery of lost audio data at a high packet loss rate.

Fig. 5 is a flowchart of an audio transmission method performed by a second terminal according to an exemplary embodiment of the present disclosure. It should be noted that the contents related to the operation performed by the second terminal mentioned above in the process of describing the audio transmission method performed by the first terminal are all applicable to the audio transmission method performed by the second terminal shown in fig. 5, and therefore, the related contents are not repeated.

Referring to fig. 5, a plurality of audio packets are received from a first terminal at step S510. Each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment. Since the audio packet includes the preset number of redundant audio frames before the current time, if a packet is lost and redundant audio frames corresponding to the audio frames in the lost packet exist in the audio packet received by the receiving end after the lost packet, the second terminal may recover the audio frames corresponding to the redundant audio frames in the lost packet by using the redundant audio frames.

In step S520, feedback for the received audio packets is sent to the first terminal. For example, when an audio packet is received, acknowledgement feedback for the audio packet is sent to the sender. For example, the packet identification of the received audio packet may be included in the acknowledgement feedback. For example, the packet identification includes index information of an audio packet received by the second terminal. The index information may be, for example, a packet sequence number of the received audio packet. For example, the second terminal can know how many audio packets are received and how many audio packets are lost according to the packet sequence numbers of the received audio packets, so that the packet loss rate can be calculated, and then the packet loss rate is fed back to the sending terminal.

Thus, optionally, although not shown in fig. 5, the method shown in fig. 5 may further comprise: and the second terminal sends feedback aiming at the packet loss rate of the audio packet sent by the first terminal to the first terminal. As an example, the second terminal may periodically send feedback of the packet loss rate to the first terminal, for example, the feedback of the packet loss rate may be sent to the first terminal every time, or the feedback of the packet loss rate may be sent to the first terminal every several times. The first terminal may adjust the preset number of redundant audio frames included in the audio packet according to the packet loss rate fed back by the receiving terminal.

In addition, in step S530, the audio packet to be retransmitted sent by the first terminal is received. Here, as mentioned in the above description about 2, the audio packet to be retransmitted is determined by the first terminal according to the feedback, and is determined according to the redundant audio frame in the audio packet received by the second terminal, and in the above description, how the first terminal determines the audio packet to be retransmitted has been mentioned in the description referring to fig. 2 to 4, and therefore, details are not repeated here.

Because the audio packet received by the second terminal includes both the audio frame at the current time and the preset number of redundant audio frames before the current time, the second terminal can also recover the audio packet corresponding to the redundant audio frame in the audio packet that is not received by the second terminal according to the redundant audio frames included in the received audio packet. Therefore, optionally, the audio transmission method shown in fig. 5 may further include: and the second terminal recovers the audio frames corresponding to the redundant audio frames in the audio packets which are not received by the second terminal according to the redundant audio frames in the received audio packets.

Since the audio transmission method shown in fig. 5 includes, in addition to the audio frame at the current time, a preset number of redundant audio frames before the current time from the audio packet received by the first terminal, and can receive the audio packet to be retransmitted sent by the first terminal (the audio packet to be retransmitted is determined by the first terminal according to the feedback and is determined according to the redundant audio frame in the audio packet received by the second terminal), the audio packet to be retransmitted selectively can be received under the condition of a high packet loss rate, so that the lost audio data can be recovered efficiently.

Fig. 6 is a block diagram illustrating a first terminal for audio transmission according to an exemplary embodiment of the present disclosure.

Referring to fig. 6, the first terminal 600 may include an audio transmitting unit 610, a feedback receiving unit 620, and a retransmission determining unit 630. Specifically, the audio transmitting unit 610 may be configured to transmit a plurality of audio packets to the second terminal, where each of the plurality of audio packets includes an audio frame at a current time and a preset number of redundant audio frames before the current time. The feedback receiving unit 620 may be configured to receive feedback of the second terminal with respect to the received audio packets. The retransmission determination unit 630 may be configured to: and determining the audio packet received by the second terminal according to the feedback, and determining the audio packet to be retransmitted according to the redundant audio frame in the audio packet received by the second terminal. In addition, the audio transmitting unit 610 may be further configured to transmit the audio packet to be retransmitted to the second terminal.

Since the audio transmission method shown in fig. 2 can be performed by the first terminal 600 shown in fig. 6, and details related to operations performed by the audio transmitting unit 610, the feedback receiving unit 620, and the retransmission determining unit 630 have been described above in the description related to fig. 2 and fig. 4, any relevant details related to the operations performed by the units in fig. 6 can be referred to the corresponding description related to fig. 2 to fig. 4, and are not repeated here.

Further, it should be noted that although the first terminal 600 is described above as being divided into units for respectively performing corresponding processes, it is clear to those skilled in the art that the processes performed by the units described above can also be performed without any specific division of the units by the first terminal 600 or without explicit demarcation between the units. In addition, the first terminal 600 may further include other units, for example, a storage unit, a data processing unit, and the like.

Fig. 7 is a block diagram illustrating a second terminal for audio transmission according to another exemplary embodiment of the present disclosure.

Referring to fig. 7, the second terminal 700 may include an audio receiving unit 710 and a feedback transmitting unit 720. Specifically, the audio receiving unit 710 may be configured to receive a plurality of audio packets from the first terminal, wherein each of the plurality of audio packets includes an audio frame at a current time and a preset number of redundant audio frames before the current time. The feedback sending unit 720 may be configured to send feedback for the received audio packets to the first terminal. Furthermore, the audio receiving unit 710 may be further configured to: and receiving an audio packet to be retransmitted sent by a first terminal, wherein the audio packet to be retransmitted is determined by the first terminal according to the feedback and is obtained according to a redundant audio frame in the audio packet received by the second terminal. Optionally, the second terminal 700 may further include a restoring unit (not shown), and the restoring unit may restore, according to a redundant audio frame included in the received audio packet, an audio frame corresponding to the redundant audio frame in an audio packet that is not received by the second terminal.

Since the audio transmission method shown in fig. 5 can be performed by the second terminal 700 shown in fig. 7, and details related to the operations performed by the audio receiving unit 710 and the feedback sending unit 720 have been described above in the description related to fig. 5, any relevant details related to the operations performed by the units in fig. 7 can be referred to the corresponding description related to fig. 5, and are not repeated here.

Further, it should be noted that although the second terminal 700 is described above as being divided into units for respectively performing corresponding processes, it is clear to those skilled in the art that the processes performed by the units described above can also be performed without any specific division of the units by the second terminal 700 or explicit demarcation between the units. In addition, the second terminal 700 may further include other units, for example, a storage unit, a data processing unit, and the like.

Referring to fig. 8, an electronic device 800 may include at least one memory 801 and at least one processor 802, the at least one memory storing computer-executable instructions that, when executed by the at least one processor, cause the at least one processor 802 to perform an audio transmission method according to an embodiment of the disclosure (the audio transmission method performed by a first terminal shown in fig. 2 or the audio transmission method performed by a second terminal shown in fig. 5).

By way of example, the electronic device may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the set of instructions described above. The electronic device need not be a single electronic device, but can be any collection of devices or circuits that can execute the above instructions (or sets of instructions) either individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).

In an electronic device, a processor may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. By way of example, and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

The processor may execute instructions or code stored in the memory, which may also store data. The instructions and data may also be transmitted or received over a network via a network interface device, which may employ any known transmission protocol.

The memory may be integral to the processor, e.g., RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, the memory may comprise a stand-alone device, such as an external disk drive, storage array, or any other storage device usable by a database system. The memory and the processor may be operatively coupled or may communicate with each other, such as through an I/O port, a network connection, etc., so that the processor can read files stored in the memory.

In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device may be connected to each other via a bus and/or a network.

According to an embodiment of the present disclosure, there may also be provided a computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform an audio transmission method according to an exemplary embodiment of the present disclosure (the audio transmission method performed by the first terminal shown in fig. 2 or the audio transmission method performed by the second terminal shown in fig. 5). Examples of the computer-readable storage medium herein include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD + R, CD-RW, CD + RW, DVD-ROM, DVD-R, DVD + R, DVD-RW, DVD + RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or compact disc memory, Hard Disk Drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a Secure Digital (SD) card or a extreme digital (XD) card), magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a magnetic tape, a magneto-optical data storage device, a hard disk, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, A solid state disk, and any other device configured to store and provide a computer program and any associated data, data files, and data structures to a processor or computer in a non-transitory manner such that the processor or computer can execute the computer program. The instructions in the computer-readable storage medium or computer program described above may be run in an environment deployed in a computer apparatus, such as a client, a host, a proxy device, a server, etc., and further, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by one or more processors or computers.

According to an embodiment of the present disclosure, there may also be provided a computer program product including computer instructions which, when executed by a processor, implement an audio transmission method according to an exemplary embodiment of the present disclosure (the audio transmission method performed by the first terminal shown in fig. 2 or the audio transmission method performed by the second terminal shown in fig. 5).

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. An audio transmission method, comprising:

the method comprises the steps that a first terminal sends a plurality of audio packets to a second terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment;

the first terminal receives feedback of the second terminal for the received audio packets;

the first terminal determines an audio packet received by the second terminal according to the feedback, and determines an audio packet to be retransmitted according to a redundant audio frame in the audio packet received by the second terminal;

and the first terminal sends the audio packet to be retransmitted to the second terminal.

2. The audio transmission method according to claim 1, wherein the determining the audio packet to be retransmitted according to the redundant audio frame in the audio packet received by the second terminal comprises:

the first terminal acquires a first number of audio frames which are not continuously received by the second terminal according to redundant audio frames in the audio packet received by the second terminal;

obtaining a second number of redundant audio frames contained in each audio packet;

and determining the audio packets to be retransmitted according to the first quantity and the second quantity.

3. The audio transmission method according to claim 2, wherein the second number is N; the determining the audio packets to be retransmitted according to the first number and the second number includes:

judging the first quantity;

when the first number is less than or equal to N +1, determining that the audio packets to be retransmitted are: taking the last audio frame in the first number of audio frames as an audio packet of the audio frame at the current moment;

when the first number is larger than N, determining that the audio packets to be retransmitted are: taking the (N + 1) th audio frame in the first number of audio frames as an audio packet of an audio frame at the current moment, acquiring the number of the remaining continuous non-received audio frames starting from the next audio frame of the (N + 1) th audio frame, updating the first number to the number of the remaining continuous non-connected audio frames, and returning to the step of judging the first number until determining that the audio packet to be retransmitted is: and taking the last audio frame in the remaining continuous unreceived audio frames as an audio packet of the audio frame at the current moment.

4. The audio transmission method according to claim 2 or 3, wherein the obtaining, by the first terminal, the first number of consecutive unreceived audio frames by the second terminal according to the redundant audio frames in the audio packet received by the second terminal, includes:

and the first terminal starts counting when determining that the second terminal does not receive a certain audio frame according to the redundant audio frame in the audio packet received by the second terminal, and stops counting when determining that the second terminal receives the audio frame again, so as to obtain the first number of the audio frames which are not received continuously by the second terminal.

5. The audio transmission method of claim 1, further comprising:

the first terminal obtains the packet loss rate of the audio packet sent to the second terminal;

and adjusting the preset quantity according to the packet loss rate.

6. An audio transmission method, comprising:

receiving a plurality of audio packets from a first terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment;

sending feedback for the received audio packets to the first terminal;

and receiving an audio packet to be retransmitted sent by a first terminal, wherein the audio packet to be retransmitted is determined by the first terminal according to the feedback and is obtained according to a redundant audio frame in the audio packet received by the second terminal.

7. A first terminal for audio transmission, comprising:

the audio transmitting unit is configured to transmit a plurality of audio packets to a second terminal, wherein each audio packet in the plurality of audio packets comprises an audio frame at the current moment and a preset number of redundant audio frames before the current moment;

a feedback receiving unit configured to receive feedback of the second terminal for the received audio packets;

a retransmission determination unit configured to: determining an audio packet received by a second terminal according to the feedback, and determining an audio packet to be retransmitted according to a redundant audio frame in the audio packet received by the second terminal;

wherein the audio transmitting unit is further configured to transmit the audio packet to be retransmitted to the second terminal.

8. A second terminal for audio transmission, comprising:

an audio receiving unit configured to receive a plurality of audio packets from a first terminal, wherein each of the plurality of audio packets includes an audio frame at a current time and a preset number of redundant audio frames before the current time;

a feedback transmitting unit configured to transmit feedback for the received audio packet to the first terminal;

the audio receiving unit is further configured to receive an audio packet to be retransmitted sent by a first terminal, where the audio packet to be retransmitted is determined by the first terminal according to the feedback and is obtained according to a redundant audio frame in the audio packet received by the second terminal.

9. An electronic device, comprising:

at least one processor;

at least one memory storing computer-executable instructions,

wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the audio transmission method of any of claims 1 to 5.

10. A computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the audio transmission method of any of claims 1 to 5.