CN110248256B

CN110248256B - Data processing method and device, storage medium and electronic device

Info

Publication number: CN110248256B
Application number: CN201910556295.3A
Authority: CN
Inventors: 胡仁成
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-06-25
Filing date: 2019-06-25
Publication date: 2021-09-10
Anticipated expiration: 2039-06-25
Also published as: EP3993436B1; EP3993436A4; WO2020259448A1; US11848969B2; US20240073263A1; CN110248256A; US20210400092A1; EP3993436A1; EP3993436C0

Abstract

The invention discloses a data processing method and device, a storage medium and an electronic device. Wherein, the method comprises the following steps: acquiring a flow control signaling sent by an interface machine, wherein the interface machine is used for acquiring audio and video data to be transmitted to a client from a server; responding to the flow control signaling to acquire a first rate of sending the audio and video data to the interface machine by the server; and sending audio data to the interface machine and discarding video data in the same picture group as the audio data when the first rate is less than a first preset threshold value. The invention solves the technical problem that the audio and video transmission is not smooth due to the poor network jitter resistance of the RTMP-based microphone connecting scheme.

Description

Data processing method and device, storage medium and electronic device

Technical Field

The invention relates to the field of computers, in particular to a data processing method and device, a storage medium and an electronic device.

Background

In the current one-to-one or many-to-many audio/video schemes, DC traffic based on a BGP (Border Gateway Protocol) network is mostly used, and UDP (User Datagram Protocol) based on a private scheme is used as a carrier for data transmission. The scheme has the characteristics of good network quality, high transmission efficiency of the private UDP protocol, strong network jitter resistance, various excellent flow control schemes and the like. However, BGP is expensive in bandwidth, i.e., the cost to the customer is high; and the application layer uses a customized private protocol, the SDK is diversified, the stickiness of a user to a merchant is too large, and the difficulty of analyzing problems is also large.

Based on this, a set of Real Time Message Protocol (RTMP) -based link-to-wire schemes is developed in the live broadcast field, expensive BGP bandwidth is replaced with low-cost edge bandwidth, and the original private UDP Protocol is replaced with the standard RTMP Protocol. The difficulty and the cost of client access are reduced. The disadvantages of this solution are also very evident: firstly, the integral end-to-end delay is about 1s higher than that of a UDP protocol; secondly, the RTMP Protocol is based on a TCP (Transmission Control Protocol) Protocol, and has poor network jitter resistance; under the condition of poor network state, data accumulation at a server end can be caused, and then the whole audio and video is not smooth and has a pause phenomenon.

In view of the above problems in the related art, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the invention provides a data processing method and device, a storage medium and an electronic device, which at least solve the technical problem that the audio and video transmission is not smooth due to the poor network jitter resistance of an RTMP-based microphone connecting scheme.

According to an aspect of an embodiment of the present invention, there is provided a data processing method, including: acquiring a flow control signaling sent by an interface machine, wherein the interface machine is used for acquiring audio and video data to be transmitted to a client from a server; responding to the flow control signaling to acquire a first rate of sending the audio and video data to the interface machine by the server; and if the first rate is less than a first preset threshold value, sending audio data to the interface machine, and discarding the video data in the same group of pictures (GOP) as the audio data.

According to another aspect of the embodiments of the present invention, there is also provided a data processing apparatus, including: the system comprises a first acquisition module, a first processing module and a second processing module, wherein the first acquisition module is used for acquiring a flow control signaling sent by an interface machine, and the interface machine is used for acquiring audio and video data to be transmitted to a client from a server; the second acquisition module is used for responding to the flow control signaling to acquire a first rate of sending the audio and video data to the interface machine by the server; and the first processing module is used for sending audio data to the interface machine and discarding video data positioned in the same group of pictures (GOP) as the audio data under the condition that the first rate is less than a first preset threshold value.

According to another aspect of the embodiments of the present invention, there is also provided a storage medium having a computer program stored therein, wherein the computer program is configured to execute the data processing method when running.

According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a memory and a processor, the memory having a computer program stored therein, the processor being configured to execute the above data processing method through the computer program.

In the embodiment of the invention, after the interface machine sends the flow control signaling, the network rate of the current terminal is detected to further determine the first rate of sending the audio and video data to the interface machine by the server, and under the condition that the first rate is less than the first preset threshold value, the audio data is sent to the interface machine, and the video data which is positioned in the same GOP (group of pictures) as the audio data is discarded, so that the aim of preferentially sending the audio and temporarily not sending the video data is fulfilled, and the technical problem that the audio and video transmission is not smooth due to the poor network jitter resistance of a RTMP-based microphone connecting scheme is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

fig. 1 is a schematic diagram of an application scenario of interactive live broadcast and live broadcast in the related art;

FIG. 2 is a diagram illustrating a structure of an interactive live broadcast and live broadcast in the related art;

FIG. 3 is a flow chart of a method for processing data according to an embodiment of the present invention;

FIG. 4 is a flow chart of a method of processing data according to an embodiment of the invention;

FIG. 5 is a schematic diagram of a RTMP-based link-wheat structure according to an embodiment of the present invention;

fig. 6 is a schematic diagram of a process of controlling current in a microphone based on the RTMP according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;

FIG. 8 is a first diagram illustrating an alternative configuration of a data processing apparatus according to an embodiment of the present invention;

FIG. 9 is a second schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present invention;

fig. 11 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

First, terms appearing in the present application are explained accordingly;

the types of video frames are divided into three types: 1) the method comprises the following steps of (1) directly decoding a key frame (I frame) and an intra-frame coding frame without reference of other frames to obtain a video image, wherein the frame data is large; 2) forward predictive coded frame (P frame): the frame needs to be decoded by referring to the previous frame, the image cannot be restored only by the frame data of the frame, and the frame data is usually small; 3) a bidirectional predictive interpolation-coded frame (B frame) requires reference to frames before and after decoding the frame, and cannot restore an image only by its own frame data, which is generally small.

GOP (Group of Pictures): a set of video frame data in a video stream, where there must be an I-frame start followed by several P-frames and B-frames until the next I-frame expires.

RTMP: an application layer protocol for addressing the multiplexing and packetization of multimedia data transport streams.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The technical scheme of the application is mainly applied to an interactive live broadcast wheat connecting scene, fig. 1 is a schematic view of an application scene of interactive live broadcast wheat connecting in the related art, and as shown in fig. 1, a user A can realize wheat connecting through a terminal A and a terminal B of a user B. Fig. 2 is a schematic view of an interactive live broadcast microphone connection structure in the related art, and as shown in fig. 2, a server in the existing live broadcast microphone connection technical scheme mainly includes the following modules: the system comprises an audio and video coding and decoding module, a flow control server and a flow pulling interface machine. Based on fig. 2, the method of the wheat connecting process in the related art includes:

step S1, the terminal A and the terminal B which connect to the wheat respectively receive and pull the flow data from the interface machine, the interface machine counts the RTT (Round Trip Time) and the packet loss rate data of the data transmission of the client;

step S2, the interface machine sends the packet loss rate and RTT to the flow control server;

step S3, the flow control server configures different coding parameters according to the network quality data of each terminal and sends the coding parameters to the interface machine;

and step S4, the interface machine issues stream data with different code rates and resolutions according to the strategy of the flow control server.

Through the steps S1 to S4, in the related art, the stream interface machine monitors in real time according to the network states of different terminals and transmits the network states to the stream control server, the stream control server calculates data issuing strategies in different scenes according to a preset stream control algorithm and returns the data issuing strategies to the stream interface machine, and the stream interface machine acquires corresponding audio and video data from the encoding and decoding server according to the issuing strategies of the different terminals and issues the audio and video data to the terminals. However, in the above-mentioned flow-control-free RTMP microphone-connecting scheme in the related art, a large delay may be caused due to a natural congestion control strategy of TCP, and thus audio and video data transmission is not smooth under a network jitter condition.

Based on the problems in the related art, the application provides a data processing method and device, a storage medium and an electronic device, and the method and device can achieve the effect that audio data are issued in real time in a scene with microphones, and important video data are issued for the second time or are not issued for the first time, so that the requirement of certain specific microphones on the video data is not high, such as insurance claims, finance and securities; under the condition that the network state is not good, remote interaction can still be smoothly realized through connecting the microphone.

According to an aspect of the embodiment of the present invention, there is provided a data processing method, and fig. 3 is a flowchart of the data processing method according to the embodiment of the present invention, as shown in fig. 3, the method includes:

step S202, a flow control signaling sent by an interface machine is obtained, wherein the interface machine is used for obtaining audio and video data to be transmitted to a client from a server;

it should be noted that, the client referred to in this application may be a handheld device (e.g., a mobile phone, a tablet), a computer, or the like. In addition, the interface machine related in the application is used for acquiring data from the server and sending the data to the client.

In an alternative embodiment of the present application, the step S202 may be implemented as follows:

step S202-11, acquiring the network state of data transmission between the interface machine and the client;

step S202-12, under the condition that the network state indicates that the client is congested, the flow control signaling sent by the interface machine is obtained.

It should be noted that, if the step S202 is applied in the continuous wheat scene, the client may refer to one of the continuous wheat clients or multiple continuous wheat clients, for example, if there are four clients in the current continuous wheat client, the network states of the four clients need to be monitored, and as long as the external network state of one of the clients indicates congestion, the interface machine will send a flow control signaling, that is, prepare for performing the flow control of the audio data and the video data as described below.

For example, an accident occurs to a user who currently owns the client a in the driving process, so that the own automobile is scratched; therefore, a user performs a microphone connecting operation to a terminal B of an agent of an insurance company through a client a, and in the process of performing claim settlement confirmation by microphone connecting, the network state of the client a is not good due to the fact that the position of the user is far away, and at this time, the client a and the client B are used for pulling off an interface machine of flow, and when the network state of the client a is not good, a flow control signaling is sent to a server. Of course, if the network status of the terminal B of the proxy is not good, the flow control signaling is also sent to the server. That is, the interface machine sends the flow control signaling to indicate that the current network state of the client is not a normal level, but as for whether flow control is to be performed, it is necessary to further determine the packet loss rate and the round trip time RRT of the data transmitted by the client, and further determine the first rate, which is referred to in the following step S204.

Certainly, the present application may also be applied to an application scenario in which more than two terminals are connected to the internet, or also take the car insurance claim as an example, after the user connects to the terminal B of the agent through the terminal a, the agent knows specific situations, and then performs corresponding processing on some of the problems, or does not have authority, and needs to report to the leader of the upper level, and then adds the leader terminal C of the agent to the internet scenario, and in the internet scenario of the three terminals, the network states of the three terminals need to be monitored to determine whether to send signaling to the server.

Step S204, responding to the flow control signaling to obtain a first rate of sending audio and video data to the interface machine by the server;

in an optional implementation manner of this step S204 in this application, the first rate is specifically determined according to a packet loss rate and a round trip time RRT of the client, and specifically may be a first rate at which the server sends the audio/video data to the interface according to the packet loss rate and the round trip time RRT of the transmission data of the client in response to the flow control signaling. If in the wheat-connecting scene, the packet loss rates and RRTs of a plurality of clients connected with wheat need to be calculated.

In step S206, when the first rate is less than the first predetermined threshold, the audio data is transmitted to the interface, and the video data in the same group of pictures GOP as the audio data is discarded.

Step S206-11, audio data and video data are split from the audio and video data, wherein the split audio data are put into an audio data queue, and the split video data are put into a video data queue;

step S206-12, adjusting the sending sequence of the audio data and the video data in the same GOP, wherein the adjusted sending sequence is used for indicating that the audio data in the same GOP is sent before the video data;

step S206-13 sends the audio data in the audio data queue to the interface machine according to the adjusted sending sequence, and discards the video data in the video data queue, which is in the same GOP as the audio data.

Through the above steps S206-11 to S206-13, when the first rate is less than the first predetermined threshold, that is, when the flow control is required for the delivered data, the video/audio data needs to be split, so that the audio data is preferentially sent later, and the video data is not first sent temporarily, thereby ensuring that the transmission of the audio data is preferentially ensured when the network state is not good.

For the above steps S206-11 to S206-13, in a specific application scenario of the car insurance claim, the interface machine may detect a network state of the client a, and then, when the first rate is calculated to be smaller than a first predetermined threshold, split audio and video data for data transmission between the client a and the client B, split the audio and video data into audio data and video data, place the audio data and video data into corresponding queues, and adjust sending sequences of the audio data and video data located in the same GOP; that is, in the scenario of vehicle insurance claim settlement, when the network status is relatively poor, the result of the last user and the agent performing the call connection through the terminal a and the terminal B is: both are connected by voice, but without video data. The operation of the whole automobile insurance claim settlement can be basically met through voice connection with the wheat, and the remote interaction of the user is not influenced. Under the condition, if the audio and video are not split and subjected to flow control, phenomena of blockage, frame dropping and the like of a microphone connecting picture can occur, video data cannot be met, and audio data cannot be guaranteed, so that the microphone connecting experience of a user is influenced.

As shown in fig. 4, in the embodiment of the present application, in the case that the network status gradually recovers, that is, after sending the audio data to the interface machine and discarding the video data located in the same GOP as the audio data, the method steps of this embodiment may further include the following method steps in addition to the method steps in fig. 3:

step S208, responding to the flow control signaling, and acquiring a second rate of sending the audio and video data to the interface machine by the server;

step S210, when the second rate is greater than the first predetermined threshold and less than the second predetermined threshold, selecting video data of a specified type frame from the video data queue located in the same GOP as the audio data, and simultaneously sending the selected video data of the specified type frame and the audio data in the audio data queue to the interface machine.

And step S212, merging the audio data queue and the video data queue under the condition that the second rate is greater than a second preset threshold value, and sending merged audio and video data to the interface machine.

As can be seen from steps S208 to S212, after the network status gradually improves (i.e., the second rate is greater than the first predetermined threshold and less than the second predetermined threshold), the audio data is preferentially transmitted, and at the same time, the video data with the frame of the specified type selected from the video data queue located in the same GOP as the audio data is also transmitted, and the frame of the specified type is preferably the I-frame data in the video data, so that the basic video link can be satisfied.

Taking the scene of the car insurance claim as an example, correspondingly exemplifying the steps S208 to S212, after a user connects a microphone with a terminal B of an agent through a terminal a, because more vehicles suffering from a traffic accident cause congestion, both a vehicle owner and a passenger on the congested vehicle use a mobile phone or other communication devices, and the network state at the moment cannot reach a normal level, based on the steps S202 to S206 in the present case, the terminal a and the terminal B can only realize audio microphone connection, but after a certain time, a traffic police processes the accident, and after the congestion is slowly relieved, a basic video call can be realized; after the traffic jam is completely solved, the network state is restored to a normal level, and at the moment, the terminal A and the terminal B can realize normal video call.

The following detailed description of the present application is provided as an alternative embodiment of the present application;

fig. 5 is a schematic diagram of an RTMP link structure according to an embodiment of the present invention, as shown in fig. 5, the link structure includes a client, a pull stream interface machine, and an RTMP server (the server referred to in this application is preferably an RTMP server); the pull interface machine is used for monitoring the packet loss rate and the RTT (round trip time) capability of the client in real time; the client is used for pulling data of the RTMP server; and the RTMP server is used for performing flow control data calculation, flow control strategy customization and flow data issuing.

Fig. 6 is a schematic diagram of a process of controlling a live traffic based on an RTMP, where based on fig. 6, the method includes:

and step S502, the client performs streaming to the streaming interface machine according to the continuous Mylar streaming address.

It should be noted that fig. 6 shows only one terminal, and of course, there are one or more other terminals connected to the microphone.

Step S504, the stream pulling interface machine tries network quality data from the computer interface machine to the client according to the network condition of the client, and judges whether a stream control strategy is started or not; and if the flow control strategy needs to be started, the flow control signal is sent to the RTMP server.

Step S506, after receiving the flow control signaling, the RTMP server counts the data issuing rate of the current client in real time, divides the original audio and video queue into two parts according to a preset algorithm, and issues the audio data in the same GOP group.

Step S508, when the flow control signal is continuously transmitted and the calculated transmission rate is smaller than a certain threshold, discarding the video data frame of the group to ensure the transmission continuity of the current audio and the subsequent audio.

Step S510, when the interface detects that the network state of the client is optimized, the interface sends corresponding network parameters to a server at the back end, and the server starts to resume sending partial video data according to a preset algorithm after sensing that the network state is optimized, wherein sending of I frame data is preferentially selected to be resumed to ensure basic video continuity.

Step S512, when the network state is restored to the normal level, the sending queues of the audio and the video are merged, and all the uplink data are issued.

Through the steps S502 to S512, based on the RTMP User control signaling, no special customization requirement of the client is required, and at the same time, all the flow control operations are completed at the server side, and the client side does not sense, so that the ordinary RTMP player can perform serial maraca streaming playing without customization. Secondly, the flow control algorithm is based on real-time network state monitoring and adjustment of the strategy of issuing audio and video data, meets the expectation of real-time audio issuing under the condition of an expected weak network, can also ensure the capability of quick recovery, and has good performance under the scenes of test environment and actual service.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

According to another aspect of the embodiment of the present invention, there is also provided a data processing apparatus for implementing the data processing method, and fig. 7 is a schematic structural diagram of the data processing apparatus according to the embodiment of the present invention, and as shown in fig. 7, the apparatus includes: the first obtaining module 62 is configured to obtain a flow control signaling sent by an interface machine, where the interface machine is configured to obtain, from a server, audio and video data to be transmitted to a client; the second obtaining module 64 is coupled with the first obtaining module 62, and is configured to obtain, in response to the flow control signaling, a first rate at which the server sends the audio/video data to the interface machine; and a first processing module 66, coupled to the second obtaining module 64, for sending the audio data to the interface machine and discarding the video data located in the same group of pictures GOP as the audio data if the first rate is less than a first predetermined threshold.

Fig. 8 is a schematic diagram illustrating an alternative structure of a data processing apparatus according to an embodiment of the present invention, and as shown in fig. 8, the first obtaining module 62 in the present application includes: a first obtaining unit 74, configured to obtain a network status of data transmission between the interface machine and the client; and the second obtaining unit 76 is coupled to the first obtaining unit 74, and is configured to obtain the flow control signaling sent by the interface machine when the network status indicates that the client is congested.

It should be noted that, if the first obtaining unit 74 and the second obtaining unit 76 are in the continuous wheat scene, the client may refer to one of the continuous wheat clients or multiple continuous wheat clients, for example, if there are four clients in the current continuous wheat client, the network states of the four clients need to be monitored, and as long as the external network state of one of the clients indicates congestion, the interface machine will send the flow control signaling.

For example, an accident occurs to a user who currently owns the client a in the driving process, so that the own automobile is scratched; therefore, a user performs a microphone connecting operation to a terminal B of an agent of an insurance company through a client a, and in the process of performing claim settlement confirmation by microphone connecting, the network state of the client a is not good due to the fact that the position of the user is far away, and at this time, the client a and the client B are used for pulling off an interface machine of flow, and when the network state of the client a is not good, a flow control signaling is sent to a server. That is to say, the interface machine sends the flow control signaling, that is, it indicates that the current network state of the client is not a normal level, but whether flow control is to be performed or not needs to further determine the packet loss rate and the round trip time RRT of data transmitted by the client, and further determine the first rate, and determine the first rate according to the result of comparing the first rate with the first predetermined threshold.

Optionally, the second obtaining module 54 in the present application is further configured to determine, in response to the flow control signaling, a first rate at which the server sends the audio/video data to the interface according to the packet loss rate and the round trip time RRT of the transmission data of the client.

The first rate is specifically determined according to the packet loss rate and the round trip time RRT of the client, and specifically may be a first rate at which the server sends the audio and video data to the interface according to the packet loss rate and the round trip time RRT of the transmission data of the client in response to the flow control signaling. If in the wheat-connecting scene, the packet loss rates and RRTs of a plurality of clients connected with wheat need to be calculated.

Fig. 9 is a schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present invention, and as shown in fig. 9, a first processing module 66 in the present application includes: the splitting unit 82 is configured to split audio data and video data from the audio and video data, where the split audio data is placed in an audio data queue, and the split video data is placed in a video data queue; an adjusting unit 84, coupled to the splitting unit 82, configured to adjust a transmission order of the audio data and the video data in the same GOP, where the adjusted transmission order is used to indicate that the audio data in the same GOP is transmitted before the video data; and the processing unit 86 is coupled to the adjusting unit 84, and is configured to send the audio data in the audio data queue to the interface according to the adjusted sending order, and discard the video data in the video data queue, which is located in the same GOP as the audio data.

As for the operations executed by the units in fig. 9, in a specific application scenario of the car insurance claim, the interface detects a network state of the client a, and then calculates that the first rate is smaller than the first predetermined threshold, the interface splits audio/video data for data transmission between the client a and the client B, splits the audio/video data into audio data and video data, places the audio data and video data into corresponding queues, and adjusts the sending sequence of the audio data and video data in the same GOP, that is, in the case that the network state is poor in the scenario of the car insurance claim, the last user and the agent perform a microphone connection through the terminal a and the terminal B, that is, the two perform microphone connection through voice, but do not have video data. The operation of the whole automobile insurance claim settlement can be basically met through voice connection with the wheat, and the remote interaction of the user is not influenced. Under the condition, if the audio and video are not split and subjected to flow control, phenomena of blockage, frame dropping and the like of a microphone connecting picture can occur, video data cannot be met, and audio data cannot be guaranteed, so that the microphone connecting experience of a user is influenced.

Fig. 10 is a schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present invention, and as shown in fig. 10, the apparatus in the present application may further include: a third obtaining module 92, coupled to the first processing module 66, configured to respond to the flow control signaling to obtain a second rate at which the server sends the audio and video data to the interface unit after sending the audio data to the interface unit and discarding the video data located in the same GOP as the audio data; a second processing module 94, coupled to the third obtaining module 92, configured to select, when the second rate is greater than the first predetermined threshold and smaller than the second predetermined threshold, video data of a specified type frame from a video data queue located in the same GOP as the audio data, and send the selected video data of the specified type frame and the audio data in the audio data queue to the interface simultaneously; and the third processing module 96, coupled to the second processing module 94, is configured to merge the audio data queue and the video data queue and send merged audio/video data to the interface machine when the second rate is greater than a second predetermined threshold.

Based on the operations performed by the modules in fig. 9, after the network status gradually improves (i.e. the second rate is greater than the first predetermined threshold and less than the second predetermined threshold), while the audio data is preferentially transmitted, the video data with the frame of the specified type selected from the video data queue located in the same GOP as the audio data is also transmitted, and the frame of the specified type is preferably the I-frame data in the video data, so as to satisfy the basic video link, and when the network body returns to the normal level (i.e. the second rate is greater than the second predetermined threshold), the audio data queue and the video data queue are merged, and the merged data is transmitted.

Taking the scene of the car insurance claim as an example, correspondingly exemplifying the operation executed by the module in fig. 9, after a user connects to a terminal B of an agent through a terminal a, because more vehicles with traffic accidents cause congestion, both owners and passengers of the congested vehicles use mobile phones or other communication devices, and the network state at the moment cannot reach a normal level, based on the steps S202 to S206 in the present case, the terminal a and the terminal B can only realize audio connection to the public, but after a certain time, a traffic police handles the accidents, and after the congestion is slowly relieved, the network state can be relieved, and a basic video call can be realized; after the traffic jam is completely solved, the network state is restored to a normal level, and at the moment, the terminal A and the terminal B can realize normal video call.

According to still another aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above-mentioned data processing method, as shown in fig. 11, the electronic device including: processor 1002, memory 1004, display 1006, user interface 1008, transmission 1010, sensor 1012, and the like. The memory has stored therein a computer program, and the processor is arranged to execute the steps of any of the above method embodiments by means of the computer program.

Optionally, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

s1, acquiring a flow control signaling sent by an interface machine, wherein the interface machine is used for acquiring audio and video data to be transmitted to a client from a server;

s2, responding to the flow control signaling, acquiring a first speed of sending audio and video data to the interface machine by the server;

s3, in the case where the first rate is less than the first predetermined threshold, transmitting the audio data to the interface, and discarding the video data located in the same group of pictures GOP as the audio data.

s1, acquiring the network state of data transmission between the interface machine and the client;

and S2, acquiring the flow control signaling sent by the interface machine under the condition that the network state indicates that the client is congested.

s1, splitting audio data and video data from the audio and video data, wherein the split audio data are put into an audio data queue, and the split video data are put into a video data queue;

s2, adjusting the sending sequence of the audio data and the video data in the same GOP, wherein the adjusted sending sequence is used for indicating that the audio data in the same GOP are sent before the video data;

and S3, sending the audio data in the audio data queue to the interface machine according to the adjusted sending sequence, and discarding the video data in the video data queue and positioned in the same GOP as the audio data.

Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 11 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 11 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 11, or have a different configuration than shown in FIG. 11.

The memory 1004 can be used for storing software programs and modules, such as program instructions/modules corresponding to the weather reproduction method and apparatus in the embodiment of the present invention, and the processor 1002 executes various functional applications and data processing, that is, implements the data processing method, by running the software programs and modules stored in the memory 1004. The memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1004 may further include memory located remotely from the processor 1002, which may be connected to the terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 1010 is used for receiving or transmitting data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 1010 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices so as to communicate with the internet or a local area Network. In one example, the transmission device 1010 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

The sensors 1012 are used to collect data. The display 1006 displays the network status, and the user interface 1008 is used for acquiring input operation instructions.

The embodiment of the invention provides a data processing scheme. After the flow control signaling is sent by the interface machine, the network rate of the current terminal is detected, the first rate of sending audio and video data to the interface machine by the server is further determined, the audio data is sent to the interface machine under the condition that the first rate is smaller than a first preset threshold value, and the video data which are located in the same GOP (group of pictures) as the audio data are discarded, so that the aim of preferentially sending the audio and temporarily not sending the video data is fulfilled, and the technical problem that the audio and video transmission is not smooth due to poor network jitter resistance of a RTMP-based microphone connecting scheme in the related technology is solved.

Embodiments of the present invention also provide a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.

Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:

Optionally, the storage medium is further arranged to store a computer program for performing the steps of:

Optionally, the storage medium is further configured to store a computer program for executing the steps included in the method in the foregoing embodiment, which is not described in detail in this embodiment.

Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A method for processing data, comprising:

acquiring a flow control signaling sent by an interface machine, wherein the interface machine is used for acquiring audio and video data to be transmitted to a client from a server;

responding to the flow control signaling to acquire a first rate of sending the audio and video data to the interface machine by the server;

under the condition that the first rate is smaller than a first preset threshold value, sending audio data to the interface machine, and discarding video data which are positioned in the same picture group with the audio data;

acquiring a flow control signaling sent by an interface machine, comprising:

acquiring a network state of data transmission between the interface machine and a client;

and acquiring the flow control signaling sent by an interface machine under the condition that the network state indicates that the client is congested.

2. The method of claim 1, wherein sending audio data to the interface engine and discarding video data in a same group of pictures as the audio data comprises:

splitting the audio data and the video data from the audio and video data, wherein the split audio data is put into an audio data queue, and the split video data is put into a video data queue;

adjusting the sending sequence of the audio data and the video data in the same picture group, wherein the adjusted sending sequence is used for indicating that the audio data in the same picture group is sent before the video data;

and sending the audio data in the audio data queue to the interface machine according to the adjusted sending sequence, and discarding the video data in the video data queue, which is positioned in the same picture group with the audio data.

3. The method of claim 1, wherein the obtaining a first rate at which the server sends the audio/video data to the interface machine in response to the flow control signaling comprises:

and responding to the flow control signaling, and determining a first rate of sending the audio and video data to the interface machine by the server according to the packet loss rate and the round trip time of the transmission data of the client.

4. The method of claim 1, wherein after sending audio data to the interface and discarding video data in the same group of pictures as the audio data, the method further comprises:

responding to the flow control signaling to acquire a second rate of sending the audio and video data to the interface machine by the server;

and under the condition that the second speed is greater than the first preset threshold and less than a second preset threshold, selecting video data of a specified type frame from a video data queue which is positioned in the same picture group with the audio data, and simultaneously sending the video data of the selected specified type frame and the audio data in the audio data queue to the interface machine.

5. The method of claim 4, wherein after sending audio data to the interface machine and discarding video data in the same group of pictures as the audio data, the method further comprises:

and under the condition that the second rate is greater than the second preset threshold value, merging the audio data queue and the video data queue, and sending merged audio and video data to the interface machine.

6. An apparatus for processing data, comprising:

the system comprises a first acquisition module, a first processing module and a second processing module, wherein the first acquisition module is used for acquiring a flow control signaling sent by an interface machine, and the interface machine is used for acquiring audio and video data to be transmitted to a client from a server;

the second acquisition module is used for responding to the flow control signaling to acquire a first rate of sending the audio and video data to the interface machine by the server;

the first processing module is used for sending audio data to the interface machine and discarding video data which are positioned in the same picture group with the audio data under the condition that the first rate is smaller than a first preset threshold value;

the first obtaining module comprises:

the first acquisition unit is used for acquiring the network state of data transmission between the interface machine and the client;

and the second obtaining unit is used for obtaining the flow control signaling sent by the interface machine under the condition that the network state indicates that the client is congested.

7. The apparatus of claim 6, wherein the first processing module comprises:

the splitting unit is used for splitting the audio data and the video data from the audio and video data, wherein the split audio data are put into an audio data queue, and the split video data are put into a video data queue;

an adjusting unit, configured to adjust a transmission order of the audio data and the video data in the same group of pictures, where the adjusted transmission order is used to indicate that the audio data in the same group of pictures is transmitted before the video data;

and the processing unit is used for sending the audio data in the audio data queue to the interface machine according to the adjusted sending sequence and discarding the video data in the video data queue, which is positioned in the same picture group with the audio data.

8. The apparatus of claim 6,

the second obtaining module is further configured to determine, in response to the flow control signaling, a first rate at which the server sends the audio and video data to the interface according to the packet loss rate and the round trip time of the transmission data of the client.

9. The apparatus of claim 6, further comprising:

the third acquisition module is used for responding to the flow control signaling to acquire a second rate of sending the audio and video data to the interface machine by the server after sending the audio data to the interface machine and discarding the video data in the same picture group as the audio data;

and the second processing module is used for selecting the video data of the specified type frame from the video data queue in the same picture group as the audio data and simultaneously sending the video data of the selected specified type frame and the audio data in the audio data queue to the interface machine under the condition that the second rate is greater than the first preset threshold and less than a second preset threshold.

10. The apparatus of claim 9, further comprising:

and the third processing module is used for merging the audio data queue and the video data queue under the condition that the second rate is greater than the second preset threshold value, and sending merged audio and video data to the interface machine.

11. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 5 when executed.

12. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 5 by means of the computer program.