CN112995697B

CN112995697B - Streaming data recovery method, server, storage medium and computer equipment

Info

Publication number: CN112995697B
Application number: CN202110483229.5A
Authority: CN
Inventors: 张梦涵
Original assignee: Wuhan Douyu Network Technology Co Ltd
Current assignee: Wuhan Douyu Network Technology Co Ltd
Priority date: 2021-04-30
Filing date: 2021-04-30
Publication date: 2021-09-07
Anticipated expiration: 2041-04-30
Also published as: CN112995697A

Abstract

The invention provides a stream data recovery method, a server, a storage medium and a computer device, comprising: receiving an initial signaling sent by a terminal; extracting key information from the streaming data based on the initial signaling; determining a target signaling based on the initial signaling and the key information, and storing the target signaling into a preset storage space; when a restart instruction is received and the restart is successful, reading a target signaling from a storage space and sending the target signaling to the target signaling; restoring the streaming data interaction between the terminal and the terminal based on the target signaling; therefore, the target signaling contains information for ensuring that the server and the terminal can normally carry out data interaction; when the live broadcast platform is restarted in the operation process, the server can read the target signaling from the storage space and send the target signaling to the server, and then the communication between the terminal and the server is recovered based on the key information; the server of the invention can not disconnect the instant communication between users when restarting, and ensure the instant communication efficiency of users in the live broadcast platform.

Description

Streaming data recovery method, server, storage medium and computer equipment

Technical Field

The invention belongs to the technical field of live instant messaging, and particularly relates to a streaming data recovery method, a server, a storage medium and computer equipment.

Background

The clients can perform audio and video Communication between Web Real-Time Communication (Webrtc) browsers, and audio and video stream management and distribution are provided by a Selective Forwarding Unit (SFU) of the server, so that media streams can be forwarded between the clients, and instant Communication between the clients is realized.

In the live platform, when version information needs to be updated or a server is unstable and needs to be restarted, the existing SFU does not support restarting. In the prior art, once the SFU server is restarted, instant messaging between users is immediately disconnected, and a call fails, which affects the instant messaging efficiency of users in a live broadcast platform.

Disclosure of Invention

Aiming at the problems in the prior art, the embodiment of the invention provides a streaming data recovery method, a server, a storage medium and computer equipment, which are used for solving the technical problem that in the prior art, when a live broadcast platform needs to perform version update or is unstable and needs to restart the server, instant messaging of a user is disconnected, and the instant messaging efficiency of the user is further influenced.

In a first aspect, the present invention provides a stream data recovery method, applied in an SFU server, where the method includes:

receiving an initial signaling sent by a terminal;

extracting key information from the streaming data based on the initial signaling; the key information is used for ensuring that the terminal and the server can normally carry out data interaction;

determining a target signaling based on the initial signaling and the key information, and storing the target signaling into a preset storage space;

when a restart instruction is received and the restart is successful, reading the target signaling from the storage space and sending the target signaling to the target signaling;

and restoring the streaming data interaction between the terminal and the terminal based on the target signaling.

Optionally, the extracting key information from the stream data based on the initial signaling includes:

if the terminal is a release terminal, extracting a first terminal key and a first server key which are mutually exchanged when the release terminal and the terminal perform handshake on the basis of the initial signaling;

when the issuing terminal transmits the issuing stream data of the current path based on a Secure real-time Transport Protocol (srtp), extracting first turning times of each serial number of issuing stream data packets in the issuing stream data of the current path based on the initial signaling.

if the terminal is a subscription terminal, extracting a second terminal key and a second server key which are mutually exchanged when the subscription terminal and the terminal perform handshake based on the initial signaling;

and when the current path of subscription streaming data is sent to the subscription terminal based on a secure real-time transport protocol srtp, extracting a synchronous source identifier in the current path of subscription streaming data and second turning times of each subscription streaming data packet serial number in the current path of subscription streaming data based on the initial signaling.

Optionally, the determining a target signaling based on the initial signaling and the key information includes:

and writing each piece of key information into a corresponding field in the initial signaling, and assigning values to each field to form the target signaling.

Optionally, the storing the target signaling into a preset storage space includes:

acquiring a release signaling and a subscription signaling in the target signaling;

determining a first active moment of the current path of the distribution stream data based on the distribution signaling;

determining a second active moment of the current path of subscription flow data based on the subscription signaling;

serializing the release signaling, the first active time, the subscription signaling and the second active time based on a preset serialization protocol;

and storing the serialized release signaling, the first active time, the subscription signaling and the second active time in the storage space in a key value table map mode.

Optionally, the restoring of streaming data interaction between the terminal and the terminal based on the target signaling includes:

extracting the key information from the target signaling;

if the terminal is a release terminal, configuring a first terminal key and a first server key in a handshaking process based on the key information, and skipping the handshaking process with the release terminal;

configuring first turnover times of each sequence number of the issuing stream data packet in a safe real-time transmission protocol based on the key information, and recovering communication between the issuing terminal and the issuing terminal; the safe real-time transmission protocol is a data transmission protocol agreed by the issuing terminal and the issuing terminal.

extracting the key information from the target signaling;

if the terminal is a subscription terminal, configuring a second terminal key and a second server key in the handshake process based on the key information; skipping a handshake process with the subscription terminal;

configuring a synchronous source identifier and a second turnover number of a sequence number of a subscription stream data packet in a secure real-time transmission protocol based on the key information, and recovering the communication between the subscription terminal and the subscription terminal; the secure real-time transmission protocol is a data transmission protocol agreed by the subscription terminal and the subscription terminal.

In a second aspect, the present invention provides a server, comprising:

a receiving unit, configured to receive an initial signaling sent by a terminal;

an extracting unit, configured to extract key information from the stream data based on the initial signaling; the key information is information for ensuring that the terminal and the server can normally carry out data interaction;

a determining unit, configured to determine a target signaling based on the initial signaling and the key information;

the storage unit is used for storing the target signaling into a preset storage space;

the sending unit is used for reading the target signaling from the storage space and sending the target signaling to the sending unit when a restart instruction is received and the restart is successful;

and the data recovery unit is used for recovering the streaming data interaction between the terminal and the data recovery unit based on the target signaling.

In a third aspect, the invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the first aspects.

In a fourth aspect, the invention provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the first aspects when executing the program.

The embodiment of the invention provides a stream data recovery method, a server, a storage medium and computer equipment, wherein the method comprises the following steps: receiving an initial signaling sent by a terminal; extracting key information from the streaming data based on the initial signaling; the key information is information for ensuring that the terminal and the server can normally carry out data interaction; determining a target signaling based on the initial signaling and the key information, and storing the target signaling into a preset storage space; when a restart instruction is received and the restart is successful, reading the target signaling from the storage space and sending the target signaling to the target signaling; restoring the streaming data interaction between the terminal and the terminal based on the target signaling; therefore, key information is extracted from the stream data according to the initial signaling, and then a target signaling is determined based on the key information and the initial signaling, so that information for ensuring that the server and the terminal can normally carry out data interaction exists in the target signaling; when the live broadcast platform is restarted in the operation process, the server can read a target signaling from the storage space and send the target signaling to the server, and then data stream interaction between the terminal and the server is recovered based on key information in the target signaling, so that communication between the terminal and the server is recovered; when the server of the live broadcast platform is restarted, no matter the anchor or audience users can not feel the audio and video data stream disconnection, the instant communication failure phenomenon of each user in the live broadcast platform is avoided, and the instant communication efficiency of the users is ensured.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a schematic diagram of a basic architecture of an SFU system according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a stream data recovery method according to an embodiment of the present invention;

fig. 3 is a schematic diagram of basic signaling used by a terminal to communicate with a server according to an embodiment of the present invention;

fig. 4 is a schematic diagram illustrating a streaming data recovery between a distribution terminal and a server according to an embodiment of the present invention;

fig. 5 is a schematic diagram illustrating a streaming data recovery between a subscriber terminal and a server according to an embodiment of the present invention;

fig. 6 is a schematic diagram illustrating an architecture of an SFU server, a main broadcasting end and a spectator end when the SFU server is applied to a live broadcasting scene in a live broadcasting platform according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a server according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a computer device according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a computer storage medium according to an embodiment of the present invention.

Detailed Description

The invention provides a streaming data recovery method, a server, a storage medium and computer equipment, and aims to solve the technical problem that in the prior art, when a live broadcast platform needs to perform version update or is unstable and needs to restart a server, instant communication of a user is disconnected, and the instant communication efficiency of the user is further influenced.

In order to better understand the technical solution of the embodiment of the present invention, the basic architecture of the SFU and the concept of hot update are introduced.

Referring to fig. 1, in the SFU system, one SFU server can access a plurality of users, and the user roles include: a publisher and a subscriber; the publisher is mainly used for publishing the audio and video stream data, and the subscriber can subscribe the corresponding audio and video stream data. The same user may be both a publisher and a subscriber, or may be only a publisher or only a subscriber. The role of the user depends on the specific business scenario.

For example, in a live broadcast and live broadcast scene, the anchor can be a publisher, and when a user terminal entering a live broadcast room of the anchor connects to the anchor, the user terminal is a subscriber.

The embodiment of the invention can complete the hot update of the SFU server in the normal operation of the live broadcast platform and the normal instant communication process among the users. For example, when the live platform needs to perform version update, the streaming data recovery method provided by the embodiment can avoid the failure of playing the audio/video stream accessed to the live platform. That is, the invention can avoid the disconnection of the instant communication between the users when the server is restarted.

The technical solution of the present invention is further described in detail by the accompanying drawings and the specific embodiments.

The embodiment provides a stream data recovery method, which is applied to an SFU server, and as shown in fig. 2, the method includes the following steps:

s210, receiving an initial signaling sent by a terminal;

in a specific implementation, when the (Webrtc) terminal needs to communicate with the (SFU) server and performs streaming data (audio or video) transmission, the terminal sends an initial signaling to the server, so that the server can receive the initial signaling sent by the terminal. In order to ensure that the server resumes streaming data interaction (which may be understood as hot update) between the terminal and the server when restarting, the initial signaling in this embodiment includes a key information field for extracting key information, and after receiving the initial signaling, the server extracts the key information from the streaming data based on the key information field in the initial signaling. First, the initial signaling needs to include basic signaling for the terminal to communicate with the server, and the basic signaling may be as shown in fig. 3.

As can be seen from fig. 3, the basic signaling includes: access signaling, create _ offer signaling (which may be understood as request signaling), create _ answer signaling (which may be understood as a result returned by the server), candidate signaling sent by the terminal to the server, candidate signaling sent by the server to the terminal, dtls handshake (handshake) signaling, and audio-video data signaling.

For convenience of description, in the embodiment of the present invention, access signaling, create _ offer signaling, create _ answer signaling, candidate signaling sent by the terminal to the server, candidate signaling sent by the server to the terminal, and dtls handshake (handhandshake) signaling are combined into one signaling, and a unified description is performed, where a basic signaling format is as follows:

message Sdpinfo// Sdp information

{

bytes sdp = 1// sdp content, the next two fields are needed when candidate is represented

bytes sdp _ mid = 2// sdp tag

int32 sdp _ mline _ index = 3// sdp number

}

Wherein Sdp information includes three fields: sdp, sdp _ mid, and sdp _ mline _ index.

The sdp represents specific signaling data for the terminal to perform streaming media data interaction. For example, sdp may be an access signaling, a create _ offer signaling, a create _ answer signaling, or the like; i.e. the content of sdp may be any of the basic signalling described above. When sdp is candidate, candidate is a part of the specific content of sdp, and two signaling data, sdp _ mid and sdp _ mline _ index, are needed. When sdp is not candidate, two signaling data, sdp _ mid and sdp _ mline _ index, are not needed subsequently.

Among them, candidate address refers to an IP address or port required for each terminal to communicate with the server.

sdp _ mid is a label for sdp content; for example, when sdp is candidate, sdp _ mid is a label for candidate.

The sdp _ mline _ index is the number of the sdp; for example, when sdp is candidate, sdp _ mline _ index is the number of candidate. Since multiple candidates may be used during streaming data transmission, the last candate will set the corresponding end number, for example, when sdp _ mline _ index is-1, it represents that the candate is the last candate.

It is to be noted that 1, 2, and 3 in the above SdpInfo represent the number of each piece of signaling data.

Further, as described above, since the SFU architecture includes the publisher and the subscriber, the terminal in this embodiment includes the publishing terminal and the subscribing terminal, and both the publishing terminal and the subscribing terminal have corresponding initial signaling.

Specifically, the initial signaling corresponding to the terminal is issued as follows:

message PublishStreamRequest

{

SdpInfo sdp_info1 = 1; // sdp offer

repeated SdpInfo ice_candidates 1= 2; // candidates

bytes client_key1 = 3; // dtls client_key

bytes server_key1 = 4; // dtls server_key

map < U > 32, U > 32> receiving _ rocs = 5;/dtls receive roc, ssrc- > roc

}

Wherein, sdp _ info1 is the above-mentioned Sdp information, and here refers to Sdp information corresponding to the distribution stream data;

ice _ candidates1 are candidate information corresponding to the streaming data issued by the issuing terminal, and the candidate information may include a plurality of candidates;

the client _ key1 is a first terminal key obtained by exchanging when dtls handshake is performed when the issuing terminal establishes connection with the server;

the Server _ key1 is a first Server key obtained by exchanging during dtls handshake when the issuing terminal establishes connection with the Server;

the receiving _ rocs is the first turnover number of the sequence number of each distribution stream data packet in the distribution stream data.

Here, when the distribution terminal transmits the current distribution stream data to the server, in order to improve the transmission efficiency, the distribution terminal transmits the data packets in a fragmentation transmission manner, and each data packet has a corresponding sequence number. The transmission protocol specifies that the range of the sequence number is 0-65535, that is, the maximum sequence number which can be used by the data packet is 65535. If the sequence number of the current data packet is 65535, the sequence number of the next data packet of the current data packet needs to be set to zero, that is, the sequence number of the next data packet of the current data packet is 0; which is considered to be a sequence number rollover. The current path of distribution stream data is any path of distribution stream data of all paths.

For example, if the sequence number of the distribution stream packet is inverted 3 times when the distribution stream data is transmitted, the receiving _ rocs is 3.

It can be seen that, in this embodiment, the key information fields carried in the initial signaling of the issuing terminal include: a first terminal key field, a first server key field, and a first number of flips field.

Correspondingly, after the server receives the initial signaling sent by the issuing terminal, there is a corresponding response signaling, which is specifically as follows:

message publishing StreamResponse// issuing terminal corresponding response signaling

{

int32 result1 = 1; // result

string stream _ id1 = 2// issue stream id

SdpInfo sdp_info 2= 3; // sdp answer

repeated SdpInfo ice_candidates2 = 5; // candidates

}

Wherein, result1 is a transmission result of the initial signaling, and different values can be used to identify the transmission result, for example, when the value of result2 is 0, it indicates that the transmission is successful; when the value of result2 is not 0, it represents a transmission error.

stream _ ID1 is the first data ID of each issue stream data;

the sdp _ info1 is Sdp information returned by the server to the publishing terminal;

ice _ candidates2 are candidates information returned from the server to the distribution terminal;

similarly, when the terminal is a subscriber terminal, the initial signaling corresponding to the subscriber terminal is as follows:

initial signaling of message SubscribeStreamRequest// subscribing terminal

{

SdpInfo sdp_info3 = 1; // sdp offer

repeated SdpInfo ice_candidates 3= 2; // candidates

string stream _ id2 = 3// subscription stream id

bytes client_key2 = 4; // dtls client_key

bytes server_key2 = 5; // dtls server_key

U int32 audio _ sink _ ssrc = 6// audio ssrc forwarded to subscribing terminal

U int32 video _ sink _ ssrc = 7// video ssrc forwarded to subscribing terminal

map < U > 32, U < U > 32> sending _ rocs = 8,// dtls sending roc, ssrc- > roc

}

Wherein, the sdp _ info3 is Sdp information corresponding to the subscription stream data;

ice _ candidates 3 are candidate information corresponding to the subscriber terminal, and may include a plurality of ones;

stream _ ID2 is the second data ID of each path of the subscription stream data;

the client _ key2 is a second terminal key obtained by exchanging when dtls handshake is performed when the subscription terminal establishes connection with the server;

server _ key2 is a second server key exchanged during dtls handshake when the subscription terminal establishes connection with the server;

the Audio _ sink _ ssrc subscribes to a synchronous source identifier ssrc in the Audio data;

the video _ sink _ ssrc subscribes to a synchronization source identifier ssrc in the video data;

the sending _ rocs is a second turning frequency of the sequence number of each subscription stream data packet in the subscription stream data.

Similarly, when the server transmits the current path of subscription stream data to the subscriber terminal, in order to improve the transmission efficiency, the data packets are transmitted in a fragmentation transmission mode, and each data packet has a corresponding serial number. The transmission protocol specifies that the range of the sequence number is 0-65535, that is, the maximum sequence number which can be used by the data packet is 65535. If the sequence number of the current data packet is 65535, the sequence number of the next data packet of the current data packet needs to be set to zero, that is, the sequence number of the next data packet of the current data packet is 0; which is considered to be a sequence number rollover. The current path subscription stream data is any path in all path subscription stream data.

For example, if the sequence number of the subscription stream data packet is inverted 3 times when the subscription stream data is transmitted, the sending _ rocs is 3.

It can be seen that, in this embodiment, the key information fields carried in the initial signaling of the subscribing terminal include: a second terminal key field, a second server key field, a second turnover number field and a synchronous source identifier ssrc field in each path of subscription stream data; wherein subscribing to streaming data comprises: subscription audio data and subscription video data.

After the server receives the initial signaling sent by the subscriber terminal, there is also a corresponding response signaling, which is specifically as follows:

response signaling corresponding to message SubscribeStreamResponse// subscribing terminal

{

int32 result2 = 1; // result

SdpInfo sdp_info4 = 2; // sdp answer

repeated SdpInfo ice_candidates4 = 3 // candidates;

}

Wherein Result 2: for the transmission result of the subscription signaling, different values may be used to identify the transmission result, such as when the value of result2 is 0, indicating that the subscription signaling is successfully transmitted; when the value of result2 is not 0, an error is signaled on behalf of the subscription.

Sdp _ info4 Sdp information returned by the server to the subscribing terminal;

ice _ candidates 4. the server returns candidate information to the subscriber terminal.

It should be noted that the numerical codes (1-8) at the right side of the equal sign in the initial signaling and the response signaling represent the number of each signaling.

In this step, since the initial signaling of the publishing terminal and the subscribing terminal carries the key information fields to be extracted, after the server receives the initial signaling, corresponding key information is extracted based on the key information fields, and is used for subsequent streaming data interaction between the terminal and the server, so as to ensure successful data interaction.

S211, extracting key information from the stream data based on the initial signaling; the key information is used for ensuring that the terminal and the server can normally carry out data interaction;

in a specific implementation of this embodiment, the stream data is audio/video stream data, and the key information is information for ensuring that the terminal and the server can perform data interaction of the audio/video stream data normally;

after the server receives the initial signaling sent by the terminal, the server can extract the key information from the stream data based on the initial signaling because the initial signaling carries the key information field to be extracted; the key information is used for ensuring that the terminal and the server can normally carry out data interaction of audio and video stream data.

In the embodiment of the present invention, when the terminal is a publishing terminal, it can be seen from the above that the key information field carried in the initial signaling of the publishing terminal includes: a first terminal key field, a first server key field, and a first number of flips field. Thus extracting key information from the streaming data based on the initial signaling, including:

if the terminal is a release terminal, extracting a first terminal key and a first server key which are mutually exchanged when the release terminal and the terminal perform handshake with each other based on the initial signaling;

when the issuing terminal transmits the issuing stream data of the current path based on the secure real-time transport protocol srtp, extracting the first turning times of the serial numbers of all issuing stream data packets in the issuing stream data of the current path based on the initial signaling.

When the terminal is a subscription terminal, the key information fields carried in the initial signaling of the subscription terminal include: a second terminal key field, a second server key field, a second turnover number field and a synchronous source identifier ssrc field in each path of subscription stream data; subscribing to streaming data includes: subscription audio data and subscription video data.

Therefore, extracting key information from the streaming data based on the initial signaling includes:

if the terminal is a subscription terminal, the server extracts a second terminal key and a second server side key which are mutually exchanged when the subscription terminal and the server perform handshake based on the initial signaling;

and when the current path of subscription streaming data is sent to the subscription terminal based on the secure real-time transport protocol srtp, extracting the synchronous source identifier in the current path of subscription streaming data and the second turnover times of each subscription streaming data packet serial number in the current path of subscription streaming data based on the initial signaling. Wherein, the synchronous source identifier in the current path subscription stream data comprises: a synchronous source identifier of the current path of subscription audio data and a synchronous source identifier of the current path of subscription video data.

In this step, the server may extract the key information in the stream data based on the initial signaling, so as to ensure that the normal communication between the server and the terminal can be recovered by using the key information when the server is restarted subsequently.

S212, determining a target signaling based on the initial signaling and the key information, and storing the target signaling into a preset storage space;

after the key information is extracted, the target signaling can be determined based on the initial signaling and the key information, and the target signaling is stored or cached in a preset storage space. The preset storage space may be a local disk of the SFU server or a third-party storage, which is not limited herein. In the embodiment, the restart efficiency is considered, the target signaling is cached in the local disk, the data reading efficiency is improved, and the restart time is shortened.

In this embodiment, determining the target signaling based on the initial signaling and the key information includes:

and the server writes each piece of key information into a corresponding field in the initial signaling to assign each field, and the assigned initial signaling forms a target signaling.

For example, for the publishing terminal, the extracted first terminal key is 23456, the first server key is abcdef, and the first number of times of flipping is 4; then "23456" is assigned to client1, "abcdef" is assigned to Server _ key1, and "4" is assigned to receiving _ rocs; and the assigned initial signaling forms the target signaling.

It can be seen that the format of the target signaling is substantially the same as that of the initial signaling, and the only difference is that the target signaling carries the key information, and the initial signaling carries the key information field; therefore, the server can analyze the target signaling and the initial signaling by using the same analysis mode, and when the server is restarted subsequently, an additional analysis mode does not need to be configured, so that the analysis efficiency of the server is improved, and the workload of developers is reduced.

After the target signaling is determined, the target signaling needs to be stored in a preset storage space, so that when the SFU server is restarted, the target signaling cached locally can be loaded, the server actively sends the target signaling to the server, and all streaming data are recovered.

In order to improve the storage efficiency and reduce the storage space, the server may utilize a serialization protocol (such as protobuf3 protocol) to serialize the target signaling into binary or json data for storage, and this storage process may also be understood as a process of persisting the target signaling.

In the embodiment of the present invention, storing a target signaling into a preset storage space includes:

the server acquires a release signaling and a subscription signaling in a target signaling;

determining a second active moment of the current path of subscription stream data based on the subscription signaling;

and storing the serialized release signaling, the first active time, the subscription signaling and the second active time in a storage space in a key value table map mode.

Specifically, each target signaling corresponds to one path of stream data, and for a current path of release stream data, signaling persistence data of the corresponding target signaling is as follows:

message publishing Persistentinfo/signaling persistent data corresponding to current path issue stream data

{

Publish StreamRequest publish _ request = 1// the current path publishes the corresponding release signaling of the stream data

int64 last _ active1_ time _ ms = 2// first active time (unix time) in unit of milliseconds

bytes last _ active1_ time _ human = 3// first active (human ready)

}

Wherein, publish _ request is a release signaling;

last _ active1_ time _ ms is the first active time (unix timestamp) of the current path publishing stream data, the first active time is the data format for communicating with the background service, and the unit is ms;

last _ active1_ time _ human is the first active time (human ready) of the current path publishing stream data, and the format is for the user to view, and is a readable format, for example, the format may be year, month, day, etc., or specific time minute and second.

For the current path subscription data, the signaling persistence data of the corresponding target signaling is as follows:

message subscription persistence info// signaling persistence data corresponding to current path subscription data

{

SubscriptionStreamRequest sub _ request = 1;/subscription signaling corresponding to current path subscription data

int64 last _ active2_ time _ ms = 2// second active time (unix time) in unit of ms

bytes last _ active2_ time _ human = 3// second active time (human ready)

}

Wherein, the subscribe _ request is a subscription signaling;

last _ active2_ time _ ms is a second active time (unix timestamp) of the current path subscription stream data, and the second active time is in a data format for communicating with the background service and has a unit of ms;

last _ active2_ time _ human is for the second active time (human ready) of the current path subscription stream data to be viewed by the user, and is in a readable format, for example, the format may be year, month, day, or the like, or specific time minute and second.

And the first active time and the second active time are the last active time before the current time.

Then the server stores the signaling persistence data corresponding to the current path of published stream data and the signaling persistence data corresponding to the current path of subscribed stream data in a map in a stream persistence mode, and the codes are implemented as follows:

message stream DataPersistence item// stream persistence data

{

PublishpersistenceInfo = 1// signaling persistent data corresponding to current path issue stream data

map<string, SubscribePersistenceInfo> subscribers = 2; // subscribr_id to subscriber

}

The Persistence Info publisher current path issues signaling persistent data corresponding to the streaming data;

the subscriptions are signaling persistent data corresponding to the current path of subscription stream data, the subscription _ ID is a subscription ID, and each subscription ID corresponds to one subscription terminal.

Furthermore, because the live broadcast platform generally includes a plurality of publishing terminals and subscribing terminals, it is necessary to continuously store all the publishing stream data of all the publishing terminals in the map for each publishing terminal; for each subscription terminal, all the path subscription stream data of each subscription terminal needs to be stored (fully persisted) in the map so as to store all the signaling persisted data, and ensure that the server can successfully recover all the stream data when restarting, and the code is implemented as follows:

message StreamDataPersistence

{

map<string, StreamDataPersistenceItem> items = 1; // stream_id to stream

}

wherein items represents signaling persistent data corresponding to all paths of release stream data and signaling persistent data corresponding to all paths of subscription stream data; and stream _ ID is an ID corresponding to one stream of data, and each stream _ ID corresponds to one piece of signaling persistent data.

In the step, the signaling persistent data corresponding to all the paths of published stream data and the signaling persistent data corresponding to all the paths of subscribed stream data are stored in a preset storage space in a serialized mode, so that when the SFU server is restarted, a target signaling cached locally can be loaded, the server actively sends the target signaling to the server, and all the stream data are recovered.

Moreover, the way of serializing and then storing the signaling data is equivalent to converting the target signaling into binary data for storage, so that the storage efficiency can be improved, the occupied space is reduced, and the serialized data support is readable, so that the server is convenient to search, the signaling persistent data corresponding to a certain path of streaming data is prevented from being leaked and stored, and the streaming data interaction between the terminal and the server can be successfully recovered.

S213, when a restart instruction is received and the restart is successful, reading the target signaling from the storage space and sending the target signaling to the self;

specifically, when the SFU server checks that the version information to be updated exists in the live broadcast platform or detects that the communication state is an unstable state, the SFU server generates a service SFU service restart instruction and sends the target signaling to the SFU server;

it can be understood that, when the live platform needs to update version information or needs to restart the server because the server is unstable, the embodiment may restart the live platform in the instant messaging process of the user. And when the server receives the restart instruction and is restarted successfully, reading the target signaling from the storage space and sending the target signaling to the server.

In this embodiment, the step of sending the target signaling to the server includes:

and creating a signaling sending thread, wherein the signaling sending thread is used for calling an external interface to send a target signaling to the signaling sending thread. The external interface is a data interface for performing data interaction between the server and the external device, for example, the external interface may be a HyperText Transfer Protocol (HTTP) interface or a Remote Procedure Call Protocol (RPC) interface.

In this step, when the SFU server is restarted, the server actively sends a target signaling to itself to recover all stream data. Since webrtc uses udp protocol for audio and video transmission, the restart of the server does not cause the failure of connection immediately, and the server can continue to interact with the terminal as long as the key information in the connection is recorded and recovered, thereby avoiding the failure of instant messaging between users in the live broadcast platform.

S214, restoring the streaming data interaction between the terminal and the terminal based on the target signaling.

And when the server receives the target signaling, restoring the streaming data interaction between the server and the terminal based on the target signaling.

If the terminal is a publishing terminal, referring to fig. 4, the method for restoring the streaming data interaction between the terminal and the server based on the target signaling in the embodiment of the present invention includes:

extracting key information from the target signaling;

if the terminal is a release terminal, configuring a first terminal key and a first server key in a handshake process based on key information; directly skipping the handshake process with the issuing terminal;

configuring first turning times of each serial number of the issuing stream data packet in a secure real-time transmission protocol srtp based on the key information to ensure that data can be decrypted normally, wherein the secure real-time transmission protocol is a data transmission protocol agreed by an issuing terminal and the issuing terminal (server); in this way, all the key information is consistent with the key information before restarting, so that the communication between the issuing terminal and the issuing terminal can be recovered; transmission and decryption of the distribution stream data is resumed.

If the terminal is a subscription terminal, referring to fig. 5, the method for restoring the streaming data interaction between the terminal and the server based on the target signaling in the embodiment of the present invention includes:

extracting key information from the target signaling;

if the terminal is a subscription terminal, configuring a synchronous source identifier in a safe real-time transmission protocol based on the key information; ensuring that the synchronization source identifier is consistent with the synchronization source identifier prior to restart;

configuring a second terminal key and a second server key in the handshake process based on the key information, and skipping the handshake process with the subscription terminal;

configuring a second turning frequency of each subscription stream data packet serial number in a secure real-time transmission protocol based on the key information, wherein the secure real-time transmission protocol is a data transmission protocol agreed by the subscription terminal and the secure real-time transmission protocol; in this way, it is equivalent to all the critical information being consistent with that before the restart, so that the communication between the subscribing terminal and itself can be restored; resuming transmission and decryption of the subscription stream data.

Wherein the synchronization source identifier comprises: a synchronization source identifier in the audio data and a synchronization source identifier in the video data.

For example, if the SFU server is applied in a live-microphone scenario of a live platform of a fish fighter, referring to fig. 6, the SFU server can be connected to one publishing terminal 61 (anchor) and can be connected to one or more subscribing terminals 62 (viewers).

Furthermore, aiming at the current path of distribution stream data and the current path of subscription stream data, the target signaling carries the first active time of the current path of distribution stream data and the second active time of the current path of subscription stream data; therefore, the activity of the current path of the published stream data can be determined according to the first active time and the activity of the current path of the subscribed stream data can be determined according to the second active time.

Specifically, the method further comprises:

determining a first active duration of the current path of the release streaming data based on the first active time and the current time aiming at the current path of the release streaming data;

for any path of subscription stream data, determining a second active duration of the current path of subscription stream data based on a second active time and the current time;

if the first active duration is determined to be smaller than a preset active threshold, deleting the current path of release stream data;

and if the second active duration is smaller than the active threshold, deleting the current path of subscription stream data.

Therefore, the streaming data with low activity is deleted in time, the memory space is increased, and the processing efficiency of the server is improved.

In this embodiment, a communication session between a publishing terminal and a subscribing terminal does not fail, when the SFU server of this embodiment restarts and restarts successfully, the SFU server triggers sending a target signaling to itself, and performs recovery of streaming data in a very short time based on the target signaling, which has the following final beneficial effects: the publishing terminal and the subscribing terminal are disconnected only temporarily, and the user at the publishing terminal side or the user at the subscribing terminal side can not perceive the data disconnection subjectively, so that the method belongs to a non-perception technical scheme; the technical scheme of the embodiment is applied to the technical field of live broadcast, and the phenomenon that the watching experience of a user is influenced because audio and video data of a user terminal is disconnected due to the restart of the service of the live broadcast platform in the instant messaging process of each user in the live broadcast platform can be avoided. When a live broadcast platform needs to be updated or an SFU server is restarted for other reasons, no matter a main broadcast or audience users cannot feel that audio and video data streams are disconnected, the instant messaging failure phenomenon of each user in the live broadcast platform is avoided, and the instant messaging efficiency of the users is ensured.

Based on the same inventive concept as the foregoing embodiment, an embodiment of the present invention further provides a server, as shown in fig. 7, where the server includes:

a receiving unit 71, configured to receive an initial signaling sent by a terminal;

an extracting unit 72 for extracting key information from the stream data based on the initial signaling; the key information is information for ensuring that the terminal and the server can normally carry out data interaction;

a determining unit 73, configured to determine a target signaling based on the initial signaling and the key information;

a storage unit 74, configured to store the target signaling into a preset storage space;

a sending unit 75, configured to, when a restart instruction is received and a restart is successful, read a target signaling from a storage space and send the target signaling to itself;

and a data recovery unit 76, configured to recover the streaming data interaction between the terminal and itself based on the target signaling.

Since the server described in the embodiment of the present invention is a server used for implementing the method in the embodiment of the present invention, based on the method described in the embodiment of the present invention, those skilled in the art can understand the specific structure and the deformation of the server, and thus, details are not described herein. All servers used in the method of the embodiment of the present invention belong to the protection scope of the present invention.

Based on the same inventive concept as the previous embodiment, the present embodiment provides a computer apparatus, as shown in fig. 8, including a memory 810, a processor 820 and a computer program 811 stored on the memory 810 and operable on the processor 820, wherein the processor 820 implements the following steps when executing the computer program 811:

receiving an initial signaling sent by a terminal;

In particular, when the processor 820 executes the computer program 811, any one of the first embodiment can be implemented.

Since the computer device described in this embodiment is a device used for implementing a stream data recovery method in the foregoing embodiment of the present invention, based on the method described in the first embodiment of the present invention, a person skilled in the art can understand a specific implementation manner of the computer device in this embodiment and various variations thereof, so that a detailed description of how the server implements the method in the embodiment of the present invention is not provided here. The equipment used by those skilled in the art to implement the method in the embodiments of the present invention is within the scope of the present invention.

Based on the same inventive concept as the previous embodiments, the present embodiment provides a computer-readable storage medium 900, as shown in fig. 9, on which a computer program 911 is stored, the computer program 911 when executed by a processor realizes the following steps:

receiving an initial signaling sent by a terminal;

In a specific implementation, the computer program 911 may be implemented to implement any one of the above embodiments when executed by a processor.

The streaming data recovery method, the server, the storage medium and the computer device provided by the embodiment of the invention have the beneficial effects that at least:

the embodiment of the invention provides a stream data recovery method, a server, a storage medium and computer equipment, wherein the method comprises the following steps: receiving an initial signaling sent by a terminal; extracting key information from the streaming data based on the initial signaling; the key information is information for ensuring that the terminal and the server can normally carry out data interaction; determining a target signaling based on the initial signaling and the key information, and storing the target signaling into a preset storage space; when a restart instruction is received and the restart is successful, reading the target signaling from the storage space and sending the target signaling to the target signaling; restoring the streaming data interaction between the terminal and the terminal based on the target signaling; therefore, key information is extracted from the stream data according to the initial signaling, and then a target signaling is determined based on the key information and the initial signaling, so that information for ensuring that the server and the terminal can normally carry out data interaction exists in the target signaling; when the live broadcast platform is restarted in the operation process, the server can read a target signaling from the storage space and send the target signaling to the server, and then data stream interaction between the recovery terminal and the server is carried out on the data based on key information in the target signaling, so that the communication between the terminal and the server is recovered; according to the embodiment of the invention, when the server of the live broadcast platform is restarted, the instant communication among the users is not disconnected, and the instant communication efficiency of the users in the live broadcast platform is ensured.

The publishing terminal and the subscribing terminal are disconnected only temporarily, and the user at the publishing terminal side or the user at the subscribing terminal side can not perceive the data disconnection subjectively, so that the method belongs to a non-perception technical scheme; the technical scheme of the embodiment is applied to the technical field of live broadcast, and the phenomenon that the watching experience of a user is influenced because audio and video data of a user terminal is disconnected due to the restart of the service of the live broadcast platform in the instant messaging process of each user in the live broadcast platform can be avoided. When a live broadcast platform needs to be updated or an SFU server is restarted for other reasons, no matter a main broadcast or audience users cannot feel that audio and video data streams are disconnected, the instant messaging failure phenomenon of each user in the live broadcast platform is avoided, and the instant messaging efficiency of the users is ensured.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

The above description is only exemplary of the present invention and should not be taken as limiting the scope of the present invention, and any modifications, equivalents, improvements, etc. that are within the spirit and principle of the present invention should be included in the present invention.

Claims

1. A stream data recovery method, applied in a selective forwarding unit SFU server, the method comprising:

receiving an initial signaling sent by a terminal;

2. The method of claim 1, wherein said extracting key information from streaming data based on said initial signaling comprises:

and when the issuing terminal transmits the issuing stream data of the current path based on a secure real-time transport protocol srtp, extracting the first turning times of the serial numbers of all issuing stream data packets in the issuing stream data of the current path based on the initial signaling.

3. The method of claim 1, wherein said extracting key information from streaming data based on said initial signaling comprises:

4. The method of claim 1, wherein the determining target signaling based on the initial signaling and the key information comprises:

5. The method of claim 1, wherein the storing the target signaling into a preset storage space comprises:

6. The method of claim 1, wherein the resuming streaming data interaction between the terminal and itself based on the target signaling comprises:

extracting the key information from the target signaling;

7. The method of claim 1, wherein the resuming streaming data interaction between the terminal and itself based on the target signaling comprises:

extracting the key information from the target signaling;

8. A server, characterized in that the server comprises:

9. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 7.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 7 when executing the program.