CN115209199A

CN115209199A - Media data processing method and device, terminal equipment and storage medium

Info

Publication number: CN115209199A
Application number: CN202210653777.2A
Authority: CN
Inventors: 谢文龙; 李云鹏; 周新海; 王艳辉
Original assignee: Visionvera Information Technology Co Ltd
Current assignee: Visionvera Information Technology Co Ltd
Priority date: 2022-06-10
Filing date: 2022-06-10
Publication date: 2022-10-18

Abstract

The embodiment of the invention provides a method, a device, terminal equipment and a storage medium for processing media data, wherein the method comprises the following steps: receiving a playing parameter change instruction sent by a server in the process of playing the media stream at a first adjusting time by the terminal equipment; determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; according to the first adjusting time and the switching time stamp, determining the playing time required by the media stream data which is not played before the switching time stamp is played; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; and according to the adjustment transition strategy, playing the media stream data which is not played before the time stamp is switched, and the terminal equipment compares the new adjustment time with the time required for playing the cached audio and video data at the original audio and video frame rate to determine the adjustment transition strategy until the terminal equipment stably plays the data at the new audio and video frame rate.

Description

Media data processing method and device, terminal equipment and storage medium

Technical Field

The present invention relates to the field of video networking technologies, and in particular, to a method and an apparatus for processing media data, a terminal device, and a storage medium.

Background

The video network is a private protocol technology under a proprietary network, and at present, the video network gathers a large number of media resources, such as monitoring resources of social environment, live network resources, conference docking resources, and terminal resources under own protocols, and terminal point location resources accumulate day by day to form huge resources. However, at present, these media resources are limited to a private protocol, and have a certain difficulty in interfacing with a public standard protocol, and the degree of fusion is relatively weak.

If the media resource is to be played on an internet page, technologies such as Real Time Messaging Protocol (RTMP), real Time Streaming Protocol (RTSP), or representation of points in an RGB color model in a cylindrical coordinate system (Hue, preservation, lighting, HSL) may be used to convert the media resource, but a Protocol with a response at an opposite end is required to receive a decapsulated interface display, or a self-built website service plays a video network monitoring resource but a website needs additional plug-in support.

The video networking resources are played on a browser webpage of the video networking equipment, and the media stream in the video networking format is directly transmitted to the webpage for playing without any plug-in by adopting the fusion of video networking and MSE (media resource extension) technical frameworks. However, the MSE technology is in a manner of waiting for playing in the same frame, that is, the audio and the video need to be played in the same frame, but in the actual playing process, for example, the video frame rate or the audio sampling rate changes, the playing video will be disturbed.

Disclosure of Invention

In view of the above problems, embodiments of the present invention are proposed to provide a media data processing method, apparatus, terminal device and storage medium that overcome or at least partially solve the above problems.

In a first aspect, an embodiment of the present invention provides a method for processing media data, where the method includes:

receiving a playing parameter changing instruction sent by a server in the process of playing a media stream at a first adjusting time by a terminal device, wherein the playing parameter changing instruction at least comprises a first playing parameter and a switching timestamp;

determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter;

according to the first adjusting time and the switching time stamp, determining the playing time required by the media stream data which is not played before the switching time stamp is played;

determining an adjustment transition strategy of a playing parameter according to the second adjustment time and the playing time required by the played media stream data;

and playing the media stream data which is not played before the switching timestamp according to the adjustment transition strategy.

Optionally, the adjusting the transition policy includes at least one of adjusting a play rate or adjusting a data frame buffering time.

Optionally, the playing, according to the transition policy for adjusting the playing rate, media stream data that is not played before the switching timestamp is played includes:

if the playing time and the second adjusting time meet a first preset condition, slowing down the playing rate of the media stream data which is not played before the time stamp is switched to a first preset value according to the first preset condition;

if the playing time and the second adjusting time meet a second preset condition, accelerating the playing speed of the media stream data which is not played before the time stamp is switched to a second preset value according to the first preset condition.

Optionally, the playing the media stream data according to the transition policy for adjusting the data frame buffering time includes:

if the playing time and the second adjusting time meet a third preset condition, increasing the caching time interval of the first playing parameter, and accelerating the playing rate of the stored media stream data to a second preset value according to a second preset value in the third preset condition;

if the playing time and the second adjusting time meet a fourth preset condition, reducing the caching time interval of the first playing parameter, and slowing down the playing rate of the stored media stream data to a first preset value according to the first preset value in the fourth preset condition.

Optionally, the determining, according to the first playing parameter, a second adjusting time for keeping audio and video synchronous playing after switching includes:

determining a first playing time length for playing each audio data packet according to the audio sampling rate;

determining a second playing time length for playing each video data packet according to the video frame rate;

determining a time difference according to the first playing time and the second playing time;

and determining second adjusting time under the condition of audio and video synchronous playing according to the time length difference and the preset audio and video synchronous playing time.

Optionally, the determining, according to the first adjustment time and the switching timestamp, a playing time required for playing media stream data that is not played before the switching timestamp is completed includes:

acquiring an initial moment for starting to play the media data within a first adjusting time;

determining the playing time under the condition that the playing parameters are not changed according to the received difference between the switching timestamp and the initial time;

acquiring unplayed media stream data stored before a switch timestamp;

determining the time required for playing the media stream data which is not played according to a second playing parameter in the first adjusting time period;

and determining the playing time required by the played media stream data according to the played time and the time required by the played media stream data.

Optionally, the playing, according to the adjustment transition policy, media stream data that is not played before the switching timestamp, includes:

acquiring media stream data corresponding to the first playing parameter;

determining the accelerated or decelerated playing speed in the transition process according to the adjustment transition strategy;

playing the media stream data which is not played before the switching timestamp in the playing time at the accelerated or decelerated playing speed in the transition process;

and playing the media stream data corresponding to the first playing parameter at the playing speed corresponding to the first playing parameter.

In a second aspect, an embodiment of the present invention provides an apparatus for processing media data, where the apparatus includes:

the terminal device comprises a receiving module, a switching module and a processing module, wherein the receiving module is used for receiving a playing parameter changing instruction sent by a server in the process of playing a media stream at a first adjusting time by the terminal device, and the playing parameter changing instruction at least comprises a first playing parameter and a switching timestamp;

the first determining module is used for determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter;

a second determining module, configured to determine, according to the first adjustment time and the switching timestamp, a playing time required for playing media stream data that is not played before the switching timestamp is played;

the adjusting module is used for determining an adjustment transition strategy of the playing parameters according to the second adjusting time and the playing time required by the playing of the media stream data;

and the playing module is used for playing the media stream data which is not played before the switching timestamp according to the adjustment transition strategy.

Optionally, the playing module is configured to:

if the playing time and the second adjusting time meet a first preset condition, slowing down the playing speed of media stream data which is not played before the time stamp is switched to a first preset value according to the first preset condition;

Optionally, the playing module is configured to:

if the playing time and the second adjusting time meet a fourth preset condition, reducing the caching time interval of the first playing parameter, and slowing down the playing rate of the stored media stream data to a first preset value according to a first preset value in the fourth preset condition.

Optionally, the first playing parameter at least includes an audio sampling rate and/or a video frame rate, and the first determining module is configured to:

Optionally, the second determining module is configured to:

determining the time which is played under the condition that the playing parameters are not changed according to the difference between the received switching timestamp and the initial time;

acquiring unplayed media stream data stored before a switch timestamp;

Optionally, the playing module is configured to:

acquiring media stream data corresponding to the first playing parameter;

In a third aspect, an embodiment of the present invention provides a terminal device, including: at least one processor and memory;

the memory stores a computer program; the at least one processor executes the computer program stored by the memory to implement the method for processing media data provided by the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed, the computer program implements the media data processing method provided in the first aspect.

The embodiment of the invention has the following advantages:

according to the media data processing method and device, the terminal device and the storage medium provided by the embodiment of the invention, the playing parameter change instruction sent by the server is received in the process of playing the media stream at the first adjusting time by the terminal device, wherein the playing parameter change instruction at least comprises the first playing parameter and the switching timestamp; determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; determining the playing time required by the media stream data which is not played before the switching timestamp is played according to the first adjusting time and the switching timestamp; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives the playing parameter change instruction, the new adjustment time for keeping the synchronization of the audio and the video is recalculated, then the time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to the comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate, and the terminal equipment can play the data stably at the new audio and video frame rate.

Drawings

FIG. 1 is a flow chart of steps of a method for processing media data according to an embodiment of the present invention;

FIG. 2 is a flow chart of steps in another embodiment of a method of processing media data in accordance with the present invention;

FIG. 3 is a flow chart of steps of a further embodiment of a method of processing media data in accordance with the present invention;

FIG. 4 is a block diagram of an embodiment of a media data processing apparatus according to the present invention;

fig. 5 is a schematic structural diagram of a terminal device of the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.

At present, the video networking and MSE technology framework directly issues media stream data in a video networking media format to a webpage of an internet terminal for playing without any plug-in, and in the playing process, audio and video synchronous playing is realized by dynamically adjusting the playing duration of each audio and video frame. However, during the playing of the media stream, the video frame rate or the audio sampling rate may change dynamically, for example, when the video frame rate of a live frame is dynamically converted from 25pfs to 5pfs desktop stream, the audio frame rate is kept unchanged at this time, and thus, the problem of unsynchronized audio playing and video playing easily occurs.

For example:

(1) assuming that the video fps is 25 frames, one frame of video lasts 40ms, the audio sampling rate is 32K, one frame is 1024 samples, and one frame is played for 1000 × 1024/32000=31.25ms. It can be seen that audio transmission is more frequent than video, so that the situation that only audio has no video can occur, and at the moment, the audio must be played by waiting for the next video, which results in asynchronous audio and video and audio delay due to long-term accumulation.

(2) Assuming that the video fps is 5 frames, one frame of video lasts 200ms, the audio sampling rate is 32K, one frame is 1024 samples, and one frame is played for 1000 × 1024/32000=31.25ms. It can be seen that audio transmission is more frequent than video transmission, so that the situation that only audio has no video can occur, and the audio must be played only after the next video comes, which results in asynchronous audio and video and audio delay due to long-term accumulation.

(3) Assuming that the video fps is 40 frames, one frame of video lasts 25ms, the audio sampling rate is 32K, one frame is 1024 samples, and one frame is played by 1000 × 1024/32000=31.25ms. The video transmission is more frequent than the audio, so that the situation that only the video has no audio can occur, the video can be played only by waiting for the next audio, the audio and the video are asynchronous and the audio is advanced due to long-term accumulation.

It can be known that the variation of the audio and video frame rates causes the audio and video asynchronism, but the asynchronism has the problem of high speed and low speed according to the frame rate difference. The frame rate variation also causes the reference to vary. (1) And (2) the number of audio packets is more than that of video packets, and the video is taken as a reference. (3) The number of the video packets is more than that of the audio packets, and the audio is taken as a reference. These all vary according to frame rate parameter variations. Meanwhile, in order to avoid desynchronization caused by frame rate change and disorder of the picture at the moment of frame rate change, a special means is required to dynamically eliminate the disorder phenomenon.

The media source extension, MSE for short, is a set of API that can continuously plug the audio and video binary data into the video tag for playing.

In view of the foregoing problems, an embodiment of the present invention provides a method for processing media data, which is used to adjust the synchronized playback of media stream data. The execution main body of the embodiment is a processing device of media data, and is arranged on a terminal device, wherein the terminal device may be a computer, a tablet computer, a mobile phone terminal, and the like.

Referring to fig. 1, a flowchart illustrating steps of an embodiment of a method for processing media data according to the present invention is shown, where the method specifically includes the following steps:

s101, in the process that the terminal equipment plays the media stream at the first adjusting time, receiving a playing parameter changing instruction sent by a server, wherein the playing parameter changing instruction at least comprises a first playing parameter and a switching timestamp;

specifically, a user plays video networking media stream data on a browser of a terminal device, and an MSE is installed on the terminal device, so that the media stream data can be played without adding a plug-in on the browser, the user selects the media stream data to be played on the browser of the terminal device, namely, the terminal device receives a media data request input by the user, the terminal device sends the media data request to an internet server, and the internet server obtains the media stream data corresponding to the media data request from a protocol server according to the media data request, wherein the media stream data at least comprises audio data, video data and second playing parameters, the relevant parameters at least comprise a video frame rate and an audio acquisition rate, and the media stream data are binary media stream data.

And the terminal equipment calculates first adjusting time for audio and video synchronous playing according to the second playing parameter, and plays the media stream data at the first adjusting time. The method comprises the steps that after an internet server receives binary media stream data returned by a protocol conversion server, the binary media stream data are analyzed, whether an audio sampling rate and a video frame rate exist in an analysis result or not is judged, if the audio sampling rate and the video frame rate exist, the binary media stream data are sent to terminal equipment, the terminal equipment respectively determines the duration of playing a frame of video and the duration of playing a frame of audio according to the audio sampling rate and the video frame rate, the time difference between the duration of playing the frame of video and the duration of playing the frame of audio is calculated, the preset audio and video synchronous playing time, namely the time when a human eye cannot see audio and video asynchronization, a numerical value is obtained by dividing the preset audio and video synchronous playing time by the time difference, namely, a plurality of difference values exist in the preset audio and video synchronous playing time, then the numerical value is multiplied by the duration of playing the frame of video, and the adjusting time under the condition of audio and video synchronous playing is determined.

In the playing process, the second playing parameter changes, for example, the video frame rate changes, or the audio sampling rate changes, at this time, the server sends a playing parameter changing instruction to the terminal device, where the playing parameter changing instruction includes a second playing parameter and a switching timestamp, and the second playing parameter may be the audio sampling rate changes or the video frame rate changes.

S102, determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter;

the second playing parameter is a playing parameter before switching, and the first playing parameter is a playing parameter after switching, and comprises an audio sampling rate and/or a video frame rate after switching.

Specifically, after receiving a playing parameter change instruction, the terminal device determines a second adjustment time for keeping audio and video synchronous playing after switching according to a first playing parameter (i.e. the switched audio sampling rate and/or video frame rate) in the instruction, that is, the first playing parameter at least includes one or more of an audio sampling rate and a video frame rate, and if the video frame rate changes, the second adjustment time for keeping audio and video synchronous playing needs to be recalculated according to the changed video frame rate and the original audio sampling rate; if the audio sampling rate changes, recalculating a second adjusting time for keeping audio and video synchronous playing according to the changed audio sampling rate and the original video frame rate; and if the video frame rate and the audio sampling rate are changed, recalculating a second adjusting time for keeping the audio and video synchronous playing according to the changed audio sampling rate and the changed video frame rate.

S103, determining the playing time required by the media stream data which is not played before the switching time stamp is played according to the first adjusting time and the switching time stamp;

specifically, the terminal device has cached a part of media stream data in the first adjustment time period, the part of media stream data is divided into audio data and video data, and the audio data and the video data are respectively stored in different arrays, that is, the audio data is stored in the audio array, the video data is stored in the video array, and both the audio data and the video data are stored in the memory, in the process of playing the stored media stream data, a change instruction of a switching playing parameter is received, in the process, it is required to calculate the time that the media stream data is played before the switching timestamp and the time that the remaining media stream data is played after the switching timestamp, and the sum of the two is the playing time required for playing the media stream data.

And playing the data with the second playing parameter before the time stamp is switched, and playing the data with the first playing parameter after the time stamp is switched.

S104, determining an adjustment transition strategy of the playing parameters according to the second adjusting time and the playing time;

specifically, the terminal device determines an adjustment transition policy for playing subsequent media stream data after obtaining a second adjustment time and a time required for actually playing the media stream data, that is, in a process of playing the media stream data, the terminal device plays the media stream data with a second playing parameter (a parameter before switching), and when the second playing parameter (a parameter before switching) is converted into a first playing parameter (a parameter after switching), so that an audio and video asynchronization phenomenon may occur during playing the media stream data, and a playing speed of the media stream data needs to be adjusted until the terminal device can play the media stream data stably with the first playing parameter.

For example, if the playing time T3 required for playing the media stream data is less than or equal to the second adjustment time T11, for example, T3< = T11& T3/m > = T11, the playing rate is slowed down, and the playing rate is slowed down to m times, where m is a first adjustment parameter of the playing rate, and 0 < m <1, that is, the originally cached media stream data is played at the original playing rate, and the new media stream data is not stored yet, the playing rate of the remaining media stream data needs to be slowed down;

if the ratio of the playing time T3 to the second adjusting time T11 required for playing the media stream data is smaller than the first adjusting parameter m of the playing rate (i.e. the ratio of the first preset value to the playing rate); for example, if T3/m is less than T11, reducing the new frame rate buffer time interval until the old data is played by m times, and inputting the new data in the time interval into the label;

therefore, enough time can be provided for caching the media stream data in the new adjusting time, and the problem of asynchronism or caching can not be caused when a user watches the browser; the playback rate cannot be reduced too low or the picture will be perceived, alternatively m =0.8.

If the playing time T3 required for playing the media stream data is longer than the second adjusting time T11, for example, T3> T11& T3/n < = T11, the playing speed is increased, and the playing speed is increased to n times, where n is a second adjusting parameter of the playing speed, and n is greater than 1, that is, the originally cached media stream data is played at the original playing speed, and the new media stream data is ready, the playing speed of the remaining media stream data needs to be increased;

if the ratio of the playing time T3 to the second adjusting time T11 required for playing the media stream data is greater than the second adjusting parameter n (i.e. the ratio of the second preset value to the playing speed); for example, T3/n > T11, the new frame rate buffering time interval is increased until the playing of the old data is completed by n times, and then the new data in the time interval is input into the tag;

in this way, the remaining media stream data can be played back quickly, and then new media stream data is played back, the user does not have the problem of asynchronism when watching the browser, the playing speed cannot be increased too low, the picture is perceived, and optionally n =1.2.

And S105, according to the adjustment transition strategy, playing the media stream data which is not played before the switching timestamp.

Specifically, after determining and adjusting the transition policy, the terminal device adjusts the playing rate through the transition policy, for example, the playing rate is increased or decreased, so that when the server detects that the playing parameter changes, for example, the video frame rate changes or the audio sampling rate changes, a playing parameter change instruction is sent to the terminal device, and the terminal device determines different transition policies according to the playing parameter in the playing parameter change instruction, so that in the process of transitioning from the second playing parameter to the first playing parameter, the audio data and the video data can be played synchronously, and then transition is gradually made to playing media stream data that is stable with the first playing parameter, and a user watching a video does not find that the playing parameter changes.

Specifically, the terminal device respectively counts the number of audio data packets and video data packets sent by the server, a first number of audio data packets and a second number of video data packets received in 1s within a second adjustment time, for example, 1s, then calculates the audio rendering playing time according to the first number, and calculates the video rendering playing time according to the second number.

That is, the terminal device counts the number of audio data packets and the number of video data packets received in the second adjustment time period, and then equally allocates the time period to each audio data packet for playing and rendering or each video data packet for playing and rendering.

For example, before switching, if the terminal device receives 100 audio packets within 1000ms, the playback rendering time allocated to one audio packet is 1000/100=10ms, and receives 50 video packets within 1000ms, the playback rendering time allocated to one video packet is 1000/50=20ms, that is, the playback rendering time of an audio packet is 10ms, and the playback rendering time of a video packet is 20ms. If the video frame rate changes, the terminal device needs to recalculate the second adjustment time according to the changed video frame rate and the changed audio sampling rate, and calculate the playing time required by playing the stored media stream data, and then according to the second adjustment time and the playing time required by playing the media stream data, determine the corresponding transition policy, if the video frame rate changes from 25 frames/second to 5 frames/second, it needs to accelerate playing the stored media stream data, and when receiving new media stream data, it gradually transitions to the video frame rate of 5 frames/second for smooth playing.

The video data packet in the embodiment of the present invention is a video without sound, for example, monitoring data without sound.

Specifically, after determining the audio rendering and playing time and the video rendering and playing time, the terminal device obtains an audio data packet from the audio array according to the audio rendering and playing time, obtains a video data packet from the video array according to the video rendering and playing time, adds the audio data packet and the video data packet into the encoder, and plays the data with a second playing parameter through the browser.

In the method for processing media data provided by the embodiment of the present invention, a playing parameter change instruction sent by a server is received in a process of playing a media stream at a first adjustment time by a terminal device, where the playing parameter change instruction at least includes a first playing parameter and a switching timestamp; determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; according to the first adjusting time and the switching time stamp, determining the playing time required by the media stream data which is not played before the switching time stamp is played; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives a playing parameter change instruction, new adjustment time for keeping synchronization of audio and video is recalculated, then time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate until the terminal equipment stably plays the data at the new audio and video frame rate.

The present invention further provides a supplementary explanation to the method for processing media data provided in the foregoing embodiment.

As shown in fig. 2, a flow chart of steps of another embodiment of a method for processing media data according to the present invention is shown, and the method for processing media data includes:

s201, in the process that the terminal equipment plays the media stream at the first adjusting time, receiving a playing parameter changing instruction sent by a server, wherein the playing parameter changing instruction at least comprises a first playing parameter and a switching time stamp;

specifically, when the playing parameter of the media stream data acquired by the server changes, for example, the original video frame rate is 50fps, and the current video frame rate is 20fps, the server sends a playing parameter change instruction to the terminal device in advance, where the playing parameter change instruction at least includes a first playing parameter and a switching timestamp.

S202, determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; the first playing parameter at least comprises an audio sampling rate and/or a video frame rate;

the method specifically comprises the following steps:

a1, determining a first playing time length for playing each audio data packet according to an audio sampling rate;

if the audio sampling rate changes, calculating a first playing time length for playing each audio data packet according to the changed audio sampling rate;

if the audio sampling rate is not changed, calculating a first playing time length for playing each audio data packet by adopting the original audio sampling rate.

Step A2, determining a second playing time length for playing each video data packet according to the video frame rate;

specifically, if the video frame rate changes, a second playing duration for playing each video data packet is calculated according to the changed video frame rate;

and if the video frame rate is not changed, calculating a second playing time length for playing each video data packet by adopting the original video frame rate.

A3, determining a time difference according to the first playing time and the second playing time;

and A4, determining second adjusting time under the condition of audio and video synchronous playing according to the time difference and the preset audio and video synchronous playing time.

Illustratively, in an initial certain time range, even if the audio-video data packets have differences, human perception is audio-video synchronization, for example, the difference of the audio-video time stamps fluctuates in a range of 300-500ms, and human perception is not provided, for example, the difference of the audio-video time stamps is considered synchronization in 300 ms. Assuming that the video frame rate is 25fps (Frames Per Second), one frame of video lasts 40ms, the audio sampling rate is 32K, one frame is 1024 samples, and one frame is played for 1000 × 1024/32000=31.25ms. The number of audio frames is greater than the number of video frames, and the video packets are played on the basis of the video packets. One frame differs by 40-31.25=8.75ms. I.e. each time a video frame is played, the audio frame is delayed by 8.75ms. Approximately 34 video packets are required within 300 ms. 34 x 40=1360ms, in which 1360ms is the adjustment time.

In the embodiment of the present invention, the adjustment time may be the maximum difference, or may be set by itself as needed, that is, less than 1360ms, and any value may be selected, and preferably, the adjustment time may be selected to be 1000ms, that is, 1s.

S203, acquiring an initial moment for starting to play the media data within a first adjusting time;

illustratively, in the first adjustment period, the terminal device obtains an initial time of the first adjustment period, for example, the first adjustment period is 1s, and the initial time is 10. The initial time is used to calculate the time when the media stream data has been played.

S204, determining the playing time under the condition that the playing parameters are not changed according to the difference between the received switching timestamp and the initial time;

specifically, the terminal device obtains an initial time in the first time terminal, obtains a switching timestamp at the same time, and calculates the time of playing the media stream data under the condition that the playing parameter is not changed according to the initial time and the switching timestamp.

S205, obtaining media stream data which is stored before the time stamp is switched and is not played;

specifically, the terminal device has stored the media stream data during the first adjustment period, and acquires the remaining unplayed media stream data.

S206, determining the time required for playing the media stream data which is not played according to the second playing parameter in the first adjusting time period;

specifically, the terminal device plays the remaining media stream data according to the changed second play parameter, and calculates the time required for the remaining media stream data.

S207, determining the playing time required by the media stream data which is played completely according to the time which is played and the time required by the media stream data which is not played completely.

S208, media stream data corresponding to the first playing parameter is obtained;

s209, determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data, and determining the accelerated or decelerated playing speed in the transition process according to the adjustment transition strategy and the adjustment transition strategy;

the over-adjustment transition strategy at least comprises one of the following strategies, specifically comprising:

as an optional implementation manner, playing media stream data that is not played before the switch timestamp according to a transition policy for adjusting the play rate includes:

if the playing time and the second adjusting time meet a first preset condition, slowing down the playing rate of media stream data which is not played before the time stamp is switched to a first preset value according to the first preset condition;

if the playing time and the second adjusting time meet a second preset condition, accelerating the playing speed of the media stream data which is not played before the timestamp is switched to a second preset value according to the first preset condition;

specifically, the first preset condition is that the playing time T3 is less than or equal to the second adjusting time T11, and a ratio of the playing time T3 to the second adjusting time T11 is greater than or equal to the playing rate first adjusting parameter m (i.e., a ratio of the first preset value to the playing rate); when the playing time and the second adjusting time meet a first preset condition, slowing down the playing speed to a first preset value, wherein the first preset value is the slowed playing speed.

The second preset condition is that the playing time T3 is greater than the second adjusting time T11, and the ratio of the playing time T3 to the second adjusting time T11 is less than or equal to the playing rate second adjusting parameter n (i.e. the ratio of the second preset value to the playing rate); and when the playing time and the second adjusting time meet a second preset condition, accelerating the playing speed to a second preset value, wherein the second preset value is the accelerated playing speed.

Exemplarily, if T3< = T11& T3/0.8> = T11, the play rate v is slowed down to the first preset value of 0.8v.

Exemplarily, if T3> T11& T3/1.2< = T11, the playing rate v is accelerated to the second preset value of 1.2v.

As another optional implementation, playing media stream data according to a transition policy for adjusting a buffering time of a data frame includes:

if the playing time and the second adjusting time meet a third preset condition, increasing the caching time interval of the first playing parameter, and accelerating the playing rate of the stored media stream data to a second preset value;

if the playing time and the second adjusting time meet a fourth preset condition, reducing the caching time interval of the first playing parameter, and slowing down the playing rate of the stored media stream data to a first preset value.

Specifically, the third preset condition is that the ratio of the playing time T3 to the second adjusting time T11 is greater than the playing rate second adjusting parameter n (i.e. the ratio of the second preset value to the playing rate); wherein n is a second adjustment parameter of the playing speed, and n is greater than 1; and when the playing time and the second adjusting time meet a third preset condition, increasing the buffer time interval of the new frame rate until the playing of the old data by n times is finished, and inputting the new data in the time interval into the tag.

The fourth preset condition is that the ratio of the playing time T3 to the second adjusting time T11 is smaller than the first adjusting parameter m of the playing rate (i.e. the ratio of the first preset value to the playing rate); and when the playing time and the second adjusting time meet a fourth preset condition, reducing the buffer time interval of the new frame rate until the playing of the old data in m times is finished, and inputting the new data in the time interval into the tag.

Illustratively, if T3/1.2> T11, the new frame rate buffer interval is increased until the old data is played at 1.2 times, and the new data for this period is entered into the tag. The above process is repeated again.

If T3/0.8< T11, the new frame rate buffer interval is decreased until the old data is played by 0.8 times, and the new data for this interval is entered into the tag. The above process is repeated again.

S210, playing media stream data which is not played before the switching timestamp in the playing time at the accelerated or decelerated playing speed in the transition process;

specifically, the terminal device plays media stream data that is not played before the timestamp is switched within the playing time at a playing rate that is increased or decreased in the transition process, obtains the media stream data corresponding to the first playing parameter, and plays the media stream data corresponding to the first playing parameter at the next adjusting time, that is, at the second adjusting time, at the playing rate corresponding to the first playing parameter.

S211, playing the media stream data corresponding to the first playing parameter at the playing rate corresponding to the first playing parameter.

Specifically, after the media stream data which is not played before the switching timestamp in the playing time is played, the media stream data is played at an accelerated or decelerated playing speed in a second adjusting time period, and the transition strategy is adjusted until the terminal device stably plays the media stream data at the playing speed corresponding to the first playing parameter.

Fig. 3 is a flowchart of steps of another embodiment of a method for processing media data according to the present invention, first obtaining a sampling rate and a frame rate of an audio/video, and calculating a required time T to possibly generate an asynchronous effect. For example, audio-video timestamps are considered synchronized within 300ms of each other. Assuming a video frame rate of 25fps, a video frame lasts 40ms, an audio sampling rate of 32K, a frame of 1024 samples, and a frame of 1000 × 1024/32000=31.25ms. The number of audio frames is greater than the number of video frames, and the video packets are played on the basis of the video packets. One frame differs by 40-31.25=8.75ms. I.e. each time a video frame is played, the audio frame is delayed by 8.75ms. Approximately 34 video packets are required within 300 ms. 34 x 40=1360ms will produce an out-of-sync effect.

The number of video data packets is small, and the number of the accumulated audio/video data packets in T =1000ms (1000ms is restricted to 1360ms) is counted respectively by taking the video data as a reference. And calculating the rendering and playing time required by each audio and video packet in the time range, and assigning values again to ensure data synchronization within 1000 ms. The playing rendering time length of each frame of audio is mainly changed by taking the video packet data as a reference.

After the playing time length of each frame of data is dynamically configured in the T time range, the media stream data and related parameters (video frame rate and audio sampling rate) are sent to a decoder to carry out H264/AAC decapsulation and FMP4 encapsulation, and then rendered in a Vedio tag. And meanwhile, dynamically configuring the next T time. And the real-time updating of data is ensured, and the parameter accumulation delay error is avoided.

When the source is switched (the frame rate or the audio sampling rate changes), the server informs the terminal equipment to finish the current stream accumulated statistics, the terminal equipment subtracts the last recorded timestamp from the current last packet timestamp to obtain T2, the rendering playing time required by the audio and video packets in the last T2 time interval when the frame rate is unchanged is calculated by the T2, and the audio and video packets are sent to the Vedio tag for rendering.

A new tsecond adjustment time is calculated based on the new frame rate and a new T11 is set (T11 < T). And caching the received media stream data in a corresponding audio array or a corresponding video array respectively.

The time T3 required for acquiring the data cached in the current video tag is T3, where T3= T2+ the original unplayed data, and comparing T3 with T11 to make a corresponding policy transition.

(1) If T3< = T11& T3/m > = T11, slow down the playing rate to m times the original playing rate (the playing rate cannot be reduced too low, the picture is perceived), preferably, m =0.8.

(2) If T3> T11& T3/n < = T11, slow the playing rate to n times the original playing rate (the playing rate cannot rise too low, picture is perceived), preferably, n =1.2.

(3) T3/n > T11, increasing the new frame rate buffer time interval until the playing of the old data is finished at n times of the original playing speed, and inputting the new data of the time interval into the tag. The above process is repeated again.

(4) T3/m < T11, reducing the new frame rate buffer time interval until the playing of the old data is finished at m times of the original playing speed, and inputting the new data of the time interval into the label. The above process is repeated again.

The specific media data processing comprises the following steps:

1. pulling the corresponding data resource according to the request;

2. obtaining the SDP, audio and video format, sampling rate, frame rate and other messages of the device and returning the messages to the terminal device (MSE);

3. initializing relevant parameters according to the audio sampling rate and the video frame rate and calculating the required time length T according to the reference possibly to generate asynchronous effect, and selecting a proper dynamic adjustment interval T1 according to the T.

4. The server pulls the video network media stream from the protocol conversion server, and converts the video network packet header into internet media stream data containing a time stamp and a data type.

5. The server continuously sends buffer data according to the request link;

6. the terminal equipment analyzes the timestamp and the format type of the received media stream data and fills the data into the corresponding array.

7. When video data is received, the difference value between the timestamp and the last dynamic modification playing time is judged after the array is added.

8. And calculating the time of rendering and playing required by each audio and video data packet, and dynamically updating the time of all frames in the last T1 time to realize the accumulated error caused by different time of each frame of the audio and video.

9. When the source is switched (the frame rate is changed), the server sends a play parameter change instruction to the terminal device.

10. And the terminal equipment finishes the current stream accumulated statistics, the current last packet timestamp is subtracted from the last recording timestamp to obtain T2, the rendering and playing time required by the audio and video packets in the last T2 time interval when the frame rate is unchanged is calculated by the T2, and the audio and video packets are sent to the Vedio tag for rendering.

11. The terminal device calculates a new T second adjustment time and sets a new T11 (T11 < T) according to the new frame rate. And the received media stream data are respectively cached in the corresponding arrays.

12. The terminal equipment needs to spend time T3 for obtaining the data cached in the current video tag, wherein T3= T2+ original data which are not played, and T3 is compared with T11 to make corresponding strategy transition.

13. If T3< = T11& T3/0.8> = T11, slow down the playing rate to 0.8 (the playing rate cannot be reduced too low, the picture is perceived); if T3> T11& T3/1.2< = T11, the playback rate is slowed to 1.2 (the playback rate cannot rise too low, the picture is perceived). T3/1.2 >. The above process is repeated again. T3/0.8 Tn 11, reducing the new frame rate buffer time interval until the playing of the old data is finished by 0.8 times, and inputting the new data of the time interval into the label. The above process is repeated again. The process aims to eliminate the disorder caused by the change of the frame rate, and gradually changes the new T11 accumulation process until the new stability is reached, and the storage of the playing picture is relatively stable.

The embodiment of the invention realizes that the webpage without the plug-in plays the audio and video pictures, and the pictures are normally played when the picture scenes are switched (the frame rate is changed).

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.

In the method for processing media data provided by the embodiment of the invention, a playing parameter change instruction sent by a server is received in the process of playing a media stream at a first adjustment time by a terminal device, wherein the playing parameter change instruction at least comprises a first playing parameter and a switching timestamp; determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; according to the first adjusting time and the switching time stamp, determining the playing time required by the media stream data which is not played before the switching time stamp is played; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives the playing parameter change instruction, the new adjustment time for keeping the synchronization of the audio and the video is recalculated, then the time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to the comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate, and the terminal equipment can play the data stably at the new audio and video frame rate.

Another embodiment of the present invention provides a device for processing media data, which is configured to execute the method for processing media data provided in the foregoing embodiment.

Referring to fig. 4, a block diagram of a media data processing apparatus according to an embodiment of the present invention is shown, and the apparatus may specifically include the following modules: a receiving module 401, a first determining module 402, a second determining module 403, an adjusting module 404, and a playing module 405, wherein:

the receiving module 401 is configured to receive a playing parameter changing instruction sent by a server in a process that a terminal device plays a media stream at a first adjustment time, where the playing parameter changing instruction at least includes a first playing parameter and a switching timestamp;

the first determining module 402 is configured to determine, according to the first playing parameter, a second adjustment time for keeping synchronous playing of the audio and video after switching;

the second determining module 403 is configured to determine, according to the first adjusting time and the switching timestamp, a playing time required for playing media stream data that is not played before the switching timestamp is played;

the adjusting module 404 is configured to determine an adjustment transition policy of the playing parameter according to the second adjusting time and the playing time required for playing the media stream data;

the playing module 405 is configured to play media stream data that is not played before the switching timestamp according to the adjustment transition policy.

The media data processing device provided by the embodiment of the invention receives a playing parameter changing instruction sent by a server in the process of playing a media stream at a first adjusting time by a terminal device, wherein the playing parameter changing instruction at least comprises a first playing parameter and a switching timestamp; determining second adjusting time for keeping synchronous playing of the audio and the video after switching according to the first playing parameter; determining the playing time required by the media stream data which is not played before the switching timestamp is played according to the first adjusting time and the switching timestamp; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives the playing parameter change instruction, the new adjustment time for keeping the synchronization of the audio and the video is recalculated, then the time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to the comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate, and the terminal equipment can play the data stably at the new audio and video frame rate.

The present invention further provides a supplementary explanation to the media data processing device provided in the above embodiment.

Optionally, adjusting the transition policy includes at least one of adjusting a play rate or adjusting a data frame buffering time.

Optionally, the playing module is configured to:

if the playing time and the second adjusting time meet a second preset condition, the playing speed of the media stream data which is not played before the timestamp is switched is increased to a second preset value according to the first preset condition.

Optionally, the playing module is configured to:

and determining second adjusting time under the condition of audio and video synchronous playing according to the time difference and the preset audio and video synchronous playing time.

Optionally, the second determining module is configured to:

determining the playing time under the condition that the playing parameters are not changed according to the difference between the received switching timestamp and the initial time;

acquiring unplayed media stream data stored before a switch timestamp;

determining the time required for playing the media stream data which is not played according to the second playing parameter in the first adjusting time period;

and determining the playing time required by the playing of the media stream data according to the time already played and the time required by the playing of the media stream data which is not played.

Optionally, the playing module is configured to:

acquiring media stream data corresponding to the first playing parameter;

determining the accelerated or decelerated playing speed in the transition process according to the transition strategy;

playing media stream data which is not played before the switching timestamp in the playing time at a playing speed which is accelerated or decelerated in the transition process;

It should be noted that the respective implementable modes in the present embodiment may be implemented individually, or may be implemented in combination in any combination without conflict, and the present application is not limited thereto.

For the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points.

Still another embodiment of the present invention provides a terminal device, configured to execute the method for processing media data provided in the foregoing embodiment.

Fig. 5 is a schematic structural diagram of a terminal device of the present invention, and as shown in fig. 5, the terminal device includes: at least one processor 501 and memory 502;

the memory stores a computer program; at least one processor executes the computer program stored in the memory to implement the media data processing method provided by the above-mentioned embodiments.

In the terminal device provided in this embodiment, a playing parameter change instruction sent by a server is received in a process that the terminal device plays a media stream at a first adjustment time, where the playing parameter change instruction at least includes a first playing parameter and a switching timestamp; determining second adjusting time for keeping synchronous playing of the audio and the video after switching according to the first playing parameter; determining the playing time required by the media stream data which is not played before the switching timestamp is played according to the first adjusting time and the switching timestamp; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives a playing parameter change instruction, new adjustment time for keeping synchronization of audio and video is recalculated, then time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate until the terminal equipment stably plays the data at the new audio and video frame rate.

Yet another embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed, the method for processing media data provided in any of the above embodiments is implemented.

According to the computer-readable storage medium of the embodiment, in the process of playing the media stream at the first adjustment time by the terminal device, a playing parameter changing instruction sent by the server is received, wherein the playing parameter changing instruction at least comprises a first playing parameter and a switching timestamp; determining second adjusting time for keeping audio and video synchronous playing after switching according to the first playing parameter; determining the playing time required by the media stream data which is not played before the switching timestamp is played according to the first adjusting time and the switching timestamp; determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time required by the playing of the media stream data; according to the adjustment transition strategy, media stream data which is not played before the time stamp is switched is played, through the embodiment of the invention, after the terminal equipment receives the playing parameter change instruction, the new adjustment time for keeping the synchronization of the audio and the video is recalculated, then the time required for playing the cached audio and video data at the original audio and video frame rate within the current adjustment time is calculated, then the adjustment transition strategy is determined according to the comparison between the new adjustment time and the time required for playing the cached audio and video data at the original audio and video frame rate, and the terminal equipment can play the data stably at the new audio and video frame rate.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing electronic device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing electronic device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing electronic devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing electronic device to cause a series of operational steps to be performed on the computer or other programmable electronic device to produce a computer implemented process such that the instructions which execute on the computer or other programmable electronic device provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "include", "including" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or electronic device including a series of elements includes not only those elements but also other elements not explicitly listed or inherent to such process, method, article, or electronic device. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or electronic device that comprises the element.

The above detailed description is provided for a media data processing method and a media data processing apparatus, and specific examples are applied herein to explain the principles and embodiments of the present invention, and the descriptions of the above embodiments are only used to help understanding the method and the core ideas of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method for processing media data, the method comprising:

determining an adjustment transition strategy of the playing parameters according to the second adjustment time and the playing time;

2. The method of claim 1, wherein the adjusting the transition policy comprises at least one of adjusting a playback rate or adjusting a data frame buffering time.

3. The method according to claim 2, wherein playing the media stream data that has not been played before the switch timestamp according to the transition policy for adjusting the playing rate comprises:

4. The method of claim 2, wherein playing the media stream data according to the transition policy for adjusting the buffering time of the data frames comprises:

if the playing time and the second adjusting time meet a fourth preset condition, reducing the caching time interval of the first playing parameter, and slowing down the playing speed of the stored media stream data to a first preset value.

5. The method according to claim 1, wherein the first playing parameter at least includes an audio sampling rate and/or a video frame rate, and the determining, according to the first playing parameter, a second adjustment time for keeping audio and video synchronous playing after switching includes:

6. The method of claim 1, wherein determining the playing time required for playing the media stream data that is not played before the switching timestamp is finished according to the first adjustment time and the switching timestamp comprises:

acquiring unplayed media stream data stored before a switch timestamp;

7. The method of claim 6, wherein playing media stream data that has not been played before the switch timestamp according to the adjusted transition policy comprises:

acquiring media stream data corresponding to the first playing parameter;

8. An apparatus for processing media data, the apparatus comprising:

the first determining module is used for determining second adjusting time for keeping synchronous audio and video playing after switching according to the first playing parameter;

the adjusting module is used for determining an adjusting transition strategy of the playing parameters according to the second adjusting time and the playing time required by the played media stream data;

9. A terminal device, comprising: at least one processor and memory;

the memory stores a computer program; the at least one processor executes the computer program stored by the memory to implement the method of processing media data according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when executed, implements the method of processing media data according to any one of claims 1 to 7.