CN118200623A

CN118200623A - Playing processing method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN118200623A
Application number: CN202410077975.8A
Authority: CN
Inventors: 李尉
Original assignee: Shenzhen TCL New Technology Co Ltd
Current assignee: Shenzhen TCL New Technology Co Ltd
Priority date: 2024-01-19
Filing date: 2024-01-19
Publication date: 2024-06-14

Abstract

The embodiment of the application discloses a play processing method, a play processing device, electronic equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring multimedia data to be played, and determining an initial playing flow for playing the multimedia data; starting an initial playing flow, and playing the current frame data of the multimedia data through at least one playing node in the initial playing flow; invoking a preset play maintenance interface to maintain a target play node in the play nodes to obtain a target play flow, wherein the target play node is a play node commonly contained in at least two play flows; and playing the residual frame data of the multimedia data according to the target playing flow. The target playing node is a universal node related to various types of playing processes, and the universal target playing node is used as a basis for maintenance processing, so that each playing process can be maintained based on the universal target playing node, and the universality of the processing of the playing process is improved.

Description

Playing processing method and device, electronic equipment and computer readable storage medium

Technical Field

The embodiment of the application relates to the field of computers, in particular to a play processing method, a play processing device, electronic equipment and a computer readable storage medium.

Background

In the field of multimedia playback, the problem of jamming is the most common problem. In order to improve the playing experience of a user, detecting the phenomenon of playing stuck, identifying root causes and repairing problems become more and more important.

At present, the katon detection technology has some defects, for example, only a certain type of detection method is described, namely, how to judge whether an abnormal detection result is obtained, or only specific scenes or partial scenes (such as game scenes) are detected, but cannot be used for all modules of a playing flow, and is difficult to be universally applied to scenes of all playing types.

Disclosure of Invention

The embodiment of the application provides a play processing method, a play processing device, electronic equipment and a computer readable storage medium, which can be widely applied to maintenance processing of various play processes and improve universality.

The embodiment of the application provides a play processing method, which comprises the following steps:

Acquiring multimedia data to be played, and determining an initial playing flow for playing the multimedia data;

Starting the initial playing flow, and playing the current frame data of the multimedia data through at least one playing node in the initial playing flow;

Invoking a preset play maintenance interface to maintain a target play node in the play nodes to obtain a target play flow, wherein the target play node is a play node commonly contained in at least two play flows;

And playing the rest frame data of the multimedia data according to the target playing flow.

Correspondingly, the embodiment of the application also provides a playing processing device, which comprises:

The acquisition module is used for acquiring the multimedia data to be played and determining an initial playing flow for playing the multimedia data;

the starting module is used for starting the initial playing flow and playing the current frame data of the multimedia data through at least one playing node in the initial playing flow;

The maintenance module is used for calling a preset play maintenance interface to maintain a target play node in the play nodes to obtain a target play flow, wherein the target play node is a play node commonly contained in at least two play flows;

and the playing module is used for playing the residual frame data of the multimedia data according to the target playing flow.

Optionally, in some embodiments of the present application, the play maintenance interface includes a node status interface, an anomaly detection interface and an anomaly repair interface, and the maintenance module includes:

The first calling unit is used for calling the node state interface to obtain a judging result that the initial playing flow contains the target playing node;

the second calling unit is used for calling the abnormality detection interface to perform abnormality detection on the target playing node if the judging result indicates that the initial playing flow contains the target playing node, so as to obtain an abnormality detection result;

And the third calling unit is used for calling the abnormality repairing interface to repair the target playing node if the abnormality detection result represents that the target playing node is abnormal, so as to obtain a target playing flow.

Wherein, in some embodiments of the application, the second call unit comprises:

the calling subunit is used for calling the abnormality detection interface to extract the input data and the output data of the target playing node;

the first judging subunit is used for obtaining an abnormality detection result according to the comparison result of the input data and the output data;

and/or the number of the groups of groups,

And the second judging subunit is used for obtaining an abnormality detection result according to the comparison result of the input data, the output data and the preset parameter threshold value.

Wherein, in some embodiments of the application, the third call unit comprises:

A determining subunit, configured to determine an anomaly type corresponding to the anomaly detection result if the anomaly detection result characterizes that the target play node is abnormal;

And the repairing subunit is used for calling the abnormal repairing interface, and repairing the target playing node according to a preset repairing strategy corresponding to the abnormal type to obtain a target playing flow.

Wherein, in some embodiments of the present application, the play maintenance interface includes at least two, and the maintenance module includes:

The sequence calling unit is used for calling the play maintenance interface to carry out maintenance processing on a target play node corresponding to the play maintenance interface in the initial play flow according to the calling sequence of the play maintenance interface to obtain a maintenance processing result, wherein the calling sequence is obtained according to the processing sequence of the play nodes in at least two play flows;

The maintenance processing unit is used for obtaining a processed target playing node corresponding to the target playing node and stopping calling a playing maintenance interface positioned behind the playing maintenance interface if the maintenance processing result meets a preset condition;

And the generating unit is used for obtaining a target playing flow according to the processed target playing node.

Wherein, in some embodiments of the present application, the apparatus further comprises a first interface generation module, the first interface generation module comprising:

the first configuration unit is used for configuring a repair base class containing an unrealized reference maintenance interface;

the first establishing unit is used for establishing a play maintenance subclass inherited from the repair base class aiming at the target play node;

The first generating unit is configured to configure implementation of the reference maintenance interface in the play maintenance subclass according to the processing logic of the target play node, so as to obtain a play maintenance interface.

Wherein, in some embodiments of the present application, the apparatus further comprises a second interface generation module, the second interface generation module comprising:

The second configuration unit is used for configuring a repair base class containing an unrealized reference maintenance interface;

The second establishing unit is used for setting the repairing base class as a parent class aiming at the target playing node and establishing an inheritance relationship between the playing processing subclass corresponding to the target playing node and the repairing base class;

And the second generating unit is used for configuring the realization of the reference maintenance interface in the playing processing subclass according to the processing logic of the target playing node to obtain a playing maintenance interface.

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the computer program when executed by the processor implements the steps in the playing processing method described above.

In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor implements the steps in the above-mentioned playing processing method.

In a fifth aspect, embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the methods provided in the various alternative implementations described in the embodiments of the present application.

In summary, after obtaining multimedia data to be played and determining an initial playing flow for playing the multimedia data, the embodiment of the application uses a plurality of playing nodes included in the initial playing flow to play current frame data of the multimedia data, and meanwhile, invokes a preset playing maintenance interface to maintain a target playing node in the playing nodes to obtain a target playing flow, wherein the target playing node is a playing node commonly included in the playing flows, and then uses the target playing flow to play the rest frame data of the multimedia data. The target playing node is a common playing node contained in a plurality of playing processes, so that the target playing node is a universal node involved in a plurality of types of playing processes, and the universal target playing node is used as a basis for maintenance processing, so that each playing process can be maintained based on the universal target playing node, and the universality of the processing of the playing process is improved.

Drawings

In order to more clearly illustrate the technical solutions of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic view of a scenario of a play processing method provided in an embodiment of the present application;

Fig. 2 is a flow chart of a play processing method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a general playback flow provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of a call sequence of a play maintenance interface according to an embodiment of the present application;

Fig. 5 is a schematic class diagram of a play processing subclass corresponding to a data source node according to an embodiment of the present application;

Fig. 6 is another flow chart of a play processing method according to an embodiment of the present application;

Fig. 7 is a schematic structural diagram of a play processing device according to an embodiment of the present application;

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made more apparent and fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the application are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application.

The embodiment of the application provides a play processing method, a play processing device, electronic equipment and a computer readable storage medium. Specifically, the embodiment of the application provides a play processing device suitable for electronic equipment, wherein the electronic equipment comprises equipment such as televisions, mobile phones, desktop computers, notebook computers or tablet computers.

Referring to fig. 1, fig. 1 is a schematic view of a scenario in which a terminal device executes the playing processing method according to an embodiment of the present application, where a specific execution procedure of the terminal device executing the playing processing method is as follows:

The terminal device 10 obtains the multimedia data to be played, determines an initial playing flow for playing the multimedia data, then starts the initial playing flow, plays the current frame data of the multimedia data by using at least one playing node contained in the initial playing flow, and meanwhile invokes a preset playing maintenance interface to maintain a target playing node in the playing nodes to obtain a target playing flow, and then plays the rest frame data of the multimedia data according to the target playing flow.

The target playing node is a playing node which is commonly contained in at least two playing flows.

The current frame data and the residual frame data are frame data corresponding to the multimedia data, for example, audio frame data or video frame data, wherein the current frame data and the residual frame data have differences in processing time.

It can be understood that, in the embodiment of the present application, after obtaining the multimedia data to be played and determining the initial play flow for playing the multimedia data, playing the current frame data of the multimedia data by using a plurality of play nodes included in the initial play flow, and meanwhile, invoking a preset play maintenance interface to perform maintenance on a target play node in the play nodes to obtain a target play flow, where the target play node is a play node commonly included in the plurality of play flows, and then playing the remaining frame data of the multimedia data by using the target play flow. The target playing node is a common playing node contained in a plurality of playing processes, so that the target playing node is a universal node involved in a plurality of types of playing processes, and the universal target playing node is used as a basis for maintenance processing, so that each playing process can be maintained based on the universal target playing node, and the universality of the processing of the playing process is improved.

For example, when playing multimedia data, no matter what the playing flow is, the maintenance processing can be performed on the target playing node related to the playing flow, namely, the maintenance processing is performed on the universal playing node related to the target playing node, so that the diversity of the maintenance processing of the playing flow is improved.

In addition, the embodiment of the application carries out maintenance processing based on the playing node without depending on the specific terminal type related to the playing flow or the operating system of the terminal, thereby achieving the effect of cross-platform playing processing maintenance.

The following will describe in detail. It should be noted that the following description order of embodiments is not a limitation of the priority order of embodiments.

Referring to fig. 2, fig. 2 is a flowchart illustrating a play processing method according to an embodiment of the application. The specific flow of the method is as follows:

101. and acquiring the multimedia data to be played, and determining an initial playing flow for playing the multimedia data.

The multimedia data refers to a data set containing various media elements (such as text, image, audio, video, etc.), and is data that needs to be played, including audio data or video data.

The initial playing process is generated based on playing requirements of the multimedia data, for example, an audio playing process or a video playing process, and the playing processes are different based on different playing requirements, for example, a part of the playing processes may include a decoding node (or a decoding module), or may not include a decoding node (or a decoding module), or a part of the playing processes may include a synchronization module, or may not include a synchronization module.

It can be understood that the play links corresponding to the play flow are generally created in advance, based on the play requirement of the multimedia data, the play processing is started or performed on the multimedia data according to the play links, for example, a plurality of play nodes or play modules possibly involved in the play process are preconfigured, and the corresponding play links are selected along with the generation of the play requirement of the multimedia data, and then the play processing is performed according to the selected play links. The playing process of the multimedia data generally includes a basic process, a post-process, and an on-screen rendering process, where the basic process includes a process of decapsulation, decoding, synchronization, and the post-process includes a process of barrage control or special effect control, and the on-screen rendering renders the video data onto a screen, and correspondingly, the on-screen rendering also includes outputting audio by an audio output device (such as a loudspeaker).

Correspondingly, the playing processes of the multimedia data can also be different based on different platforms or different terminal devices, but the current playing detection maintenance processing scheme is mainly only used for detecting a certain module or detecting processing based on a specific scene of playing processing, and is difficult to be suitable for various playing processes or various playing platforms, so that the current playing process detection scheme is relatively chaotic and not universal enough.

Based on the above, the embodiment of the application designs a play processing method, which aims to solve the problem of abnormal detection and maintenance processing of different play processes of different platforms, improve the universality and application range of the play maintenance processing, and improve the reliability and accuracy of the play processes.

102. And starting the initial playing flow, and playing the current frame data of the multimedia data through at least one playing node in the initial playing flow.

It should be noted that, the play node is a play module or a processing module included in the play flow, for example, the play flow may include a data source module, a decapsulation module, a decoding module, a synchronization module, an output module, and the like, and correspondingly, the play node may include a data source node, a decapsulation node, a decoding node, a synchronization node, an output node, and the like.

After obtaining the multimedia data to be played and determining an initial playing flow for playing the multimedia data, the playing node involved in the initial playing flow can play the multimedia data.

The multimedia data is generally divided into data frames during processing, and each frame of data is sequentially played through a plurality of playing nodes related to an initial playing flow. Correspondingly, different frame data of the multimedia data are correspondingly processed according to different moments of playing processing. And the current frame data is the frame data processed at the current time.

In a general case, based on the playing processing of the multimedia data by each playing node in the initial playing flow, playing of the multimedia data, including video playing, audio playing and the like, can be realized. However, in the embodiment of the present application, in order to optimize a play flow, in a play processing procedure, a maintenance process is performed on a play node in the play flow, so as to expect to obtain an optimized target play flow.

103. And calling a preset play maintenance interface to maintain a target play node in the play nodes to obtain a target play flow, wherein the target play node is a play node commonly contained in at least two play flows.

It should be noted that, the play maintenance interface is an interface configured for the target play node, where the play maintenance interface is used to perform maintenance processing on the target play node, and the play maintenance interface corresponds to a code component to perform maintenance processing on the target play node according to the logic of the maintenance processing.

The target playing node is a part of playing nodes in the initial playing process, for example, the target playing node may be a decoding node or a synchronization node in the initial playing process.

It should be emphasized that the target playing node is also a playing node commonly included in a plurality of playing processes, for example, for different playing devices, different device operating systems or different playing applications, the playing node commonly included in the corresponding playing processes is taken as the target playing node, and it is understood that the target playing node is also a general playing node commonly included in a plurality of playing processes, that is, a playing node involved in most of the playing processes.

It can be understood that by taking the target playing node as the basis of the maintenance processing of the playing flow, the maintenance processing of each playing flow is generalized, for example, for different playing flows, the maintenance processing can be performed on the universal playing node involved in the playing flow, which is applicable to a plurality of different playing flows.

In the embodiment of the present application, the target playing node may perform similar or identical statistics on the playing nodes related to multiple playing flows, for example, playing nodes included in more than three playing flows are used as the target playing node.

It can be understood that the target playing node after the maintenance processing is obtained through the maintenance processing of the target playing node in the initial playing process, and correspondingly, the optimized target playing process is obtained based on the target playing node.

104. And playing the rest frame data of the multimedia data according to the target playing flow.

Correspondingly, after the optimized target playing flow is obtained, the playing processing can be carried out on the rest frame data in the multimedia data based on the target playing flow, so that the dynamic maintenance processing in the playing process of the multimedia data is realized.

Correspondingly, the multimedia data can be test type multimedia data in the process of maintaining and processing the playing flow, or can be multimedia data which is specifically played based on the actual requirement of the user, namely, the maintenance processing in the process of playing can be applied to the test link of the playing flow, or can be applied to the actual playing processing.

It may be understood that the play maintenance interface is configured for a target play node, and accordingly, in order to detect or determine whether the initial play flow includes the target play node, a node status interface for detecting whether the target play node exists may be configured, and based on execution of the node status interface, whether the initial play flow includes the target play node is determined, that is, optionally, in some embodiments of the present application, the play maintenance interface includes a node status interface, an anomaly detection interface and an anomaly repair interface, and the step of "calling a preset play maintenance interface to perform maintenance processing on the target play node in the play node to obtain a target play flow" includes:

invoking the node state interface to obtain a judgment result that the initial playing flow contains the target playing node;

If the judging result represents that the initial playing flow contains the target playing node, calling the abnormality detection interface to perform abnormality detection on the target playing node to obtain an abnormality detection result;

and if the abnormality detection result indicates that the target playing node is abnormal, calling the abnormality repairing interface to repair the target playing node to obtain a target playing flow.

The node state interface is used for judging whether the initial playing flow contains the target playing node or not, and judging results of whether the target playing node exists or not are obtained through execution of the node state interface.

Correspondingly, the maintenance processing of the target playing node comprises abnormality detection and abnormality restoration aiming at the target playing node, the abnormality detection is realized through an abnormality detection interface, the abnormality restoration is realized through an abnormality restoration interface, the abnormality detection interface can be called to carry out abnormality detection on the target playing node if the judgment interface obtained by the node state interface indicates that the target playing node exists, an abnormality detection result is obtained, and the abnormality restoration interface is called to carry out restoration processing on the target playing node if the abnormality detection result represents that the target playing node is abnormal, so that an optimized target playing node is obtained, and the optimized target playing flow is obtained.

Correspondingly, in the embodiment of the present application, different play maintenance interfaces may be configured correspondingly for different target play nodes, for example, different anomaly detection interfaces and anomaly repair interfaces are configured correspondingly for different target nodes, for example, an anomaly detection interface and an anomaly repair interface for decoding processing are configured for a decoding node, and an anomaly detection interface and an anomaly repair interface for synchronous processing are configured for a synchronous node.

The step of "calling the anomaly detection interface to perform anomaly detection on the target play node to obtain an anomaly detection result" in some embodiments of the present application includes:

invoking the abnormality detection interface to extract input data and output data of the target playing node;

obtaining an abnormality detection result according to the comparison result of the input data and the output data;

and/or the number of the groups of groups,

And obtaining an abnormality detection result according to the comparison result of the input data, the output data and the preset parameter threshold value.

By extracting the input data and the output data of the target playing node, whether the processing of the target playing node by using the input data and the output data accords with the expected detection analysis can be performed, so that an abnormal detection result can be obtained.

The preset parameter threshold is a preset parameter threshold, and a corresponding preset parameter threshold can be configured based on each target playing node.

Correspondingly, for different target playing nodes, different exception repairing strategies can be correspondingly configured, and the corresponding repairing strategies perform exception repairing processing on the target playing node with the detected exception, that is, optionally, in some embodiments of the present application, the step of "if the exception detection result indicates that the target playing node is abnormal, invoking the exception repairing interface to perform repairing processing on the target playing node to obtain a target playing flow" includes:

If the abnormality detection result represents that the target playing node is abnormal, determining an abnormality type corresponding to the abnormality detection result;

and calling the abnormal repair interface, and performing repair processing on the target play node according to a preset repair strategy corresponding to the abnormal type to obtain a target play flow.

Aiming at the target playing node, analyzing the type of the abnormality according to the detected abnormality detection result, and carrying out abnormality repair processing based on a pre-configured repair strategy.

For example, referring to fig. 3, fig. 3 is a schematic diagram of a general play flow provided in an embodiment of the present application, where several general play nodes involved in the general play flow are shown as an example, and the several general play nodes include a data source node a, a decapsulation node B, a decoding node C, a synchronization node D, and an output node E, respectively, specifically:

The data Source node a, also called Source node or data Source module, is the Source part of the data. Such as a local file source, HTTP source, HLS source, RTP source, FTP source, etc. For play-like scenes, source modules are often indispensable. The Source module needs to provide parsing of Source data, taking HLS Source as an example, the module needs to parse the Source according to HLS protocol, and then read out the Source data and transmit the Source data to the following module. Therefore, the Source module implements at least one read interface for reading the Source data.

The decapsulation node B, also called Demux node or decapsulation module, is responsible for parsing raw data from the container and delivering it to the subsequent modules. For example: TS_demux is responsible for resolving raw data such as video/audio/subtitle from TS file containers; MP4_Demux is responsible for parsing raw data such as video/audio/subtitle from MP4 file container. Thus, the Demux module implements at least one Demux interface. The Demux module is not necessary for a play-like scene.

The decoding node C, also called a Decoder node or a decoding module, is responsible for decoding raw data into data before encoding and compression. For example: the Video Decoder is responsible for decoding Video raw data into yuv or rgb frames; the Audio Decoder is responsible for decoding Audio raw data into PCM FRAMES; the Subtitle Decoder is responsible for decoding Subtitle raw data into text or picture data of subtitles. Thus, the Decoder module implements at least one Decoder interface. The Decoder module is not necessary for playback class scenarios.

The synchronization node D, also called A/V/S Sync node, or synchronization module, is mainly responsible for synchronous control of PTS of audio/video/subtitle to decide which frame to output. For example: the synchronization mechanism using audio as a master updates the system time stamp with audio PTS and compares the system time stamp with video PTS to decide whether to display or discard the frame video frame; similarly, the same is true for subtitle. Thus, the Sync module implements at least one render interface and one drop interface. The Sync module is not necessary for a playback class scenario.

The output node E, also called an A/V/S Sink node or an output module, outputs audio/video/subtitle frames to the corresponding device through operating device driving. For example: the device corresponding to the Video Sink may be a screen; the device corresponding to the Audio Sink may be a horn; the device corresponding to the Subtitle Sink may be a screen. Therefore, the Sink module needs to implement at least one show interface. For play-like scenes, sink modules are often indispensable.

Accordingly, a corresponding abnormality detection node interface (check ()) and abnormality repair interface (repair ()) are respectively configured for each node, for example,

1. For a data Source (Source) node:

The corresponding anomaly detection interface is denoted as source. Check (), which functions as: detecting whether the processing of the data source node is abnormal or not, wherein the design thought is as follows: the average bit rate (unit: b/s) read by the Source module is compared with the bit rate parameter in the stream container, and if the average bit rate is lower than a certain threshold (e.g., 20%), the Source module is indicated to be stuck. The bit rate parameters of the code stream are usually in the header information of the container and are easily parsed.

The corresponding exception repair interface is expressed as: source. Reply (), which acts to: repairing the data source node, wherein the design thought is as follows: for a network scene, repairing can be performed by improving network bandwidth and throughput; for the local scene, the IO rate of the disk can be improved for repairing.

2. For a decapsulation (Demux) node:

The corresponding anomaly detection interface is denoted demux. Check (), which functions as: the method comprises the following steps of detecting the abnormality of the unpacking node, wherein the design thought is as follows: the output of the Demux module is raw data (or ES data), which is the input of the next-stage Decoder module. The number inputCount of ES data sent to the Decoder and the number outputCount of yuv data acquired from the Decoder can be counted, and then the following steps are performed: when inputCount is less than outputCount and inputCount is less than a threshold (e.g., 50%) of the total, then the Demux module is considered stuck.

Its corresponding exception repair interface is denoted demux.repair (), which functions to: and carrying out exception repair treatment on the unpacking node, wherein the design thought is as follows: if the Demux module is stuck, delay may exist in the algorithm mechanism for explaining the decapsulation, so that a better idea is to directly optimize the algorithm flow. But if the priority is to be dynamically adjusted, the priority can be dynamically adjusted for the unpacked thread to achieve the repairing effect.

3. For a decoding (Decoder) node:

The corresponding anomaly detection interface is denoted as decoder. Check (), which functions as: detecting decoding nodes, wherein the design thought is as follows: the output of the Decoder module is yuv frame, which is the input of the next stage Sync module. The expected number of frames framesExpected for a period of time may be calculated by counting the number of yuv frames framesCount for the period of time and then calculating the expected number of frames framesExpected for the period of time based on the frame rate, and if framesCount is determined to be less than framesExpected and framesCount is less than a threshold (e.g., 80%) of the total number, then the Decoder module is considered to be stuck.

The corresponding exception repair interface is denoted as decoder.repair (), which functions as: performing exception repair processing on the decoding node, wherein the design thought is as follows: if soft solution is used, the stuck state can be repaired by switching to hard solution; if hard solution is used, the repairing effect can be achieved by increasing the number of output round buffers of decoding, dynamically adjusting the thread priority related to decoding, and the like.

4. For synchronization (Sync) nodes:

The corresponding abnormality detection interface is expressed as sync (), and has the functions of: the abnormal detection is carried out on the synchronous nodes, and the design thinking is as follows: the Sync synchronization module typically compares the PTS of Audio, video, subtitle with the system timestamp to determine how to process the frame. The Sync module has a synchronization interval, taking Video as an example, and when the difference between the PTS and the system time stamp is smaller than lowBound (lower limit) of the synchronization interval, the Video frame is drop (discarded); when the difference is greater than highBound (upper limit) of the sync interval, then the video frame will hold. Both of these conditions cause jamming to occur. Therefore, we can determine if stuck is occurring by checking if the frame synchronization state is drop or hold.

The corresponding exception repair interface is expressed as sync. Performing exception repair processing on the synchronous nodes, wherein the design thinking is as follows: the Sync module can accelerate the frame consumption by dynamically adjusting the synchronization interval so as to achieve the purpose of quick synchronization and repair the problem of stuck.

5. For output (Sink) nodes:

The corresponding abnormality detection interface is expressed as sink. Check (), and has the functions of: the output node is subjected to abnormality detection, and the design thought is as follows: the Sink module is a module related to a peripheral device, such as Window or AudioTrack, which may itself have buffers to buffer frame data. Therefore, by counting framesCount frames at the Sink end in a period of time, and then calculating the expected frame number framesExpected in the period of time according to the frame rate, if framesCount is determined to be greater than a threshold value (e.g. 30%) of framesExpected, it is considered that too many buffers are held by the Sink module, which may cause a jam to occur.

The corresponding abnormality repair interface is expressed as sink. Repair (), and has the functions of: performing exception repair processing on the output node, wherein the design thought is as follows: the repair can be performed by means of fast frame loss, fast cache clearing, dynamic adjustment of device bandwidth, dynamic adjustment of thread priority, etc.

In summary, for different target playing nodes, an abnormality detection interface and an abnormality repair interface can be correspondingly configured to realize abnormality detection and abnormality repair of the target playing nodes.

Accordingly, in the embodiment of the present application, the initial play flow may include a plurality of target play nodes, and accordingly, a plurality of play maintenance nodes are required to perform maintenance processing on the initial play flow, and according to a policy that an abnormality is detected first and a probability is a source of the abnormality, each play maintenance interface may be called according to a sequence of the target play node in the initial play flow, that is, optionally, in some embodiments of the present application, the play maintenance interfaces include at least two, and the step of "calling a preset play maintenance interface to perform maintenance processing on a target play node in the play node to obtain a target play flow" includes:

according to the calling sequence of the play maintenance interface, the play maintenance interface is called to carry out maintenance processing on a target play node corresponding to the play maintenance interface in the initial play flow, and a maintenance processing result is obtained, wherein the calling sequence is obtained according to the processing sequence of the play nodes in at least two play flows;

If the maintenance processing result meets the preset condition, obtaining a processed target playing node corresponding to the target playing node, and stopping calling a playing maintenance interface positioned behind the playing maintenance interface;

and obtaining a target playing flow according to the processed target playing node.

It can be understood that the calling sequence of the play maintenance interface is configured based on the play sequence of the target play node in the initial play process, and correspondingly, the sequence of the target play node in the initial play process also basically accords with the sequence in various other types of play processes.

Accordingly, when the maintenance processing result of a certain play maintenance interface meets a preset condition, the call to the subsequent play maintenance interface can be canceled, wherein the preset condition can be predefined based on the actual maintenance processing requirement, for example, in the embodiment of the present application, the preset condition can be that the abnormality detection interface of the play maintenance interface detects that the corresponding target play node is abnormal, and repair is completed based on the abnormality repair interface.

For example, referring to fig. 4, fig. 4 is a schematic diagram of a call sequence of a play maintenance interface provided by an embodiment of the present application, where a call sequence of a play maintenance interface a of a data source node and a play maintenance interface B corresponding to a decapsulation node is shown, where the play maintenance interface a is called first, and then the play maintenance interface B is called, where a node state interface A1, an anomaly detection interface A2, and an anomaly repair interface A3 are sequentially called when the play maintenance interface a is called, and a node state interface B1, an anomaly detection interface B2, and an anomaly repair interface B3 are sequentially called when the play maintenance interface B is called, where if the node state interface A1 does not detect a target play node (herein, the data source node) in an initial play flow, the play maintenance interface B corresponding to the decapsulation node is directly called; or if the node state interface A1 detects a target playing node (here, a data source node) in the initial playing flow, but the abnormality detection interface A2 does not detect an abnormality, directly calling a playing maintenance interface B corresponding to the decapsulation node.

Correspondingly, if the node state interface A1 detects a target playing node (here, a data source node) in the initial playing process and detects an abnormality through the abnormality detection interface A2, an abnormality repair interface A3 is called to repair the target playing node (here, the data source node) so as to obtain an optimized target playing node (here, the data source node).

Correspondingly, when the continuous repair of the abnormal repair interface fails, the next play maintenance interface is tried to be called for play maintenance processing, and the prompt information of the current target play node repair failure is returned.

It can be understood that the maintenance processing result includes an abnormality detection result and an abnormality repair result, which correspond to the abnormality detection result and the abnormality repair result, respectively.

Optionally, in the embodiment of the present application, a repair base class for a play maintenance interface may be established, and the play maintenance interface may be obtained by inheriting the repair base class and implementing a reference maintenance interface in the repair base class, that is, optionally, in some embodiments of the present application, the configuration of the play maintenance interface may include:

Configuring a repair base class containing an unrealized reference maintenance interface;

Aiming at the target playing node, establishing a playing maintenance subclass inherited from the repairing base class;

And according to the processing logic of the target playing node, configuring the implementation aiming at the reference maintenance interface in the playing maintenance subclass to obtain a playing maintenance interface.

The repair base class is a configured abstract class, and includes an unrealized reference maintenance interface, that is, the reference maintenance interface does not include a method body, and by configuring a play maintenance subclass for the target play node, and establishing inheritance of the play maintenance subclass for the repair base class (the repair base class is a parent class and the play maintenance subclass is a subclass), and implementing the reference maintenance interface in the play maintenance subclass, the play maintenance interface is obtained.

Correspondingly, the playing maintenance interface comprises a node state interface, an abnormality detection interface and an abnormality repair interface, when the initial playing process is started, each playing maintenance interface is sequentially called according to the calling sequence, whether the initial playing process contains a corresponding target playing node or not is judged through the node state interface in the playing maintenance interface, after the target playing node is detected, the abnormality detection interface is called to carry out abnormality detection on the execution of the target playing node, and the abnormality repair interface is called to carry out abnormality repair on the target playing node, so that the optimized target playing node is obtained, and the optimized target playing process is obtained.

Optionally, in the embodiment of the present application, after a general target playing node is specified, a playing processing subclass corresponding to the target playing node included in an initial playing flow may be inherited, so that the playing processing subclass inherits a corresponding repairing base class, to obtain a functional interface including a function of the functional interface, and further includes a node state interface, an abnormality detection interface and an abnormality repairing interface corresponding to a playing maintenance interface, that is, optionally, in some embodiments of the present application, the configuration of the playing maintenance interface further includes:

Setting the repair base class as a parent class aiming at the target play node, and establishing an inheritance relationship between play processing subclasses corresponding to the target play node and the repair base class;

and according to the processing logic of the target playing node, configuring the realization of the reference maintenance interface in the playing processing subclass to obtain a playing maintenance interface.

The playing processing subclass is a functional class or an object class corresponding to the target playing node, and includes a corresponding functional processing interface, for example, a decoding class of the decoding node includes a decoding interface, and a synchronization class of the synchronization node includes a synchronization interface.

Accordingly, after the playback processing subclass of the target playback node inherits the repair base class, the playback processing subclass includes a playback maintenance interface and its own functional interface, for example, refer to fig. 5, where fig. 5 is a schematic class diagram of the playback processing subclass corresponding to the data source node according to the embodiment of the present application, where the playback processing subclass includes a node status interface (source. Ismoduleexist ()), an anomaly detection interface (source. Check ()) and an anomaly repair interface (source. Repair ()) in addition to a data reading interface (read ()) of the data source node.

For example, when playing multimedia data, no matter what the playing flow is, the maintenance processing can be performed on the target playing node related to the playing flow, namely, the maintenance processing is performed on the universal playing node related to the target playing node, so that the diversity of the maintenance processing of the playing flow is improved. In addition, the embodiment of the application carries out maintenance processing based on the playing node without depending on the specific terminal type related to the playing flow or the operating system of the terminal, thereby achieving the effect of cross-platform playing processing maintenance.

The method comprises the steps of detecting whether a general target playing node is contained in an initial playing flow or not through a configuration node state interface, detecting the abnormality of the target playing node after the target playing node is detected through a configuration abnormality detection interface, and performing abnormality repair processing when the target playing node is detected to be abnormal through the abnormality detection interface through a configuration abnormality repair interface, so that maintenance processing of the general target playing node contained in the initial playing flow is realized.

In order to facilitate understanding of the embodiments of the present application, the following will generally describe the embodiments of the present application, and in particular, refer to fig. 6, where fig. 6 is another flow chart of a playing processing method provided by the embodiment of the present application, and the playing processing method includes:

201. The reference playing node is obtained by comparing differences of the playing nodes contained in each playing flow and selecting the commonly contained playing nodes from the differences;

202. Configuring a play maintenance interface for each reference play node, wherein the play maintenance interface comprises a node state interface, an abnormality detection interface and an abnormality repair interface;

203. after receiving the multimedia data to be played, determining an initial playing flow for playing the multimedia data;

204. Performing playing processing on the current frame data of the multimedia data according to the playing node in the initial playing flow;

205. according to each reference playing node, calling a corresponding node state interface to determine whether a target playing node corresponding to the reference playing node is included in the initial playing flow;

206. if the target playing node is included, calling an abnormality detection interface corresponding to the target playing node to perform abnormality detection on the target playing node, and obtaining an abnormality detection result;

207. if the abnormality detection result represents that the target playing node is abnormal, an abnormality repair interface corresponding to the target playing node is called to repair the target playing node, so that an optimized target playing node is obtained, and a target playing flow of an initial playing flow after optimization is obtained;

208. and playing the rest frame data of the multimedia data according to the target playing flow.

The maintenance processing of the playing flow is performed based on the passing target playing node, so that the maintenance processing is suitable for each playing flow, and is applicable to a plurality of playing flows because the maintenance processing is aimed at the playing node in the playing flow and is not aimed at playing equipment, playing application and terminal operation platform, so that the maintenance processing method in the playing process has wider application range and is generally applicable to a plurality of playing flows.

In order to facilitate better implementation of the playing processing method, the application also provides a playing processing device based on the playing processing method. The meaning of the nouns is the same as that in the playing processing method, and specific implementation details can be referred to the description in the method embodiment.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a playback processing device according to an embodiment of the present application, where the playback processing device may specifically be as follows:

The obtaining module 301 is configured to obtain multimedia data to be played, and determine an initial playing flow for playing the multimedia data;

The starting module 302 is configured to start the initial play procedure, and perform play processing on current frame data of the multimedia data through at least one play node in the initial play procedure;

The maintenance module 303 is configured to invoke a preset play maintenance interface to perform maintenance processing on a target play node in the play nodes, so as to obtain a target play flow, where the target play node is a play node that is commonly included in at least two play flows;

and the playing module 304 is configured to play the remaining frame data of the multimedia data according to the target playing procedure.

Optionally, in some embodiments of the present application, the play maintenance interface includes a node status interface, an anomaly detection interface, and an anomaly repair interface, and the maintenance module 303 includes:

and/or the number of the groups of groups,

Wherein, in some embodiments of the application, the third call unit comprises:

Wherein, in some embodiments of the present application, the play maintenance interface includes at least two, and the maintenance module 303 includes:

In the embodiment of the application, firstly, the acquisition module 301 acquires the multimedia data to be played, and determines an initial playing process for playing the multimedia data, then, the starting module 302 starts the initial playing process, the current frame data of the multimedia data is played through at least one playing node in the initial playing process, then, the maintenance module 303 invokes a preset playing maintenance interface to maintain a target playing node in the playing nodes, so as to obtain a target playing process, wherein the target playing node is a playing node commonly contained in at least two playing processes, and then, the playing module 304 plays the remaining frame data of the multimedia data according to the target playing process.

In the embodiment of the application, the target playing node is a playing node commonly contained in a plurality of playing processes, so that the target playing node is a universal node related to a plurality of types of playing processes, and the universal target playing node is used as a basis for maintenance processing, so that each playing process can be maintained and processed based on the universal target playing node, and the universality of the processing of the playing process is improved.

In addition, the present application further provides an electronic device, as shown in fig. 8, which shows a schematic structural diagram of the electronic device according to the present application, specifically:

The electronic device may include one or more processing cores 'processors 401, one or more computer-readable storage media's memory 402, power supply 403, and input unit 404, among other components. It will be appreciated by those skilled in the art that the electronic device structure shown in fig. 8 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. Wherein:

The processor 401 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 402, and calling data stored in the memory 402, thereby performing overall monitoring of the electronic device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, etc., and the modem processor mainly processes wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 401.

The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by executing the software programs and modules stored in the memory 402. The memory 402 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the electronic device, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

The electronic device further comprises a power supply 403 for supplying power to the various components, preferably the power supply 403 may be logically connected to the processor 401 by a power management system, so that functions of managing charging, discharging, and power consumption are performed by the power management system. The power supply 403 may also include one or more of any of a direct current or alternating current power supply, a recharging system, a power device commissioning circuit, a power converter or inverter, a power status indicator, etc.

The electronic device may further comprise an input unit 404, which input unit 404 may be used for receiving input digital or character information and generating keyboard, mouse, joystick, optical or trackball signal inputs in connection with user settings and function control.

Although not shown, the electronic device may further include a display unit or the like, which is not described herein. In this embodiment, the processor 401 in the electronic device loads executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application programs stored in the memory 402, so as to implement the steps in any play processing method provided in the embodiment of the present application.

After acquiring multimedia data to be played and determining an initial playing flow for playing the multimedia data, the embodiment of the application uses a plurality of playing nodes contained in the initial playing flow to play the current frame data of the multimedia data, and meanwhile, calls a preset playing maintenance interface to maintain a target playing node in the playing nodes to obtain a target playing flow, wherein the target playing node is a playing node commonly contained in the playing flows, and then uses the target playing flow to play the residual frame data of the multimedia data.

The target playing node is a common playing node contained in a plurality of playing processes, so that the target playing node is a universal node involved in a plurality of types of playing processes, and the universal target playing node is used as a basis for maintenance processing, so that each playing process can be maintained based on the universal target playing node, and the universality of the processing of the playing process is improved.

The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, the present application provides a computer readable storage medium having stored thereon a computer program that can be loaded by a processor to perform the steps of any of the playback processing methods provided by the present application.

Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

Because the instructions stored in the computer readable storage medium can execute the steps in any of the playing processing methods provided by the present application, the beneficial effects that any of the playing processing methods provided by the present application can be achieved, and detailed descriptions of the foregoing embodiments are omitted herein.

The foregoing has described in detail a playback processing method, apparatus, electronic device and computer readable storage medium, and specific examples have been applied to illustrate the principles and embodiments of the present application, and the above description of the embodiments is only for aiding in understanding the method and core idea of the present application; meanwhile, as those skilled in the art will vary in the specific embodiments and application scope according to the ideas of the present application, the present description should not be construed as limiting the present application in summary.

It should be noted that, in the specific embodiment of the present application, related data such as broadcast code stream data, interaction data, update status data of interaction data, storage status data, etc. are related, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and collection, use and processing of related data need to comply with related laws and regulations and standards of related countries and regions.

Claims

1. A play processing method, the method comprising:

2. The playback processing method of claim 1, wherein the playback maintenance interface comprises a node status interface, an anomaly detection interface, and an anomaly repair interface;

The calling the preset play maintenance interface to maintain the target play node in the play nodes to obtain a target play flow, which comprises the following steps:

3. The play processing method according to claim 2, wherein the calling the anomaly detection interface to perform anomaly detection on the target play node to obtain an anomaly detection result includes:

and/or the number of the groups of groups,

4. The playback processing method as claimed in claim 2, wherein if the abnormality detection result indicates that the target playback node is abnormal, invoking the abnormality repair interface to repair the target playback node to obtain a target playback flow, includes:

5. The play processing method according to claim 1, wherein the play maintenance interfaces include at least two, and the invoking the preset play maintenance interface performs maintenance processing on a target play node in the play nodes to obtain a target play flow, including:

6. The playback processing method of claim 1, wherein the configuration of the playback maintenance interface comprises:

7. The playback processing method of claim 1, wherein the configuration of the playback maintenance interface further comprises:

8. A playback processing apparatus, characterized in that the apparatus comprises:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the playback processing method of any one of claims 1-7 when the computer program is executed by the processor.

10. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the playback processing method as claimed in any one of claims 1 to 7.