CN113225605A

CN113225605A - Video playing processing method and device, electronic equipment and storage medium

Info

Publication number: CN113225605A
Application number: CN202010080911.5A
Authority: CN
Inventors: 胡玮; 张晓文; 吴桂盛
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-02-05
Filing date: 2020-02-05
Publication date: 2021-08-06
Anticipated expiration: 2040-02-05
Also published as: CN113225605B

Abstract

The invention provides a video playing processing method, a video playing processing device, electronic equipment and a storage medium; the method comprises the following steps: responding to a switching video operation aiming at a player, acquiring a target video of the switching video operation, and determining a target decoder instance which is adapted to the target video from decoder instances stored in a shared space; calling the target decoder instance in the shared space to decode media data included in the target video, and outputting a decoding result of the media data to be presented in a display interface of the player. The invention can effectively reduce the delay during video switching.

Description

Video playing processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to multimedia technologies, and in particular, to a video playing method and apparatus, an electronic device, and a storage medium.

Background

Multimedia forms, represented by videos (in particular short videos), are widely spread by means of the internet, in particular the mobile internet. Due to the characteristics of massive video, a very high-frequency use scene is a video which is switched and played in the process of watching the video, and if delay occurs in the switching process, the user experience is directly influenced. The related art lacks a general and efficient solution for reducing the delay of video playback switching.

Disclosure of Invention

The embodiment of the invention provides a video playing processing method and device, electronic equipment and a storage medium, which can effectively reduce the delay in video switching.

The technical scheme of the embodiment of the invention is realized as follows:

the embodiment of the invention provides a video playing processing method, which comprises the following steps:

responding to a switching video operation aiming at a player, acquiring a target video of the switching video operation, and determining a target decoder instance which is adapted to the target video from decoder instances stored in a shared space;

calling the target decoder instance in the shared space to decode media data included in the target video, and outputting a decoding result of the media data to be presented in a display interface of the player.

An embodiment of the present invention provides a video playing processing apparatus, including:

the device comprises an acquisition module, a video switching module and a video switching module, wherein the acquisition module is used for responding to video switching operation of a player and acquiring a target video of the video switching operation;

a determining module, configured to determine, from among the decoder instances stored in the shared space, a target decoder instance adapted to the target video;

a calling module for calling the target decoder instance in the shared space to decode media data included in the target video;

and the output module is used for outputting the decoding result of the media data to present in a display interface of the player.

In the above scheme, the apparatus further comprises a storage module,

the storage module is used for responding to the video switching operation of a player and storing the running decoder instance in the shared space when the historical video is decoded;

wherein the history video is a video played before the switching video operation video is received.

In the above scheme, the storage module is further configured to store, in the memory space of the shared space, the decoder resource corresponding to the decoder instance and the corresponding display interface output layer that are called when the historical video is decoded;

and the display interface output layer is used for presenting according to the decoding result of the received media data.

In the above scheme, the apparatus further comprises an initialization module,

the initialization module is configured to initialize and invoke a new decoder instance adapted to the target video when the target decoder instance adapted to the target video is not stored in the shared space, and output a decoding result to be presented in a display interface of the player.

In the above scheme, the device further comprises a release module,

the releasing module is configured to, when the memory space occupied by the shared space reaches a memory space threshold, release at least part of the decoder instances stored in the shared space until the memory space occupied by the shared space reaches the memory space threshold

The number of decoder instances in the shared space does not exceed a number threshold, or the memory space occupied by the shared space does not exceed a memory space threshold.

In the foregoing solution, the releasing module is further configured to perform at least one of the following releasing operations:

releasing decoder resources corresponding to the decoder instances entering the shared space and a corresponding display interface output layer according to a first-in first-out mode;

and releasing the decoder resources corresponding to the decoder instances in the shared space and the corresponding display interface output layers according to the descending order of the frequency of the decoders in the shared space used in the monitoring time.

In the above solution, the apparatus further comprises a preloading module,

the preloading module is configured to determine a decoder instance meeting a preloading condition when the player is initialized or when the shared space is empty during operation of the player, and determine the decoder instance meeting the preloading condition and execute the preloading condition

Preloading the decoder instances that satisfy a preload condition in the shared space.

In the above solution, the preloading module is further configured to determine a decoder instance that meets at least one of the following preloading conditions:

using a plurality of decoder instances preceding in descending order of frequency;

a plurality of decoder instances last called by the player before it was last turned off;

decoder instance adapted to the preferred video type.

In the above scheme, the preloading module is further configured to obtain context features related to the player and user features of a player user;

fusing the context features and the user features, and predicting the obtained fusion features by combining a machine learning model to obtain preloading probabilities corresponding to a plurality of candidate decoder instances;

preloading a plurality of candidate decoder instances which are in front in descending order of preloading probability into the shared memory;

wherein the contextual characteristics include at least one of: information of a video that the player has played; video switching frequency;

wherein the information of the video comprises at least one of: the type and name of the video; and the available memory space of the player is used when the video is played.

In the above solution, the calling module is further configured to intercept an application program interface of the decoder instance;

proxying the functionality of the intercepted application program interface by a proxy decoder instance comprising the same application program interface as the intercepted decoder instance.

In the above solution, the determining, from among the decoder instances stored in the shared space, a target decoder instance adapted to the target video includes:

acquiring configuration parameters of a decoder instance stored in a shared space and video parameters of the target video;

and determining the decoder example corresponding to the configuration parameter matched with the video parameter as a target decoder example adapted to the target video.

In the above solution, the invoking the target decoder instance in the shared space to decode media data included in the target video includes:

inserting preset type coding information into media data included in the target video to obtain edited media data, wherein the coding information includes at least one of the following: a sequence parameter set; a picture parameter set;

invoking the target decoder instance in the shared space to decode the edited media data.

An embodiment of the present invention provides a video playing processing device, including:

a memory for storing executable instructions;

and the processor is used for realizing the video playing processing method provided by the embodiment of the invention when the executable instructions stored in the memory are executed.

The embodiment of the present invention provides a computer-readable storage medium, which is characterized in that executable instructions are stored, and the executable instructions are used for causing a processor to execute the method for processing video playing provided by the embodiment of the present invention.

The embodiment of the invention has the following beneficial effects:

in the process of switching and playing the target video by the player, the decoder instance adapted to the target video can be called from the shared space, so that the decoder instance instantiated by the player can be flexibly multiplexed, a new decoder instance does not need to be created all the time, and the delay in video switching is effectively reduced.

Drawings

FIG. 1 is a schematic diagram of a decoder lifecycle according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating a decoder-based short video playing method according to an embodiment of the present invention;

fig. 3 is a schematic flowchart of decoder multiplexing in a player according to an embodiment of the present invention;

fig. 4 is a scene diagram of video switching definition according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an alternative architecture of a video playback processing system according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of a video playback processing apparatus according to an embodiment of the present invention;

fig. 7A is an alternative flowchart of a video playing processing method according to an embodiment of the present invention;

FIG. 7B is a flowchart illustrating a method for determining an example of a target decoder according to an embodiment of the present invention;

FIG. 7C is a flowchart illustrating a method for invoking an example of a target decoder according to an embodiment of the present invention;

FIGS. 8A-8B are schematic diagrams of an interface for video switching scenarios according to an embodiment of the present invention;

FIG. 9 is a flowchart illustrating a cross-player multiplexing decoder according to an embodiment of the present invention;

FIG. 10A is a diagram illustrating the multiplexing condition of a decoder according to an embodiment of the present invention;

fig. 10B is a schematic diagram of a multiplexing condition during adaptive playing according to an embodiment of the present invention;

fig. 10C is a schematic diagram of a non-adaptive play-time multiplexing condition according to an embodiment of the present invention;

FIG. 11 is a block diagram of a class design according to an embodiment of the present invention;

fig. 12 is a schematic diagram of an alternative architecture of a video playing system according to an embodiment of the present invention;

FIG. 13 is a diagram illustrating operation of a decoder multiplexing preamble according to an embodiment of the present invention;

FIG. 14 is a diagram illustrating a module call relationship according to an embodiment of the present invention;

fig. 15 is a comparison diagram of the video time consumption provided by the embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail with reference to the accompanying drawings, the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.

Before further detailed description of the embodiments of the present invention, terms and expressions mentioned in the embodiments of the present invention are explained, and the terms and expressions mentioned in the embodiments of the present invention are applied to the following explanations.

1) Video first frame: the process of reading a video from a player to render a first frame of the video for display on a display interface.

2) The player: multimedia playing frameworks that can provide playing capabilities, such as MediaPlayer, ExoPlayer, ijkplay.

3) A decoder: namely, a video decoder, which is used in a decoding link of video data in a player, for example, a MediaCodec decoder on an Android platform, creates each class in a static code of the decoder as an object in a memory space, which is called an instantiation process of the decoder, where the instantiated decoder is a decoder in a running state, which is also called a decoder instance, and occupies a certain memory resource (memory space), and when the decoder instance is released, object data in the memory space is emptied.

4) Proxy decoder (TMediaCodec): the function of the original decoder (MediaCodec) used by the proxy player can intercept MediaCodec and selectively multiplex, thereby reducing unnecessary initialization time.

5) Adaptive Playback (Adaptive Playback Support) refers to directly decoding the next video source content without resetting a decoder after switching the video source (video + audio) under certain conditions.

6) The decoder multiplexes: that one video is to be played is that the created decoder instance is multiplexed to decode other videos.

7) The shared space, also called shared pool, is used to store the memory space of the already instantiated decoders.

For example, a shared pool (KeepPool) is maintained, and when one video finishes playing, the decoder does not directly destroy the decoder instance, but puts it in the shared pool and maintains its state so that other player instances (i.e. when the player plays other videos) can be multiplexed.

The inventor finds that, in the process of implementing the video playing processing method provided by the embodiment of the present invention, the conventional short video playing method is based on a decoder (MediaCodec), and implements the function of coding and decoding by accessing the underlying decoder, and the MediaCodec processes data in an asynchronous manner and uses a set of input and output buffers (buffers). Referring to fig. 1, fig. 1 is a schematic diagram of a life cycle of a decoder according to an embodiment of the present invention, as shown in fig. 1, in a process of encoding and decoding, a life cycle of MediaCodec includes three states: a stop state (Stopped), an execution state (Executing), and a release state (Released), wherein the stop state includes: the system comprises three sub-states of an uninitialized state, a configured state, an error state and the like, and an execution state comprises three sub-states of a refreshing state, a running state, a flow ending and the like.

The rule switching process of the three sub-states of the stop state is as follows: when a MediaCo dec object is created, which is in an uninitialized state, the MediaCodec can be reset in any state to return to the uninitialized state, and the MediaCodec can be configured to go to a configured state. The error state is entered when the MediaCodec encounters an error, which may be caused by an error or exception returned during the queue operation.

The rule switching process of the three sub-states of the execution state is as follows: after calling the start function, the MediaCode c immediately enters a refresh substate, and at the moment, the MediaCode c has all caches and can return to the refresh substate by refreshing at any time of the execution state; once the first input buffer is removed from the queue, Me diaCodec is transferred to a run sub-state, which occupies most of the life cycle of MediaCodec, and is transferred to an uninitialized state by calling a stop function; when an input with an end-of-stream flag is buffered in the queue, MediaCodec transitions to the end-of-stream sub-state.

After the MediaCodec is used up, the release function is called to release the resources of the MediaCodec, and the terminal release state is entered. Referring to fig. 2, fig. 2 is a schematic flowchart of a decoder-based short video playing method according to an embodiment of the present invention, and since one player only corresponds to one decoder and can only decode videos in configured coding formats, as shown in fig. 2, in the short video playing provided by the embodiment of the present invention, when one short video is played each time, the player restarts a new decoder, specifically:

1) creating a decoder:

this process corresponds to the uninitialized state shown in fig. 1, and in actual implementation, the create decoder is implemented by the function MediaCodec.

2) Configuring a decoder:

this process corresponds to the configuration state shown in fig. 1, and in practical implementation, the decoder is configured through mediacodec.config ure (), and video coding parameters such as coding format, video width, and the like to be played are configured. Typically, a decoder with configured parameters can only decode the corresponding encoded format video.

3) Turning on a decoder:

this process corresponds to the operational state shown in fig. 1, and in actual implementation, the decoder is started by mediacodec.start (), enters the operational state, and has decoding capability.

Through the above mode, the coding and decoding process based on MediaCodec is as follows: first, an empty input buffer is requested from the MediaCodec, and the buffer is transferred to the MediaCodec for processing after being filled with data; then Me diaCodec finishes processing the data and outputs the processing result to an empty output cache; finally, obtaining the data of the output buffer from the MediaCodec, consuming the data in the data, and releasing the data of the output buffer back to the decoder after using the data of the output buffer; the switching process of video playback based on this is time consuming.

Further, the embodiment of the present invention improves time consumption of the above conventional mode by using ExoPlayer, and provides a multiplexing scheme for a decoder in a player, see fig. 3, and fig. 3 is a schematic diagram of a multiplexing process of a decoder in a player provided by the embodiment of the present invention, as shown in fig. 3, if a decoder played last time can be just multiplexed by this playing in the same player, an initialization process of the decoder is not required. Therefore, when a new short video is played each time, whether a decoder played last time in the player can be multiplexed or not is judged, and if yes, the decoder is directly used; if not, the current decoder is released and the new decoder is reinitialized.

However, since ExoPlayer is an intra-player decoder multiplexing scheme that employs a decoder that does not destroy previous decoders, and the decoder is only internal to the player instance, it can only be determined that the last decoder cannot be multiplexed, and cannot be multiplexed across players. Moreover, one player hardware only corresponds to one decoder, and only can decode the video with the adaptive coding format and width and height, and can be well multiplexed only when the currently played video source is consistent with the last played video source; however, in an actual short video scene, the video characteristics such as the encoding format are random and are not fixed, and therefore, the scheme based on Ex oPlayer results in a low multiplexing rate. In addition, due to the randomness of the short video source, the scheme based on the multiplexing decoder in the player is more suitable for the scene with video switching definition as shown in fig. 4, the video can be switched to the scene with different definition according to the network speed, the improvement effect on the first frame is more limited, and the delay of video playing switching cannot be effectively reduced.

In view of the above, embodiments of the present invention provide a video playing processing method, an apparatus, an electronic device, and a storage medium, which can reduce delay in video switching, and an exemplary application of the video playing processing device provided in the embodiments of the present invention is described below. In the following, exemplary applications of the electronic device will be explained.

Referring to fig. 5, fig. 5 is an alternative architecture diagram of the video playback processing system 100 according to the embodiment of the present invention, in order to support a video playback processing application, the terminal 400 (exemplary shown are the terminal 400-1 and the terminal 400-2) is connected to the server 200 through the network 300, and the network 300 may be a wide area network or a local area network, or a combination of the two.

In practical applications, the form of the player may be a dedicated player client (APP, Native APP), or a Web page player in a browser, or may be a player in the form of an applet (Web APP), for example, the terminal 400 is provided with a player for displaying the short video to be switched acquired from the server 200 on a graphical interface 410 (the graphical interface 410-1 and the graphical interface 410-2 are exemplarily shown), when a target user clicks the short video to be switched, a codec manager of the player on the terminal 400 acquires a target video for switching the video operation in response to the video switching operation for the player, and determines a target decoder instance adapted to the target video from the decoder instances stored in the shared pool; and calling a target decoder instance in the sharing pool to decode the media data included in the target video and output the media data to be presented in a display interface of the player.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a video playback processing apparatus 500 according to an embodiment of the present invention, where the video playback processing apparatus 500 shown in fig. 6 includes: at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components in video playback processing device 500 are coupled together by a bus system 540. It is understood that the bus system 540 is used to enable communications among the components. The bus system 540 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration the various buses are labeled as bus system 540 in figure 6.

The Processor 510 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 530 includes one or more output devices 531 enabling presentation of media content, including one or more speakers and/or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components to facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 550 optionally includes one or more storage devices physically located remote from processor 510.

The memory 550 may comprise volatile memory or nonvolatile memory, and may also comprise both volatile and nonvolatile memory. The nonvolatile memory may be a Read Only Memory (ROM), and the volatile memory may be a Random Access Memory (RAM). The memory 550 described in connection with embodiments of the invention is intended to comprise any suitable type of memory.

In some embodiments, memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 551 including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

a network communication module 552 for communicating to other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

a presentation module 553 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more output devices 531 (e.g., a display screen, speakers, etc.) associated with the user interface 530;

an input processing module 554 to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

In some embodiments, the apparatus provided by the embodiments of the present invention may be implemented in software, and fig. 6 shows an apparatus 555 stored in a memory 550, which may be software in the form of programs and plug-ins, and the like, and includes the following software modules: the obtaining module 5551, the determining module 5552, the calling module 5553 and the outputting module 5554 are logical modules, and thus may be arbitrarily combined or further split according to the implemented functions.

The functions of the respective modules will be explained below.

In other embodiments, the video playing processing apparatus provided in the embodiments of the present invention may be implemented in hardware, and for example, the video playing processing apparatus provided in the embodiments of the present invention may be a processor in the form of a hardware decoding processor, which is programmed to execute the video playing processing method provided in the embodiments of the present invention, for example, the processor in the form of the hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components.

The video playing processing method provided by the embodiment of the present invention will be described in conjunction with exemplary applications and implementations of the terminal provided by the embodiment of the present invention.

Referring to fig. 7A, fig. 7A is an optional flowchart of a video playing processing method according to an embodiment of the present invention, and the steps shown in fig. 7A are described by taking a player running in a terminal as an example.

The form of the player can be a special player APP, a webpage player in a browser, or a small program type player.

Step 301: and the coding and decoding manager of the player responds to the video switching operation aiming at the player and acquires a target video of the video switching operation.

For example, a player APP is provided on the terminal, and when a user clicks a switch key for the player, a corresponding target video is obtained from the terminal or the server.

Step 302: from the decoder instances stored in the shared space, a target decoder instance adapted to the target video is determined.

Referring to fig. 7B, fig. 7B is a flowchart illustrating a method for determining a target decoder instance according to an embodiment of the present invention, and in some embodiments, the step 302 shown in fig. 7A can be implemented by

steps

3021 and 3022 as shown in fig. 7B:

step 3021: and acquiring configuration parameters of a decoder instance stored in a shared space and video parameters of the target video.

Step 3022: and determining the decoder example corresponding to the configuration parameter matched with the video parameter as a target decoder example adapted to the target video.

Here, adaptation refers to video size (width and height), encoding format, angle, and buffer size supported by the decoder configuration, and is compatible with the size, encoding format, angle, and buffer size of the target video. In actual implementation, whether the decoder instance is adapted to the target video is determined by comparing the decoder configuration parameters with the video parameters of the target video. Generally, when the angle, the encoding format, and the angle and the encoding format of the target video are the same, the decoder buffer size is larger than or equal to the buffer size of the target video, and the video width and height also satisfy a certain condition, the decoder instance is considered to be adapted to the target video.

For a decoder supporting adaptive playing capability, as long as the maximum video width of the configured decoder is greater than or equal to the video width of the target video and the maximum video height of the configured decoder is greater than or equal to the video height of the target video, the decoder instance is considered to be adapted to the target video. For a decoder that does not support adaptive playback capability, the decoder instance is considered to be adapted to the target video only if the video width and height of the configured decoder are completely the same as the video width and height of the target video.

In some embodiments, prior to determining a target decoder instance that fits the target video from among the decoder instances stored in the shared pool, the codec manager of the player further stores, in the shared pool, decoder instances that were run when decoding the historical video in response to a switch video operation for the player; wherein, the history video is the video played after receiving the switching video operation video.

In practical implementation, processor resources occupied by the decoder instance operated during decoding of the historical video can be kept, the decoder instance operated during decoding of the historical video is migrated to the memory space corresponding to the shared pool from the originally occupied memory space, and the originally occupied memory space is released, and the shared pool in the mode does not occupy the special memory space in the RAM; or the decoder running when the historical video is decoded is mapped to the memory space corresponding to the shared pool from the originally occupied memory space without migration, and the shared pool in the mode can occupy a section of continuous memory space in the RAM.

In some embodiments, the codec manager of the player may store the instance of the decoder that was run when decoding the historical video in the shared pool by:

storing decoder resources corresponding to decoder instances called when the historical videos are decoded and corresponding display interface output layers into a memory space of a shared pool; and the display interface output layer displays according to the decoding result of the received media data.

Here, the decoder resources are instantiated, including objects created in memory space for each class in the decoder's code.

In some embodiments, when a target decoder instance adapted to the target video is not stored in the shared pool, the server also initializes and invokes a new decoder instance adapted to the target video, and outputs the decoding result for presentation in the display interface of the player.

Here, that the target decoder instance adapted to the target video is not stored in the shared pool means that the decoder instance is not stored in the shared pool, or the stored configuration parameters of the decoder instance and the target video parameters do not satisfy the multiplexing condition.

Step 303: a target decoder instance in the shared pool is invoked to decode media data included in the target video.

Referring to fig. 7C, fig. 7C is a flowchart illustrating a method for invoking an example of a target decoder according to an embodiment of the present invention, in some embodiments, the invoking of the example of the target decoder in the shared pool in step 303 shown in fig. 7A may be implemented by

steps

3031 and 3032 shown in fig. 7C:

step 3031: intercepting an application program interface of a decoder example by a coding and decoding manager of the player;

step 3032: proxying the functionality of the intercepted application program interface by a proxy decoder instance comprising the same application program interface as the intercepted decoder instance.

Here, the proxy decoder instance has an interface that is completely consistent with the intercepted decoder instance, and the proxy decoder instance proxies the functionality of the intercepted original decoder instance.

In some embodiments, a target decoder instance in the shared space may be invoked to decode media data included in the target video by:

inserting preset type coding information into media data included in a target video to obtain edited media data, wherein the coding information includes at least one of the following: a sequence parameter set; a picture parameter set; a target decoder instance in the shared space is invoked to decode the edited media data.

Here, when the decoder does not satisfy the multiplexing condition, processing may be performed as necessary so that the decoder instance can understand the content of the new video, for example, before decoding the video, decoding information such as a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), and the like is inserted so that the decoder instance can understand that the video source is switched; or, for a decoder supporting the adaptive playing capability, if the maximum video width height of the configured decoder is less than or equal to the video width height of the video to be decoded, a 32 × 32 special construction frame is inserted before the video to be decoded is decoded, so that the decoder can understand that the video source is switched; or call a refresh function flush () of the decoder to clear buffer data of the previously decoded video.

Step 304: outputting the decoding result of the media data for presentation in a display interface of the player.

In some embodiments, when the memory space occupied by the shared pool reaches the memory space threshold, the codec manager of the player further releases at least part of the decoder instances already stored in the shared pool until the number of decoder instances of the shared pool does not exceed the number threshold, or the memory space occupied by the shared pool does not exceed the memory space threshold.

Here, the memory space threshold may be a preset value; may be dynamically set according to the actual available memory space of the electronic device running the player, e.g. not exceeding 5% of the available memory space.

In some embodiments, the codec manager of the player may release at least some of the decoder instances already stored in the shared pool by: performing at least one of the following release operations: releasing decoder resources corresponding to the decoder instances entering the shared pool and corresponding display interface output layers according to a first-in first-out mode; and releasing the decoder resources corresponding to the decoder instances in the shared pool and the corresponding display interface output layers according to the descending order of the frequency of the decoders in the shared pool used in the monitoring time.

Here, when the shared pool is operated, its state is judged, and when it exceeds the maximum memory of the shared pool, the decoder therein is eliminated. The elimination strategy here can be set by itself, such as First-in First-out (FIFO), which means that the First-in decoder is eliminated First, and then the second-in decoder is eliminated. The obsolete decoder is removed from the shared pool, really freeing the memory occupied by the resources.

In addition, elimination strategies such as a Least Recently Used (LRU) algorithm, a Least Recently Used (LFU) algorithm and the like may be adopted to eliminate decoder resources corresponding to the Least Recently Used or Least Recently Used decoder instances and corresponding display interface output layers within a preset time period of the last time in the shared pool.

In some embodiments, when the player is initialized or when the shared pool is empty during the running of the player, the codec manager of the player determines the decoder instance satisfying the preloading condition and preloads the decoder instance satisfying the preloading condition in the shared pool.

In some embodiments, the codec manager of the player may determine the decoder instance that satisfies the preload condition by: determining decoder instances that satisfy at least one of the following pre-load conditions: using a plurality of decoder instances preceding in descending order of frequency; a plurality of decoder instances last called by the player before it was last turned off; decoder instance adapted to the video type preferred by the player user.

Here, in consideration of the similarity of short videos, it is preferable to determine a decoder instance that has been used most frequently within a last preset time as a decoder instance satisfying the preload condition, or to determine a decoder instance that was last called before the player was last turned off as a decoder instance satisfying the preload condition, or to determine a decoder instance of a video type that conforms to the user preference of the player as a decoder instance satisfying the preload condition based on portrait information of the user of the player.

In some embodiments, the codec manager of the player may also determine the decoder instance that satisfies the preload condition by: obtaining context characteristics related to a player and user characteristics of a player user; fusing the context characteristics and the user characteristics, and predicting the obtained fusion characteristics by combining a machine learning model to obtain preloading probabilities corresponding to a plurality of candidate decoder instances; preloading a plurality of candidate decoder instances at the front in the descending order of the preloading probability into a shared memory; wherein the contextual characteristics include at least one of: information of a video that the player has played; video switching frequency; wherein the information of the video comprises at least one of: the type and name of the video; the available memory space of the player when playing video.

The machine learning technology is used for predicting the context characteristics such as the information of the video played by the player or the video switching frequency and the fusion characteristics of the user characteristics of the player user to obtain the preloading probability corresponding to a plurality of candidate decoder examples, and the candidate decoder examples with the preloading probability exceeding the probability threshold are selected and preloaded into the shared memory.

By the mode, in the process of switching and playing the target video by the player, the decoder instance adapted to the target video is called from the shared pool, so that the instantiated decoder instance of the player can be flexibly multiplexed, a new decoder instance does not need to be created all the time, and the delay in video switching is effectively reduced.

In the following, an exemplary application of the embodiments of the present invention in a practical application scenario will be described.

Aiming at the problems that the switching process of video playing is time-consuming when the short video playing method based on MediaCodec provided by the embodiment of the invention is used for carrying out video playing, and the initialization time consumption of a decoder is consumed when the video playing is switched due to the fact that a scheme based on ExoPlayer can only judge that the last decoder can not be multiplexed and can not be multiplexed across players, the embodiment of the invention provides a video playing processing method, wherein a coding and decoding manager of a player responds to the video switching operation aiming at the player to obtain a target video of the video switching operation, and determines a target decoder example adaptive to the target video from the decoder examples stored in a sharing pool; and calling a target decoder instance in the sharing pool to decode the media data included in the target video and output the media data to be presented in a display interface of the player.

The video playing processing method provided by the embodiment of the invention is based on a cross-player decoder multiplexing scheme, has a decoder sharing pool, considers multiplexing from the global perspective, and can preload a proper decoder in the sharing pool due to the existence of the decoder sharing pool, thereby being more suitable for short video playing and switching scenes of decoder instances. When the decoder is selected, due to the shared pool concept breaking the limitation of the player, more choices can be provided, and more decisions can be made.

By the video playing processing method provided by the embodiment of the invention, in a short video playing scene, because the short video is switched relatively frequently, and the adaptive playing characteristic of the decoder is utilized, when the next player plays, a proper decoder can be directly selected from the shared pool without initializing the decoder every time, and the flow of the first frame of the video is optimized, so that the time required by the first frame is greatly reduced, and the video playing experience is optimized finally.

For example, referring to fig. 8A-8B, fig. 8A-8B are schematic interface diagrams of a video switching scene according to an embodiment of the present invention, as shown in fig. 8A-8B, when a user watches a short video and switches from the video a shown in fig. 8A to the next video B, when the playing of the video a is stopped, the decoder for the video a is not directly destroyed, but is placed in the sharing pool; when the B video starts to play, a new decoder instance is not directly reinitialized, and a proper decoder instance is selected from the shared pool for multiplexing, so that the time required by the first frame of the video is saved. Likewise, when switching from the C video shown in fig. 8B to the next video D, the decoder for the C video is put into the shared pool when the playback of the C video is stopped; when the D video starts to be played, a proper decoder instance is selected from the shared pool for multiplexing, so that the time required by the first frame of the video is saved.

In practical applications, when switching videos, the process of multiplexing the decoders among videos is greatly changed, and the description continues with a video switching scene in which a video is switched to B video shown in fig. 8A, and when switching videos, the player a is stopped first, and then the player B is started to play, referring to fig. 9, where fig. 9 is a schematic flow diagram of a cross-player multiplexing decoder provided by an embodiment of the present invention, and will be described with reference to fig. 9:

first, when the a player stops, the decoder is "recycled", where "recycling" means that the decoder is not used any more, and does not mean that the memory space occupied by the decoder resource is released. On the contrary, the decoder resources and the output layer (i.e. Surface) carrying pictures bound with the decoder are reserved so that the pictures of the video are rendered on the Surface, generally speaking, the decoder needs to bind one Surface before playing the video to output the pictures, and when stopping the video, the binding is released. With the above "reclamation," the decoders of the a-player are hosted by a global shared pool and the state of the decoders is maintained.

Next, before the player B starts playing, a suitable decoder is selected from the shared pool, and here, whether the decoder selected from the shared pool is suitable is determined by determining whether the decoder satisfies the multiplexing condition, referring to fig. 10A, fig. 10A is a schematic diagram of the multiplexing condition of the decoder provided in the embodiment of the present invention, and as shown in fig. 10A, the configuration parameters of the decoder are respectively compared with the video parameters of the video to be decoded to determine whether the decoder is suitable for the video to be decoded. In practical implementation, when the angle and the encoding format of the decoder configuration and the angle and the encoding format of the video to be decoded are respectively the same, the size of the decoder buffer is greater than or equal to the size of the buffer of the video to be decoded, and the width and the height of the video also meet certain conditions, the decoder is considered to be suitable for the video to be decoded.

Next, two cases are used to describe the parameter of the video width and height, referring to fig. 10B, fig. 10B is a schematic diagram of multiplexing conditions during adaptive playing provided by the embodiment of the present invention, as shown in fig. 10B, for a decoder supporting adaptive playing capability, as long as the maximum video width of the configured decoder is greater than or equal to the video width of the video to be decoded, and the maximum video height of the configured decoder is greater than or equal to the video height of the video to be decoded, the decoder is considered to be suitable for the video to be decoded. Referring to fig. 10C, fig. 10C is a schematic diagram of multiplexing conditions during non-adaptive playing according to an embodiment of the present invention, as shown in fig. 10C, for a decoder that does not support adaptive playing capability, only when the video width and height of the configured decoder are completely the same as the video width and height of the video to be decoded, the decoder is considered to be suitable for the video to be decoded.

By the above manner, a suitable decoder can be selected from the shared pool to decode the video to be decoded, and it should be noted that when a suitable decoder cannot be found from the shared pool, a new decoder needs to be created by itself.

The following description will be made of a shared pool, generally speaking, the shared pool is a multiplexing pool of managed decoders, which is established when playing APP starts, and is a storage structure similar to an array, and is stored in a memory. When the shared pool is operated, the state of the shared pool is judged, and when the state exceeds the maximum memory of the shared pool, the decoder in the shared pool is eliminated. The elimination strategy can be set by itself, such as LRU algorithm, LFU algorithm, F IFO algorithm and the like. The obsolete decoder is removed from the shared pool, really freeing the memory occupied by the resources.

Here, the maximum memory of the shared pool may be a preset value, for example, the maximum number of decoders, and the maximum value of memory space occupied by the decoders; the number of the available memory spaces and the number of video switching times/frequency can be determined comprehensively, and the parameters are positively correlated with the value of the memory size of the shared pool (at least one of the parameters can be used to establish a linear/non-linear positive correlation with the memory), so that the fluency and the operation stability (the memory cannot overflow) during frequent switching are ensured.

When the shared pool is empty, that is, when the player app is started and video playing is not performed, in order to solve the time delay of video switching caused by the time taken for the player to acquire decoder initialization later, preloading is performed here, that is, an instance of a decoder that may be used is created in advance, for example, the historical use times/frequencies of different decoders are counted, the decoder is arranged in a descending order, and the part of the decoder instance which is arranged in the front and is adapted to the preferred video type is preloaded.

In practical applications, as to what kind of decoder is preloaded, machine learning can be adopted to predict, specifically, context features related to the player and user features of the player user are obtained; fusing the context characteristics and the user characteristics, and predicting based on the machine learning model and the obtained fusion characteristics to obtain preloading probabilities corresponding to a plurality of candidate decoder instances; preloading a plurality of candidate decoder instances at the front in the descending order of the preloading probability into a shared memory; wherein the contextual characteristics include at least one of: information of a video that the player has played; video switching frequency; the information of the video includes at least one of: the type and name of the video; the available memory space of the player when playing video.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a class design provided by an embodiment of the present invention, and as shown in fig. 11, an interception module intercepts an interface of a raw decoder (MediaCodec), and proxies the raw decoder (MediaCodec) function by a proxy decoder (TMediaC codec); a proxy decoder management module (tcodecmanager) globally manages a proxy decoder map (TMediaCodecMap), decoder multiplexing processing logic (R euseHandler), Policy configuration module (Policy), and configuration module (Config); the decoder wrapper module includes a decoder wrapper class (CodecWrapper) including a video decoder wrapper class (videocodecwrapper) and an audio decoder wrapper class (AudioCodecWrapper) and a direct encoder wrapper class (directcwrrapper). The decoder pool includes a run pool (Runi ngPool), a shared pool (KeepPool), and a recycle pool (recycled pool). The function of each class will be explained next.

Referring to fig. 12, fig. 12 is a schematic diagram of an alternative architecture of a video playing system according to an embodiment of the present invention, and as shown in fig. 12, the video playing system according to the embodiment of the present invention includes: the interception module, the strategy configuration module, the coding and decoding management module, the decoder packing module and the decoder pool are explained one by one as follows:

1. interception module

The interception module is composed of three parts, namely a proxy decoder example (TMediCodec), a texture view controller (THookte xtureView) and a third-party interception library, and is mainly used for intercepting an interface of an original decoder example (MediaCodec) and proxying the functions of the original decoder example.

1) Proxy decoder (TMediaCodec): having a completely consistent interface with the original decoder (MediaCodec), acting on the functionality of the MediaCodec, performs the decoding.

2) Texture view controller (THookTextureView): the proxy raw texture view controller (Textu view) ensures that the state of the output layer is not released when being unbound by the player, because the MediaCodec state is abnormal after the output layer is released and cannot be decoded again, that is, cannot be multiplexed again, the timing of releasing the output layer needs to be controlled here to ensure that the MediaCodec decoding state is normal.

3) A third party interception library: generally, the function of the proxy class requires replacing the class with the proxy class. The interception library can directly proxy the functions of the classes under the condition of not making code intrusion.

2. Policy configuration module

1) Elimination strategy: the memory size of the shared pool in the decoder pool is 2 by default, when the decoder is full, the elimination strategy is needed, FIFO by default, and configuration can be carried out.

2) The decoder configuration strategy is as follows: the MediaCodec is configured in the configuration stage not directly according to the parameters of the video, but according to the policy, so as to adapt to more video plays, here default is one each of the 2k video H264, H265 encoding configurations.

3. Agent decoder management module (TCodecManager)

1) Proxy decoder mapping: original decoder instance (MediaCodec) and proxy decoder (tmedicodec) mappings are maintained to facilitate lookup.

2) Decoder allocation and reclamation: decoder allocation and reclamation is managed by the TCodecManager, deciding to which player instance a decoder instance is allocated, and also deciding when a decoder instance is reclaimed.

3) Decoder multiplexing processing logic: in some cases, due to the problem of model compatibility, processing is required before multiplexing, and a specific processing manner can be seen in fig. 13, fig. 13 is an operation schematic diagram of a decoder multiplexing front-end provided in the embodiment of the present invention, as shown in fig. 13, a video and an audio in a target video are separated into elementary streams, and when a decoder meets a multiplexing condition, the video and the audio are directly multiplexed without special processing; when the decoder does not meet the multiplexing condition, processing can be performed as required to enable the decoder instance to understand the content of the new video, for example, for a decoder supporting the adaptive playing capability, if the maximum video width and height of the configured decoder is less than or equal to the video width and height of the video to be decoded, a 32 × 32 special construction frame is inserted before the video to be decoded is decoded, so that the decoder can understand that the video source is switched; or calling a refresh function flush () of a decoder to clear the buffer data of the previously decoded video; alternatively, decoding information such as SPS, PPS, etc. is inserted before decoding the video so that the decoder instance can understand that the video source is switched. Likewise, the audio data is processed accordingly.

4. Monitoring module

The monitoring module is used for generally failing to decode if an abnormality occurs in the whole process of the proxy MediaCodec function. The monitoring module can record the site and report the log.

5. Decoder packaging module

For packaging MediaCodec, different packaging classes are used in different situations.

1) Decoder wrapper class (CodecWrapper): the wrapper class of a decoder instance refers to MediaCodec, not TMediaCodec, the base class of all wrapper classes, containing the decoder instance.

2) Multiplex decoder wrapper class (reusecodwrapper): a wrapper class of decoder instances that possess multiplexing capabilities.

3) Video decoder wrapper class (VideoCodecWrapper): for packaging of video decoders.

4) Audio decoder wrapper class (AudioCodecWrapper): for packaging of audio decoders.

5) Direct encoder wrapper class (DirectCodecWrapper), the wrapper class of the decoder instance that does not multiplex.

6. Decoder pool

1) Running pool (RunningPool): the decoder instance that has been bound to the output layer is mapped, where mapping refers to storing in memory in the form of key value pairs Map.

2) Shared pool (KeepPool): the unbound decoder instances are retained for reuse by other players.

Referring to fig. 14, fig. 14 is a schematic diagram of a module call relationship provided by the embodiment of the present invention, as shown in fig. 14, all places used in MediaCodec in the embodiment of the present invention are replaced by a proxy decoder (tmediacodec), when in use, a proxy decoder management module (tcodecmager) globally manages a decoder pool and the like in the present solution, and a target video processed by a video/audio decoder determines whether an original decoder can be multiplexed or not and how to process a multiplexed decoder example through a decoder multiplexing processing logic. In actual implementation, the proxy decoder management module maps the decoder instance bound with the output layer into the running pool, and when the decoder instance needs to be recycled and released, the decoder instance is not directly recycled but is stored in the shared pool for use by other players. The shared pool determines the reclamation period of the decoder instance, i.e., whether the decoder instance needs to be reclaimed. In general, when the memory space occupied by the shared pool reaches a memory space threshold, part of the decoder instances stored in the shared pool needs to be recycled until the number of the decoder instances in the shared pool does not exceed the number threshold, or the memory space occupied by the shared pool does not exceed the memory space threshold. When the player App is started or the shared pool is empty in the running process of the player App, some decoder instances meeting the preloading condition can be preloaded to the shared pool so as to ensure that the shared pool always has reusable decoder instances.

Referring to fig. 15, fig. 15 is a schematic diagram illustrating comparison of video time consumption provided by an embodiment of the present invention, as shown in fig. 15, when the original scheme is not multiplexed, 95% of the initialization time of the machine decoder is within 200ms, and only 25% of the initialization time of the machine decoder is within 50 ms; after the scheme is used for multiplexing, 90% of the time required by the machine decoder is within 22ms, and 99% of the time required by the machine decoder is within 50 ms.

Through the way, the video playing processing method provided by the embodiment of the invention has the following technical effects:

1. the optimization effect of the first frame of the scheme is obvious

1) When the scheme is not used, the initialization of decoders of all models needs 84ms on average, and the scheme only needs 8ms on average in the multiplexing process, so that 76ms is saved.

2) On low end models, decoder initialization requires 200ms, whereas the present scheme requires only 20 ms.

2. The scheme has good portability and is easily adapted to any Android platform and a player based on a MediaCodec decoder.

3. The scheme is decoupled from the player, and the decoder can be shared among the players.

4. The decoder can be selected in the shared pool before playing, more decoders can be selected, and the reuse rate is improved.

Continuing with the exemplary structure of the video playback processing apparatus 555 provided by the embodiment of the present invention implemented as a software module, in some embodiments, as shown in fig. 6, the software module stored in the video playback processing apparatus 555 in the memory 540 may include: an acquisition module 5551, a determination module 5552, a call module 5553, and an output module 5554.

An obtaining module 5551, configured to, in response to a switching video operation for a player, obtain a target video of the switching video operation;

a determining module 5552, configured to determine a target decoder instance adapted to the target video from among the decoder instances stored in the shared space;

an invoking module 5553 for invoking the target decoder instance in the shared space to decode media data comprised by the target video;

an output module 5554, configured to output a decoding result of the media data for presentation in a display interface of the player.

In some embodiments, the apparatus further comprises a storage module,

wherein the history video is a video played before the switching video operation is received.

In some embodiments, the storage module is further configured to store, in the memory space of the shared space, a decoder resource corresponding to a decoder instance and a corresponding display interface output layer that are called when the historical video is decoded;

In some embodiments, the apparatus further comprises an initialization module,

In some embodiments, the apparatus further comprises a release module,

The number of decoder instances in the shared pool does not exceed a number threshold, or the memory space occupied by the shared space does not exceed a memory space threshold.

In some embodiments, the releasing module is further configured to perform at least one of the following releasing operations:

In some embodiments, the apparatus further comprises a preload module,

the preloading module is configured to determine a decoder instance meeting a preloading condition when the player is initialized or the shared space is empty during operation of the player, and determine the decoder instance meeting the preloading condition, and execute the preloading condition when the shared space is empty

In some embodiments, the preload module is further configured to determine decoder instances that satisfy at least one of the following preload conditions:

decoder instance adapted to the preferred video type.

In some embodiments, the preloading module is further configured to obtain contextual features related to the player and user features of a player user;

In some embodiments, the calling module is further configured to intercept an application program interface of the decoder instance;

In some embodiments, the determining module is further configured to obtain configuration parameters of a decoder instance stored in a shared space, and video parameters of the target video;

In some embodiments, the invoking module is further configured to insert a preset type of coding information into the media data included in the target video, so as to obtain the edited media data, where the coding information includes at least one of: a sequence parameter set; a picture parameter set;

a memory for storing executable instructions;

In some embodiments, the computer comprises various computing devices including intelligent terminals and servers, and the storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (H TML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

In summary, the video playing processing method provided by the embodiment of the present invention has the following beneficial effects:

1. in the process of switching and playing the target video by the player, the decoder instance adapted to the target video is called from the shared pool, so that the instantiated decoder instance of the player can be flexibly multiplexed, a new decoder instance does not need to be created again all the time, and the delay in video switching is effectively reduced;

2. the portability is good, and the player can be easily adapted to any player of an Android platform based on a MediaCodec decoder;

3. the decoder can be shared among the players before playing, more decoders can be selected before playing, and the multiplexing rate is improved.

The above description is only an example of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

Claims

1. A video playback processing method, the method comprising:

responding to the video switching operation of a player, acquiring a target video of the video switching operation, and

determining a target decoder instance adapted to the target video from among the decoder instances stored in the shared space;

invoking the target decoder instance in the shared space to decode media data included in the target video, and

outputting the decoding result of the media data to be presented in a display interface of the player.

2. The method of claim 1, further comprising:

in response to a switch video operation for a player, storing in the shared space a decoder instance that is run when decoding historical video;

3. The method of claim 2, wherein storing, in the shared space, an instance of a decoder that is running when decoding historical video comprises:

storing decoder resources corresponding to decoder instances called when the historical videos are decoded and corresponding display interface output layers into a memory space of the shared space;

4. The method of claim 1, further comprising

Initializing and invoking a new decoder instance adapted to the target video when the target decoder instance adapted to the target video is not stored in the shared space, and

and outputting the decoding result to be presented in a display interface of the player.

5. The method of claim 1, further comprising:

when the memory space occupied by the shared space reaches a memory space threshold, releasing at least part of the decoder instances stored in the shared space until

6. The method of claim 5, wherein the releasing at least a portion of decoder instances already stored in the shared space comprises:

performing at least one of the following release operations:

7. The method of claim 1, further comprising:

when the player is initialized or the shared space is empty in the running process of the player, determining a decoder instance meeting a preloading condition, and

8. The method of claim 7, wherein determining the decoder instance that satisfies the preload condition comprises:

determining decoder instances that satisfy at least one of the following pre-load conditions:

decoder instance adapted to the preferred video type.

9. The method of claim 7, wherein determining the decoder instance that satisfies the preload condition comprises:

obtaining context characteristics related to the player and user characteristics of a player user;

10. The method of any of claims 1 to 9, wherein said invoking said target decoder instance in said shared space comprises:

intercepting an application program interface of the decoder instance;

11. The method according to any one of claims 1 to 9, wherein the determining a target decoder instance adapted to the target video from among the decoder instances stored in the shared space comprises:

12. The method of any of claims 1 to 9, wherein said invoking said target decoder instance in said shared space to decode media data included in said target video comprises:

13. A video playback processing apparatus, comprising:

14. An electronic device for video playback processing, the device comprising:

a memory for storing executable instructions;

a processor, configured to execute the executable instructions stored in the memory to implement the video playback processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium having stored thereon executable instructions for causing a processor to perform the video playback processing method of any one of claims 1 to 12 when executed.