CN117834982A

CN117834982A - Video playing processing method and device, electronic equipment and storage medium

Info

Publication number: CN117834982A
Application number: CN202211184451.6A
Authority: CN
Inventors: 张燕
Original assignee: Tencent Cloud Computing Beijing Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2022-09-27
Filing date: 2022-09-27
Publication date: 2024-04-05

Abstract

The application provides a video playing processing method, a video playing processing device, electronic equipment and a storage medium; the method comprises the following steps: acquiring a video file; decoding the video file and playing the video frame data obtained by decoding in a third-party player interface; and in response to a first switching instruction indicating the third-party player to switch from the foreground to the background, hiding the third-party player interface, and sending the video frame data to a native player in the terminal equipment so that the native player continues the playing progress of the third-party player, and continuing to play the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of a display screen of the terminal equipment. According to the method and the device, the continuity of video playing can be guaranteed when the third-party player is switched from the foreground to the background, and therefore user experience of seamless watching is achieved.

Description

Video playing processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of internet technologies, and in particular, to a video playing processing method, a device, an electronic device, and a storage medium.

Background

For various mobile operating systems (including iOS operating systems and android operating systems), due to the limitation of the operating systems, after the third party player falls back to the background for a period of time, the third party player is suspended by the operating systems and cannot continue playing the video. In addition, if the third party player still performs video rendering in the background after backing up to the background, a flash backing-up phenomenon is easy to occur.

If the native Player of the operating system (for example, AV-Player in iOS operating system) is used to play in order to avoid the above-mentioned situation of the third-party Player, only the third-party Player can be guaranteed to play the video normally due to the limitation of the native Player's own hard-decoding capability, but if the playback is switched to the native Player (for example, the AV-Player is enabled to play), the playback fails, resulting in a poor user experience.

In summary, there is no effective solution in the related art for how to ensure the continuity of video playback when a third party player switches from the foreground to the background.

Disclosure of Invention

The embodiment of the application provides a video playing processing method, a video playing processing device, electronic equipment, a computer readable storage medium and a computer program product, which can ensure the continuity of video playing when a third-party player is switched from a foreground to a background, thereby realizing the user experience of seamless watching.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a video playing processing method, which is applied to a third party player in terminal equipment and comprises the following steps:

acquiring a video file;

decoding the video file and playing the video frame data obtained by decoding in a third-party player interface;

in response to a first switching instruction indicating that the third-party player is switched from the foreground to the background, hiding the third-party player interface, and sending the video frame data to a native player in the terminal equipment so that

And continuing the playing progress of the third-party player by the original player, and continuing to play the video frame data in an original player interface, wherein the size of the original player interface is smaller than that of a display screen of the terminal equipment.

The embodiment of the application provides a video playing processing device, which is applied to a third party player in terminal equipment, and comprises:

the acquisition module is used for acquiring the video file;

the decoding module is used for decoding the video file;

the playing module is used for playing the video frame data obtained by decoding in the third-party player interface;

The hiding module is used for hiding the third-party player interface in response to a first switching instruction for indicating the third-party player to switch from the foreground to the background;

and the sending module is used for sending the video frame data to a native player in the terminal equipment so that the native player can continue the playing progress of the third-party player, and continuously playing the video frame data in a native player interface, wherein the size of the native player interface is smaller than that of a display screen of the terminal equipment.

An embodiment of the present application provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for realizing the video playing processing method provided by the embodiment of the application when executing the executable instructions stored in the memory.

The embodiment of the application provides a computer readable storage medium, which stores computer executable instructions for implementing the video playing processing method provided by the embodiment of the application when being executed by a processor.

Embodiments of the present application provide a computer program product, including a computer program or computer executable instructions, configured to implement, when executed by a processor, a video playing processing method provided by the embodiments of the present application.

The embodiment of the application has the following beneficial effects:

when the third-party player is switched from the foreground to the background, the decoding capability of the third-party player is continuously multiplexed to decode the video file, and video frame data obtained by decoding is sent to the original player, so that the original player is not required to decode the video file, and only the video frame data sent by the third-party player is continuously played in the original player interface, the occurrence of the situation of playing failure when the third-party player is switched from the third-party player to the original player is avoided, the function of seamlessly playing the video under the situation that the third-party player is switched from the foreground to the background is realized, and the viewing experience of a user is improved.

Drawings

Fig. 1 is a schematic architecture diagram of a video playback processing system 100 according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of an electronic device 500 according to an embodiment of the present application;

fig. 3 is a flowchart of a video playing processing method according to an embodiment of the present application;

fig. 4 is a flowchart of a video playing processing method according to an embodiment of the present application;

fig. 5 is a flowchart of a video playing processing method according to an embodiment of the present application;

Fig. 6 is a flowchart of a video playing processing method according to an embodiment of the present application;

fig. 7 is an application scenario schematic diagram of a video playing processing method provided in an embodiment of the present application;

fig. 8 is a schematic diagram of an overall architecture of a video playing processing method according to an embodiment of the present application;

fig. 9 is a schematic diagram of playing video frame data according to an embodiment of the present application;

fig. 10 is a schematic diagram of playing audio frame data according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings, and the described embodiments should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.

It will be appreciated that in the embodiments of the present application, related data such as user information is referred to, and when the embodiments of the present application are applied to specific products or technologies, user permissions or consents need to be obtained, and the collection, use and processing of related data need to comply with related laws and regulations and standards of related countries and regions.

In the following description, the term "first/second/is referred to merely as distinguishing between similar objects and not as representing a particular ordering of the objects, it being understood that the" first/second/may be interchanged with a particular order or sequence, as permitted, to enable embodiments of the application described herein to be practiced otherwise than as illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

Before further describing embodiments of the present application in detail, the terms and expressions that are referred to in the embodiments of the present application are described, and are suitable for the following explanation.

1) In response to: for representing a condition or state upon which an operation is performed, one or more operations performed may be in real-time or with a set delay when the condition or state upon which the operation is dependent is satisfied; without being specifically described, there is no limitation in the execution sequence of the plurality of operations performed.

2) The native player, i.e. the player preinstalled in the operating system of the terminal device, is typically the default player.

3) Third party player: the player is downloaded and installed from an application store by a user, is different from a native player, is used for improving the deficiency of an operating system and realizing personalized functions, and provides richer use experience for the user.

4) Picture-in-Picture (PiP): the method is a video content presentation mode, and is characterized in that one video is played in full screen and the other video is played in a small area of a picture.

5) Software development kit (SDK, software Development Kit): also known as a software development kit is a collection of development tools that build application software for a particular software package, software framework, hardware platform, operating system, etc. Software development kits broadly refer to a collection of related documents, examples, and tools that facilitate the development of a certain class of software.

6) Background Task (Background Task): by a process provided by the operating system that can run in the background, a background task belonging to an application can continue to silently perform related operations even if the application has been suspended or is no longer running. It needs to cooperate with background mode when in use, for example, for iOS operating systems, it needs to be set in iOS development IDE tool (e.g. XCode) Project Setting.

7) Background mode: a process execution mode does not occupy display equipment and input equipment and cannot interact with a user. In order to enable the application to continue running after switching from the foreground to the background, it is necessary to set the running condition of the application in the background (for example, to add the application to a white list that is not suspended by the operating system), which is called a background mode.

8) FFmpeg: is a set of open source computer programs that can be used to record, convert digital audio, video, and convert it into streams. LGPL or GPL licenses are employed which provide a complete solution for recording, converting and streaming audio and video. It contains very advanced audio and video codec library libavcodec, in which many codes (codes) have been developed from scratch in order to guarantee high portability and codec quality.

9) Open graphics library (OpenGL, open Graphics Library): is a cross-language, cross-platform application programming interface for rendering 2D, 3D vector graphics, which consists of nearly 350 different function calls for drawing a scene from simple graphics bits to complex three dimensions.

10 Metal): the method is a low-level rendering application programming interface, provides the lowest layer required by software, and ensures that the software can run on different graphic chips.

11 Dynamic picture experts group (MPEG, moving Picture Experts Group): the MPEG file format is an international standard of a moving image compression algorithm, and adopts a lossy compression method to reduce redundant information in a moving image, and simultaneously ensures an image dynamic refresh rate of 30 frames per second.

12 Group of pictures (GOP): is a set of consecutive pictures consisting of an I frame, which is an intra-coded frame (also called a key frame), and several B/P frames, which are basic units accessed by video encoders and decoders, and whose sequence repeats until the end of the picture, wherein the P frames are forward predicted frames and the B frames are bi-directionally interpolated frames.

13 HTTP Streaming (HLS), which is an adaptive rate Streaming protocol, describes a set of tools and programs that provide audio and video services over the internet. A video file may be partitioned into a plurality of video slices (ts) whose delivery locations and order are in a set of XML files called playlists, which end in a file extension m3u 8. Assuming, for example, that the duration of a video slice is 10 seconds, an hour of a movie may be cut into 360 video slices of 10 seconds length, and then a file called a playlist is created that contains the video name, location, and slice play sequence (along with metadata describing the codec, resolution, and stream, etc.), the process of creating these video slices being called video slicing.

The embodiment of the application provides a video playing processing method, a video playing processing device, electronic equipment, a computer readable storage medium and a computer program product, which can ensure the continuity of video playing when a third-party player is switched from a foreground to a background, thereby realizing the experience of seamless watching of a user. An exemplary application of the electronic device provided by the embodiment of the present application is described below, where the electronic device provided by the embodiment of the present application may be implemented as a terminal device, or may be implemented cooperatively by the terminal device and a server.

The following describes an example of a video playing processing method provided in the embodiment of the present application implemented by the terminal device and the server cooperatively.

Referring to fig. 1, fig. 1 is a schematic architecture diagram of a video playing processing system 100 according to an embodiment of the present application, in order to support an application capable of ensuring continuity of video playing when a third party player switches from a foreground to a background, as shown in fig. 1, the video playing processing system 100 includes: server 200, network 300 and terminal device 400, wherein third party player 410 and operating system self-contained native player 420 are running on terminal device 400, network 300 may be a wide area network or a local area network, or a combination of both.

In some embodiments, taking a network playing scenario as an example, the third party player 410 may receive, through the network 300, a video file sent by the server 200, for example, after the server 200 receives a download request sent by the third party player 410, the third party player 410 may send a play list (for example, m3u8, where m3u8 is a text file in UTF-8 coding format) to the third party player 410, so that the third party player 410 sequentially downloads, according to the play list, a plurality of fragments of the video file from the server 200 (TS, TS is in a packaging format, all of which is named MPEG-TS, the file is divided into three layers, namely, a TS layer, a pes layer, and an es layer, where the es layer is audio-video data, the pes layer is description information of a data frame added with a timestamp and the like on the audio-video data, the TS layer is added with necessary information for identifying and transmitting a data stream, each fragment may be independently decoded (i.e., independent of other fragments), and the third party player 410 decodes the fragments in sequence according to a receiving order, and decodes the fragments into the third party video data in the third party player interface to obtain the third party video frame; the third party player 410 then conceals the third party player interface (i.e., switches from foreground to background) and sends video frame data to the native player 420 in the terminal device 400 in response to a first switch instruction instructing the third party player to switch from foreground to background (e.g., receiving a user click operation on the Home key for returning to Home screen function). After receiving the video frame data sent by the third party player 410, the native player 420 continues the playing progress of the third party player 410, and continues to play the video frame data in the native player interface in the picture-in-picture display mode, so that seamless switching of video frame data playing is realized, and the viewing experience of a user is improved.

In other embodiments, taking a local play scenario as an example, the video play processing method provided in the embodiments of the present application may also be separately executed by the terminal device 400 shown in fig. 1, for example, the third party player 410 responds to a play request triggered by a user to read a video file from a local storage location of the terminal device 400, and then the third party player 410 invokes a built-in SDK to decode the video file and play the audio frame data obtained by decoding in a third party player interface; then, after the third party player 410 receives the first switching instruction sent by the operating system of the terminal device 400, the third party player interface is hidden (i.e. the operation is switched from the foreground to the background), and video frame data is sent to the native player 420 in the terminal device 400, so that the native player 420 is connected with the playing progress of the third party player 410 (for example, if the third party player 410 is playing video frame data corresponding to the third second of the video file in the third party player interface when receiving the first switching instruction sent by the operating system of the terminal device 400, the native player 420 is continuously playing video frame data corresponding to the subsequent time period from the third second in the native player interface), and the video frame data is continuously played in the native player interface in the picture-in-picture display mode, so that the continuity of video playing can be ensured when the third party player is switched from the foreground to the background, and the viewing experience of the user is improved.

In other embodiments, the embodiments of the present application may also be implemented by means of Cloud Technology (Cloud Technology), which refers to a hosting Technology that unifies serial resources such as hardware, software, networks, etc. in a wide area network or a local area network, so as to implement calculation, storage, processing, and sharing of data.

The cloud technology is a generic term of network technology, information technology, integration technology, management platform technology, application technology and the like based on cloud computing business model application, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical network systems require a large amount of computing and storage resources.

By way of example, the server 200 in fig. 1 may be a stand-alone physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN, content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 400 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc. The terminal device 400 and the server 200 may be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the present application.

The following continues to describe the structure of the electronic device provided in the embodiment of the present application. Taking an electronic device as an example of a terminal device, referring to fig. 2, fig. 2 is a schematic structural diagram of an electronic device 500 provided in an embodiment of the present application, and the electronic device 500 shown in fig. 2 includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in electronic device 500 are coupled together by bus system 540. It is appreciated that the bus system 540 is used to enable connected communications between these components. The bus system 540 includes a power bus, a control bus, and a status signal bus in addition to the data bus. The various buses are labeled as bus system 540 in fig. 2 for clarity of illustration.

The processor 510 may be an integrated circuit chip with signal processing capabilities such as a general purpose processor, such as a microprocessor or any conventional processor, or the like, a digital signal processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.

The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and/or one or more visual displays. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, and the like. Memory 550 may optionally include one or more storage devices physically located remote from processor 510.

Memory 550 includes volatile memory or nonvolatile memory, and may also include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a random access Memory (RAM, random Access Memory). The memory 550 described in embodiments herein is intended to comprise any suitable type of memory.

In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 551 including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB, universal Serial Bus), etc.;

A presentation module 553 for enabling presentation of information (e.g., a user interface for operating a peripheral device and displaying content and information) via one or more output devices 531 (e.g., a display screen, speakers, etc.) associated with the user interface 530;

the input processing module 554 is configured to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

In some embodiments, the video play processing device provided in the embodiments of the present application may be implemented in a software manner, and fig. 2 shows a video play processing device 555 stored in a memory 550, which may be software in the form of a program, a plug-in, or the like, including the following software modules: the acquisition module 5551, the decoding module 5552, the playing module 5553, the hiding module 5554, the sending module 5555, the creation module 5556, the delivery module 5557, the stopping module 5558 and the decapsulation module 5559 are logical, so that any combination or further splitting may be performed according to the implemented functions. It should be noted that, in fig. 2, all the above modules are shown once for convenience of expression, but it should not be considered that the implementation of the acquisition module 5551, the decoding module 5552, the playing module 5553, the hiding module 5554, and the transmitting module 5555 may be included only in the video playback processing apparatus 555, and the functions of the respective modules will be described below.

The video playing processing method provided in the embodiment of the present application will be specifically described below with reference to exemplary applications and implementations of the terminal device provided in the embodiment of the present application.

It should be noted that, in the terminal device provided in the embodiment of the present application, a third party player and a native player with an operating system are operated, where the operating system installed in the terminal device includes, but is not limited to: the video playing processing method provided by the embodiment of the application will be specifically described from the interaction angle between the third-party player and the original player running in the terminal device.

Referring to fig. 3, fig. 3 is a flowchart of a video playing processing method according to an embodiment of the present application, and will be described with reference to the steps shown in fig. 3.

In step 101, a third party player obtains a video file.

In some embodiments, taking a local play scenario as an example, the third party player responds to a video play request to read a video file to be played from a storage location local to the terminal device. For example, at least one name corresponding to each video file to be played is displayed in a third-party player interface, and when a clicking operation of a user for the name of the video file A is received, the third-party player reads the video file A from a storage position local to the terminal device. That is, the package format of the video file stored locally in the terminal device may be a non-streaming media format, that is, the video file may be decoded and played after the video file needs to be downloaded completely, for example, a window media video (WMV, windows Media Video) file, a video file in a MKV file format (MKV), and the like.

In other embodiments, taking a network play scenario as an example, the third party player sends a video play request to the server in response to a user-triggered video play operation (e.g., receiving a click operation from a user on a "start play" button displayed in the third party player interface). The server responds to a video playing request sent by the third-party player, sends a play list of the video file to be played to the third-party player, and then the third-party player sequentially downloads a plurality of video clips of the video file from the server according to the play list, wherein each video clip carries information required for decoding, and therefore the video clip can be independently decoded, namely independent of other video clips. That is, for video files in the network playing scene, the package format of the video files may be a streaming media format, that is, the video files may be decoded and played without being completely downloaded or additionally transcoded, for example, the video files in FLV (Flash Video) format, etc.

In some embodiments, the third party player may also obtain the video file by: acquiring a multimedia file, wherein the multimedia file is obtained by packaging a video file, an audio file and a subtitle file; and decapsulating the multimedia file to obtain a video file. For example, the server may encapsulate the rmvb video file, the mp3 audio file, and the srt subtitle file into one mp4 or mkv multimedia file in advance, and after the third party player obtains the multimedia file, the third party player may first decapsulate the multimedia file, for example, invoke a decapsulator (demux) to decompose the synthesized multimedia file into the video file, the audio file, and the subtitle file, where the subtitles in the subtitle file are used for playing in synchronization with the video frame data in the third party player interface, for example, assuming that the video frame data corresponding to the 90 th second of the video file is currently being played in the third party player interface, the subtitles corresponding to the 90 th second of the subtitle file may be played in synchronization.

It should be noted that, when the third party player switches from the foreground to the background, the third party player may send the subtitle file to the native player, so that the native player continues the playing progress of the third party player, and the subtitles in the subtitle file are continuously played in the native player interface.

In step 102, the third party player decodes the video file and plays the decoded video frame data in the third party player interface.

In some embodiments, a software development kit (i.e., SDK) may be built in advance into the third party player, where the software development kit integrates a variety of different types of decoders (e.g., FFMPEG decoder, video ToolBox decoder, mediaCodec decoder, dolby Video private decoder, etc.), then the third party player may decode the acquired Video file in the following manner: calling a built-in software development suite to execute the following processes: determining the coding format of a video file; and calling a decoder matched with the coding format to decode the video file. For example, assuming that the SDK built in the third party player determines that the video file to be played currently is encoded by MPEG, the FFMPEG decoder may be invoked to decode the video file to obtain corresponding video frame data. After the video frame data is obtained, the third-party player can render the video frame data obtained by decoding the SDK through tools such as OpenGL or Metal, so that a video picture obtained based on the video frame data rendering can be displayed in a third-party player interface.

In other embodiments, the third party player may play the decoded audio frame data while playing the decoded video frame data in the third party player interface, and then after step 102 shown in fig. 3 is performed, step 106 shown in fig. 4 may also be performed, which will be described in connection with the steps shown in fig. 4.

In step 106, the third party player decodes the audio file and plays the audio frame data synchronized with the currently playing video frame data according to the currently playing video frame data in the third party player interface.

In some embodiments, after the multimedia file is unpackaged to obtain the audio file, the third party player may call the built-in SDK to decode the audio file, for example, the SDK may first determine an encoding format of the audio file obtained by unpacking, then call a decoder matching the encoding format to decode the audio file, for example, assuming that the encoding format of the audio file is dolby audio, the SDK may call the dolby audio decoder to decode the audio file. After the SDK decodes the audio file to obtain audio frame data, the third party player may play audio frame data synchronized with the currently played video frame data in play time according to the currently played video frame data in the third party player interface, for example, assuming that video frame data corresponding to the 70 th second of the video file is currently being played in the third party player interface, the third party player may synchronously play audio frame data corresponding to the 70 th second.

By way of example, a third party player may play synchronized audio frame data by: filling the decoded audio frame data into at least one buffer area (also called a buffer, which is a part of the memory space, that is, a certain storage space is reserved in the memory space, and the storage space is used for buffering input or output data, and the reserved space is called a buffer area), and storing the at least one buffer area into an audio queue; and calling an audio output device (such as a built-in loudspeaker of the terminal equipment or an external earphone, such as an earphone connected with the terminal equipment in a wired or wireless way, and the like) to read the audio frame data filled in the at least one buffer area from the audio queue, so that the audio output device outputs a corresponding audio signal. For example, taking iOS as an example, the AV-Player may first read Audio frame data decoded by the SDK into a buffer of the Audio Queue from the hard disk (for example, assuming that there are 3 buffers of the Audio Queue, namely, buffer 1, buffer 2 and buffer 3, and each buffer may store Audio frame data for 10 seconds, then Audio frame data for 1 second to 10 seconds may be stored in buffer 1, audio frame data for 11 seconds to 20 seconds may be stored in buffer 2, audio frame data for 21 seconds to 30 seconds may be stored in buffer 3), and the buffers may be sent to the Audio Queue; the AV-Player can tell the playback device (i.e. the Audio output device) through the interface provided by the Audio Queue, for example, a speaker built in the terminal device, or an earphone connected with the terminal device through USB or bluetooth, where the buffer area has Audio frame data, and can take out the Audio frame data (for example, the playback device reads the Audio frame data stored in the buffer area 1 in the Audio Queue first for playing, reads the Audio frame data stored in the buffer area 2 in the Audio Queue for playing, and so on); after the Audio frame data in one buffer is played, the Audio Queue tells the AV-Player that an empty buffer is currently available for filling the Audio frame data (for example, when the Audio frame data from 1 st to 10 th seconds stored in the buffer 1 is played, the Audio frame data from 31 st to 40 th seconds may be continuously filled into the buffer 1), and the above process is repeated until all the Audio frame data are played.

With continued reference to fig. 3, in step 103, the third party player conceals the third party player interface in response to a first switching instruction instructing the third party player to switch from the foreground to the background.

In some embodiments, the first switching instruction may be generated by an operating system of the terminal device and sent to the third party player, and generated when at least one of: receiving a trigger operation for instructing the third party player to switch from the foreground to the background (for example, receiving a click operation of a user on a start key "Home" key for returning to a Home screen function, or receiving a sliding operation triggered by the user for exiting the third party player); receiving a voice call request (for example, when the third party player plays video frame data in a third party player interface in a full screen mode, an operating system of the terminal equipment receives the voice call request, the operating system can generate a first switching instruction for indicating the third party player to switch from a foreground to a background and send the first switching instruction to the third party player so as to enable the third party player to switch from the foreground to the background, thereby being convenient for a user to continuously watch the video frame data while processing the voice call request); receiving a video call request (for example, when the third party player plays the decoded video frame data in the third party player interface in a full screen mode, the operating system of the terminal device receives the video call request, the operating system may generate a first switching instruction indicating that the third party player switches from the foreground to the background, and send the first switching instruction to the third party player, so that the third party player switches from the foreground to the background, thereby facilitating a user to continuously watch the video frame data while processing the video call request); the currently played video frame data comprises recommendation information (such as advertisements), the played time length of the recommendation information exceeds a time length threshold, for example, when the decoded video frame data is played in a third-party player interface, an operating system of the terminal equipment can call a machine learning model to identify the video frame data, when the operating system detects that the advertisements are currently played in the third-party player interface, and the played time length of the advertisements exceeds the time length threshold, a first switching instruction for indicating the third-party player to switch from the foreground to the background can be generated, and the first switching instruction is sent to the third-party player, so that the third-party player can switch from the foreground to the background, interference caused by the recommendation information to a user can be reduced, and viewing experience of the user is improved.

It should be noted that, after receiving the first switching instruction sent by the operating system of the terminal device, the third party player may hide the interface of the third party player (i.e. switch from the foreground to the background), and continue to operate in the background mode (for example, the third party player may be added to a white list allowed to operate in the background, so that the third party player may not be suspended by the operating system when switching from the foreground to the background, and thus may continue to operate in the background mode), for example, continue to call the built-in SDK to decode the video file, or continue to play the audio frame data obtained by decoding, and so on.

In other embodiments, the third party player may further perform the following processing in response to a first switching instruction instructing the third party player to switch from the foreground to the background: according to the progress of the video frame data that the native player continues to play in the native player interface (e.g., may be characterized by the timestamp of the currently playing video frame), audio frame data synchronized with the video frame data that is played in the native player interface (at the play time) is played, thereby ensuring that the timestamp of the currently playing audio frame is synchronized with the timestamp of the video frame.

For example, when the third party player receives the first switching instruction sent by the operating system of the terminal device, after the first switching instruction is switched from the foreground to the background, the audio frame data decoded by the SDK may be continuously played in the background mode, for example, the third party player may continuously play the audio frame data synchronized with the video frame data played in the native player interface in the background mode according to the video frame data continuously played in the native player interface by the native player (for example, it is assumed that the video frame data corresponding to the 90 th second is currently being played in the native player interface), that is, the third party player continuously plays the audio frame data corresponding to the 90 th second in the background mode.

In step 104, the third party player transmits video frame data to the native player in the terminal device.

In some embodiments, the decoding of the video file by the third party player may be pre-decoding, where the range of pre-decoding may be: a predetermined length of time (e.g., 10 seconds) or a predetermined number of decoding units (i.e., a minimum decoding unit, e.g., one GOP in MPEG, one ts in HLS, etc.) after the current play time point, the third party player may transmit video frame data to the native player in the terminal device by: transmitting the following data among video frame data of a predetermined length to a native player: and playing the video frame data with the time after the first moment, wherein the first moment is the moment when the third party player switches from the foreground to the background.

Taking the format of the video file as MPEG as an example, the third party player may call the built-in SDK to decode the subsequent partial period in advance, taking MP4 as an example, where the decoding unit is GOP (for example, the SDK enters a waiting state after decoding the 1 st GOP, decodes the 2 nd GOP after playing the video frame data obtained by decoding the 1 st GOP, and so on), assuming that the third party player has already completed decoding the 7 th GOP when switching from the foreground to the background (assuming that the playing period corresponding to the video frame data obtained by decoding the 7 th GOP is the third 70 seconds to the third 90 seconds), and at the same time, assuming that the video frame data corresponding to the 80 th second of the video file is currently being played in the third party player interface, the third party player may transmit the video frame data obtained by decoding the 7 th GOP but not being played (i.e., the video frame data corresponding to the 80 th second to the 90 th second) to the native player, and at the same time, the third party player may transmit the video frame data obtained by decoding the subsequent GOP to the native player, so that the native data may be reduced.

It should be noted that, the predetermined time period may be a fixed value (for example, 10 seconds), or may be a dynamic value adaptively adjusted according to a hardware configuration of the terminal device, for example, when the hardware configuration of the terminal device is higher than a configuration threshold, it indicates that the computing capability of the terminal device is stronger, and the corresponding predetermined time period may be longer (for example, 20 seconds); when the hardware configuration of the terminal device is lower than the configuration threshold, which indicates that the operation capability of the terminal device is weaker, the corresponding predetermined time period may be shorter (for example, 5 seconds), which is not particularly limited in the embodiment of the present application.

In other embodiments, the decoding of the video file by the third party player may be pre-decoding, where the range of pre-decoding may be: the third party player may send video frame data to the native player in the terminal device by: and sending video frame data corresponding to a subsequent time period to the original player, wherein the subsequent time period is a time period positioned after a first moment in a playing time axis of the video file, and the first moment is a moment when the third party player switches from the foreground to the background.

For example, for a video file with a playing duration less than a duration threshold (for example, 5 minutes), or with a resolution lower than a resolution threshold, the third party player may invoke the built-in SDK to decode the video file once to obtain video frame data corresponding to all playing time periods of the video file (assuming that the duration of the video file is 5 minutes, all playing time periods are 1 st to 300 nd seconds), and assuming that the third party player is switching from the foreground to the background, the third party player may send video frame data corresponding to a subsequent time period (i.e., 60 th to 300 th seconds) to the native player, so that the native player continues the playing progress of the third party player, and continue playing video frame data in the native player interface (i.e., the native player may continue playing video frame data corresponding to the subsequent time period from 60 th to 300 th seconds in the native player interface).

It should be noted that, when the third party player sends the video frame data obtained by decoding the SDK to the native player, the third party player may also still store the video frame data in a memory corresponding to an instance running the third party player, for example, may store the video frame data whose distance from the current playing time point is less than the time threshold in the memory, and delete the video frame data after exceeding the storage time, for example, assuming that the current playing time point is 90 seconds of the video file, the third party player may also store the video frame data in the memory while sending the video frame data corresponding to the subsequent time period to the native player, for example, store the video frame data corresponding to the 90 th to 100 th seconds obtained by decoding the SDK, and then delete the video frame data corresponding to the 90 th to 100 th seconds stored in the memory when the video file is played to the 100 th second in the native player, so that the normal playing of the video file is ensured and precious memory space is saved.

In other embodiments, the third party player may also perform the following processing when sending video frame data to the native player in the terminal device: and sending the audio frame data obtained by decoding the SDK to the original player so that the original player continues the playing progress of the third-party player and continues to play the audio frame data.

By way of example, the third party player may send audio frame data to the native player by: one of the following processes is performed: transmitting the following data among the audio frame data of a predetermined length to the native player: audio frame data whose playing time is after the first time (corresponding to the case where the range of the SDK pre-decoding is a predetermined length of time after the current playing time point, where the predetermined length of time may be a playing time period corresponding to video frame data obtained by the SDK decoding for 1 GOP in the video file, for example, 10 seconds); the audio frame data located after the first time in the full play timeline of the video file (corresponding to the case where the extent of SDK pre-decoding is the full play period of the audio file, e.g., for audio files having a play duration less than the duration threshold, e.g., audio files having a total duration of only 5 minutes) is sent to the native player.

It should be noted that, the transmission manner of the audio frame data is similar to the transmission manner of the video frame data, and may be implemented by referring to the transmission manner of the video frame data, which is not described herein.

In step 105, the native player continues the playing progress of the third party player, and continues playing the video frame data in the native player interface.

Here, the size of the native player interface is smaller than the size of the display screen of the terminal device, that is, the native player continues the playing progress of the third party player in the picture-in-picture display mode, and video frame data continues to be played in the native player interface.

In some embodiments, after receiving the video frame data sent by the third party player, the native player may continue to play the video frame data of the third party player in the native player interface, for example, if the third party player receives the first switching instruction sent by the operating system of the terminal device and switches from the foreground to the background, and when the third party player is playing the video frame data corresponding to the 70 th second of the video file in the third party player interface, the native player may continue to play the video frame data of the subsequent time period from the 70 th second of the video file in the native player interface, so that the video playing continuity can be maintained when the third party player switches from the foreground to the background, and a seamless switching viewing experience can be realized.

In other embodiments, when the native player continues the playing progress of the third party player and continues playing the video data frames in the native player interface, the third party player may further perform the following processing: and sending the video frame data with the playing time before the first time to the original player so that the original player responds to the instruction of playing from the second time, and playing from the video frame data corresponding to the second time based on the received video frame data with the playing time before the first time in the original player interface, wherein the second time is earlier than the first time.

For example, after the third party player switches from the foreground to the background, in addition to the video frame data whose playing time is located after the first time, the video frame data whose playing time is located before the first time may be sent to the native player, so, when the native player receives an instruction to start playing from the second time (i.e., the user wants to view video frame data corresponding to an earlier time in the native player), the native player may start playing from video frame data corresponding to the second time (e.g., the playing time is 40 th seconds of the video file) based on the received video frame data whose playing time is located before the first time (e.g., the playing time is 90 th seconds of the video file) in the native player interface (i.e., the playing time period is video frame data corresponding to from 1 st second to 90 th seconds).

In other embodiments, referring to fig. 5, fig. 5 is a flowchart of a video playing processing method provided in the embodiment of the present application, as shown in fig. 5, a native player may further execute step 107 and step 108 shown in fig. 5 before executing step 105 shown in fig. 3, and will be described with reference to the steps shown in fig. 5.

In step 107, the native player creates a picture-in-picture controller object.

In step 108, the native player passes a reference to the PIP controller object for the native player interface to bind the native player interface to the PIP controller object.

Here, the picture-in-picture controller object is used to control the native player interface to employ a display mode that is smaller than the size of the display screen.

Taking a native Player as an example, the AV-Player is a Player that implements a video playing function based on the iOS operating system, the AV-Player cannot display the played video content separately, and the video content can be displayed by adding graphics to a Layer to be displayed by means of the AV-Player Layer (i.e., a native Player interface), so the AV-Player can perform the following processing before continuing to play video frame data in the AV-Player Layer: a new avpicturelnpicturecon object is created and a reference to the avpicturelnpicturecon object for displaying video content is delivered, i.e., the avpicturelnpicturecon object needs to be strongly referenced to implement the picture-in-picture function, that is, after binding the AV-Player Layer with the avpicturelnpicturecon object, the AV-Player Layer can be managed by the avpicturelnpicturecon object, for example, the AV-Player Layer is controlled to use a display mode smaller than the size of the display screen of the terminal device (i.e., the AV-Player Layer is controlled to display in the picture-in-picture display mode).

In other embodiments, referring to fig. 6, fig. 6 is a flowchart of a video playing processing method according to the embodiment of the present application, as shown in fig. 6, after step 105 shown in fig. 3 is performed, step 109 and step 110 shown in fig. 6 may also be performed, and the description will be made with reference to the steps shown in fig. 6.

In step 109, the third party player displays a third party player interface in response to a second switch instruction instructing the third party player to switch from the background back to the foreground.

In some embodiments, the second switching instruction may be generated by an operating system of the terminal device and sent to the third party player, and generated when at least one of: receiving a trigger operation (for example, receiving a click operation of a user on an icon of the third-party player displayed on a desktop) for instructing the third-party player to switch back from the background to the background; the voice call ends (for example, when the operating system of the terminal device detects that the voice call ends, a second switching instruction for instructing the third party player to switch back to the foreground from the background may be generated, and the second switching instruction is sent to the third party player, so that the third party player switches back to the foreground from the background); ending the video call (for example, when the operating system of the terminal device detects that the video call is ended, a second switching instruction for instructing the third party player to switch back to the foreground from the background may be generated, and the second switching instruction is sent to the third party player so as to cause the third party player to switch back to the foreground from the background); the currently played video frame data does not include recommendation information (for example, when the operating system of the terminal device detects that the advertisement has been played, a second switching instruction for instructing the third party player to switch from the background to the foreground may be generated, and the second switching instruction may be sent to the third party player, so that the third party player switches back from the background to the foreground again).

In other embodiments, after receiving the second switching instruction sent by the operating system of the terminal device, the third party player may display the third party player interface (i.e. switch from the background to the foreground), and send a hiding notification to the native player, so that the native player hides the native player interface, where a manner of hiding the native player interface includes: closing the native player (namely closing the instance of running the native player and recovering the memory corresponding to the instance); the instance of the native player remains running, closing only the native player interface.

For example, with the above, when the native player interface is hidden (e.g., an instance of running the native player is closed and memory corresponding to the instance is reclaimed), in which case there is less likelihood of a subsequent switch from the third party player to the native player, the third party player may also perform the following: the transmission of video frame data to the native player is stopped.

For example, with the above, when the native player interface is hidden (e.g., an instance of the native player remains running, only the native player interface is turned off, in which case there is a greater likelihood that a subsequent switch from the third party player to the native player will occur, and the native player will automatically delete older video frame data, e.g., video frame data greater than a predetermined length of time from the current play point in time), if the video frame data has not been transmitted to the native player, the third party player may also perform the following processing: and continuously sending the video frame data with the playing time after a third time to the original player, wherein the third time is the time when the third-party player switches from the background to the foreground. For example, assuming that the third party player is playing the 100 th second of the video file in the native player interface when switching back to the foreground from the background, the third party player can continue to send the video frame data after the 100 th second decoded by the SDK to the native player while keeping the instance of the native player running and only closing the native player interface, so when switching back from the third party player to the native player again, the native player can play in the native player interface directly based on the received video frame data, and the switching speed is further improved.

In step 110, the third party player continues the playing progress of the native player, and continues playing the video frame data in the third party player interface.

In some embodiments, after the third party player receives the second switching instruction sent by the operating system of the terminal device and switches back to the foreground from the background, the third party player interface may be redisplayed, and then the playing progress of the native player is continued, and video frame data continues to be played in the third party player interface, for example, when the third party player switches back to the foreground from the background, video frame data corresponding to the 120 th second of the video file is being played in the native player interface, and then the third party player may continue to play video frame data corresponding to the subsequent time period from the 120 th second in the third party player interface.

It should be noted that, for the case that when the third party player switches from the foreground to the background, the audio frame data obtained by decoding the SDK is continuously played by the native player, when the third party player switches back to the foreground from the background, the third party player may stop sending the audio frame data obtained by decoding to the native player; of course, in consideration of the possibility of the subsequent re-switching, the third party player may also continue to send the decoded audio frame data to the native player, which is not specifically limited in the embodiment of the present application.

According to the video playing processing method, when the third-party player is switched from the foreground to the background, the decoding capability of the third-party player is continuously multiplexed to decode the video file, and video frame data obtained by decoding is sent to the original player, so that the original player does not need to decode the video file, only the video frame data sent by the third-party player is continuously played in the original player interface, the occurrence of the situation of playing failure when the third-party player is switched to the original player is avoided, the function of seamlessly playing the video under the situation that the third-party player is switched from the foreground to the background is realized, and the viewing experience of a user is improved.

An exemplary application of the embodiments of the present application in an actual application scenario is described below using the iOS operating system as an example.

With the development of internet technology, video playing applications (corresponding to the third party player described above) have been widely used in daily life of people, and have become an integral part of people's lives. These video playback applications use native players (hereinafter simply referred to as system players) that are self-contained based on the operating system or implement functions that are similar to "picture-in-picture" by themselves to perform video playback. When the user performs the background-returning operation, the playing of the video is paused by setting the state machine of the player (for example, switching from the playing state to the pause state) without exception. It is desirable for users of video experience to be able to seamlessly conduct the video viewing experience. While facing this need, the solutions provided by the related art are implemented in dependence on the picture-in-picture capability provided by the iOS operating system. However, this implementation has a drawback that the hardware decoding capability associated with the iOS operating system is limited, which easily results in normal video playback in the application, but fails to switch to the system player for playback, and the user experience is very bad for the user of the video.

In view of this, the embodiment of the present application provides a video playing processing method, which implements seamless switching playing of video based on an iOS operating system by embedding a video playing component (i.e., SDK) in a video playing application program, and the scheme provided in the embodiment of the present application may implement seamless playing of video under the condition of switching from a foreground to a background, so that user experience and effect are not affected at all, and user experience when a user uses the video playing application program is greatly improved.

For example, referring to fig. 7, fig. 7 is a schematic view of an application scenario of a video playing processing method provided in an embodiment of the present application, as shown in fig. 7, a playing interface 701 of a third party player is displayed on a display screen 700 of a terminal device (e.g. a mobile phone); when the third party Player is switched from the foreground to the background, the playing interface 701 of the third party Player is hidden on the display screen 700, and the AV-Player (i.e., the system Player) of the iOS operating system is enabled, so that video frame data is continuously output in the playing interface 702 of the AV-Player in a picture-in-picture display mode, and seamless continuous playing of video can be realized under the condition of switching from the foreground to the background.

The following specifically describes a video playing processing method provided in the embodiment of the present application.

The technical scheme provided by the embodiment of the application is realized based on the playing frame of the iOS, and the picture-in-picture capability of the system player is relied on, namely, when the third party player is switched from the foreground to the background, the system player can be started to realize the capability of continuously playing the video.

The following describes a specific implementation procedure of the video play processing method according to the embodiment of the present application with reference to fig. 8. As shown in fig. 8, firstly, a playback component (i.e., an SDK, including a demux API, a Decoder API, and a render API) built in a third party player is used, where the demux API includes an FFMPEG demux and a self-grinding demux, for example, the demux API may be called by a demux thread to decapsulate a multimedia file into a video file and an Audio file, the demux API includes an FFMPEG soft solution, mediaCodec, videoToolBox, audioToolBox, dolby Audio private decoding, and the like, for example, the demux API may be called by the demux thread to decode the decapsulated video file and the Audio file to obtain video frame data and Audio frame data, the render API includes OpenGL, metal, audio Track, openSL, audioQueue, for example, the OpenGL in the render API or the Metal may be called by the render thread to decode the video file, and then process the video frame data and the Audio frame data obtained by calling the decoding of the demux, where the processing procedure for the video frame data is as follows: the method comprises the steps of providing an AV-Player provided by an iOS operating system for outputting, binding a picture-in-picture controller (AVPictures controller), enabling a picture-in-picture display mode, and continuing to output video frame data; the processing procedure for the audio frame data is as follows: the method adopts a low-cost mode based on Audio Queue, which is applicable to recording and playing music on an iOS operating system and a Mac OS X operating system, and comprises the following steps: firstly, audio frame data is read into a buffer area of an Audio Queue, the buffer area is sent into an Audio Queue, then, the buffer area is told to an external device, the Audio frame data is already in the buffer area, the external device can take out and play the Audio frame data, finally, after the external device finishes playing, the buffer area is emptied, and the steps are repeated until all the Audio frame data are completely played.

That is, the SDK part built in the third-party Player on the left side of fig. 8 is responsible for decoding video files and Audio files and outputting decoded video frame data and Audio frame data, wherein for the video frame data part, when the third-party Player switches from the foreground to the background, the AV-Player with the iOS operating system can be started, and the video frame data decoded by the SDK is continuously played through the AV-Player, while for the Audio frame data part, the Audio Queue can be continuously called for playing output.

The video portion will be specifically described with reference to fig. 9.

For example, referring to fig. 9, fig. 9 is a schematic diagram of playing video frame data provided in the embodiment of the present application, as shown in fig. 9, after a multimedia file is unpacked (demux) to obtain a video file, the video file may be decoded (Decoder) to obtain video frame data, and then the video frame data may be displayed through OpenGL or Metal. When the third-party Player is switched from the foreground to the background, the AV-Player with the iOS operating system can be started to continue to display the video frame data, so that the aim of seamless switching playing is fulfilled, the video frame data played by the AV-Player is ensured to be still obtained by multiplexing the decoding capability of the SDK, and the advantage of the method is far higher than the hardware decoding capability of the iOS operating system.

The audio portion will be described in detail with reference to fig. 10.

For example, referring to fig. 10, fig. 10 is a schematic diagram of the principle of playing Audio frame data provided in the embodiment of the present application, as shown in fig. 10, for an Audio portion, the Audio portion may be based on an Audio processing frame encapsulated in an iOS operating system (the Audio Queue is an Audio processing frame encapsulated in the iOS operating system and may be used for playing or recording Audio, and supports encoding and decoding of platform-level Audio formats, the Audio Queue is a software object used for playing or recording Audio in the iOS operating system or a Mac OS X operating system, and is an Audio Queue ref data type defined in the Audio queue.h, and includes a plurality of buffers, each Buffer is a Buffer of some Audio frame data, and a Buffer Queue (Buffer Queue), that is an ordered Queue composed of a plurality of buffers, and is used for notifying that Audio frame data in the Buffer has been played completely, and may be filled with new Audio frame data in the Buffer), and includes the following steps: 1. reading audio frame data from a Hard disk (Hard disk); 2. storing the audio frame data in a plurality of buffers (e.g., 3 buffers, buffer 1, buffer 2, and buffer 3, respectively); 3. 3 buffer areas filled with audio frame data are sent to an audio queue; 4. notifying the playback device to play the audio frame data in the audio queue; 5. after the playing is finished, related resources are cleaned, wherein the operation flow for the Buffer zone (Buffer) is as follows: firstly, calling a corresponding method (such as a Callback), reading Audio frame data from a hard disk into a buffer zone of an Audio Queue, and sending the buffer zone filled with the Audio frame data into an Audio Queue; then, through the interface provided by the Audio Queue, the play-out device (such as a microphone of the terminal device) is notified, and the buffer area already stores Audio frame data and can be taken out for play; after the Audio frame data in one buffer is played, the Audio Queue may go through the Callback, and an empty buffer (e.g., buffer 1 shown in fig. 10) may be currently used to fill new Audio frame data, and the above steps are repeated until all the Audio frame data is played.

According to the video playing processing method, the function of seamlessly playing the video when the third-party player is switched from the foreground to the background can be achieved, the hardware decoding capability improved by the operating system is effectively utilized, the software decoding and self-defined decoding capability provided by the SDK built-in the third-party player can be utilized, normal output of the audio and video is not affected when the foreground is switched from the background, and therefore viewing experience of a user is effectively improved.

Continuing with the description below of an exemplary architecture of the video playback processing device 555 implemented as a software module provided in embodiments of the present application, in some embodiments, as shown in fig. 2, the software modules stored in the video playback processing device 555 of the memory 550 may include: an acquisition module 5551, a decoding module 5552, a playing module 5553, a hiding module 5554, and a transmitting module 5555.

An acquisition module 5551, configured to acquire a video file; a decoding module 5552 for decoding a video file; a playing module 5553, configured to play the video frame data obtained by decoding in the third-party player interface; a hiding module 5554, configured to hide the third-party player interface in response to a first switching instruction that instructs the third-party player to switch from the foreground to the background; and the sending module 5555 is configured to send the video frame data to a native player in the terminal device, so that the native player continues the playing progress of the third-party player, and continue playing the video frame data in a native player interface, where the size of the native player interface is smaller than the size of the display screen of the terminal device.

In some embodiments, the third party player has a software development suite built into it that integrates a variety of different types of decoders; the decoding module 5552 is further configured to invoke the software development suite to perform the following processing: determining the coding format of a video file; and calling a decoder matched with the coding format to decode the video file.

In some embodiments, the sending module 5555 is further configured to send the following data in the video frame data for a predetermined duration to the native player: and playing the video frame data with the time after the first moment, wherein the first moment is the moment when the third party player switches from the foreground to the background.

In some embodiments, the sending module 5555 is further configured to send video frame data corresponding to a subsequent time period to the native player, where the subsequent time period is a time period located after a first time in a playing time axis of the video file, and the first time is a time when the third party player switches from the foreground to the background.

In some embodiments, when the native player continues the playing progress of the third party player and continues playing the video frame data in the native player interface, the sending module 5555 is further configured to send the video frame data with the playing time before the first time to the native player, so that the native player starts playing from the video frame data corresponding to the second time based on the received video frame data with the playing time before the first time in the native player interface in response to the instruction to start playing from the second time, where the second time is earlier than the first time.

In some embodiments, the video play processing device 555 further comprises a creation module 5556 and a delivery module 5557, wherein the creation module 5556 is configured to create a picture-in-picture controller object before continuing to play video frame data in the native player interface; the transfer module 5557 is configured to transfer a reference to the pd controller object for the native player interface, so that the native player interface is bound to the pd controller object, where the pd controller object is configured to control the native player interface to adopt a display mode smaller than a size of the display screen.

In some embodiments, the first switching instruction is generated by an operating system of the terminal device and sent to the third party player, and is generated when at least one of: receiving triggering operation for indicating the third-party player to switch from the foreground to the background; receiving a voice call request; receiving a video call request; the currently played video frame data includes recommendation information.

In some embodiments, the hiding module 5554 is further configured to hide the native player interface in response to a second switch instruction that instructs the third party player to switch from the background back to the foreground; the playing module 5553 is further configured to display a third-party player interface, and continue the playing progress of the native player, and continue playing the video frame data in the third-party player interface.

In some embodiments, the video play processing device 555 further includes a stop module 5558 for stopping sending video frame data to the native player when the native player interface is hidden.

In some embodiments, the sending module 5555 is further configured to, when the native player interface is hidden, if the video frame data has not been transmitted to the native player, continue to send the video frame data with a playing time after a third time to the native player, where the third time is a time when the third party player switches from the background to the foreground.

In some embodiments, the obtaining module 5551 is further configured to obtain a multimedia file, where the multimedia file is obtained by encapsulating a video file, an audio file, and a subtitle file; the video playing processing device 555 further comprises a decapsulation module 5559, configured to decapsulate the multimedia file to obtain a video file; the decapsulation module 5559 is further configured to decapsulate the multimedia file to obtain an audio file and a subtitle file, where subtitles in the subtitle file are used for playing in synchronization with video frame data in the third-party player interface.

In some embodiments, the decoding module 5552 is further configured to decode an audio file; the playing module 5553 is further configured to play audio frame data synchronized with the video frame data currently played according to the video frame data currently played in the third party player interface.

In some embodiments, the playing module 5553 is further configured to fill the audio frame data into at least one buffer area, and store the at least one buffer area into the audio queue; the audio output device is called to read the audio frame data filled in the at least one buffer area from the audio queue so that the audio output device outputs a corresponding audio signal.

In some embodiments, the playing module 5553 is further configured to, in response to a first switching instruction instructing the third party player to switch from the foreground to the background, play audio frame data synchronized with the video frame data played in the native player interface according to the progress of the video frame data that the native player continues to play in the native player interface.

In some embodiments, the sending module 5555 is further configured to send, when sending the video frame data to the native player in the terminal device, the audio frame data to the native player, so that the native player continues the playing progress of the third party player, and continues to play the audio frame data.

In some embodiments, the sending module 5555 is further configured to perform one of the following: transmitting the following data among the audio frame data of a predetermined length to the native player: playing the audio frame data with the time after the first time; and sending the audio frame data positioned after a first moment in a complete playing time axis of the video file to the original player, wherein the first moment is the moment when the third party player switches from the foreground to the background.

It should be noted that, the description of the apparatus in the embodiment of the present application is similar to the description of the embodiment of the method described above, and has similar beneficial effects as the embodiment of the method, so that a detailed description is omitted. The technical details of the video play processing device provided in the embodiment of the present application may be understood from the description of any one of fig. 3 to fig. 6.

Embodiments of the present application provide a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the video playing processing method according to the embodiment of the application.

The embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions, in which the computer-executable instructions are stored, which when executed by a processor, cause the processor to perform the video playback processing method provided by the embodiments of the present application, for example, the video playback processing method as illustrated in any one of fig. 3 to 6.

In some embodiments, the computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; but may be a variety of devices including one or any combination of the above memories.

In some embodiments, the executable instructions may be in the form of programs, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and they may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

As an example, executable instructions may be deployed to be executed on one electronic device or on multiple electronic devices located at one site or, alternatively, on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and scope of the present application are intended to be included within the scope of the present application.

Claims

1. A video playing processing method, which is applied to a third party player in a terminal device, the method comprising:

acquiring a video file;

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the third-party player is internally provided with a software development suite, and the software development suite is integrated with a plurality of decoders of different types;

the decoding the video file includes:

invoking the software development suite to perform the following:

determining the coding format of the video file;

and calling the decoder matched with the coding format to decode the video file.

3. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the sending the video frame data to a native player in the terminal device includes:

transmitting the following data in video frame data of a predetermined duration to the native player: and playing the video frame data with the time after the first moment, wherein the first moment is the moment when the third party player is switched from the foreground to the background.

4. The method of claim 1, wherein the step of determining the position of the substrate comprises,

and sending the video frame data corresponding to a subsequent time period to the original player, wherein the subsequent time period is a time period located after a first time in a playing time axis of the video file, and the first time is a time when the third-party player is switched from a foreground to a background.

5. The method of claim 3 or 4, wherein when the native player continues to play the video frame data in a native player interface following the play progress of the third party player, the method further comprises:

transmitting video frame data with playing time before the first time to the original player so that

And the native player responds to an instruction for starting playing from a second moment, and starts playing from video frame data corresponding to the second moment based on the received video frame data with playing time before the first moment in the interface of the native player, wherein the second moment is earlier than the first moment.

6. The method of claim 1, wherein before continuing to play the video frame data in the native player interface, the method further comprises:

creating a picture-in-picture controller object;

and transmitting a reference to the PIP controller object for the native player interface so that the native player interface is bound with the PIP controller object, wherein the PIP controller object is used for controlling the native player interface to adopt a display mode smaller than the size of the display screen.

7. The method of claim 1, wherein the first switching instruction is generated by an operating system of the terminal device and sent to the third party player, and is generated when at least one of:

receiving a triggering operation for indicating the third-party player to switch from the foreground to the background;

Receiving a voice call request;

receiving a video call request;

the video frame data currently played includes recommendation information.

8. The method according to claim 1, wherein the method further comprises:

in response to a second switching instruction indicating that the third-party player is switched from the background to the foreground, hiding the native player interface, displaying the third-party player interface, and

continuing the playing progress of the original player, and continuing to play the video frame data in the third-party player interface.

9. The method of claim 8, wherein when hiding the native player interface, the method further comprises:

stopping sending the video frame data to the native player.

10. The method of claim 8, wherein when hiding the native player interface, the method further comprises:

and responding to the video frame data which is not transmitted to the original player, and continuing to transmit the video frame data with the playing time after a third time to the original player, wherein the third time is the time when the third-party player switches from the background to the foreground.

11. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the obtaining the video file includes:

acquiring a multimedia file, wherein the multimedia file is obtained by packaging the video file, the audio file and the subtitle file;

decapsulating the multimedia file to obtain the video file;

the method further comprises the steps of:

and decapsulating the multimedia file to obtain the audio file and the subtitle file, wherein the subtitles in the subtitle file are used for being played in synchronization with the video frame data in the third-party player interface.

12. The method of claim 11, wherein when decoding the video file and playing the decoded video frame data in a third party player interface, the method further comprises:

and decoding the audio file, and playing the audio frame data synchronized with the video frame data currently played according to the video frame data currently played in the third-party player interface.

13. The method of claim 12, wherein said playing said audio frame data synchronized with said video frame data currently being played comprises:

Filling the audio frame data into at least one buffer area, and storing the at least one buffer area into an audio queue;

and calling an audio output device to read the audio frame data filled in the at least one buffer area from the audio queue so that the audio output device outputs a corresponding audio signal.

14. The method of claim 12, wherein in response to a first switch instruction instructing the third party player to switch from the foreground to the background, the method further comprises:

and playing the audio frame data synchronized with the video frame data played in the native player interface according to the progress of the video frame data which is continuously played in the native player interface by the native player.

15. The method of claim 12, wherein when transmitting the video frame data to a native player in the terminal device, the method further comprises:

and sending the audio frame data to the original player so that the original player can continue to play the audio frame data according to the playing progress of the third-party player.

16. The method of claim 15, wherein the sending the audio frame data to the native player comprises:

One of the following processes is performed:

transmitting the following data in audio frame data of a predetermined length to the native player: playing the audio frame data with the time after the first time;

and sending the audio frame data positioned after a first moment in a complete playing time axis of the video file to the original player, wherein the first moment is the moment when the third party player switches from a foreground to a background.

17. A video play processing apparatus, characterized by being applied to a third party player in a terminal device, comprising:

the acquisition module is used for acquiring the video file;

the decoding module is used for decoding the video file;

18. An electronic device, comprising:

a memory for storing executable instructions;

a processor configured to implement the video playback processing method of any one of claims 1 to 16 when executing the executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the video playback processing method of any one of claims 1 to 16.

20. A computer program product comprising a computer program or computer executable instructions which, when executed by a processor, implement the video playback processing method of any one of claims 1 to 16.