CN117768660A

CN117768660A - Data processing method, device, apparatus, medium and program product

Info

Publication number: CN117768660A
Application number: CN202211133653.8A
Authority: CN
Inventors: 张佳; 曹洪彬; 黄永铖; 杨小祥; 曹健; 陈思佳; 宋美佳
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-09-16
Filing date: 2022-09-16
Publication date: 2024-03-26

Abstract

The embodiment of the application discloses a data processing method, a device, equipment, a medium and a program product, which are applied to the field of cloud technology. The method comprises the following steps: and receiving the encoded first picture frame, calling a first decoder to decode the encoded first picture frame, sending a decoding switching request to the data encoding equipment when the switching condition of the decoder is met, and calling a second decoder to decode the encoded switching picture frame when the encoded switching picture frame is received to obtain the switching picture frame. By adopting the embodiment of the application, the decoding process of the picture data can be optimized, and the display effect of the picture data is further improved.

Description

Data processing method, device, apparatus, medium and program product

Technical Field

The present disclosure relates to the field of cloud technologies, and in particular, to a data processing method, apparatus, device, medium, and program product.

Background

Currently, in a scene of transmitting picture data (such as game picture data), it is common to encode the picture data for transmission and decode the encoded picture data to restore and display the corresponding picture data. In the decoding process, the display effect of the picture data is poor due to the influence of some factors, so that the user experience is affected. For example, a picture data is stuck at the time of display due to an abnormality of the decoder. Therefore, how to optimize decoding of picture data becomes a hot problem of research.

Disclosure of Invention

The embodiment of the application provides a data processing method, a device, equipment, a medium and a program product, which can optimize the decoding process of picture data and improve the display effect of the picture data.

In one aspect, an embodiment of the present application provides a data processing method, including:

receiving the encoded first picture frame, and calling a first decoder to decode the encoded first picture frame;

when the decoder switching condition is met, a decoding switching request is sent to the data encoding equipment; the decoding switching request is used for indicating the data encoding equipment to perform full-frame encoding processing on the switching picture frame so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when the data coding equipment receives a decoding switching request;

and when the coded switching picture frame is received, invoking a second decoder to decode the coded switching picture frame to obtain the switching picture frame.

In one aspect, an embodiment of the present application provides another data processing method, including:

acquiring an encoded first picture frame and transmitting the encoded first picture frame to data decoding equipment;

When receiving a decoding switching request sent by data decoding equipment, performing full-frame coding processing on the switching picture frame to obtain a coded switching picture frame; the switching picture frame is a picture frame to be coded when a decoding switching request is received;

and transmitting the encoded switching picture frame to a data decoding device.

In one aspect, an embodiment of the present application provides a data processing apparatus, including:

the calling module is used for receiving the encoded first picture frame and calling the first decoder to decode the encoded first picture frame;

a transmitting module, configured to transmit a decoding switching request to the data encoding apparatus when a decoder switching condition is satisfied; the decoding switching request is used for indicating the data encoding equipment to perform full-frame encoding processing on the switching picture frame so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when the data coding equipment receives a decoding switching request;

and the calling module is also used for calling the second decoder to decode the coded switching picture frame to obtain the switching picture frame when the coded switching picture frame is received.

In one aspect, an embodiment of the present application provides another data processing apparatus, including:

The acquisition module is used for acquiring the encoded first picture frame and transmitting the encoded first picture frame to the data decoding equipment;

the processing module is used for carrying out full-frame coding processing on the switching picture frame when receiving a decoding switching request sent by the data decoding equipment to obtain a coded switching picture frame; the switching picture frame is a picture frame to be coded when a decoding switching request is received;

and the processing module is also used for sending the coded switching picture frame to the data decoding equipment.

In one aspect, embodiments of the present application provide a data decoding device, including a processor and a memory, where the memory is configured to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform some or all of the steps in the above method.

In one aspect, embodiments of the present application provide a data encoding apparatus, including a processor and a memory, where the memory is configured to store a computer program, the computer program including program instructions, the processor being configured to invoke the program instructions to perform some or all of the steps of the above-described method.

In one aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program comprising program instructions for performing part or all of the steps of the above method when executed by a processor.

Accordingly, according to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions which, when executed by a processor, implement some or all of the steps of the above method.

In the embodiment of the application, the first picture frame after encoding is received, and the first decoder is called to decode the first picture frame after encoding, when the switching condition of the decoder is met, a decoding switching request is sent to the data encoding equipment, and the switching condition of the decoder can be used for indicating whether the currently used decoder is suitable or not; when the coded switching picture frame is received, a second decoder is called to decode the coded switching picture frame to obtain the switching picture frame; the coded switching picture frame is obtained based on full-frame coding processing and is independently decodable data, so that timely switching to a proper decoder can be realized, the decoding process of picture data can be optimized, and the display effect of the picture data can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an application architecture according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application;

fig. 3 is a schematic diagram of a scenario of a switching decoder according to an embodiment of the present application;

fig. 4 is a flow chart of a data processing method according to an embodiment of the present application;

fig. 5 is a schematic view of a scene of an encoded picture frame according to an embodiment of the present application;

fig. 6 is a schematic view of a scene of an encoded picture frame according to an embodiment of the present application;

fig. 7 is a schematic diagram of a scenario of a switching decoder according to an embodiment of the present application;

FIG. 8 is an interaction schematic diagram of a data processing method according to an embodiment of the present application;

fig. 9a is a schematic diagram of a frame of decoding a frame of a picture based on an end cloud coordination policy according to an embodiment of the present application;

fig. 9b is a schematic diagram of a frame of decoding a frame of a picture based on an end cloud coordination policy according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of a data decoding device according to an embodiment of the present application;

Fig. 13 is a schematic structural diagram of a data encoding apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

The data processing method provided by the embodiment of the application is implemented in a data decoding device, and the data decoding device can be a server or a terminal. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like. The terminal may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc.

Next, technical terms related to the technical field of possible applications of the solution of the embodiment of the present application are described in the following:

1. Cloud technology:

cloud technology (Cloud technology) is a generic term of network technology, information technology, integration technology, management platform technology, application technology and the like based on Cloud computing business model application, and can form a resource pool, so that the Cloud computing business model application system is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

In one possible implementation manner, the data processing method provided by the embodiment of the application may be combined with cloud technology. Such as may be combined with cloud application technology in the cloud technology field. The method can be particularly applied to Cloud game (Cloud game) scenes, the picture frames can belong to game picture data in the Cloud game, the data decoding equipment can be equipment (such as a terminal) for operating the Cloud game, and the data encoding equipment can be equipment (such as a Cloud server) for operating the Cloud game. The cloud game may also be called game on demand (game on demand), which is an online game technology based on a cloud computing technology. Cloud gaming technology enables lightweight devices (thin clients) with relatively limited graphics processing and data computing capabilities to run high quality games. In a cloud game scene, the game is not run in a player game terminal, but is run in a cloud server, the cloud server renders the game scene into a video and audio stream, and the video and audio stream is transmitted to the player game terminal through a network. The player game terminal does not need to have strong graphic operation and data processing capability, and only needs to have basic streaming media playing capability and the capability of acquiring player input instructions and sending the player input instructions to the cloud server.

2. Decoding picture data:

picture data decoding, also called video data decoding, means that after picture frames (also called video frames, image frames) in picture data (also called video data, image data) are encoded, the encoded picture frames are decoded by a decoder to restore the corresponding picture frames. Wherein the decoder includes a hardware decoder and a software decoder. Decoding by a hardware decoder, also known as hard decoding, means decoding by a graphics card core GPU (Graphics Processing Unit, graphics processor), the CPU (Central Processing Unit ) occupancy of the hard decoding is lower, and the decoding rate is faster. Decoding by a software decoder, also known as soft decoding, means decoding by the CPU, the decoding rate is typically less than that of hard decoding. The method and the device can realize real-time dynamic switching of the decoder in the decoding process, and can increase the proportion of hard decoding as much as possible.

In some embodiments, please refer to fig. 1, fig. 1 is a schematic diagram of an application architecture provided in an embodiment of the present application, through which the data processing method proposed in the present application may be executed. As shown in fig. 1, among others, a data decoding apparatus and a data encoding apparatus. The data encoding device can encode the picture frame and transmit the encoded picture frame to the data decoding device, and when the data decoding device receives the encoded first picture frame, the data decoding device calls the first decoder to decode the encoded first picture frame, and the first decoder represents the currently used decoder; when the decoder switching condition is met, the data decoding device sends a decoding switching request to the data encoding device so as to instruct the data encoding device to perform full-frame encoding processing on the switching picture frame to obtain an encoded switching picture frame, wherein the switching picture frame is a picture frame to be encoded; when the data decoding device receives the coded switching picture frame, a second decoder is called to decode the coded switching picture frame, so that real-time switching of the decoders is realized. After the switching picture frame is obtained, the switching picture frame may be displayed. It will be appreciated that after a decoder switch, and before the next decoder switch condition is met, the received encoded picture frame is then decoded by the switched second decoder. Furthermore, the data encoding apparatus may be a terminal or a server. In the following description, a data decoding device is taken as a terminal (e.g., a user terminal), and a data encoding device is taken as a server as an example.

It should be understood that fig. 1 is merely an exemplary representation of an application architecture that may exist in the present application technical solution, and is not limited to a specific architecture of the present application technical solution, that is, the present application technical solution may also provide other application architectures.

Optionally, in some embodiments, the computer device (such as the data encoding device and the data decoding device) may perform the data processing method according to actual service requirements to improve the display effect of the picture data. The technical scheme can be applied to decoding scenes of any picture data. For example, the method can be applied to different types of video transmission in a transmission scene of video data, such as online watched video, live broadcast and the like. In this scenario, the data encoding device encodes video picture frames in the video data, and transmits each video picture frame to the data decoding device after encoding is completed, and the data decoding device calls the corresponding decoder to decode each time the encoded video picture frame is received. As another example, the method can be applied to different types of game transmission in a transmission scene of game data, such as cloud games, applet games and the like. In this scenario, the data encoding device may render the corresponding game picture frame based on the game operation to obtain the game picture data, encode and transmit each game picture frame after the rendering is completed to the data decoding device, and when each encoded game picture frame is received, the data decoding device invokes the corresponding decoder to perform decoding. In some embodiments, in the process that the data decoding device invokes the first decoder to decode the picture data, if the decoder switching condition is met, timely switching of the decoder can be achieved according to the technical scheme of the application, that is, a decoding switching request is sent to the data encoding device to obtain an encoded switching picture frame, and the encoded switching picture frame is a frame that can be independently decoded, so that a second decoder to be switched can be invoked to decode the encoded switching picture frame, and optimization of the decoding process is achieved.

Optionally, the data related to the present application, such as a frame of a picture, a decoding rate of a decoder, etc., may be stored in a database, or may be stored in a blockchain, such as by a blockchain distributed system, which is not limited in this application.

It should be noted that, in the specific embodiment of the present application, if related data to collect user information, such as the decoding rate of a decoder on a user terminal, is involved, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with related laws and regulations and standards of related countries and regions.

It can be understood that the above scenario is merely an example, and does not constitute a limitation on the application scenario of the technical solution provided in the embodiments of the present application, and the technical solution of the present application may also be applied to other scenarios. For example, as one of ordinary skill in the art can know, with the evolution of the system architecture and the appearance of new service scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

Based on the above description, the embodiments of the present application propose a data processing method, which can be performed by the above-mentioned data decoding apparatus. Referring to fig. 2, fig. 2 is a flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 2, the flow of the data processing method in the embodiment of the present application may include the following:

S201, receiving the encoded first picture frame, and calling a first decoder to decode the encoded first picture frame.

In some embodiments, the first picture frame may be acquired or rendered by a data encoding device, and encoded based on certain picture encoding parameters to obtain an encoded first picture frame, which is decoded by a data decoding device. Further, the first picture frame may be one or more picture frames in picture data, and the picture data may be any type of data, such as video picture data, or game picture data, or the like. For example, the data decoding apparatus views a video online, and the data encoding apparatus may acquire corresponding video picture data, encode video picture frames in the video picture data, and transmit the video picture frames to the data decoding apparatus. In another example, the data decoding device performs game operations in the game application, the game application is run by the data encoding device, the data decoding device sends corresponding operation information to the data encoding device, and the data encoding device performs corresponding game control and game picture frame rendering in the game application according to the operation information, encodes the game picture frames and transmits the game picture frames to the data decoding device. Optionally, after the first decoder is called to decode the encoded first picture frame to obtain the first picture frame, the first picture frame may be displayed.

In some embodiments, the first decoder is a decoder used in the current decoding process, and the first decoder may be a hardware decoder or a software decoder. It may be that the data encoding apparatus transmits one encoded picture frame every time it encodes a picture frame, and the data decoding apparatus invokes the first decoder to perform a real-time decoding process on the received encoded first picture frame (i.e., the picture frame transmitted in real time). Or the data encoding device may transmit the encoded specified number of picture frames once every time the specified number (more than or equal to 2) of picture frames (for example, 10 frames), and the data decoding device invokes the first decoder to perform decoding processing on the received encoded first picture frames (i.e., the transmitted specified number of picture frames once). And are not limited herein.

In some embodiments, during coded transmission of picture data, the transmission is typically performed in a picture stream, such as a picture stream with a specified number of frames (e.g., 100 frames) or a specified duration (e.g., 1 minute). The starting picture frame of a picture stream is usually encoded as an I-frame (an encoded frame representing the complete picture of a picture frame), also called a key frame, i.e. a frame that is encoded based on the entire data of the starting picture frame and can be decoded independently without the need for further picture frames. Intra picture frames other than the starting picture frame in a picture stream may be encoded as P frames (encoded frames representing difference pictures of a current picture frame and a previous picture frame) or B frames (encoded frames representing difference pictures of a current picture frame and a previous and subsequent picture frames), i.e., frames encoded based on data in the intra picture frame and adjacent picture frames of the intra picture frame, which require reference to the adjacent picture frames or the encoded data of the adjacent picture frames for decoding. The first picture frame may be a picture frame in a picture stream, and the first picture frame may include a start picture frame and/or an intra picture frame in a picture stream, that is, the encoded first picture frame may be obtained based on encoding as an I frame or may be obtained based on encoding as a P frame or a B frame.

In addition, in the decoding process of the current decoder, there may be a scene where the decoder needs to be switched, such as that the current decoder is not available or can be switched to a decoder with better decoding efficiency, etc. When the decoder is switched, the display of the picture data is easy to be interrupted or blocked by the perception of the user, so that the timely switching of the decoder can be realized through the technical scheme, and the user experience is improved.

S202, when the decoder switching condition is met, a decoding switching request is sent to the data encoding device.

The decoding switching request may be used to instruct the data encoding device to perform full-frame encoding processing on the switching picture frame to obtain an encoded switching picture frame.

In some embodiments, the data decoding apparatus may detect whether a decoder switching condition is currently satisfied, and upon satisfaction of the decoder switching condition, transmit a data switching request to switch a currently used first decoder to a second decoder.

In some embodiments, if the first decoder is a hardware decoder, the second decoder to be switched may be a software decoder; if the first decoder is a software decoder, the second decoder to be switched is a hardware decoder. Determining whether the decoder switching condition is currently satisfied may be determining that the decoder switching condition is satisfied if the first decoder is a hardware decoder and it is detected that the hardware decoder is updated from an available state to an unavailable state; if the first decoder is a software decoder and it is detected that the hardware decoder is updated from an unavailable state to an available state, it is determined that a decoder switching condition is satisfied. Since the decoding rate of a hardware decoder is typically significantly higher than that of a software decoder and the CPU occupancy of the hardware decoder is low, it is desirable to use the hardware decoder as much as possible in the decoding process in order to ensure a user experience and low latency of the display of picture data. That is, the data decoding apparatus may detect the state of the hardware decoder, use the software decoder when in an unavailable state (e.g., hardware decoder abnormality, etc.), and switch to the hardware decoder in time when detecting that in an available state.

In some embodiments, the hardware decoder may therefore be disabled or (temporarily) unavailable for reasons of device resource occupancy, device operation, device version, etc. If the hardware decoder is in the disabled state, the data decoding device does not have the capability of calling the hardware decoder to decode, and is realized by using the software decoder when decoding the picture data. If the hardware decoder is in an unavailable state, the data decoding device has the capability of calling the hardware decoder to decode, but the hardware decoder cannot be used at the current decoding moment, and when the hardware decoder can be used, the software decoder is switched to the hardware decoder, so that the hardware decoding proportion is improved.

For example, when it is detected that the hardware decoder satisfies the use prohibition condition, it is determined that the hardware decoder is in the use prohibition state. The use prohibition condition may be that the data decoding device does not have a hardware decoder, or that a device driving version of the hardware decoder is lower than a preset version, or the like. The present invention is not limited thereto, and may be specifically set according to actual scenes.

For another example, upon detecting that the hardware decoder satisfies the unavailable condition, it is determined that the hardware decoder is in an unavailable state. For example, it may be determined that an unavailability condition is met when an abnormal operation of the hardware decoder is detected. Or may be to obtain the device resource occupancy rate of the hardware encoder, and if the device resource occupancy rate is greater than or equal to the occupancy rate threshold, determine that the unavailable condition is satisfied.

For another example, when the hardware encoder is detected to meet the available condition, it is determined that the hardware decoder is in an available state. For example, it may be determined that the availability condition is met when it is detected that the hardware decoder is functioning properly. Or may be to obtain the device resource occupancy rate of the hardware encoder, and if the device resource occupancy rate is less than the occupancy rate threshold, determine that the available condition is met.

In some embodiments, the decoder switching condition may also be determined based on decoding rates of the first decoder and the second decoder if both are in an available state. Since the higher the decoding rate, the higher the decoding effect. It is therefore possible to use a decoder having a higher decoding rate as much as possible in the decoding process of picture data. Thus, satisfying the decoder switching condition may mean that the decoding rate of the second decoder is greater than the decoding rate of the first decoder. The decoding rate may be acquired by the data decoding device in real time or periodically.

Thus, the decoder switching request may be used to indicate that the currently used decoder is appropriate. For example, if the currently used decoder is a software decoder and a hardware decoder can be used, the currently used decoder is not suitable, and the currently used decoder can be switched to the hardware decoder; if the currently used decoder is a hardware decoder and the hardware decoder cannot be used, the currently used decoder is not suitable, and the currently used decoder can be switched to a software decoder and timely switched to a suitable decoder for decoding, so that the decoding process is optimized. For another example, both the hardware decoder and the software decoder may be used, if the currently used decoder is a software decoder and the decoding rate of the hardware decoder is greater than the decoding rate of the software decoder, the currently used decoder is not suitable, and the currently used decoder may be switched to the hardware decoder; if the currently used decoder is a hardware decoder and the decoding rate of the hardware decoder is smaller than that of the software decoder, the currently used decoder is not suitable, and the currently used decoder can be switched to the software decoder and timely switched to the suitable decoder for decoding, so that the decoding process is optimized.

In some embodiments, the data decoding apparatus may perform the detection of the decoder switching condition at a point in time when a preset time condition is satisfied. The preset time condition may be that the current time point satisfies a specified time interval from the last detection time point, i.e. periodic detection. For example, the detection of the decoder switching condition may be performed every 2 seconds from the start of the decoding process of the picture data. Alternatively, the preset time condition may be a specific decoding progress point at which the current time point satisfies one picture frame, for example, a start time of decoding processing for one picture frame after encoding, or a specific time after the start (for example, 3 th second, 5 th second, etc. after the start), or an end time, or a specific time after the end (for example, 3 rd second, 5 th second, etc. after the end), and the detection of the decoder switching condition may be performed. And are not limited herein.

In some embodiments, the data decoding apparatus may send a decoding switching request to the data encoding apparatus when the decoder switching condition is satisfied, the decoding switching request being for instructing the data encoding apparatus to perform full-frame encoding processing on the switching picture frame to obtain the encoded switching picture frame. The switching picture frame may be a picture frame to be encoded by the data encoding apparatus upon receiving the decoding switching request. The picture frame to be encoded may be a picture frame being encoded (i.e., an encoding uncompleted picture frame) or an uncoded picture frame, such as the first uncoded picture frame after receiving a decoding switch request. The switching frame may be an adjacent frame after the first frame or may be a frame after the first frame and spaced apart by a certain frame.

In some embodiments, when an encoded picture frame is transmitted once per picture frame, the picture frame to be encoded may be, in particular, a picture frame being encoded (i.e., a picture frame being encoded but not encoded) or a picture frame to be encoded (i.e., a picture frame to be subjected to an encoding process, such as the first picture frame not encoded described above). Alternatively, when the encoded picture frame is transmitted once every encoding of the specified number of picture frames, the picture frame to be encoded may specifically be the picture frame being encoded or the picture frame to be encoded among the specified number of picture frames, or may be the first picture frame among the specified number of picture frames being encoded or the first picture frame among the specified number of picture frames to be encoded. Further, the full-frame encoding process means that all data of the switching picture frame is encoded, that is, the switching picture frame is encoded as an I frame.

Since the switching of the decoder needs to be done by I-frames, i.e. the decoder needs to start decoding from the start frame of a picture stream. When the decoder switching condition is satisfied, the picture frame to be encoded by the data encoding apparatus may be an intra picture frame of one picture stream, that is, when the encoded intra picture frame is received, the switching of the decoder cannot be performed. Since the conventional I frames have a certain interval during the encoding process of the picture data, if the conventional I frames are switched, the decoder cannot be switched for a long time easily, which affects the decoding and display of the picture data. Therefore, the data decoding device and the data encoding device can actively request the I frame to realize negotiation and cooperation of the end cloud (for example, the end can refer to a client, the cloud can refer to a server such as a cloud server).

S203, when the coded switching picture frame is received, a second decoder is called to decode the coded switching picture frame, and the switching picture frame is obtained.

In some embodiments, the encoded switching picture frame is a picture frame that can be independently decoded, so the data decoding apparatus may invoke the second decoder to perform decoding processing on the encoded switching picture frame when the encoded switching picture frame is received, and may display the switching picture frame after the switching picture frame is obtained.

In some embodiments, if the encoded second picture frame sent by the data encoding device is received, the switched second decoder is called to perform decoding processing on the encoded second picture frame to obtain the second picture frame, and the second picture frame is displayed. That is, encoded data corresponding to a picture frame after switching the picture frame is decoded by the second decoder. Wherein the second picture frame is a picture frame after the switching picture frame. I.e. in the belonging picture data the position of the second picture frame is after the switching picture frame. It will be appreciated that after switching to the second decoder in the current decoding process, the second decoder may be regarded as the first decoder in the next decoding process, and determine whether the decoder switching condition is satisfied in the above manner, and generate a decoding switching request when the decoder switching condition is satisfied, so as to implement switching of the decoders.

For example, as shown in fig. 3, fig. 3 is a schematic view of a switching decoder according to an embodiment of the present application; the data decoding device receives the encoded first picture frame, invokes the hardware decoder to decode the encoded first picture frame, detects the state of the hardware decoder, and sends a decoding switching request to the data encoding device when detecting that the hardware decoder is in an unavailable state, such as detecting that the hardware decoder is in an unavailable state at a preset time point; the data encoding device acquires a switching picture frame 1 when receiving a decoding switching request, wherein the switching picture frame is a picture frame to be encoded, for example, if a picture frame is being encoded at the moment, the picture frame is the picture frame to be encoded, and if the previous picture frame is already encoded and encoding of a next picture frame is to be performed at the moment, the picture frame to be encoded is the picture frame to be encoded; the data encoding device carries out full-frame encoding processing on the switched picture frames; when the data decoding equipment receives the coded switching picture frame 1, a software decoder is called to perform decoding processing so as to display the switching picture frame 1; the data encoding device encodes the second picture frame 1 (namely, the subsequent picture frame after switching the picture frame 1), and the data decoding device calls a software decoder to perform decoding when receiving the encoded second picture frame 1; in the process of calling the software decoder for decoding, if the hardware decoder is detected to be in an available state, for example, the hardware decoder is detected to be in an available state at a preset time point, a decoding switching request is sent, and the data encoding equipment acquires a switching picture frame 2 and performs full-frame encoding processing; the data decoding device receives the encoded switching picture frame 2, invokes the hardware decoder to perform decoding processing, and invokes the hardware decoder to perform decoding processing when receiving the encoded second picture frame 2 (i.e., a subsequent picture frame after the switching picture frame 2). Through the process, when the hardware decoder is unavailable, the software decoder can be switched to in time, so that the problem of damage degree of user experience is reduced as much as possible. And when the hardware decoder is available, the hardware decoder is switched in time to decode the picture frames at a higher decoding rate so as to reduce the display delay of the picture data.

In the embodiment of the application, the first encoded picture frame can be received, and the first decoder is called to perform decoding processing on the first encoded picture frame, and when the decoder switching condition is met, a decoding switching request is sent to the data encoding device, and the decoder switching condition can be used for indicating whether the currently used decoder is suitable or not; when the coded switching picture frame is received, a second decoder is called to decode the coded switching picture frame to obtain the switching picture frame; the coded switching picture frame is obtained based on full-frame coding processing and is independently decodable data, so that timely switching to a proper decoder can be realized, the decoding process of picture data can be optimized, and the display effect of the picture data, such as display fluency, can be improved.

Referring to fig. 4, fig. 4 is a flowchart of a data processing method according to an embodiment of the present application, where the method may be performed by the above-mentioned data encoding apparatus. As shown in fig. 4, the flow of the data processing method in the embodiment of the present application may include the following:

s401, acquiring an encoded first picture frame, and sending the encoded first picture frame to a data decoding device.

In some embodiments, the first frame may be a frame acquired or rendered by the data encoding device, and the encoded first frame is decoded by the data decoding device calling the first decoder, where the data decoding device may call a client application to which the frame belongs, for example, the frame belongs to game frame data, and the data decoding device may call a game application to perform a game operation. The first picture frame may be one or a specified number of picture frames in the picture data. The data encoding apparatus may transmit the encoded picture frame (i.e., the encoded first picture frame) to the data decoding apparatus every time one picture frame is encoded, or may transmit the encoded picture frame (i.e., the encoded first picture frame) to the data decoding apparatus every time a specified number of picture frames are encoded.

In some embodiments, the data encoding device typically transmits in a picture stream to encode a starting picture frame in a picture stream as an I-frame and an intra picture frame in a picture stream as a P-frame or a B-frame, and sequentially transmits the encoded picture frames in a picture stream. The picture data corresponds to one or more picture streams. For example, the picture data is video picture data, and the picture stream may be video picture data corresponding to every 3 seconds. As another example, the frame data is game frame data, and the frame stream may be game frame data rendered every 10 seconds. The first picture frame may include a starting picture frame and/or an intra picture frame in a picture stream, i.e. the first picture frame may be encoded by performing an encoding process according to a corresponding encoding manner, e.g. in a manner of encoding as an I frame and/or in a manner of encoding as a P frame or a B frame.

And S402, when a decoding switching request sent by the data decoding equipment is received, performing full-frame coding processing on the switching picture frame to obtain a coded switching picture frame.

Wherein the decoding switching request is generated when the data decoding apparatus detects that the decoder switching condition is satisfied at a preset point of time. For a specific description of the decoder switching condition, reference may be made to the description of the above embodiments.

In some embodiments, the switch picture frame may be a picture frame to be encoded upon receipt of a decode switch request. The picture frame to be encoded may be the picture frame being encoded, or the picture frame to be encoded, or the first picture frame of a specified number of picture frames being encoded, or the first picture frame of a specified number of picture frames to be encoded. And are not limited herein. Since the switching of the decoder requires an I frame, the switching picture frame can be subjected to full-frame encoding processing to obtain an encoded switching picture frame as an I frame. The data encoding apparatus may perform switching of the decoder by switching the picture frames after encoding.

In some embodiments, the full-frame encoding processing of the switching picture frame may be that a picture encoding parameter matched with a decoder to be switched indicated by the decoding switching request is obtained, and the full-frame encoding processing is performed on the switching picture frame by using the picture encoding parameter matched with the decoder to be switched, so as to obtain the encoded switching picture frame. The picture coding parameter matched with the decoder to be switched can be preset, for example, the picture coding parameter is a coding frame rate, when the decoder to be switched is a hardware decoder, the coding frame rate is a preset frame rate, and when the decoder to be switched is a software decoder, the coding frame rate is 1/2 of the preset frame rate; alternatively, it may be determined based on the decoding rate of the decoder to be decoded.

Since the performance of different decoders is different, the decoding rate is also different, so the picture coding parameters can be adaptively adjusted based on the decoding rates of different decoders. For example, when the picture coding parameter is the coding frame rate, if the decoding rate of the decoder to be switched is higher than that of the decoder before switching, the current coding frame rate can be improved, so that the display in the picture data is richer and smoother; if the decoding rate of the decoder to be switched is lower than that of the decoder before switching, the current coding frame rate can be reduced, so that decoding congestion caused by the reduction of the decoding rate due to the unchanged frame rate is avoided. Better decoding results can be obtained by encoding picture encoding parameters that match the decoder.

In some embodiments, the picture coding parameters that match the decoder to be switched may be determined by the data decoder and carried in the decoding switch request. Or the decoding rate of the decoder may be obtained by the data encoding device to determine the matching picture coding parameters. The decoding rate may be carried in a request for switching between decoding and decoding, or may be obtained by the data encoding apparatus in real time or periodically.

In some embodiments, the decoder in use is set to be a first decoder and the decoder to be switched is set to be a second decoder. The picture coding parameters matched with the decoder to be switched can be that a first decoding rate of a first decoder and a second decoding rate of a second decoder are obtained, and the picture coding parameters matched with the second decoder are determined according to the first decoding rate and the second decoding rate; the picture coding parameters matched to the second decoder are used to code the switching picture frames.

In some embodiments, if the picture coding parameters include a coding frame rate, determining a picture coding parameter matching the second decoder according to the first decoding rate and the second decoding rate may be to obtain a current coding frame rate; the current coding frame rate is a coding frame rate used by the data coding device when coding the first picture frame, i.e. the last coding frame rate used when coding the picture frame, and may be a coding frame rate matched when the first encoder is invoked for coding; and determining the encoding frame rate matched with the second encoder according to the first decoding rate, the second decoding rate and the current encoding frame rate, or adjusting the current encoding frame rate according to the rate difference value between the first decoding rate and the second decoding rate to obtain the encoding frame rate matched with the second encoder. The method specifically comprises the following steps:

wherein P1 represents a first encoding rate, P2 represents a second encoding rate, P1-P2 represents a rate difference, W1 represents a current encoding frame rate, and W2 represents a matching encoding frame rate.

In some embodiments, the picture coding parameters may further include a coding resolution, and the manner of determining the coding resolution that matches the second encoder may be the same as the manner of determining the matching coding frame rate.

Therefore, end cloud negotiation and cooperation can be realized through data interaction between the data decoding device and the data encoding device, so that the data decoding device and the data encoding device negotiate whether encoding parameters used next need to be adjusted to proper encoding parameters or not, and the switched decoder is more suitable for decoding the picture frames. Further, when the data decoding apparatus starts decoding, the decoding rate of the decoder used may be acquired, and the encoding parameter to be currently used may be determined from the mapping relationship between the decoding rate and the picture encoding parameter.

S403, the coded switching picture frame is sent to a data decoding device.

In some embodiments, the data decoding device may switch the decoder by an encoded switch picture frame, i.e. the encoded switch picture frame is invoked by the data decoding device to perform the decoding process by the switched second decoder. And after the data encoding device encodes the second picture frame after switching the picture frame by using the picture encoding parameter matched with the second decoder, invoking the switched second decoder by the data decoding device to decode the encoded second picture frame.

It is understood that the first frame, the switching frame and the second frame may be frames belonging to the same frame stream, i.e. the switching frame and the second frame are intra frames in the frame stream. It is therefore possible to realize switching of decoders in the decoding process of one picture stream by performing full-frame encoding processing on the switching picture frames, and to perform decoding of the switching picture frames and the second picture frames using the switched decoders.

In some embodiments, when the encoded picture frame is transmitted once for each encoding of the specified number of picture frames, the data encoding apparatus may transmit the encoded switching picture frame to the data decoding apparatus after performing the full frame encoding process on the switching picture frame, and transmit the encoded switching picture frame to the data decoding apparatus after performing the encoding process on the specified number of picture frames after the switching picture frame; the encoding process may be continued for the subsequent picture frames, and the encoded picture frames (including the switching picture frames) may be obtained and then transmitted to the data decoding apparatus at once.

In some embodiments, in the encoding of the subsequent picture frame, the picture frame after the switching picture frame may be encoded with the switching picture frame as a start frame of a picture stream, or may still be encoded in the original encoding manner. For example, as shown in fig. 5, fig. 5 is a schematic view of a scene of an encoded picture frame according to an embodiment of the present application; if one picture frame is encoded, transmitting the encoded picture frame once, setting the switching picture frame as the last picture frame in a picture stream (set as 20 picture frames), setting the next picture frame (picture frame a) of the switching picture frame as the first picture frame in the picture stream, and when the switching picture frame is encoded into an I frame through full-frame encoding, encoding the next picture frame in the original encoding mode, namely, performing full-frame encoding as the initial frame in the picture stream, and performing non-full-frame encoding as the internal picture frame corresponding to the next picture frame in the next picture frame; alternatively, the next picture frame and the 18 subsequent picture frames may be used as intra-picture frames corresponding to the switching picture frames, and subjected to non-full-frame encoding (for example, a manner of encoding as P-frames or B-frames).

As another example, as shown in fig. 6, fig. 6 is a schematic view of a scene of an encoded picture frame according to an embodiment of the present application; if the coded picture frame is transmitted once for every specified number of picture frames, the switching picture frame may be the picture frame being coded or to be coded or the first picture frame of the specified number of picture frames, the switching picture frame is set to be the 3 rd picture frame of the specified number (e.g. 10) of picture frames, and is the last picture frame of a picture stream (set to be 20 picture frames), and the 4 th picture frame (picture frame B) of the specified number of picture frames is the first picture frame of the picture stream, then the coded switching picture frame may be sent to the data decoding device, and the 10 picture frames after the switching picture frame are coded and sent to the data decoding device; or 9 picture frames after the switching picture frame are coded, the 9 picture frames are transmitted together with the coded switching picture frame, and the subsequent 10 picture frames are continuously coded and then transmitted to the data decoding equipment; when the 4 th picture frame is coded, the 4 th picture frame can still be coded according to the original coding mode, namely, the 4 th picture frame is used as a starting frame in a picture stream to carry out full-frame coding processing, and 19 subsequent picture frames of the 4 th picture frame can be used as internal picture frames corresponding to the 4 th picture frame; alternatively, the 4 th frame and the 18 subsequent frames may be used as intra frames corresponding to the switching frame.

For example, as shown in fig. 7, fig. 7 is a schematic view of a switching decoder according to an embodiment of the present application; the data encoding device encodes the first picture frame by using picture encoding parameters 1, wherein the picture encoding parameters 1 are encoding parameters matched with the first decoder, and the encoded first picture frame is sent to the data decoding device; the data decoding device calls a first decoder to decode the encoded first picture frame, and generates a decoding switching request when the condition of decoder switching is detected to be met; when the data encoding device receives a decoding switching request, acquiring a picture encoding parameter 2 and a switching picture frame matched with a second decoder, wherein the picture encoding parameter can be preset or determined based on the decoding rate of the decoder; performing full-frame encoding processing on the switched picture frame by using picture encoding matched with the second decoder to obtain an encoded second picture frame; the data decoding device calls a second decoder to decode the encoded second picture frame; the data encoding device acquires the second picture frame after switching the picture frame, performs encoding processing by using the picture encoding parameter 2 matched with the second decoder, and calls the second decoder to perform decoding processing on the encoded second picture frame.

In this embodiment of the present application, an encoded first picture frame may be obtained, and the encoded first picture frame may be sent to a data decoding device, and when a decoding switching request sent by the data decoding device is received, a full-frame encoding process is performed on the switching picture frame, so as to obtain an encoded switching picture frame, and the encoded switching picture frame is sent to the data decoding device. By the method, the coded switching picture frames can be independently decoded by performing full-frame coding processing on the switching picture frames, and the data decoding equipment can realize the switching of the decoders based on the coded switching picture frames, so that the timely switching to the proper decoders can be realized, the decoding process of the picture data can be optimized, and the display effect of the picture data can be improved.

Referring to fig. 8, fig. 8 is an interaction diagram of a data processing method according to an embodiment of the present application, where the method may be performed by the above-mentioned data decoding device and data encoding device. As shown in fig. 8, the interaction procedure of the data processing method in the embodiment of the present application may include the following:

s801, the data encoding apparatus transmits the encoded first picture frame.

In some embodiments, the first picture frame may be one or more picture frames, and the data encoding apparatus encodes the first picture frame using picture transmission parameters matched to the first decoder. The picture transmission parameters matched with the first decoder may be preset or determined based on the decoding rate of the first decoder. The data encoding device may transmit the encoded picture frame once every picture frame is encoded, or may transmit the encoded picture frame once every prescribed number of picture frames are encoded.

S802, the data decoding device calls a first decoder to decode the encoded first picture frame, and sends a decoding switching request when the decoder switching condition is met.

In some embodiments, the data decoding device invokes the first decoder to decode the encoded first picture frame, and when it is detected that a decoder switching condition is satisfied, sends a decoding switching request, where the decoding switching request may carry picture transmission parameters matched with the decoder to be switched, and may also carry decoder information, where the decoder information indicates the second decoder to be switched by the data decoding device.

In some embodiments, it may be detected at a preset point in time whether the decoder switching condition is satisfied. The preset time point may be a time point of periodic detection, or may be any time point when the first encoded picture frame is decoded. For example, the time point when the encoded first picture frame is received may be any time point during the decoding process, or may be a time point when the decoding process is completed. And are not limited herein.

In some embodiments, when the first decoder is a software decoder, the decoder switching condition may be that the hardware decoder is updated from an unavailable state to an available state; when the first decoder is a hardware decoder, the decoder switching condition may be that the hardware decoder is updated from an available state to an unavailable state. The method can switch to soft decoding in time when the hard decoding is unavailable, reduce the display interruption condition of the picture data, and switch to the hard decoding in time when the hard decoding is available, increase the proportion of the hard decoding and reduce the display time delay of the picture data. The detection of the decoder switching condition can be to dynamically switch to a decoder capable of decoding or to a decoder with higher decoding efficiency when decoding the picture data, so that the display effect of the picture data, such as the smoothness of picture display, is ensured, and the blocking or interruption of the user experience is avoided.

S803, the data encoding device performs full-frame encoding processing on the switching picture frame to obtain an encoded switching picture frame, and returns the encoded switching picture frame.

In some embodiments, the data decoding apparatus may acquire, as the switching picture frame, a picture frame to be decoded, which may be a picture frame being decoded or a picture frame to be decoded, upon receiving the decoding switching request. The full frame coding process is to code all data of the switching picture frame, and the switching picture frame after coding is coded into an I frame.

In some embodiments, the data decoding apparatus may perform full frame encoding processing on the switching picture frame using picture encoding parameters matched with the decoder to be switched, improving flexibility of encoding at the time of decoder switching. The data decoding device may acquire the picture coding parameters which are carried by the decoding switching request and matched with the decoder to be switched, or may determine the matched picture coding parameters according to the decoder information carried by the decoding switching request, for example, may acquire the decoding rates of the hardware decoder and the software decoder, and determine the picture coding parameters matched with the indicated decoder to be switched according to the decoding rates of the hardware decoder, the decoding rates of the software decoder, the current coding parameters and the decoder information.

S804, the data decoding device calls a second decoder to decode the coded switching picture frame to obtain the switching picture frame.

In some embodiments, the encoded switching frame is a frame that can be independently decoded without combining information of other frames, so the data decoding device may invoke the second decoder to perform decoding processing, implement switching of the decoder, and display the switching frame after decoding is completed.

In some embodiments, the data encoding device obtains a second picture frame, the second picture frame being a picture frame after switching picture frames, encodes the second picture frame with the matched picture encoding parameters, and the data decoding device invokes a second decoder to decode the encoded second picture frame. Subsequently, when the second decoder is called to perform decoding processing, the second decoder can be regarded as the first decoder, at this time, the second picture frame after the switching picture frame can be regarded as the first picture frame, when the decoder switching condition is satisfied, a new switching picture frame is acquired, and the switching of the decoder is performed by using the new switching picture frame according to the above-mentioned procedure.

In some embodiments, when the data decoding device starts to decode, it may be detected whether the capability of calling the hardware decoder is provided, and if the capability of calling the hardware decoder is not provided, that is, the hardware decoder is in a disabled state, the software decoder is always used in the decoding process. If the capability of calling the hardware decoder is provided, the switching of the decoder can be performed according to whether the hardware decoder is in an available state or an unavailable state, so that a better decoding effect is realized.

For example, in a cloud game scenario, a cloud game needs to perform real-time transmission on huge amounts of game picture data, usually, a cloud server (data encoding device) firstly renders game picture frames and encodes the game picture frames, the encoded game picture frames are transmitted to a terminal (data decoding device) through a network, the terminal decodes and displays pictures, generally, the decoding speed of a hardware decoder is obviously faster than that of a software decoder, in order to ensure low delay and user experience of the cloud game, the hardware decoder is used as much as possible, but the hardware decoder is temporarily unavailable due to reasons such as equipment occupation, or the hardware decoder recovers the available state, in order to ensure that the game picture data can be successfully decoded, a certain end cloud cooperation strategy is needed to be adopted to realize timely switching of the decoder, namely, the terminal sends a decoding switching request to the cloud server to actively negotiate to encode the picture frames to be encoded into I frames to switch the decoder, and adjust to proper picture transmission parameters, so that the hardware decoder and the software decoder can be reasonably used adaptively, and the hardware decoder can be used as much as possible, and the user experience can be improved.

For example, as shown in fig. 9 a-9 b, fig. 9 a-9 b are schematic diagrams of frames of decoding frames of pictures based on an end cloud matching policy according to an embodiment of the present application; fig. 9a is an end cloud collaboration framework implemented based on an end cloud coordination policy, where a data encoding device always detects whether a data decoding device sends a decoding switching request, and instructs to perform a corresponding operation when receiving the decoding switching request. The decoding switching request may be understood as including a parameter adjustment request and an I-frame request, where the I-frame request is used to instruct the data encoding device to acquire a switching picture frame and perform full-frame encoding processing, and the parameter adjustment request is used to instruct the data encoding device to adjust a current encoding parameter to a picture encoding parameter matched with a decoder to be switched to perform encoding of the picture frame, and the picture encoding parameter may be, for example, an encoding frame rate, an encoding resolution, and the like.

Referring to fig. 9b, s1, the data decoding device performs detection of the hard decoding capability at the decoding initial stage, for example, when an application to which the picture data belongs is started, and determines whether the capability of performing hard decoding is available based on the detection result; s2, if the hardware decoder does not have the hard decoding capability, starting the software decoder, and sending a parameter adjustment request to the data encoding equipment so that the data encoding equipment adjusts the picture encoding parameters to parameters matched with the software decoder, wherein the parameters can be encoding parameters preset for the software decoder or encoding parameters determined based on the mapping relation between the decoding rate of the software decoder and the encoding parameters; s3, if the hardware decoder has the hard decoding capability and is in an available state, starting the hardware decoder, and sending a parameter adjustment request to the data encoding equipment so that the data encoding equipment adjusts the picture encoding parameters to parameters matched with the hardware decoder; s4, if the hardware decoder has the hard decoding capability and is in an unavailable state, starting the software decoder, and sending a parameter adjustment request to the data encoding equipment so that the data encoding equipment adjusts the picture encoding parameters to parameters matched with the software decoder; s5, after starting a hardware decoder or a software decoder, periodically detecting, namely detecting whether a time detection point is reached; s6, when the time monitoring point is reached, detecting whether the state of the hardware decoder is available; s7, if the state of the hardware decoder is in an available state, detecting whether the hardware decoder is currently used or not; s8, if the currently used decoder is not a hardware decoder, the decoder switching condition is met, namely, the software decoder is switched to the hardware decoder, and then a decoding switching request is sent; s9, the data encoding equipment receives a decoding switching request, adjusts picture encoding parameters, namely adjusts current encoding parameters to picture encoding parameters matched with a hardware decoder, and carries out full-frame encoding processing on the switching picture frames according to the matched picture encoding parameters; s10, the data decoding equipment calls a hardware decoder to decode the encoded switching picture frame, so that switching of the decoder is realized;

S11, if the state of the hardware decoder is in an unavailable state, detecting whether the hardware decoder is currently used or not;

s12, if the currently used decoder is a hardware decoder, which means that the decoder switching condition is met, namely, the hardware decoder is switched to a software decoder, a decoding switching request is sent; s13, the data encoding equipment receives a decoding switching request, adjusts picture encoding parameters, namely adjusts current encoding parameters to picture encoding parameters matched with a software decoder, and carries out full-frame encoding processing on the switching picture frames according to the matched picture encoding parameters; s14, the data decoding equipment calls a software decoder to decode the encoded switching picture frame, so that switching of the decoder is realized; in addition, in the above process, the data decoding apparatus periodically detects the state of the hardware decoder at all times to determine the timeliness of the decoder switching. And the decoder switching is realized through the end cloud cooperation strategy, so that the optimal experience of a user is ensured.

In this embodiment of the present application, the data encoding device may send the encoded first picture frame, the data decoding device may invoke the first decoder to perform decoding processing on the encoded first picture frame, and when a decoder switching condition is satisfied, send a decoding switching request, the data encoding device may obtain a switching picture frame, and perform full-frame encoding processing on the switching picture frame, to obtain an encoded switching picture frame, and return to the encoded switching picture frame, and the data decoding device may invoke the second decoder to perform decoding processing on the encoded switching picture frame to obtain the switching picture frame. By the method, the switching of the decoder can be realized through the data interaction between the data decoding equipment and the data encoding equipment, so that the decoding process is optimized, and the display effect of the picture data is improved.

The foregoing details of the method of embodiments of the present application are set forth in order to provide a better understanding of the foregoing aspects of embodiments of the present application, and accordingly, the following provides a device of embodiments of the present application. FIG. 10 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure; the data processing apparatus may be for a computer program (including program code) running in a computer device such as a data decoding device, for example the data processing apparatus may be an application program in a computer device; the data processing device may be used to perform some or all of the steps in the method embodiments shown in fig. 2 and 8. Referring to fig. 10, the data processing apparatus includes the following modules:

a calling module 1001, configured to receive the encoded first picture frame, and call a first decoder to decode the encoded first picture frame;

a sending module 1002, configured to send a decoding switching request to the data encoding apparatus when a decoder switching condition is satisfied; the decoding switching request is used for indicating the data encoding equipment to perform full-frame encoding processing on the switching picture frame so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when the data coding equipment receives a decoding switching request;

The calling module 1001 is further configured to, when receiving the encoded switching picture frame, call the second decoder to perform decoding processing on the encoded switching picture frame, so as to obtain the switching picture frame.

In some embodiments, if the first decoder is a hardware decoder, the second decoder is a software decoder; if the first decoder is a software decoder, the second decoder is a hardware decoder; the sending module 1002 is further configured to:

if the first decoder is a hardware decoder and the hardware decoder is detected to be updated from an available state to an unavailable state, determining that a decoder switching condition is met;

if the first decoder is a software decoder and it is detected that the hardware decoder is updated from an unavailable state to an available state, it is determined that a decoder switching condition is satisfied.

In some embodiments, the sending module 1002 is further configured to:

acquiring the equipment resource occupancy rate of a hardware encoder;

if the occupancy rate of the equipment resources is smaller than the occupancy rate threshold value, determining that the hardware encoder is in an available state;

and if the occupancy rate of the equipment resources is greater than or equal to the occupancy rate threshold value, determining that the hardware encoder is in an unavailable state.

In some embodiments, the calling module 1001 is further configured to:

Receiving an encoded second picture frame sent by the data encoding device; the second picture frame is a picture frame after the switching picture frame;

and calling a second decoder to decode the encoded second picture frame to obtain the second picture frame.

In some embodiments, the sending module 1002 is further configured to:

acquiring a first decoding rate of a first decoder and a second decoding rate of a second decoder;

determining picture coding parameters matched with a second decoder according to the first decoding rate and the second decoding rate; the decode switch request carries picture coding parameters that are used to code the switch picture frame.

In some embodiments, the picture coding parameters include a coding frame rate; the transmitting module 1002, when configured to determine a picture coding parameter matching the second decoder according to the first decoding rate and the second decoding rate, is specifically configured to:

acquiring a current coding frame rate; the current coding frame rate is the coding frame rate when the first picture frame is coded;

and adjusting the current coding frame rate according to the rate difference value between the first decoding rate and the second decoding rate to obtain the coding frame rate matched with the second coder.

According to one embodiment of the present application, each module in the data processing apparatus shown in fig. 10 may be separately or completely combined into one or several other modules to form a structure, or some (some) of the modules may be further split into a plurality of modules with smaller functions to form a structure, which may achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logic functions, and in practical application, the functions of one module may be implemented by a plurality of modules, or the functions of a plurality of modules may be implemented by one module. In other embodiments of the present application, the data processing apparatus may also include other modules, and in practical applications, these functions may also be implemented with assistance from other modules, and may be implemented by cooperation of a plurality of modules.

According to another embodiment of the present application, a data processing apparatus as shown in fig. 10 may be constructed by running a computer program (including program code) capable of executing the steps involved in the respective methods as shown in fig. 2 and 8 on a general-purpose computing device such as a computer including a processing element such as a central processing module (CPU), a random access storage medium (RAM), a read only storage medium (ROM), and the like, and a storage element, and implementing the data processing method of the embodiments of the present application. The computer program may be recorded on, for example, a computer-readable recording medium, and loaded into and run in the above-described computing device through the computer-readable recording medium.

In the embodiment of the application, the calling module receives the encoded first picture frame and calls the first decoder to decode the encoded first picture frame; the transmitting module transmits a decoding switching request to the data encoding equipment when the decoder switching condition is met; the decoder switching condition may be used to indicate whether the currently used decoder is suitable; and when the calling module receives the coded switching picture frame, the second decoder is called to decode the coded switching picture frame, so that the switching picture frame is obtained. By the device, the coded switching picture frame can be obtained based on full-frame coding processing and is independently decodable data, so that timely switching to a proper decoder can be realized, the decoding process of a picture can be optimized, and the display effect of the picture can be improved.

The foregoing details of the method of embodiments of the present application are set forth in order to provide a better understanding of the foregoing aspects of embodiments of the present application, and accordingly, the following provides a device of embodiments of the present application. FIG. 11 is a schematic diagram of a data processing apparatus according to an embodiment of the present application; the data processing apparatus may be for a computer program (including program code) running on a computer device such as a data encoding device, for example the data processing apparatus may be an application program in the computer device; the data processing device may be used to perform some or all of the steps in the method embodiments shown in fig. 4 and 8. Referring to fig. 11, the data processing apparatus includes the following modules:

an obtaining module 1101, configured to obtain an encoded first picture frame, and send the encoded first picture frame to a data decoding device;

the processing module 1102 is configured to perform full-frame encoding processing on the switching picture frame when receiving a decoding switching request sent by the data decoding device, so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when a decoding switching request is received;

the processing module 1102 is further configured to send the encoded switching picture frame to a data decoding device.

In some embodiments, the processing module 1102 is configured to, when configured to perform full frame encoding processing on the switching picture frame to obtain an encoded switching picture frame, specifically:

acquiring picture coding parameters matched with a decoder to be switched indicated by a decoding switching request;

and carrying out full-frame coding processing on the switching picture frame by using picture coding parameters matched with the decoder to be switched to obtain the coded switching picture frame.

According to one embodiment of the present application, each module in the data processing apparatus shown in fig. 11 may be separately or completely combined into one or several other modules to form a structure, or some (some) of the modules may be further split into a plurality of modules with smaller functions to form a structure, which may achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logic functions, and in practical application, the functions of one module may be implemented by a plurality of modules, or the functions of a plurality of modules may be implemented by one module. In other embodiments of the present application, the data processing apparatus may also include other modules, and in practical applications, these functions may also be implemented with assistance from other modules, and may be implemented by cooperation of a plurality of modules.

According to another embodiment of the present application, a data processing apparatus as shown in fig. 11 may be constructed by running a computer program (including program code) capable of executing the steps involved in the respective methods as shown in fig. 4 and 8 on a general-purpose computing device such as a computer including a processing element such as a central processing module (CPU), a random access storage medium (RAM), a read only storage medium (ROM), and the like, and a storage element, and implementing the data processing method of the embodiments of the present application. The computer program may be recorded on, for example, a computer-readable recording medium, and loaded into and run in the above-described computing device through the computer-readable recording medium.

In the embodiment of the application, the acquisition module acquires the encoded first picture frame and sends the encoded first picture frame to the data decoding equipment; when receiving a decoding switching request sent by data decoding equipment, the processing module carries out full-frame coding processing on the switching picture frame to obtain a coded switching picture frame; the processing module sends the encoded switching picture frame to the data decoding device. Through the device, the full-frame coding processing can be carried out on the switching picture frames so that the coded switching picture frames can be independently decoded, and the data decoding equipment can realize the switching of the decoders based on the coded switching picture frames, so that the timely switching to the proper decoders can be realized, the decoding process of picture data can be optimized, and the display effect of the picture data can be improved.

Referring to fig. 12, fig. 12 is a schematic structural diagram of a data decoding apparatus according to an embodiment of the present application. As shown in fig. 12, the data decoding apparatus 1200 includes: at least one processor 1201, memory 1202. Optionally, the data decoding device may further comprise a network interface. Data may be interacted among the processor 1201, the memory 1202 and a network interface, the network interface being controlled by the processor 1201 for receiving and sending messages, the memory 1202 being for storing a computer program comprising program instructions, the processor 1201 being for executing the program instructions stored by the memory 1202. Wherein the processor 1201 is configured to invoke the program instructions to perform the method described above.

The memory 1202 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory 1202 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a Solid State Drive (SSD), etc.; memory 1202 may also include a combination of two or more of the above-described types of memory.

The processor 1201 may be a central processing unit (central processing unit, CPU), among others. In one embodiment, the processor 1201 may also be a graphics processor (Graphics Processing Unit, GPU). The processor 1201 may also be a combination of a CPU and a GPU.

In one possible implementation, the memory 1202 is configured to store program instructions that the processor 1201 may call to perform the following steps:

In some embodiments, if the first decoder is a hardware decoder, the second decoder is a software decoder; if the first decoder is a software decoder, the second decoder is a hardware decoder; the processor 1201 is also configured to:

In some embodiments, the processor 1201 is further to:

acquiring the equipment resource occupancy rate of a hardware encoder;

In some embodiments, the processor 1201 is further to:

In some embodiments, the picture coding parameters include a coding frame rate; the processor 1201, when configured to determine picture coding parameters matching the second decoder according to the first decoding rate and the second decoding rate, is specifically configured to:

In specific implementation, the above-described devices, processors, memories, etc. may perform the implementation described in the above-described method embodiments, or may perform the implementation described in the embodiment of the present application, which is not described herein again.

Referring to fig. 13, fig. 13 is a schematic structural diagram of a data encoding apparatus according to an embodiment of the present application. As shown in fig. 13, the data encoding apparatus 1300 includes: at least one processor 1301, a memory 1302. Optionally, the data encoding device may further comprise a network interface. The processor 1301, the memory 1302, and the network interface may exchange data, the network interface is controlled by the processor 1301 to send and receive messages, the memory 1302 is used for storing a computer program, the computer program includes program instructions, and the processor 1301 is used for executing the program instructions stored in the memory 1302. Wherein the processor 1301 is configured to invoke the program instructions to perform the above method.

Wherein the memory 1302 may include volatile memory (RAM), such as random-access memory (RAM); the memory 1302 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a Solid State Drive (SSD), etc.; the memory 1302 may also include a combination of two or more of the above types of memory.

The processor 1301 may be a central processing unit (central processing unit, CPU). In one embodiment, processor 1301 may also be a graphics processor (Graphics Processing Unit, GPU). Processor 1301 may also be a combination of a CPU and GPU.

In one possible implementation, the memory 1302 is configured to store program instructions that the processor 1301 may call to perform the following steps:

acquiring an encoded first picture frame and transmitting the encoded first picture frame to data encoding equipment;

when receiving a decoding switching request sent by data encoding equipment, performing full-frame encoding processing on a switching picture frame to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when a decoding switching request is received;

And sending the coded switching picture frame to a data coding device.

In some embodiments, the processor 1301, when configured to perform full frame encoding processing on the switching picture frame, is specifically configured to:

Also provided in this embodiment is a computer (readable) storage medium storing a computer program, where the computer program includes program instructions, and when the program instructions are executed by a processor, the program instructions enable the processor to perform some or all of the steps performed in the method embodiments described above. The computer storage medium may be volatile or nonvolatile. The computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created from the use of blockchain nodes, and the like.

Embodiments of the present application also provide a computer program product comprising computer instructions (program instructions) which, when executed by a processor, implement some or all of the steps of the above-described picture data processing method. Alternatively, the computer instructions may be stored in a computer-readable storage medium, from which a processor of a computer device reads the program instructions, which the processor executes, causing the computer device to execute the above-provided picture data processing method. For example, the computer device may be a data encoding device or a data decoding device.

References herein to "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable devices. The computer instructions may be stored in or transmitted across a computer-readable storage medium.

The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or data storage devices, such as servers, data centers, etc., that contain an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

The foregoing is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present disclosure, and all changes and substitutions are intended to be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data processing, the method comprising:

receiving an encoded first picture frame, and calling a first decoder to decode the encoded first picture frame;

when the decoder switching condition is met, a decoding switching request is sent to the data encoding equipment; the decoding switching request is used for indicating the data encoding equipment to perform full-frame encoding processing on the switching picture frame so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when the data coding equipment receives the decoding switching request;

2. The method of claim 1, wherein if the first decoder is a hardware decoder, the second decoder is a software decoder; if the first decoder is a software decoder, the second decoder is a hardware decoder; the method further comprises the steps of:

if the first decoder is the hardware decoder and the hardware decoder is detected to be updated from an available state to an unavailable state, determining that the decoder switching condition is met;

if the first decoder is the software decoder and the hardware decoder is detected to be updated from the unavailable state to the available state, the decoder switching condition is determined to be met.

3. The method according to claim 2, wherein the method further comprises:

acquiring the equipment resource occupancy rate of the hardware encoder;

if the equipment resource occupancy rate is smaller than an occupancy rate threshold value, determining that the hardware encoder is in the available state;

and if the equipment resource occupancy rate is greater than or equal to the occupancy rate threshold value, determining that the hardware encoder is in the unavailable state.

4. The method according to claim 1, wherein the method further comprises:

Receiving the encoded second picture frame sent by the data encoding equipment; the second picture frame is a picture frame after the switching picture frame;

and calling the second decoder to decode the encoded second picture frame to obtain the second picture frame.

5. The method according to claim 1, wherein the method further comprises:

acquiring a first decoding rate of the first decoder and a second decoding rate of the second decoder;

determining picture coding parameters matched with the second decoder according to the first decoding rate and the second decoding rate; the decoding switching request carries the picture coding parameter, and the picture coding parameter is used for coding the switching picture frame.

6. The method of claim 5, wherein the picture coding parameters comprise a coding frame rate; the determining picture coding parameters matched to the second decoder according to the first decoding rate and the second decoding rate includes:

and adjusting the current coding frame rate according to the rate difference between the first decoding rate and the second decoding rate to obtain the coding frame rate matched with the second coder.

7. A method of data processing, the method comprising:

acquiring an encoded first picture frame and transmitting the encoded first picture frame to a data decoding device;

when receiving a decoding switching request sent by the data decoding equipment, performing full-frame coding processing on the switching picture frame to obtain a coded switching picture frame; the switching picture frame is a picture frame to be coded when the decoding switching request is received;

and transmitting the encoded switching picture frame to the data decoding device.

8. The method of claim 7, wherein the performing full frame encoding on the switching picture frame to obtain the encoded switching picture frame comprises:

acquiring picture coding parameters matched with a decoder to be switched indicated by the decoding switching request;

9. A data processing apparatus, the apparatus comprising:

A transmitting module, configured to transmit a decoding switching request to the data encoding apparatus when a decoder switching condition is satisfied; the decoding switching request is used for indicating the data encoding equipment to perform full-frame encoding processing on the switching picture frame so as to obtain an encoded switching picture frame; the switching picture frame is a picture frame to be coded when the data coding equipment receives the decoding switching request;

and the calling module is also used for calling a second decoder to decode the coded switching picture frame when the coded switching picture frame is received, so as to obtain the switching picture frame.

10. A data processing apparatus, the apparatus comprising:

the processing module is used for carrying out full-frame coding processing on the switching picture frame when receiving a decoding switching request sent by the data decoding equipment to obtain a coded switching picture frame; the switching picture frame is a picture frame to be coded when the decoding switching request is received;

the processing module is further configured to send the encoded switching picture frame to the data decoding device.

11. A data decoding device comprising a processor and a memory, wherein the memory is for storing a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-6.

12. A data encoding device comprising a processor and a memory, wherein the memory is for storing a computer program, the computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 7-8.

13. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of any of claims 1-8.

14. A computer program product comprising computer instructions which, when executed by a processor, implement the method of any of claims 1-8.