CN115348451A

CN115348451A - Encoding processing method, device, server and medium

Info

Publication number: CN115348451A
Application number: CN202211023783.6A
Authority: CN
Inventors: 陈思佳; 曹洪彬; 曹健; 黄永铖; 宋美佳; 杨小祥; 张佳
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-08-24
Filing date: 2022-08-24
Publication date: 2022-11-15

Abstract

The embodiment of the application discloses a coding processing method, a device, a server and a medium, wherein the method comprises the following steps: when the decoding abnormal condition existing in the terminal equipment is detected, according to the corresponding relation between different decoding abnormal conditions and the coding strategy, taking the coding strategy corresponding to the decoding abnormal condition as a recommended coding strategy; acquiring a reference decoding capacity which can be provided by a terminal device when the terminal device decodes the coded data generated by the recommended coding strategy, and acquiring an expected decoding capacity which the terminal device needs to have when the terminal device normally decodes the coded data generated by the recommended coding strategy; and matching the reference decoding capability and the expected decoding capability to obtain a matching result, and when the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, coding the game frame to be processed in the target cloud game by adopting a recommended coding strategy, so that the smoothness of decoding of the terminal equipment and the stable output of the picture can be ensured.

Description

Encoding processing method, device, server and medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to an encoding processing method, apparatus, server, and medium.

Background

With the development of video encoding and decoding technology, in some encoding and decoding scenarios (such as a scenario in which a cloud and a terminal cooperate), compressed video data can be transmitted from a video content producer (such as a server) to a video content consumer (such as a terminal device). In this process, the output of the picture is often unstable due to the decoding jam of the video content consumer, for example, the picture jam occurs. The traditional processing mode usually directly discards a frame which fails to be transmitted or is not ready to be decoded, and practice shows that the problem that terminal equipment is stuck in the decoding process cannot be effectively avoided by adopting the traditional frame processing mode, so that how to ensure the decoding fluency of the terminal equipment becomes a current research hotspot.

Disclosure of Invention

The embodiment of the application provides a coding processing method, a coding processing device, a server and a medium, which can effectively utilize the decoding capability provided by a terminal device to perform coding processing based on a coding strategy fully adaptive to the terminal device, can better avoid decoding abnormity, and ensure the smoothness of decoding of the terminal device and the stable output of pictures.

In one aspect, an embodiment of the present application provides an encoding processing method, including:

when the decoding abnormal condition existing in the terminal equipment is detected, taking the coding strategy corresponding to the decoding abnormal condition as a recommended coding strategy according to the corresponding relation between different decoding abnormal conditions and the coding strategy; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different;

acquiring reference decoding capacity which can be provided by the terminal equipment when the terminal equipment performs decoding processing on the coded data generated by the recommended coding strategy, and acquiring expected decoding capacity which the terminal equipment needs to have when the terminal equipment performs normal decoding processing on the coded data generated by the recommended coding strategy;

and matching the reference decoding capability and the expected decoding capability to obtain a matching result, and when the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, adopting a recommended coding strategy to code the game frame to be processed in the target cloud game.

In one aspect, an embodiment of the present application provides an encoding processing apparatus, including:

the determining module is used for taking the coding strategy corresponding to the decoding abnormal condition as a recommended coding strategy according to the corresponding relation between different decoding abnormal conditions and the coding strategy when the decoding abnormal condition existing in the terminal equipment is detected; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different;

the acquisition module is used for acquiring the available reference decoding capability when the terminal equipment decodes the coded data generated by the recommended coding strategy and acquiring the expected decoding capability required by the terminal equipment when the terminal equipment normally decodes the coded data generated by the recommended coding strategy;

the matching module is used for matching the reference decoding capability and the expected decoding capability to obtain a matching result;

and the encoding module is used for adopting a recommended encoding strategy to encode the game frames to be processed in the target cloud game when the matching result indicates that the reference decoding capability is successfully matched with the expected decoding capability.

Accordingly, an embodiment of the present application provides a server, including: a processor, memory, and a network interface; the processor is connected with the memory and the network interface, wherein the network interface is used for providing a network communication function, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the coding processing method in the embodiment of the application.

Accordingly, embodiments of the present application provide a computer-readable storage medium, where a computer program is stored, where the computer program includes program instructions, and when the program instructions are executed by a processor, the program instructions execute a coding processing method in embodiments of the present application.

Accordingly, the present application provides a computer program product, which includes a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the encoding processing method of the present application is implemented.

In the embodiment of the application, a decoding abnormal situation of the terminal device can be detected, an encoding strategy corresponding to the detected decoding abnormal situation of the terminal device can be used as a recommended encoding strategy through corresponding relations between different decoding abnormal situations and encoding strategies, so that the corresponding encoding strategy can be adaptively recommended for the decoding abnormal situation, then a reference decoding capability provided by the terminal device for decoding encoded data generated by using the recommended encoding strategy can be obtained, an expected decoding capability required to be provided when the terminal device normally decodes the encoded data generated by using the recommended encoding strategy is obtained, whether the recommended encoding strategy can be adopted is determined based on the obtained reference decoding capability and the expected decoding capability, specifically, the judgment is carried out by matching the reference decoding capability and the expected decoding capability, when the matching result indicates that the reference decoding capability and the expected decoding capability are matched, the terminal device supports normal decoding of the encoded data generated by using the recommended encoding strategy, the decoding abnormal situation is overcome, the recommended encoding strategy is fully adapted to the decoding capability provided by the terminal device, the game frame to be processed of the target cloud game is coded by using the encoding strategy, the decoding capability of the terminal device can be fully utilized, the encoded data can be well, and the normal decoding of the terminal device can be well, so that the decoded picture can be stably output.

Drawings

FIG. 1 is an architecture diagram of an encoding processing system provided by an exemplary embodiment of the present application;

fig. 2 is a schematic flowchart of an encoding processing method according to an exemplary embodiment of the present application;

FIG. 3 is a flow chart illustrating another encoding processing method provided by an exemplary embodiment of the present application;

fig. 4a is a schematic diagram illustrating a network probe data transmission within a time window according to an exemplary embodiment of the present application;

fig. 4b is a schematic diagram of a scenario where a terminal device is overloaded according to an exemplary embodiment of the present application;

FIG. 5a is a schematic diagram illustrating an effect of masking a reference frame according to an exemplary embodiment of the present application;

FIG. 5b is a schematic illustration of a flow chart of an encoding process provided by an exemplary embodiment of the present application;

fig. 6 is a schematic structural diagram of an encoding processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

The application provides an encoding processing method, a server as a content (such as video content) producer can encode the game frame to be processed of a target cloud game, an abnormal decoding condition of a terminal device (a consumer of content (such as video content)) can be detected, and after the abnormal decoding condition existing in the terminal device is detected, obtaining a corresponding recommended coding strategy according to the corresponding relationship between different decoding abnormal conditions and the coding strategy, and after obtaining the recommended coding strategy, the server can normally decode the coding data generated by the recommended coding strategy according to the requirement, the terminal equipment needs to have the expected decoding capability, and the terminal device matches the decoding capability of the reference decoding capability that can be provided when decoding the encoded data generated by using the recommended encoding strategy, so that the computer device can determine whether the terminal device can normally decode the encoded data obtained by using the recommended encoding strategy based on the matching result, when the matching result indicates that the desired decoding capability matches the reference decoding capability available from the terminal device, the terminal device is stated to support normal decoding of encoded data generated using the recommended encoding strategy, i.e. the recommended encoding strategy is stated to be sufficiently adapted to the decoding capabilities provided by the terminal device, which, in this case, the server can also adopt the recommended coding strategy to code the game frame to be processed of the target cloud game, the decoding capability of the terminal equipment can be fully utilized, the terminal equipment can normally decode the received coded data, and the smoothness of decoding and stable output of pictures of the terminal equipment are ensured. Under the condition that the matching result indicates that the expected decoding capability is not matched with the reference decoding capability provided by the terminal equipment, the server can perform coding processing on the game frame to be processed of the target cloud game by selecting a new coding reference frame and adopting the new coding reference frame as an immediate refresh frame of the game frame to be processed of the target cloud game, so that the server can realize effective evaluation of the decoding capability of the terminal equipment, realize self-adaptive flexible adjustment of the coding processing mode of the game frame to be processed of the target cloud game, and improve the flexibility of the server in the coding process.

In an embodiment, when the server may be a cloud server, then, the efficient encoding process for the cloud game is implemented based on the reference of the cloud server to the decoding capability of the terminal device, that is, the end cloud cooperative encoding process for the cloud game between the cloud server and the terminal device, where the cloud game refers to an online game technology based on a cloud computing technology. The Cloud computing technology belongs to a Cloud technology, and the Cloud technology refers to a hosting technology for unifying series resources such as hardware, software, networks and the like in a wide area network or a local area network to realize the computation, storage, processing and sharing of data. Among them, cloud computing (cloud computing) is a computing model that distributes computing tasks over a resource pool formed by a large number of computers, so that various application systems can obtain computing power, storage space, and information services as needed. The network that provides the resources is called the "cloud". Resources in the "cloud" appear to the user as if they are infinitely expandable and can be acquired at any time, used on demand, expanded at any time, and paid for use. The cloud computing resource pool mainly comprises computing equipment (which is a virtualization machine and comprises an operating system), storage equipment and network equipment, wherein the cloud server has powerful computing resources and has the characteristics of high virtualization and high distribution, and the computing capacity is relatively strong.

The encoding processing method provided in the embodiment of the present application may be applied to an encoding processing system shown in fig. 1, where the encoding processing system is an encoding processing system formed by the terminal device and a cloud server, as shown in fig. 1, the terminal device may be any one marked by 11 in fig. 1, and the cloud server (or servers) may be any one or more marked by 12 in a server cluster shown in fig. 1. In an embodiment, when the decoder included in the terminal device 11 is different and the corresponding decoding capability is different, for example, when the decoder included is a decoder supporting H264 (a Video Coding standard) or the decoder included is a decoder supporting HEVC (High Efficiency Video Coding), the Video codec standards adopted by the corresponding decoders (i.e., decoding chips) are different, so that the decoding capability of the terminal device 11 is different, that is, the server 12 may determine the decoding capability that can be provided by the terminal device 11 based on the type of the decoder included in the terminal device 11.

In addition to the influence of the type including the decoder on the decoding capability of the terminal device 11, the decoding capability of the terminal device 11 may be limited due to poor network conditions (e.g., network fluctuation) between the terminal device 11 and the server 12, and in this case, not only the decoding capability of the terminal device is poor, but also abnormal decoding of the terminal device is caused, such as packet loss and jamming of the terminal device. Based on the connection relationship between the terminal device 11 and the server 12, the server 12 can detect the abnormal decoding condition of the terminal device in time, and when the abnormal decoding condition occurs, the encoding policy of the local terminal is adjusted through the decoding capability actually provided by the terminal device 11 to ensure that the terminal device 11 performs normal decoding on the encoded data issued by the server 12, so that the server 12 can effectively avoid the abnormal decoding condition of the terminal device 11 due to the fluctuation of the network, and further ensure the fluency of the presentation of the corresponding picture data at the terminal device side. It is understood that the terminal device 11 includes, but is not limited to: the smart phone, the tablet computer, the wearable device of intelligence, intelligent voice interaction device, intelligent household electrical appliances, personal computer, vehicle mounted terminal and so on, this application does not limit this. The number of the terminal devices is not limited in the present application. The server 12 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a web service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content delivery Network), and a big data and artificial intelligence platform, but is not limited thereto.

Please refer to fig. 2, which illustrates an encoding processing method according to an exemplary embodiment of the present application. The method may be performed by the server described above. The encoding processing method includes the following steps S201 to S203:

s201, when the decoding abnormal condition existing in the terminal equipment is detected, according to the corresponding relation between different decoding abnormal conditions and the coding strategy, the coding strategy corresponding to the decoding abnormal condition is taken as a recommended coding strategy.

The abnormal decoding condition refers to a condition that an abnormality occurs in the process of decoding the encoded data by the terminal device, and can be used for reflecting the actual decoding condition of the terminal device, for example, the terminal device is jammed in decoding. The decoding abnormal condition existing in the terminal device may be caused by unstable network transmission or insufficient hardware capability of the terminal device, that is, the decoding abnormal condition may occur due to network fluctuation or fluctuation of real-time decoding capability of the terminal device, so that different decoding abnormal conditions may include an abnormal condition of network fluctuation and an abnormal condition of overload (that is, an excessive amount of encoded data waiting for decoding by the terminal device). The decoding abnormal condition existing in the terminal device is an abnormal condition of network fluctuation or an abnormal condition of overload, and is usually represented as picture pause presented by the terminal device. In one implementation, the abnormal condition of the network fluctuation includes any one of a frequent fluctuation abnormal condition (i.e., an abnormal condition in which the network fluctuates frequently) and a sporadic fluctuation abnormal condition (i.e., an abnormal condition in which the network fluctuates instantaneously), and the abnormal condition of the overload includes any one of a frequent overload and a sporadic overload. At this time, the decoding abnormal condition may be a frequent fluctuation abnormal condition or an occasional fluctuation abnormal condition, and may also be a frequent overload condition or an occasional overload condition.

The correspondence between different decoding abnormal situations and encoding strategies is used to indicate an encoding strategy to which one decoding abnormal situation is adapted. One decoding abnormal condition can correspond to one coding strategy, different decoding abnormal conditions can correspond to different coding strategies, and further, for different decoding abnormal conditions, recommended coding strategies determined based on the corresponding relation between the decoding abnormal conditions and the coding strategies are different. For example, the encoding policy includes an encoding policy a and an encoding policy B, the encoding policy a may be corresponding to the decoding abnormal condition when the decoding abnormal condition is accidental overload, and the encoding policy a may be used as a recommended encoding policy, the encoding policy B may be corresponding to the decoding abnormal condition when the decoding abnormal condition is frequent fluctuation abnormal, and the encoding policy B may be used as the recommended encoding policy.

Different decoding anomalies may also correspond to the same recommended encoding strategy. For example, if the decoding abnormal condition is accidental overload or accidental fluctuation abnormal, the corresponding encoding strategy A can be adopted as the recommended encoding strategy. In addition, different abnormal situations of network fluctuation (including frequent network fluctuation and occasional network fluctuation), or different abnormal situations of overload (including frequent overload and occasional overload) can also correspond to different recommended coding strategies. For example, if it is detected that the decoding abnormal condition existing in the terminal device is accidental overload, the determined encoding policy a may be used as a recommended encoding policy, and if the decoding abnormal condition existing in the terminal device is frequent overload, the determined encoding policy B may be used as a recommended encoding policy.

The encoding strategy comprises at least two types, the encoding strategy is used for indicating a mode of encoding processing on the data to be encoded, and the encoding reference frame can be managed through the encoding strategy, specifically, the dependency relationship between the encoding reference frame and the frame to be encoded is managed. The encoding reference frame refers to a frame to which the to-be-encoded data refers, and usually, a frame to which the to-be-encoded data refers can also be used as a decoding reference frame after being decoded in a decoding stage, and the dependency relationship is used for indicating which encoding reference frames the to-be-encoded frame can refer to for encoding. In one implementation, the encoding strategy may be a layered encoding strategy or a frame skipping encoding strategy, and the decoding abnormal condition may correspond to any one of the layered encoding strategy and the frame skipping encoding strategy. The layered coding strategy is a strategy of coding by adopting a layered coding mode, the layered coding mode can be divided into an enhancement layer and a base layer and supports the discarding of frames of the enhancement layer, the frame skipping coding strategy is a strategy of coding by shielding a specific reference frame or a reference frame appointed for reference, the shielded specific reference frame or the reference frame appointed for reference can be flexibly set based on actual conditions, and the reference frames mentioned in the two coding strategies are coding reference frames.

When the server detects the decoding abnormal condition existing in the terminal equipment, the encoding strategy corresponding to the detected decoding abnormal condition can be selected from at least two encoding strategies based on the preset corresponding relation between the decoding abnormal condition and the encoding strategy, and then the encoding strategy corresponding to the detected decoding abnormal condition is used as the recommended encoding strategy. In this way, an encoding strategy adapted to the detected decoding abnormal situation is adaptively recommended based on the correspondence between the decoding abnormal situation and the encoding strategy. The server can recommend corresponding encoding strategies according to decoding abnormal conditions existing in different terminal devices, and reference frames required by the frames to be encoded can be managed through the encoding strategies, for example, partial reference frames are discarded, so that the discarded frames are not required to be referred to in the encoding and decoding processes. If a decoding abnormity problem occurs in a reference frame during decoding, the reference frame is discarded to realize normal decoding, the decoding abnormity condition can be relieved, and the probability of the decoding abnormity can be reduced.

S202, acquiring the reference decoding capacity which can be provided by the terminal equipment when the terminal equipment performs decoding processing on the coded data generated by the recommended coding strategy, and acquiring the expected decoding capacity which the terminal equipment needs to have when the terminal equipment performs normal decoding processing on the coded data generated by the recommended coding strategy.

Before detecting the decoding abnormal condition of the terminal device, the server can detect the terminal device to obtain the reference decoding capability provided by the terminal device, and obtain the reference decoding capability from the database of the server when the server needs to be used. The reference decoding capability is used to reflect an actual decoding capability of the terminal device at the time of decoding processing of the encoded data generated by the recommended encoding policy. The reference decoding capability may include any one or more of a static decoding capability and a dynamic decoding capability, where the static decoding capability is detected in a static manner, for example, the static decoding capability may be obtained by sending query information to a hardware interface (specifically, a communication structure of a decoder) of the terminal device, for example, a chip type supported by the terminal device, where the chip type is divided according to a supported video codec standard, for example, an H264 chip, an HEVC chip, whether the chip of the terminal device supports layered codec, a threshold value of a number of reference frames used by the chip of the terminal device for decoding one frame, and the like. The dynamic decoding capability is obtained by detecting in a dynamic manner, for example, sending code streams generated by different encoding strategies, and determining whether the terminal device can decode normally (i.e., whether decoding fails or is stuck). The dynamic decoding capability and the static decoding capability are determined by the hardware device (such as a decoder) of the terminal device, and are not usually affected by other factors (such as network fluctuation) to change.

The server can use the recommended coding strategy to code the data to be coded to obtain coded data, and when the terminal equipment normally decodes the coded data generated by the recommended coding strategy, the abnormal decoding condition can be overcome. If the terminal device is required to normally decode the encoded data generated by the recommended encoding strategy (that is, the terminal device does not have a jam or an error prompt during decoding), the terminal device is required to have a decoding capability capable of supporting normal decoding, that is, a decoding capability is required. Therefore, when the terminal device is enabled to perform normal decoding processing on the encoded data generated by the recommended encoding strategy, the terminal device can obtain the expected decoding capability required by the terminal device, wherein the expected decoding capability is the capability of the terminal device to perform normal decoding on the encoded data generated based on the recommended encoding strategy so as to overcome the abnormal decoding condition. The expected decoding capability can be reflected by the number of reference frames supported by the terminal device for decoding one frame, and also can be reflected by the decoding condition of the encoded data generated by one encoding strategy of the decoding supported by the terminal device. Whether the recommended coding strategy is available can be further determined by a matching result between the reference decoding capability actually possessed by the terminal device and the expected decoding capability possessed by the expected terminal device.

And S203, matching the reference decoding capability and the expected decoding capability to obtain a matching result, and when the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, adopting a recommended coding strategy to code the game frame to be processed in the target cloud game.

The server may perform matching processing on the reference decoding capability and the expected decoding capability, specifically, the information included in the reference decoding capability and the expected decoding capability (for example, the maximum number of reference frames supported by actually decoding one frame and the maximum number of reference frames reached by expecting to decode one frame) is compared, and a matching result obtained by matching between the reference decoding capability and the expected decoding capability may indicate whether the two decoding capabilities are successfully matched, that is, whether the decoding capability supported by the terminal device reaches the decoding capability expected by the server for the terminal device, and different processing may be performed according to different contents indicated by the matching result.

When the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, the decoding capability provided by the terminal device meets the requirement of the expected normal decoding capability, and the server can adopt a recommended encoding strategy to encode the data to be encoded, wherein the data to be encoded is a game frame to be processed in a target cloud game, the target cloud game is any one of one or more cloud games operated in the server, and the game frame to be processed is a game picture to be encoded in the target cloud game. Whether the server uses the recommended coding strategy is judged by comparing the reference decoding capacity provided by the terminal equipment with the expected decoding capacity required by the terminal equipment, when the recommended coding strategy is determined to be adopted to code the game frame to be processed, and the coded game frame is sent to the terminal equipment, the terminal equipment can normally decode the game frame processed based on the recommended coding strategy without abnormal decoding, and the game picture of the cloud game is stably and smoothly output in the terminal equipment.

In one implementation, when the matching result indicates that the reference decoding capability is unsuccessfully matched with the expected decoding capability, it indicates that the decoding capability provided by the terminal device does not meet the requirement of the decoding capability expected to be normally decoded, and at this time, the server may use another encoding strategy to encode the data to be encoded. If the matching result indicates that the target cloud game frame and the target cloud game frame are not matched, the new coding reference frame is selected to code the game frame to be processed of the target cloud game, and therefore the recommended coding strategy can be adaptively adjusted based on the decoding capability of the terminal device, and the flexibility of use of the coding strategy is high.

It should be noted that, for different terminal devices, the encoding processing method provided by the present application may be adopted, whether the recommended encoding policy is supported or not may be intelligently determined based on the decoding capability of the terminal device itself, and for a terminal device that supports the recommended encoding policy, the recommended encoding policy may be used to perform reference frame management, and the original management manner of encoding reference frames is modified to perform encoding processing. Even if the decoding capabilities of different terminal devices are different, the original management mode for the coding reference frame can be adaptively modified for coding according to the decoding capabilities of the terminal devices and by combining with the decoding abnormal conditions, so that the different decoding abnormal conditions and the decoding capabilities of the terminal devices are comprehensively considered, and the decoding abnormal conditions can be effectively avoided.

According to the encoding processing method provided by the embodiment of the application, the server can detect the decoding abnormal condition of the terminal equipment, the encoding strategy corresponding to the detected decoding abnormal condition of the terminal equipment can be used as the recommended encoding strategy through the corresponding relation between different decoding abnormal conditions and the encoding strategy, the recommended encoding strategies corresponding to different decoding abnormal conditions are different, therefore, the encoding strategy can be adaptively recommended for the decoding abnormal condition, the reference decoding capability provided by the terminal equipment when the terminal equipment decodes the encoded data generated by using the recommended encoding strategy can be obtained, the expected decoding capability provided when the terminal equipment normally decodes the encoded data generated by using the recommended encoding strategy can be obtained, whether the recommended encoding strategy can be adopted or not can be further determined based on the reference decoding capability and the expected decoding capability, specifically, the difference between the two decoding capabilities can be compared through matching the reference decoding capability and the expected decoding capability to obtain the matching result, when the matching result indicates that the terminal equipment matches with the reference decoding capability and the expected decoding capability, the terminal equipment supports normal decoding of the encoded data generated by using the recommended encoding strategy, and the encoded data can be stably output by the game terminal equipment.

Referring to fig. 3, fig. 3 is a coding processing method according to an exemplary embodiment of the present application. The method may be performed by a server. The encoding processing method includes the following steps S301 to S305:

s301, when the decoding abnormal condition existing in the terminal equipment is detected, according to the corresponding relation between different decoding abnormal conditions and the coding strategy, the coding strategy corresponding to the decoding abnormal condition is taken as a recommended coding strategy.

In one embodiment, the determination manner of the decoding abnormal condition existing in the terminal device may be as described in the following (1) to (3):

(1) And sending the network detection data to the terminal equipment within the time window. Specifically, the server may send network probe data to the terminal device within a time window, where the network probe data may be encoded data of the target cloud game, and specifically may be one or more encoded game frames; the network probe data may also be encoded data unrelated to the target cloud game, such as any encoded video frame. The time window may be a sliding window of time length T. The time window comprises a plurality of time points, and one or more network detection data are sent to the terminal equipment at one time point. The continuous detection can be performed by sending network detection data at different time points in a time window, and the time points can be periodic time points or non-periodic time points, namely corresponding to periodic detection or non-periodic detection. For example, a frame of encoded game frames may be transmitted to the terminal device every 100 milliseconds (ms) during a time window having a duration of 1 second, such that 10 detections may be made during the time window, as shown in fig. 4 a. By detecting in the time window, the information about the network detection data can be collected in time window units, and the detection of whether the terminal equipment has the decoding abnormal condition in the time window is realized.

(2) And acquiring detection feedback data corresponding to the network detection data sent at each time point of the time window, and generating a network detection result at the corresponding time point according to the detection feedback data.

The server sends the network detection data to the terminal device at each time point in the time window, and can obtain detection feedback data corresponding to the network detection data at each time point, wherein the detection feedback data can be used for feeding back the transmission condition of the network detection data. The detection feedback data corresponding to a time point can generate a network detection result corresponding to the time point. The network detection result corresponding to any time point is used for indicating whether network fluctuation occurs at any time point.

In one implementation, taking any time point as a target time point as an example, generating the network probing result of the target time point according to the probing feedback data of the target time point may be: comparing the detection feedback data of the target time point with a preset fluctuation threshold value, and generating a network detection result for indicating that network fluctuation occurs at the target time point when the detection feedback data of the target time point reaches the preset fluctuation threshold value; and when the detection feedback data of the target time point does not reach the preset fluctuation threshold value, generating a network detection result for indicating that the network fluctuation does not occur at the target time point. Optionally, the detection feedback data includes any one or more of packet loss times and network transmission delay, where the packet loss times refer to times lost when the network detection data is transmitted to the terminal device, and the network transmission delay refers to time delayed when the network detection data is transmitted to the terminal device through the network, for example, the packet loss times are 3, and the network transmission delay is 200 milliseconds. The preset fluctuation threshold includes a packet loss frequency threshold and a transmission delay threshold, and the detection feedback data reaching the preset fluctuation threshold refers to: and the packet loss times in the detection feedback data are more than or equal to a packet loss time threshold, and the network transmission delay is more than or equal to one or two of a transmission delay threshold. When the detection feedback data of the target time point reaches a preset fluctuation threshold value, the server can determine that the current network fluctuates at the target time point, and the generated detection result is used for indicating that the network fluctuation occurs at the target time point. Otherwise, determining that the current network does not fluctuate at the target time point, and using the generated detection result to indicate that the network fluctuates at the target time point.

The network fluctuation occurring at any time point can be in a long-term unstable scene of the network and can also be in a transient network fluctuation scene. Therefore, the network detection results of all time points in the time window can be integrated to judge the network fluctuation in which scene occurs, and further the decoding abnormal condition of the terminal equipment is determined.

(3) And counting the times of network fluctuation in the time window according to the network detection results respectively corresponding to each time point of the time window, and determining the decoding abnormal condition existing in the terminal equipment according to the counted times.

And one time point in the time window corresponds to one network detection result, the network detection result is used for indicating whether network fluctuation occurs at the corresponding time point, and one network fluctuation corresponds to one time point. Network fluctuations may occur at multiple points in time within the time window, or may not occur at various points in time. And counting the times of network fluctuation in the time window according to the network detection result, wherein the counted times of the network fluctuation can be used for determining the decoding abnormal condition existing in the terminal equipment.

In one implementation, determining the decoding abnormal condition existing in the terminal device according to the counted times includes: acquiring a fluctuation index value; when the counted times are more than or equal to the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent fluctuation abnormality; and when the counted times are more than or equal to 1 and less than the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental fluctuation abnormality.

The fluctuation index value is a preset threshold value for judging whether the network fluctuation is frequent or not. The counted number of times is the counted number of times of network fluctuation, and when the counted number of times is greater than or equal to the fluctuation index value, it indicates that the number of times of network fluctuation in the time window exceeds a preset fluctuation threshold value, so that it can be determined that the decoding abnormal condition existing in the terminal device is a frequent fluctuation abnormal condition, i.e., the abnormal condition of network fluctuation is frequent, for example, the fluctuation index value is 2, the counted number of times of network fluctuation in the time window T1 is 3, and the number of times of network fluctuation is greater than the fluctuation index value, so that it can be determined that the current network fluctuation is frequent. Further, it can be known that the network environment where the terminal device is located may be an environment where the network is unstable for a long time. On the contrary, when the counted times is more than or equal to 1 and less than the fluctuation index value, it is indicated that network fluctuation occurs in the time window, but the times of the network fluctuation does not exceed the preset fluctuation time threshold value, and further, the decoding abnormal condition existing in the terminal equipment can be determined as accidental fluctuation abnormality, namely the abnormal condition of the network fluctuation which happens occasionally. When occasional fluctuation is abnormal, the network environment in which the terminal device is located may be an environment in which instantaneous network fluctuation or jamming occurs. The occasional fluctuation anomaly and the frequent fluctuation anomaly both belong to decoding anomaly conditions, and the two decoding anomaly conditions can be collectively referred to as network fluctuation anomaly conditions. When the decoding abnormal condition existing in the terminal equipment comprises an abnormal condition of network fluctuation, the abnormal condition is usually referred to as occasional fluctuation abnormality or frequent fluctuation abnormality.

It can be understood that when the counted number is equal to 0, it indicates that no network fluctuation occurs in the time window, and it can be determined that the terminal device does not have a decoding abnormal condition. Further, under the condition that network fluctuation does not occur in a time window, the time window can be updated, the network environment where the terminal device is located can be continuously detected, and when the decoding abnormal condition of the terminal device is detected, the current decoding abnormal condition is overcome and the subsequent decoding abnormal condition is avoided based on the scheme. Therefore, whether the terminal equipment has abnormal decoding conditions or not can be followed up in the whole encoding process, and encoding processing is carried out by adopting a corresponding encoding strategy, so that the terminal equipment can decode smoothly, and stable output of pictures is realized.

In another embodiment, the determination manner of the decoding abnormal condition existing in the terminal device may also include the following: (1) And acquiring load information of the terminal equipment at different time points of the time window. (2) And determining the overload times of the terminal equipment in the time window according to the load information respectively corresponding to the terminal equipment at different time points of the time window. (3) And determining the decoding abnormal condition of the terminal equipment according to the overload times.

The different time points of the time window are the time points of the server for detecting the terminal device, the time window comprises at least two time points, and the different time points can be the time points of periodic detection or the time points of non-periodic detection. In one approach, the respective time points may be historical time points prior to a time point of encoding processing one frame. For example, within a time window of 1s, the current frame is the 4 th frame encoded at 400ms, then the respective time points may be the 100 th, 200 th and 300 th ms. Therefore, the decoding load condition of the terminal equipment before encoding a frame can be acquired in real time, and a reference is provided for the use of the encoding strategy. It is to be understood that, for the time point of the transmission of the network probe data, a historical time point before encoding one frame may be used. By acquiring the network fluctuation situation (possibly network accidental fluctuation or network frequent fluctuation) before one frame is coded and adapting to different network fluctuation situations, a corresponding coding strategy is provided so as to intelligently manage the reference frame.

The server can obtain load information of the terminal device at each time point in the time window, wherein the load information is the data volume to be decoded of the terminal device at the corresponding time point. The data amount to be decoded refers to the data amount of encoded data waiting for decoding by the terminal device, for example, the number of encoded frames waiting for decoding by the terminal device. One time point corresponds to one load information and also corresponds to one detection of the terminal equipment by the server.

And when the data volume to be decoded of the terminal equipment at any time point is greater than or equal to the data volume threshold value, the terminal equipment is overloaded once at any time point. If the data volume to be decoded is the number of the encoded frames waiting for the terminal device to decode, the threshold value of the data volume is the threshold value of the number of the frames waiting for the terminal device to decode. When the number of the coded frames waiting for the terminal device to decode is greater than or equal to the threshold value, it indicates that the terminal device has a temporary decoding load that is too high, i.e. an overload occurs. It will be appreciated that when the amount of data to be decoded by the terminal device at any point in time is less than the data amount threshold, the terminal device is not overloaded at any point in time.

The overload of the terminal device may be caused by unstable network or by the limitation of the hardware capability of the terminal device itself, and the real-time decoding capability of the terminal device may fluctuate due to unstable network transmission or the influence of the hardware device of the terminal device, resulting in too high decoding load and overload, which appears as decoding stuck and picture stuck in the terminal device. For example, because the network card fails to transmit the encoded 2 nd frame to the terminal device, the server may try to retransmit the 2 nd frame a plurality of times, and during the retry process of the server, the 3 rd frame, which is arranged after the 2 nd frame in the encoding sequence, is also encoded and waits to be transmitted to the terminal device, and after the network is stabilized at a certain time, the terminal device may receive the encoded data of the 2 nd frame and the 3 rd frame at the same time, so that overload occurs, as shown in fig. 4 b. In addition, the decoding speed of the terminal device may not be equal to the encoding speed of the server, and the encoded data sent by the server to the terminal device in a unit time may not be decoded in time in the unit time due to the limitation of the computing capacity of the hardware, thereby causing overload.

And judging and determining whether each time point is overloaded according to the above mode for the load information corresponding to each time point in the time window, and finally obtaining the overload times of the terminal equipment in the time window. For example, the terminal device is detected at 10 time points within a time window of 1 second, to obtain data volumes to be decoded at 10 time points, the data volumes to be decoded and the data volume threshold are respectively compared, to obtain that the terminal device is overloaded at the first time point, the 3 rd time point and the 5 th time point of the time window, and the remaining time is not overloaded, that is, it is determined that the terminal device is overloaded 3 times within the time window.

An overload index value may be set to determine the decoding anomaly present at the terminal device by determining whether the number of overloads within the time window reaches the overload index value. In one implementation manner, when the number of times of overload reaches the overload index value, it is determined that the decoding abnormal condition existing in the terminal device is frequent overload, and when the number of times of overload is greater than 1 and does not reach the overload index value, it is determined that the decoding abnormal condition existing in the terminal device is accidental overload.

When the overload times reach the overload index value, namely the overload times are greater than or equal to the overload index value, the situation that the decoding load of the terminal equipment is too high for multiple times in a time window is described, and the decoding abnormal situation existing in the terminal equipment can be determined to be frequent overload, and the current decoding pressure of the terminal equipment is too high; when the overload times do not reach the overload index value, namely the overload times are smaller than the overload index value and larger than 1, the terminal equipment is overloaded (for example, the overload is only carried out once) in the time window, but the decoding load is too high occasionally, the decoding abnormal condition existing in the terminal equipment can be determined to be the accidental overload, and the decoding pressure of the terminal equipment is accidentally too high. It can be understood that, when the number of times of overload is equal to 0, it is determined that the terminal device does not have a decoding abnormal condition, i.e., the terminal device does not have an overload condition. The above frequent overload and accidental overload both belong to decoding abnormal situations, and these two decoding abnormal situations can be collectively referred to as overload abnormal situations, and when the decoding abnormal situation existing in the terminal device includes overload abnormal situations, it is usually referred to as frequent overload or accidental overload.

It can be understood that the above described detection method for detecting the decoding abnormal condition existing in the terminal device can also be applied in the terminal device to detect the decoding load condition and the network fluctuation condition of the terminal device, so as to obtain the decoding abnormal condition existing in the terminal device. The decoding abnormal condition of the terminal device may be directly sent to the server by the terminal device, or the terminal device may periodically send related information (such as load information) to the server, and the server determines the decoding abnormal condition of the terminal device, for example, when the terminal device detects that the load information exceeds a load threshold, it may be determined that the terminal device is overloaded, and the counted number of times of overload is sent to the server after a time window is over, or an information indicating that the terminal device is overloaded is sent to the server when an overload occurs within a time window, and the server counts the number of times of overload occurring within the time window.

In one implementation, the encoding strategy includes a layered encoding strategy and a frame skipping encoding strategy; when the decoding abnormal condition is the abnormal condition of network fluctuation, the corresponding recommended coding strategy comprises the layered coding strategy; and when the decoding abnormal condition is an overload abnormal condition, the corresponding recommended coding strategy comprises a frame skipping coding strategy.

Specifically, when the decoding abnormal condition is an abnormal condition of network fluctuation, the decoding abnormal condition may be a frequent fluctuation abnormal condition or an occasional fluctuation abnormal condition, and when the decoding abnormal condition is an abnormal condition of overload, the decoding abnormal condition may be a frequent overload condition or an occasional overload condition. The correspondence between different decoding abnormal situations and encoding strategies can be set as follows: the coding strategy corresponding to the decoding abnormal condition being frequent fluctuation abnormality is a layered coding strategy, and the coding strategy corresponding to the decoding abnormal condition being accidental overload is a frame skipping coding strategy; the coding strategy corresponding to the decoding abnormal condition being accidental fluctuation abnormal is a frame skipping coding strategy, and the coding strategy corresponding to the decoding abnormal condition being frequent overload is a layered coding strategy.

Therefore, the encoding strategies corresponding to different decoding abnormal conditions are different, so that when the terminal equipment has the decoding abnormal conditions, the corresponding encoding strategy can be used as the recommended encoding strategy in a targeted manner based on the corresponding relation, and the different decoding abnormal conditions are overcome. For example, when the decoding abnormal condition is a frequent fluctuation abnormal condition included in an abnormal condition of network fluctuation, the corresponding recommended coding strategy includes a layered coding strategy, and when the decoding abnormal condition is an accidental overload included in an abnormal condition of overload, the corresponding recommended coding strategy is a frame skipping coding strategy.

The hierarchical coding strategy refers to a strategy for coding in a hierarchical coding mode. Scalable Video Coding (SVC) is a Video Coding technique that codes a Video signal into layers, and transmits and decodes only a code stream of a base layer when a bandwidth is insufficient, but the decoded Video quality is not high. When the bandwidth becomes slowly larger, the code stream of the enhancement layer can be transmitted and decoded to improve the decoding quality of the video. The video frame positioned in the base layer is a video frame which cannot be discarded in the layered coding and is used as a reference of a subsequent video frame, and if the video frame is discarded, decoding failure or screen splash can be caused; the video frame in the enhancement layer is a video frame that can be dropped at will in the layered coding, and is not used as a reference for the subsequent video frame, if the dropping has no effect on the decoder supporting the SVC. When the layered coding strategy is adopted, the video frames of the base layer and the video frames of the enhancement layer can be transmitted in channels, so that the network transmission pressure is reduced.

The frame skipping coding strategy is a strategy for coding by adopting a frame skipping coding mode. Frame skipping coding refers to: before a certain frame is coded, the reference frame dependency relationship of the current frame is temporarily modified in a mode of specifying a specific reference frame or shielding the specific reference frame. For example, the current frame is defaulted to refer to the previous 2 frames, and the flexible frame skipping can modify the current frame to refer to the previous third frame and the previous second frame, or mask the previous 1 frame of the current frame, so that the current frame refers to the previous third frame and the previous second frame. The different coding strategies can embody different management modes for the reference frame, and specifically correspond to different frame loss schemes.

In the unstable scene of the network, the configuration of layered coding can be started, the coding is carried out by using a layered coding strategy, and partial reference frames positioned in an enhancement layer can be discarded in the transmission process, so that the pressure of network transmission is reduced. Because the starting of the layered coding can cause the reduction of the coding quality during the starting period, and the starting of the layered coding can cause the reduction of the coding quality of redundant time for instant network fluctuation or accidental overload, namely, the use of the layered coding strategy is more suitable for the scenes that the network is unstable for a long time or the overload is frequent, therefore, the coding strategy corresponding to the frequent fluctuation abnormity or the frequent overload can be set as the layered coding strategy, and the accidental decoding load is overhigh or the instant network fluctuation can discard part of the reference frames through flexible frame skipping coding. Under the condition of coping with accidental decoding blockage, only instantaneous coding quality is slightly reduced when frame skipping coding is used, and coding quality at other time is not influenced, so that the overall picture quality of the terminal equipment is ensured.

The coding strategy corresponding to the decoding abnormal condition of frequent fluctuation abnormity or frequent overload is a layered coding strategy, and the coding strategy corresponding to the decoding abnormal condition of occasional fluctuation abnormity or occasional overload is a frame skipping coding strategy. It can be seen that the coding strategies corresponding to different decoding abnormal conditions can also be the same, that is, both frequent fluctuation abnormal conditions and frequent overload conditions can correspond to layered coding strategies, and both occasional fluctuation abnormal conditions and occasional overload conditions can correspond to frame skipping coding strategies. When the decoding abnormal condition existing in the terminal equipment is detected to be frequent network abnormality or frequent overload, the corresponding layered coding strategy can be used as a recommended coding strategy, and when the decoding abnormal condition existing in the terminal equipment is detected to be occasional fluctuation abnormality or occasional overload, the corresponding frame skipping coding strategy can be used as the recommended coding strategy. The recommended encoding strategy is an encoding strategy recommended to be adopted by the prior attempt, and whether the recommended encoding strategy can be used finally needs to be further judged based on a matching result between the reference decoding capability and the expected decoding capability of the terminal equipment. In the above manner, the decoding abnormal condition can reflect the terminal decoding speed (which can be represented by the decoding load condition) and the current network condition (such as the network fluctuation condition), and for different decoding abnormal conditions (including the overload abnormal condition and the network fluctuation abnormal condition), the server can automatically determine the coding strategy corresponding to the decoding abnormal condition according to the corresponding relationship between the different decoding abnormal conditions and the coding strategy, so as to avoid the blockage caused by the network fluctuation or the capability limitation of the terminal equipment. It can be understood that, in the encoding process of the whole target cloud game, the encoding strategy is not fixed, but the currently adopted encoding strategy can be flexibly adjusted based on the currently detected decoding abnormal condition existing in the terminal device. For example, if the frame skipping coding strategy is adopted for coding the 4 th frame, and frequent overload of the terminal device is detected before the 7 th frame is coded, the frame skipping coding strategy can be changed into the layered coding strategy.

In a feasible manner, the abnormal situation of network fluctuation and the abnormal situation of overload can also exist simultaneously, that is, when the detected abnormal decoding situation existing in the terminal device includes frequent fluctuation abnormity and accidental overload, or includes frequent overload and accidental fluctuation abnormity, the hierarchical coding strategy can be used as the recommended coding strategy. Namely, under the condition of comprehensively considering the network fluctuation condition and the decoding load of the terminal equipment, determining the layered coding strategy as the recommended coding strategy once the decoding abnormal condition corresponding to the layered coding strategy occurs no matter what decoding abnormal condition exists. Therefore, the situation that the network is unstable for a long time or the decoding load is too high can be solved preferentially, and the unpredictable decoding risk is avoided.

S302, acquiring a reference decoding capability which can be provided by the terminal device when the terminal device performs decoding processing on the coded data generated by the recommended coding strategy, and acquiring a desired decoding capability which the terminal device needs to have when the terminal device performs normal decoding processing on the coded data generated by the recommended coding strategy.

In one embodiment, the reference decoding capability that the terminal device can provide includes a static decoding capability, and for the detection process of the static decoding capability, the following may be included: (1) Determining a communication interface corresponding to a decoder from the terminal equipment, and acquiring description information of the decoder through the communication interface; (2) When the description information indicates that the decoder supports decoding processing of the encoded data generated by the recommended encoding strategy, the number of decoding reference frames provided by the description information is used as the static decoding capability provided by the terminal equipment.

In particular, the decoder in the terminal device may be a hardware chip supporting decoding of encoded data. Due to different video coding and decoding standards, types of decoders supported by the terminal device may also be different, for example, the decoder may be an H264 chip, an HEVC chip, or the like. Under different video coding and decoding standards, the supported coding strategies are also different, so that when terminal equipment supports different types of decoders, the available decoding capabilities are also different, for example, an H264 chip supports layered coding and decoding, but other chips do not support layered coding and decoding. The server can send an acquisition request to the communication interface through the determined communication interface corresponding to the decoder, and the terminal device can respond to the acquisition request and feed back the description information about the decoder to the server. The description information includes: information indicating whether a decoder supports a decoding process for encoded data generated by the recommended encoding policy, and information indicating the number of decoding reference frames provided when supported.

Whether or not the decoder supports the decoding processing of the encoded data generated by the recommended encoding policy and the number of decoding reference frames provided by the decoder when the decoding processing of the encoded data generated by the recommended encoding policy is supported can be known based on the description information of the decoder. Wherein, the recommended coding strategy can be one or more of a layered coding strategy and a frame skipping coding strategy. The number of decoded reference frames refers to the number of reference frames required for decoding one frame, and generally, the number of reference frames required for decoding one frame may be 0 to 8, or 0 to 16. The number of decoding reference frames provided when supported by the terminal device may include a maximum number of decoding reference frames provided when supported by the terminal device.

When the description information indicates that the decoder supports decoding processing on the encoded data generated by the recommended encoding policy, the decoder in the terminal device is described to have the capability of decoding the encoded data generated by the recommended encoding policy. Further, to more accurately determine whether the recommended coding strategy is available, the server may use the number of decoding reference frames provided by the terminal device when supported as the static decoding capability that can be provided by the terminal device. When the description information indicates that the decoder does not support the decoding processing of the coded data generated by the recommended coding strategy, the description information indicates that the decoder in the terminal device does not have the capability of decoding the coded data generated by the recommended coding strategy, at this time, the dynamic decoding capability of the terminal device can be acquired, and whether the recommended coding strategy is available or not is further determined through the dynamic decoding capability. For the acquisition of dynamic decoding capability, see the following embodiments.

In another embodiment, the reference decoding capability available for the terminal device includes a dynamic decoding capability, and for the probing process of the dynamic decoding capability, the following may be included: firstly, a server can send a first detection code stream generated by adopting a layered coding strategy and a second detection code stream generated by adopting a frame skipping coding strategy to a terminal device; then, the server can obtain a first decoding result for respectively decoding each detection segment of the first detection code stream and a second decoding result for decoding each detection segment of the second detection code stream from the terminal equipment; and finally, the server can generate the dynamic decoding capability which can be provided by the terminal equipment according to the first decoding result and the second decoding result.

The server may send, to the terminal device, a detection code stream generated by different coding strategies, where the detection code stream may be a video code stream used for detecting the dynamic decoding capability of the terminal device, and may be obtained by coding and compressing any video data. The first detection code stream and the second detection code stream both comprise a plurality of detection fragments, and different detection fragments are obtained by coding with different coding reference frame numbers. The number of the detection segments contained in the first detection code stream and the number of the detection segments contained in the second code stream may be the same or different, the number of the coding reference frames of different detection segments in the same detection code stream is different, the coding reference frame refers to a reference frame required by coding one frame, and the coding reference frame may be an I frame or a P frame. For example, the first detection code stream includes 2 detection segments generated by using layered coding, and the number of coding reference frames used by each detection segment is 4 and 8 respectively; the second detection code stream comprises 3 detection segments generated by adopting frame skipping coding, and the number of coding reference frames adopted by each detection segment is respectively 4, 8 and 16.

After receiving each detection segment of the first detection code stream and each detection segment of the second detection code stream, the terminal device may perform decoding processing on each detection segment included in different detection code streams to obtain a decoding result. The decoding result is used to indicate whether the corresponding probe segment is decoded normally (i.e., whether an error cue or stuck occurs). The server may obtain a first decoding result and a second decoding result from the terminal device, where the first decoding result includes a decoding result obtained by the terminal device decoding each detection segment of the first detection code stream, and the second decoding result includes a decoding result obtained by the terminal device decoding each detection segment of the second detection code stream. The server can determine whether the terminal equipment supports normal decoding of the coded data generated by the recommended coding strategy based on the obtained decoding result obtained by decoding each detection segment of the first detection code stream and the obtained decoding result obtained by decoding each detection segment of the second detection code stream, and further generate the dynamic decoding capability provided by the terminal equipment.

In one implementation, generating the dynamic decoding capability that can be provided by the terminal device according to the first decoding result and the second decoding result includes: storing the decoding description information of the terminal equipment on each detection segment of the first detection code stream recorded in the first decoding result and the number of the coding reference frames corresponding to each detection segment of the first detection code stream in an associated manner as the dynamic decoding capability provided by the terminal equipment when supporting the layered coding strategy; and storing the decoding description information of the terminal equipment on each detection segment of the second detection code stream recorded in the second decoding result and the number of the coding reference frames corresponding to each detection segment of the second detection code stream in an associated manner as the dynamic decoding capability provided by the terminal equipment when the frame skipping coding strategy is supported. The correlated storage refers to storing the decoding description information of whether one detection segment is successfully decoded and the number of the coding reference frames corresponding to the detection segment after being correlated, and is convenient for acquiring all information about one detection segment in one station through the correlated storage, thereby being beneficial to judging whether a recommended coding strategy is available. The available dynamic decoding capabilities include one or more of decoding capabilities that the terminal device can provide when supporting a layered coding strategy and decoding capabilities that the terminal device can provide when supporting a frame skipping coding strategy. Thus, the dynamic decoding capabilities that may be provided include one or more of: (1) the decoding description information of each detection segment in the first detection code stream, the number of coding reference frames of each detection segment (2), the decoding description information of each detection segment in the second detection code stream, and the number of coding reference frames of each detection segment. Wherein the decoding description information is used for indicating whether the terminal equipment successfully decodes the corresponding detection segment.

The dynamic decoding capability may indicate that a decoder in the terminal device may support decoding of encoded data for the layered coding strategy and the maximum number of encoded reference frames when decoding is successful (i.e., no stuck occurs). For example, the available dynamic decoding capability is used to indicate: and the terminal equipment successfully decodes the detection segment D generated by the layered coding strategy, and the number of the coding reference frames corresponding to the detection segment D is 8. If the decoding description information recorded in the first decoding result includes information of successfully decoding at least one probe segment of the first probe code stream, it indicates that the terminal device supports a layered coding strategy, that is, supports decoding of encoded data obtained by using the layered coding strategy, but how large the number of encoded reference frames that the terminal device supports for decoding can reach is determined by the probe segment corresponding to the maximum number of encoded reference frames in the at least one probe segment that is successfully decoded, that is, the maximum number of decoded reference frames that the decoder supports is equal to the maximum number of encoded reference frames of the stored probe segment that is successfully decoded. That is, for the same recommended coding strategy (e.g., layered coding strategy), in the dynamic decoding capability provided by the terminal device in supporting the recommended coding strategy (e.g., layered coding strategy), when each probe segment is successfully decoded or in the case that part of the probe segments are successfully decoded, the maximum number of decoded reference frames supported by the terminal device may be the same or different due to the difference in successfully decoded probe segments. For example, the first detection code stream obtained by encoding different numbers of reference frames by using a layered coding strategy includes 3 detection segments, which are respectively a detection segment D1, a detection segment D2, and a detection segment D3, the numbers of corresponding reference frames are respectively 8, 6, and 4, decoding description information of the detection segment D2 and the detection segment D3, and decoding description information of the detection segment D1 that fails to decode, and the number of reference frames corresponding to each detection segment are stored as dynamic decoding capabilities that can be provided when a terminal device supports the layered coding strategy, if only the detection segment D2 and the detection segment D3 are successfully decoded, the maximum number of reference frames supported by a decoder is 6, and if the above 3 detection segments can be successfully decoded, the maximum number of reference frames supported by the decoder is 8.

It can be understood that, if the decoding description information recorded in the first decoding result is information indicating that the decoding of each probe segment of the first probe code stream fails, that is, the terminal device does not support the layered coding policy, at this time, only the decoding description information in the first decoding result may be stored as the dynamic decoding capability that can be provided by the terminal device, where the dynamic decoding capability indicates that the decoder does not support the layered coding policy. The same applies to the second decoding result. When all the detection segments are successfully decoded or when part of the detection segments are successfully decoded, the number of the coding reference frames stored in the dynamic decoding capability provided by the terminal device when the frame skipping coding strategy is supported is different, and the maximum number of the supported decoding reference frames may be the same or different.

It should be noted that, the detection of the static decoding capability and the dynamic decoding capability of the terminal device may be performed before encoding, and the information obtained by the detection is stored in the database of the server and is obtained from the corresponding database when needed. Before processing a frame to be coded, the decoding abnormal condition existing in the terminal equipment needs to be detected, and the static decoding capability and the dynamic decoding capability are inherent capabilities of the terminal equipment and can be repeatedly used after one-time detection.

In addition to obtaining the reference decoding capability of the terminal device, the server may further obtain a desired decoding capability that the terminal device needs to have when the terminal device performs normal decoding processing on the encoded data generated by the recommended encoding policy. When the terminal equipment carries out normal decoding processing on the coded data generated by the recommended coding strategy, decoding blockage or error prompt does not occur in the decoding processing process, so that the abnormal decoding condition existing in the terminal equipment can be overcome. The expected decoding capability is a basic decoding capability required by the terminal device to overcome the decoding abnormal condition, and the expected decoding capability can be set by the server based on the corresponding recommended coding strategy, for example, when the recommended coding strategy is a layered coding strategy, the server requires that the number of reference frames corresponding to the number of reference frames is required for starting the layered coding strategy, and then the maximum number of decoding reference frames supported by the terminal device at least needs to reach the required number of reference frames, so that the encoded data obtained by using the layered coding strategy can be normally decoded.

In one embodiment, before performing the step S303 of performing the matching process on the reference decoding capability and the expected decoding capability to obtain the matching result, it may be determined whether the recommended coding policy can be supported by the server according to the information about the recommended coding policy.

If the recommended coding strategy is a layered coding strategy, the following steps can be performed: and acquiring support information about the layered coding strategy, and triggering execution to match the reference decoding capability with the expected decoding capability when the support information of the layered coding strategy indicates that the layered coding strategy is supported. Specifically, the support information about the layered coding policy is used to characterize whether the encoder in the server supports the layered coding policy. When the support information indicates that the server supports the layered coding policy, a step of matching the reference decoding capability and the desired decoding capability may be performed, and availability of the layered coding policy may be determined by comparing the actual decoding capability of the reference terminal device with the decoding capability achieved by the desired terminal device.

In one implementation, when the support information on the layered coding strategy indicates that the layered coding strategy is not supported, the frame skipping coding strategy is taken as a new recommended coding strategy. That is, when the recommended coding strategy is a layered coding strategy but the layered coding strategy is not supported to be used, in order to solve the decoding abnormal situation of the terminal device and ensure stable output of the picture in the terminal device, a new coding strategy, which is a frame skipping coding strategy, may be adopted, and then, the determination as to whether the coding process can be finally performed using the frame skipping coding strategy is the same as the layered coding strategy, that is: when the obtained support information about the frame skipping coding strategy indicates that frame skipping coding is supported, the step of matching the reference Decoding capability with the expected Decoding capability may be triggered, otherwise, other schemes may be adopted for coding, for example, before the server requests an IDR frame (i.e. an instant Refresh frame) to be inserted into the game frame to be processed, the game frame to be processed only refers to the IDR frame for coding. It should be noted that, because the size of the IDR frame is usually significantly larger than the size of the P frame, reapplication of the IDR frame may cause further network congestion, and therefore, when the network congestion or decoding congestion occurs, a recommended coding strategy (either a hierarchical coding strategy or an adjustment coding strategy) is preferably adopted to replace the IDR frame with the P frame, thereby avoiding that the IDR frame may cause further network congestion, and at the same time, allowing the encoded partial frame to be discarded. However, if the server does not support the recommended encoding strategy, the IDR frame may be used to solve the stuck problem as much as possible in order to advance the decoding progress.

If the recommended coding strategy is a frame skipping coding strategy, the method further comprises the following steps (1) to (3):

(1) and determining the reference quantity corresponding to the reference frame to be shielded and a reference frame list corresponding to the target cloud game according to the current network condition of the terminal equipment.

The server can judge the current network condition of the network environment where the terminal device is located, wherein the current network condition comprises any one of a network fluctuation condition and a network stable condition, the network stable condition indicates that the network environment is stable, and the network fluctuation condition indicates that the network is possibly blocked. For each coded reference frame transmitted to the terminal device, whether to mask (i.e., discard) or not can be determined by the number of packet losses or the transmission delay, so as to determine the reference frames to be masked (i.e., the reference frames to be discarded) and the reference number (i.e., the number of reference frames to be discarded). The reference frame list corresponding to the target cloud game comprises reference frames required by the game frames to be processed. For example, as shown in fig. 5a, before the 6 th frame is encoded, the reference frame list includes 4 encoded reference frames (i.e., 2 nd to 5 th frames) required by the 6 th frame, and due to the influence of the current network condition (e.g., network congestion occurs), the 5 th frame is lost when being transmitted to the terminal device, and is lost 3 times after attempting retransmission, and reaches the threshold of the number of lost packets, it may be determined that the 5 th frame is a reference frame to be masked, and each frame is sequentially determined, it is determined that the 5 th frame and the 4 th frame need to be masked, and the 3 rd frame and the 2 nd frame are normally transmitted to the terminal device, that is, the reference number corresponding to the reference frame to be masked is 2. After the frame to be masked is masked, the frame to be encoded is encoded independent of the masked encoded reference, i.e., the 5 th frame is encoded dependent on the 2 nd and 3 rd frames. The reference frame to be masked is not used as a reference frame in encoding and decoding, and the masked frame can be transmitted to the terminal equipment and not used as a reference frame in decoding, or can be directly not transmitted to the terminal equipment.

(2) When the frame skipping coding strategy is determined to be supported, whether the reference frames with the reference number in the target cloud game are shielded or not is determined according to the total number of the reference frames and the reference number contained in the reference frame list.

The method includes the steps that support information used for indicating whether an encoder supports a frame skipping encoding strategy or not can be acquired through a communication interface of the encoder in a server, when the frame skipping encoding strategy is determined to be supported, the server is indicated to have the capacity of supporting specific reference frames or shielding reference frames, whether a reference frame list supports reference frames with the number of the reference frames for achieving shielding reference can be judged, and specifically, the total number of the reference frames contained in the reference frame list and the reference number corresponding to the reference frames to be shielded can be compared to determine the reference frame list. If the reference number corresponding to the reference frame to be masked is greater than or equal to the total number of the reference frames contained in the reference frame list, i.e. after the reference frames with the reference number are masked, no reference frame exists in the reference frame list, and as the normal encoding of the frame to be encoded is to be ensured, the server can apply for an IDR frame to serve as the reference frame of the frame to be encoded at the moment, so as to ensure the normal output of the picture. Therefore, in order to ensure the use of the frame skipping coding strategy, after reference frames with reference number are masked, it is usually required to ensure that reference frames still exist in the reference frame list and that IDR frames do not appear, and the reference number corresponding to the reference frames to be masked is required to be less than the total number of reference frames in the reference frame list.

The server supporting the frame skipping coding strategy correspondingly supports the reference frames with the preset shielding number, and the preset number of the shielding reference frames supported by different servers may have differences. For any server, if the reference number is smaller than the total number of reference frames in the reference frame list and the server has the limited number of masked reference frames, even if the reference number is smaller than the total number of reference frames in the reference frame list, the server does not support the reference frames with the masked reference number because the server does not support the masking larger than the number of reference frames limited by the server, that is, the server does not support the reference frames with the masked reference number, specifically, the reference frame list does not support the reference frames with the masked reference number, and the frame skipping coding strategy is not available.

(3) And when the reference frames supporting the reference number in the target cloud game are determined to be shielded, triggering to perform matching processing of the reference decoding capability and the expected decoding capability. Specifically, when the server determines that the reference frames with the reference number in the target cloud game can be shielded, it indicates that the server supports the frame skipping coding strategy, and further, the reference decoding capability and the expected decoding capability can be matched, specifically, the matching between any one or more of the dynamic decoding capability and the static decoding capability and the expected decoding capability is performed, so as to obtain a matching result.

Whether the strategy is a layered coding strategy or a frame skipping coding strategy, the coded partial frames can be discarded, so that the P frame coding does not depend on the frames needing to be discarded, and meanwhile, when any one of the two coding strategies can be used, the scheme of inserting IDR frames can be avoided. Because the size of the IDR frame is obviously larger than that of the P frame, the reapplication of the IDR frame causes the problem of further network congestion, and when both coding strategies are not available, in order to make the terminal device not congestion in the decoding of a certain frame, the decoding progress of the terminal device is promoted, therefore, the scheme of inserting the IDR frame can be adopted when the layered coding strategy and the frame skipping coding strategy are not available.

In one implementation, when it is determined that masking of reference frames of a reference number in the target cloud game is not supported, the method may: and selecting a new coding reference frame from the processed game frames of the target cloud game, and taking the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frame of the target cloud game to perform coding processing on the to-be-processed game frame based on the immediate refresh frame.

The method for selecting a new coding reference frame from the processed game frames of the target cloud game may be as follows: determining a frame sequence group where processed game frames of the target cloud game are located, and determining a first intra-frame coding reference frame in the frame sequence group as a new coding reference frame. The frame sequence group comprises a plurality of frame images, and the processed game frame can be generally divided into a plurality of frame sequence groups in the encoding process and transmitted, and the frame sequence group can also be called as an image group. Illustratively, if a frame sequence group contains IPPPP, the currently encoded 4 th frame refers to the first 3P frames, and when decoding pause occurs, the first I frame can be regarded as an immediate refresh frame (IDR frame). Starting from the IDR frame, a new sequence can be recalculated to start encoding. By taking the selected new coding reference frame as the immediate refresh frame, the immediate refresh frame can be inserted before the game frame to be processed of the target cloud game, so that the game frame to be processed does not refer to the original reference frame any more, but only refers to the IDR frame for coding processing, and through setting of the IDR, the coding reference frame before the insertion position of the IDR frame can be discarded at will.

And S303, matching the reference decoding capability and the expected decoding capability to obtain a matching result.

In one embodiment, the reference decoding capability that the terminal device can provide includes: static decoding capabilities and available dynamic decoding capabilities. The server can match the static decoding capability with the expected decoding capability, match the dynamic decoding capability with the expected decoding capability, and synthesize respective matching results of the static decoding capability and the dynamic decoding capability to obtain a final matching result. The dynamic decoding capabilities that may be provided may include dynamic decoding capabilities provided in support of a layered coding strategy and dynamic decoding capabilities provided in support of a frame skipping coding strategy.

In the case that the reference decoding capability includes a static decoding capability and a dynamic decoding capability, the reference decoding capability and the expected decoding capability are matched, and the matching result may be obtained by:

first, the acquisition desired decoding capability indicates a desired number of decoding reference frames, and the number of decoding reference frames providable by the terminal device is acquired from the static decoding capability. Specifically, the desired decoding capability indicates that the desired number of decoding reference frames is a condition for ensuring that the terminal device normally decodes the encoded data generated at the recommended encoding policy. The matching between the desired decoding capability and the static decoding capability specifically refers to: the number of available decoding reference frames of the terminal equipment reaches the expected number of decoding reference frames. The desired number of decoding reference frames may be determined by the number of encoding reference frames required under the encoding policy supported by the server. For example, under the supporting of the frame skipping coding strategy, the server supports a number A1 of coding reference frames in the masked reference frame list, and then the desired number of decoding reference frames is A1, and under the supporting of the layered coding strategy, the server sets a number A2 of coding reference frames required by the layered coding, and then the desired number of decoding reference frames is A2.

Then, a number matching result obtained by matching the number of the decoding reference frames that can be provided by the terminal device with the number of the expected decoding reference frames is used as a matching result of the static decoding capability and the expected decoding capability. Specifically, the server may compare the number of available decoding reference frames with the number of expected decoding reference frames to obtain a number matching result, where the number matching result is used to indicate whether the number of available decoding reference frames reaches the number of expected decoding reference frames, and reaching the number of expected decoding reference frames refers to: the number of available decoding reference frames is greater than or equal to the desired number of decoding reference frames, and the failure of the desired number of decoding reference frames means: the number of available decoding reference frames is less than the desired number of decoding reference frames. The number matching result can be used as a matching result obtained by matching the static decoding capability and the expected decoding capability, so that whether the static decoding capability and the dynamic decoding capability are successfully matched or not can be known.

And finally, matching target detection fragments with the number of the coding reference frames being more than or equal to the expected number of the decoding reference frames and decoding description information of the target detection fragments from the available dynamic decoding capacity, and taking the matched decoding description information as a matching result of the dynamic decoding capacity and the expected decoding capacity. Specifically, the server may match a target probe segment from each probe segment included in the dynamic decoding capability that may be provided by the terminal device, where the number of coding reference frames included in the target probe segment is greater than or equal to the number of decoding reference frames expected by the expected decoding capability, if the recommended coding policy is a layered coding policy, the matched target probe segment is a probe segment in a first probe code stream generated by using the layered coding policy, and if the recommended coding policy is a frame skipping coding policy, the matched target probe segment is a probe segment in a second probe code stream generated by using the frame skipping coding policy. The decoding description information of the matched target detection segment is used to indicate whether the target detection segment is successfully decoded, and may be used as a matching result between the dynamic decoding capability and the expected decoding capability to indicate whether the dynamic decoding capability and the expected decoding capability are successfully matched. When the decoding description information of the matched target detection fragment indicates that the target detection fragment is successfully decoded, the matching result indicates that the dynamic decoding capability and the expected decoding capability are successfully matched; otherwise, the matching result indicates that the dynamic decoding capability and the expected decoding capability failed to match.

In a possible implementation manner, when the number matching result indicates that the number of decoding reference frames that the terminal device can provide is greater than or equal to the expected number of decoding reference frames, and the matched decoding description information indicates that the terminal device successfully decodes the target sounding reference frame, it indicates that the terminal device can successfully decode at least the encoded data that includes the expected number of decoding reference frames under the recommended encoding policy, and it is determined that the matching result indicates that the reference decoding capability is successfully matched with the expected decoding capability.

When the number matching result indicates that the number of decoding reference frames which can be provided by the terminal equipment is greater than or equal to the expected number of decoding reference frames, the matching between the static decoding capability and the expected decoding capability is successful, when the matched decoding description information indicates that the terminal equipment successfully decodes the target detection segment, the matching between the dynamic decoding capability and the expected decoding capability is successful, and when the static decoding capability and the dynamic decoding capability which are included in the reference decoding capability are successfully matched with the expected decoding capability, the reference decoding capability and the expected decoding capability are successfully matched. Conversely, when any one of the static decoding capability and the dynamic decoding capability included in the reference decoding capability fails to match the desired decoding capability, it is determined that the reference decoding capability and the desired decoding capability fail to match.

S304, when the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, adopting a recommended coding strategy to code the game frames to be processed in the target cloud game.

When the recommended encoding policy is a layered encoding policy, the encoding processing on the to-be-processed game frame in the target cloud game may be to discard a proper amount of encoding reference frames belonging to the enhancement layer and referred to by the to-be-processed game frame, and transmit the encoding reference frames of different layers in channels to reduce network transmission pressure. When the recommended encoding strategy is a frame skipping encoding strategy, the encoding reference frames referred to by the to-be-processed game frames in the target cloud game can be appropriately adjusted, for example, one frame of the 4 frames of encoding reference frames referred to by the to-be-processed game frames is shielded, and then the to-be-processed game frames can be encoded by referring to the rest 3 frames.

S305, when the matching result indicates that the matching between the reference decoding capacity and the expected decoding capacity fails, selecting a new coding reference frame from the processed game frames of the target cloud game, and taking the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frames of the target cloud game to perform coding processing on the to-be-processed game frames based on the immediate refresh frame.

When the matching result indicates that the reference decoding capability and the expected decoding capability are unsuccessfully matched, the server is indicated to be incapable of adopting a layered coding strategy and incapable of adopting a frame skipping coding strategy, and when the two coding strategies cannot be adopted, the game frame to be processed can be coded by immediately refreshing the frame. For specific processing principles, reference may be made to the foregoing description, which is not repeated herein. For the above, see the flow diagram of the encoding process shown in fig. 5 b.

The encoding processing scheme provided by the embodiment of the application can detect the decoding abnormal condition of the terminal equipment, specifically can determine the decoding abnormal condition according to the corresponding network fluctuation times or overload times in a time window, can reflect the real-time decoding state of the terminal equipment, can flexibly recommend an encoding strategy to perform encoding processing according to the real-time decoding state, realizes intelligent management of a reference frame, and ensures the decoding fluency of the terminal equipment and stable output of pictures. Wherein, for different abnormal conditions of network fluctuation (such as frequent fluctuation and accidental fluctuation) or different abnormal conditions of overload (including frequent overload and accidental overload), different coding strategies can be provided for frame dropping so as to adapt to different network fluctuation conditions and real-time decoding capability. In addition, the static decoding capability and the dynamic decoding capability can be detected for the decoding capability of the terminal equipment, the static decoding capability and the dynamic decoding capability are obtained, the server can self-adaptively determine the availability of the recommended coding strategy according to the decoding capability and the network condition provided by the terminal equipment, and then the decoding capability of the terminal equipment is fully utilized and the abnormal decoding condition is avoided. When the matching result indicates that the matching between the reference decoding capability and the expected decoding capability fails, other coding strategies can be adopted to code the game frame to be processed of the target cloud game, and therefore the recommended coding strategy can be adaptively adjusted based on the decoding capability of the terminal equipment, and the use flexibility of the coding strategy is high.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an encoding processing apparatus according to an embodiment of the present disclosure. The encoding processing device may be a computer program (including program code) running in a server, for example, the encoding processing device is an application software; the encoding processing device can be used for executing corresponding steps in the method provided by the embodiment of the application. As shown in fig. 6, the encoding processing apparatus 600 may include at least one of the following: a determination module 601, an acquisition module 602, a matching module 603, and an encoding module 604.

A determining module 601, configured to, when a decoding abnormal condition existing in a terminal device is detected, take an encoding policy corresponding to the decoding abnormal condition as a recommended encoding policy according to a correspondence between different decoding abnormal conditions and encoding policies; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different;

an obtaining module 602, configured to obtain a reference decoding capability that the terminal device can provide when decoding the encoded data generated by the recommended encoding policy, and obtain an expected decoding capability that the terminal device needs to have when the terminal device performs normal decoding on the encoded data generated by the recommended encoding policy;

a matching module 603, configured to perform matching processing on the reference decoding capability and the expected decoding capability to obtain a matching result;

the encoding module 604 is configured to, when the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched, perform encoding processing on the to-be-processed game frame in the target cloud game by using the recommended encoding policy.

In one embodiment, the determining module 601 is configured to: sending network detection data to terminal equipment in a time window, wherein the time window comprises a plurality of time points, and one or more network detection data are sent to the terminal equipment at one time point; acquiring detection feedback data corresponding to the network detection data sent at each time point of the time window, and generating a network detection result at the corresponding time point according to the detection feedback data; the network detection result corresponding to any time point is used for indicating whether network fluctuation occurs at any time point; and counting the times of network fluctuation in the time window according to the network detection results respectively corresponding to each time point of the time window, and determining the decoding abnormal condition existing in the terminal equipment according to the counted times.

In one embodiment, the determining module 601 is configured to: acquiring a fluctuation index value; when the counted times is more than or equal to the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent fluctuation abnormity; and when the counted times are more than or equal to 1 and less than the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental fluctuation abnormality.

In one embodiment, the determining module 601 is configured to: acquiring load information of terminal equipment at different time points of a time window, wherein the load information is the data volume to be decoded of the terminal equipment at corresponding time points; determining the overload times of the terminal equipment in the time window according to the load information respectively corresponding to the terminal equipment at different time points of the time window; when the data volume to be decoded of the terminal equipment at any time point is larger than or equal to a data volume threshold value, the terminal equipment is overloaded once at any time point; determining the decoding abnormal condition of the terminal equipment according to the overload times; when the overload frequency reaches an overload index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent overload, and when the overload frequency is more than 1 and does not reach the overload index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental overload.

In one embodiment, the encoding strategy includes a layered encoding strategy and a frame skipping encoding strategy; when the decoding abnormal condition is the abnormal condition of network fluctuation, the corresponding recommended coding strategy comprises the layered coding strategy; and when the decoding abnormal condition is an overload abnormal condition, the corresponding recommended coding strategy comprises a frame skipping coding strategy.

In one embodiment, the obtaining module 602 is configured to: determining a communication interface corresponding to a decoder from the terminal device, and acquiring description information of the decoder through the communication interface, wherein the description information includes: information indicating whether the decoder supports decoding processing of encoded data generated by the recommended encoding policy, and information indicating the number of decoding reference frames provided when supported; and when the description information indicates that the decoder supports decoding processing on the encoded data generated by the recommended encoding strategy, the number of decoding reference frames indicated and provided in the description information is used as the static decoding capability which can be provided by the terminal equipment.

In one embodiment, the obtaining module 602 is configured to: sending a first detection code stream generated by adopting a layered coding strategy and a second detection code stream generated by adopting a frame skipping coding strategy to the terminal equipment, wherein the first detection code stream and the second detection code stream both comprise a plurality of detection segments, and different detection segments are obtained by coding with different coding reference frame numbers; acquiring a first decoding result for respectively decoding each detection segment of the first detection code stream and a second decoding result for decoding each detection segment of the second detection code stream from the terminal equipment; and generating the dynamic decoding capability which can be provided by the terminal equipment according to the first decoding result and the second decoding result.

In an embodiment, the obtaining module 602 is specifically configured to: storing the decoding description information of the terminal device to each detection segment of the first detection code stream recorded in the first decoding result and the number of coding reference frames corresponding to each detection segment of the first detection code stream in an associated manner as the dynamic decoding capability which can be provided by the terminal device when the terminal device supports a layered coding strategy; storing the decoding description information of the terminal equipment on each detection segment of the second detection code stream recorded in the second decoding result and the number of the coding reference frames corresponding to each detection segment of the second detection code stream in an associated manner as the dynamic decoding capability provided by the terminal equipment when the frame skipping coding strategy is supported; wherein the decoding description information is used to indicate whether the terminal device successfully decodes the corresponding sounding reference.

In an embodiment, if the recommended coding policy is a hierarchical coding policy, the obtaining module 602 is configured to: acquiring support information about the layered coding strategy, and triggering to execute matching processing of the reference decoding capability and the expected decoding capability when the support information of the layered coding strategy indicates that the layered coding strategy is supported; a determining module 601, configured to: and when the support information on the hierarchical coding strategy indicates that the hierarchical coding strategy is not supported, taking the frame skipping coding strategy as a new recommended coding strategy.

In an embodiment, if the recommended coding strategy is a frame skipping coding strategy, the determining module 601 is configured to: determining the reference number corresponding to the reference frame to be shielded and a reference frame list corresponding to the target cloud game according to the current network condition of the terminal equipment; when the frame skipping coding strategy is determined to be supported, determining whether the reference frames with the reference number in the target cloud game are supported to be shielded or not according to the total number of the reference frames contained in the reference frame list and the reference number; when the reference frames supporting the reference number in the target cloud game are determined to be shielded, triggering to perform matching processing on the reference decoding capacity and the expected decoding capacity; otherwise, selecting a new coding reference frame from the processed game frames of the target cloud game, and taking the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frame of the target cloud game, so as to perform coding processing on the to-be-processed game frame based on the immediate refresh frame.

In one embodiment, the reference decoding capability that the terminal device can provide includes: static decoding capabilities and available dynamic decoding capabilities; a matching module 603 configured to: acquiring the number of decoding reference frames expected by the indication of the expected decoding capability and acquiring the number of decoding reference frames available by the terminal equipment from the static decoding capability; a quantity matching result obtained by quantity matching between the quantity of the decoding reference frames which can be provided by the terminal equipment and the quantity of the expected decoding reference frames is used as a matching result of the static decoding capability and the expected decoding capability; matching target detection fragments with the number of coding reference frames being larger than or equal to the expected number of decoding reference frames and decoding description information of the target detection fragments from the available dynamic decoding capability, and taking the matched decoding description information as a matching result of the dynamic decoding capability and the expected decoding capability.

In one embodiment, the determining module 601 is further configured to: when the number matching result indicates that the number of decoding reference frames that can be provided by the terminal device is greater than or equal to the number of expected decoding reference frames and the matched decoding description information indicates that the terminal device successfully decodes the target sounding segment, it is determined that the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched.

In one embodiment, the encoding module 604 is further configured to: and when the matching result indicates that the reference decoding capability and the expected decoding capability are unsuccessfully matched, selecting a new coding reference frame from the processed game frames of the target cloud game, and using the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frames of the target cloud game so as to perform coding processing on the to-be-processed game frames based on the immediate refresh frame.

It can be understood that the functions of the functional modules of the encoding processing apparatus described in the embodiment of the present application may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the description related to the foregoing method embodiment, which is not described herein again. In addition, the beneficial effects of the same method are not described in detail.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a server according to an embodiment of the present disclosure. The server 700 may comprise a standalone device (e.g., one or more of a server, a node, a terminal, etc.) or may comprise a component (e.g., a chip, a software module, or a hardware module, etc.) within a standalone device. The server 700 may comprise at least one processor 701 and a communication interface 702, further optionally, the server 700 may further comprise at least one memory 703 and a bus 704. The processor 701, the communication interface 702, and the memory 703 are connected by a bus 704.

The processor 701 is a module for performing arithmetic operation and/or logical operation, and may specifically be one or a combination of multiple processing modules, such as a Central Processing Unit (CPU), a picture processing Unit (GPU), a Microprocessor (MPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Complex Programmable Logic Device (CPLD), a coprocessor (which assists the central processing Unit to complete corresponding processing and Application), and a Micro Control Unit (MCU).

Communication interface 702 may be used to provide information input or output to at least one processor. And/or, the communication interface 702 may be used to receive and/or transmit data externally, and may be a wired link interface such as an ethernet cable, and may also be a wireless link (Wi-Fi, bluetooth, general wireless transmission, vehicle-mounted short-range communication technology, other short-range wireless communication technology, and the like) interface. The communication interface 702 may serve as a network interface.

The memory 703 is used to provide a storage space in which data such as an operating system and computer programs can be stored. The memory 703 may be one or a combination of Random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable read-only memory (CD-ROM), among others.

The at least one processor 701 in the server 700 is configured to call up a computer program stored in the at least one memory 703 for executing the encoding processing method described in the embodiment shown in the present application.

In one possible implementation, the processor 701 in the server 700 is configured to invoke a computer program stored in the at least one memory 703 for performing the following operations: when the decoding abnormal condition existing in the terminal equipment is detected, according to the corresponding relation between different decoding abnormal conditions and the coding strategy, taking the coding strategy corresponding to the decoding abnormal condition as a recommended coding strategy; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different; acquiring a reference decoding capacity which can be provided by the terminal equipment when the terminal equipment performs decoding processing on the coded data generated by the recommended coding strategy, and acquiring an expected decoding capacity which the terminal equipment needs to have when the terminal equipment performs normal decoding processing on the coded data generated by the recommended coding strategy; matching the reference decoding capability and the expected decoding capability to obtain a matching result; and when the matching result indicates that the reference decoding capacity is successfully matched with the expected decoding capacity, adopting the recommended coding strategy to code the game frames to be processed in the target cloud game.

In one embodiment, the processor 701 is configured to: sending network detection data to terminal equipment in a time window, wherein the time window comprises a plurality of time points, and one or more network detection data are sent to the terminal equipment at one time point; acquiring detection feedback data corresponding to the network detection data sent at each time point of the time window, and generating a network detection result at the corresponding time point according to the detection feedback data; the network detection result corresponding to any time point is used for indicating whether network fluctuation occurs at any time point; and counting the times of network fluctuation in the time window according to the network detection results respectively corresponding to each time point of the time window, and determining the decoding abnormal condition existing in the terminal equipment according to the counted times.

In one embodiment, the processor 701 is configured to: acquiring a fluctuation index value; when the counted times is more than or equal to the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent fluctuation abnormity; and when the counted times are more than or equal to 1 and less than the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental fluctuation abnormality.

In one embodiment, the processor 701 is configured to: acquiring load information of terminal equipment at different time points of a time window, wherein the load information is the data volume to be decoded of the terminal equipment at the corresponding time point; determining the overload times of the terminal equipment in the time window according to the load information respectively corresponding to the terminal equipment at different time points of the time window; when the data volume to be decoded of the terminal equipment at any time point is larger than or equal to a data volume threshold value, the terminal equipment is overloaded once at any time point; determining the decoding abnormal condition of the terminal equipment according to the overload times; when the overload frequency reaches an overload index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent overload, and when the overload frequency is more than 1 and does not reach the overload index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental overload.

In one embodiment, the processor 701 is configured to: determining a communication interface corresponding to a decoder from the terminal device, and acquiring description information of the decoder through the communication interface, wherein the description information includes: information indicating whether the decoder supports decoding processing of the encoded data generated by the recommended encoding policy, and information indicating the number of decoding reference frames provided when supported; and when the description information indicates that the decoder supports decoding processing on the coded data generated by the recommended coding strategy, taking the number of decoding reference frames provided by the description information as the static decoding capability provided by the terminal equipment.

In one embodiment, the processor 701 is configured to: sending a first detection code stream generated by adopting a layered coding strategy and a second detection code stream generated by adopting a frame skipping coding strategy to the terminal equipment, wherein the first detection code stream and the second detection code stream both comprise a plurality of detection segments, and different detection segments are obtained by coding with different coding reference frame numbers; acquiring a first decoding result for respectively decoding each detection segment of the first detection code stream and a second decoding result for decoding each detection segment of the second detection code stream from the terminal equipment; and generating the dynamic decoding capability which can be provided by the terminal equipment according to the first decoding result and the second decoding result.

In an embodiment, the processor 701 is specifically configured to: storing the decoding description information of the terminal equipment on each detection segment of the first detection code stream recorded in the first decoding result and the number of coding reference frames corresponding to each detection segment of the first detection code stream in an associated manner as the dynamic decoding capability provided by the terminal equipment when a layered coding strategy is supported; storing the decoding description information of the terminal device to each detection segment of the second detection code stream recorded in the second decoding result and the number of the coding reference frames corresponding to each detection segment of the second detection code stream in an associated manner as the dynamic decoding capability which can be provided by the terminal device when the frame skipping coding strategy is supported; wherein the decoding description information is used for indicating whether the terminal device successfully decodes the corresponding detection segment.

In an embodiment, if the recommended coding strategy is a layered coding strategy, the processor 701 is configured to: acquiring support information about the layered coding strategy, and triggering to execute matching processing of the reference decoding capability and the expected decoding capability when the support information of the layered coding strategy indicates that the layered coding strategy is supported; a processor 701 configured to: and when the support information on the hierarchical coding strategy indicates that the hierarchical coding strategy is not supported, taking the frame skipping coding strategy as a new recommended coding strategy.

In an embodiment, if the recommended coding strategy is a frame skipping coding strategy, the processor 701 is configured to: determining the reference number corresponding to the reference frame to be shielded and a reference frame list corresponding to the target cloud game according to the current network condition of the terminal equipment; when the frame skipping coding strategy is determined to be supported, determining whether the reference frames with the reference number in the target cloud game are supported to be shielded or not according to the total number of the reference frames contained in the reference frame list and the reference number; triggering to perform matching processing on the reference decoding capability and the expected decoding capability when the reference frames of the reference number in the target cloud game are determined to be supported to be shielded; otherwise, selecting a new coding reference frame from the processed game frames of the target cloud game, and taking the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frame of the target cloud game, so as to perform coding processing on the to-be-processed game frame based on the immediate refresh frame.

In one embodiment, the reference decoding capability that the terminal device can provide includes: static decoding capabilities and available dynamic decoding capabilities; a processor 701 configured to: acquiring the number of decoding reference frames expected by the expected decoding capability indication, and acquiring the number of decoding reference frames available by the terminal equipment from the static decoding capability; a quantity matching result obtained by quantity matching of the number of decoding reference frames which can be provided by the terminal equipment and the number of expected decoding reference frames is used as a matching result of the static decoding capability and the expected decoding capability; matching target detection fragments with the number of coding reference frames being larger than or equal to the expected number of decoding reference frames and decoding description information of the target detection fragments from the available dynamic decoding capability, and taking the matched decoding description information as a matching result of the dynamic decoding capability and the expected decoding capability.

In one embodiment, the processor 701 is further configured to: when the number matching result indicates that the number of decoding reference frames that can be provided by the terminal device is greater than or equal to the number of expected decoding reference frames and the matched decoding description information indicates that the terminal device successfully decodes the target sounding segment, it is determined that the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched.

In one embodiment, the processor 701 is further configured to: and when the matching result indicates that the reference decoding capability and the expected decoding capability are unsuccessfully matched, selecting a new coding reference frame from the processed game frames of the target cloud game, and using the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frames of the target cloud game so as to perform coding processing on the to-be-processed game frames based on the immediate refresh frame.

It should be understood that the server 700 described in this embodiment may perform the description of the encoding processing method in the embodiment corresponding to the foregoing description, and may also perform the description of the encoding processing apparatus 600 in the embodiment corresponding to the foregoing fig. 6, which is not described herein again. In addition, the beneficial effects of the same method are not described in detail.

In addition, it should be further noted that an exemplary embodiment of the present application further provides a storage medium, where the storage medium stores a computer program of the foregoing encoding processing method, where the computer program includes program instructions, and when one or more processors load and execute the program instructions, the description of the encoding processing method in the embodiment may be implemented, which is not described herein again, and beneficial effects of using the same method are also described herein without details. It will be understood that the program instructions may be deployed to be executed on one server or on multiple servers that are capable of communicating with each other.

The computer readable storage medium may be the coding processing device provided in any of the foregoing embodiments or an internal storage unit of the server, such as a hard disk or a memory of the server. The computer readable storage medium may also be an external storage device of the server, such as a plug-in hard disk (hdd), a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash memory card (flash card), and the like, provided on the server. Further, the computer-readable storage medium may also include both an internal storage unit of the server and an external storage device. The computer readable storage medium is used for storing the computer program and other programs and data required by the server. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.

In one aspect of the application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the server reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the server executes the method provided by one aspect of the embodiment of the present application.

In one aspect of the present application, another computer program product is provided, which includes a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the steps of the encoding processing method provided by the embodiment of the present application are realized.

The steps in the method of the embodiment of the application can be sequentially adjusted, combined and deleted according to actual needs.

The modules in the device can be merged, divided and deleted according to actual needs.

While the invention has been described with reference to a number of embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. An encoding processing method, comprising:

when the decoding abnormal condition existing in the terminal equipment is detected, according to the corresponding relation between different decoding abnormal conditions and the coding strategy, taking the coding strategy corresponding to the decoding abnormal condition as a recommended coding strategy; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different;

acquiring a reference decoding capability which can be provided by the terminal equipment when the terminal equipment performs decoding processing on the coded data generated by the recommended coding strategy, and acquiring an expected decoding capability which the terminal equipment needs to have when the terminal equipment performs normal decoding processing on the coded data generated by the recommended coding strategy;

and matching the reference decoding capacity and the expected decoding capacity to obtain a matching result, and when the matching result indicates that the reference decoding capacity and the expected decoding capacity are successfully matched, adopting the recommended coding strategy to code the game frames to be processed in the target cloud game.

2. The method of claim 1, wherein the detecting manner of the abnormal decoding condition existing in the terminal device comprises:

sending network detection data to terminal equipment in a time window, wherein the time window comprises a plurality of time points, and one or more network detection data are sent to the terminal equipment at one time point;

acquiring detection feedback data corresponding to the network detection data sent at each time point of the time window, and generating a network detection result at the corresponding time point according to the detection feedback data; the network detection result corresponding to any time point is used for indicating whether network fluctuation occurs at any time point;

and counting the times of network fluctuation in the time window according to the network detection results respectively corresponding to each time point of the time window, and determining the decoding abnormal condition of the terminal equipment according to the counted times.

3. The method as claimed in claim 2, wherein said determining the decoding abnormal condition existing in the terminal device according to the counted number comprises:

acquiring a fluctuation index value;

when the counted times is more than or equal to the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent fluctuation abnormity;

and when the counted times are more than or equal to 1 and less than the fluctuation index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental fluctuation abnormality.

4. The method of claim 1, wherein the detecting manner of the abnormal decoding condition existing in the terminal device comprises:

acquiring load information of terminal equipment at different time points of a time window, wherein the load information is the data volume to be decoded of the terminal equipment at corresponding time points;

determining the overload times of the terminal equipment in the time window according to the load information respectively corresponding to the terminal equipment at different time points of the time window; when the data volume to be decoded of the terminal equipment at any time point is larger than or equal to a data volume threshold value, the terminal equipment is overloaded once at any time point;

determining the decoding abnormal condition of the terminal equipment according to the overload times;

when the overload frequency reaches an overload index value, determining that the decoding abnormal condition existing in the terminal equipment is frequent overload, and when the overload frequency is more than 1 and does not reach the overload index value, determining that the decoding abnormal condition existing in the terminal equipment is accidental overload.

5. The method of any of claims 1-4, wherein the coding strategies include a layered coding strategy and a frame-skipping coding strategy;

when the decoding abnormal condition is an abnormal condition of network fluctuation, the corresponding recommended coding strategy comprises the layered coding strategy; and when the decoding abnormal condition is an overload abnormal condition, the corresponding recommended coding strategy comprises the frame skipping coding strategy.

6. The method of claim 1, wherein the obtaining of the reference decoding capability that the terminal device can provide when performing decoding processing on the encoded data generated by the recommended encoding policy comprises:

determining a communication interface corresponding to a decoder from the terminal device, and acquiring description information of the decoder through the communication interface, wherein the description information includes: information indicating whether the decoder supports decoding processing of the encoded data generated by the recommended encoding policy, and information indicating the number of decoding reference frames provided when supported;

and when the description information indicates that the decoder supports decoding processing on the coded data generated by the recommended coding strategy, taking the number of decoding reference frames provided by the description information as the static decoding capability provided by the terminal equipment.

7. The method of claim 1, wherein the obtaining of the reference decoding capability that the terminal device can provide when performing decoding processing on the encoded data generated by the recommended encoding policy comprises:

sending a first detection code stream generated by adopting a layered coding strategy and a second detection code stream generated by adopting a frame skipping coding strategy to the terminal equipment, wherein the first detection code stream and the second detection code stream both comprise a plurality of detection segments, and different detection segments are obtained by coding with different coding reference frame numbers;

acquiring a first decoding result for respectively decoding each detection segment of the first detection code stream and a second decoding result for decoding each detection segment of the second detection code stream from the terminal equipment;

and generating the dynamic decoding capability which can be provided by the terminal equipment according to the first decoding result and the second decoding result.

8. The method of claim 7, wherein the generating the dynamic decoding capability that can be provided by the terminal device according to the first decoding result and the second decoding result comprises:

storing the decoding description information of the terminal equipment on each detection segment of the first detection code stream recorded in the first decoding result and the number of coding reference frames corresponding to each detection segment of the first detection code stream in an associated manner as the dynamic decoding capability provided by the terminal equipment when a layered coding strategy is supported;

storing the decoding description information of the terminal device to each detection segment of the second detection code stream recorded in the second decoding result and the number of the coding reference frames corresponding to each detection segment of the second detection code stream in an associated manner as the dynamic decoding capability which can be provided by the terminal device when the frame skipping coding strategy is supported; wherein the decoding description information is used for indicating whether the terminal device successfully decodes the corresponding detection segment.

9. The method of claim 1, wherein if the recommended coding strategy is a layered coding strategy, the method further comprises:

acquiring support information about the layered coding strategy, and triggering and executing matching processing on the reference decoding capability and the expected decoding capability when the support information of the layered coding strategy indicates that the layered coding strategy is supported;

and when the support information on the hierarchical coding strategy indicates that the hierarchical coding strategy is not supported, taking the frame skipping coding strategy as a new recommended coding strategy.

10. The method of claim 1, wherein if the recommended coding strategy is a frame skip coding strategy, the method further comprises:

determining the reference number corresponding to the reference frame to be shielded and a reference frame list corresponding to the target cloud game according to the current network condition of the terminal equipment;

when the frame skipping coding strategy is determined to be supported, determining whether the reference frames with the reference number in the target cloud game are shielded or not according to the total number of the reference frames contained in the reference frame list and the reference number;

triggering to perform matching processing on the reference decoding capability and the expected decoding capability when the reference frames of the reference number in the target cloud game are determined to be supported to be shielded;

otherwise, selecting a new coding reference frame from the processed game frames of the target cloud game, and taking the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frame of the target cloud game, so as to perform coding processing on the to-be-processed game frame based on the immediate refresh frame.

11. The method of claim 1, wherein the reference decoding capabilities that the terminal device can provide include: static decoding capabilities and available dynamic decoding capabilities; the matching the reference decoding capability and the expected decoding capability to obtain a matching result includes:

acquiring the number of decoding reference frames expected by the expected decoding capability indication, and acquiring the number of decoding reference frames available by the terminal equipment from the static decoding capability;

a quantity matching result obtained by quantity matching of the number of decoding reference frames which can be provided by the terminal equipment and the number of expected decoding reference frames is used as a matching result of the static decoding capability and the expected decoding capability;

matching target detection fragments with the number of coding reference frames being more than or equal to the expected number of decoding reference frames and decoding description information of the target detection fragments from available dynamic decoding capabilities, and taking the matched decoding description information as a matching result of the dynamic decoding capabilities and the expected decoding capabilities.

12. The method of claim 11, wherein the method further comprises:

when the number matching result indicates that the number of decoding reference frames that can be provided by the terminal device is greater than or equal to the number of expected decoding reference frames and the matched decoding description information indicates that the terminal device successfully decodes the target sounding segment, it is determined that the matching result indicates that the reference decoding capability and the expected decoding capability are successfully matched.

13. The method of any one of claims 1-12, further comprising:

and when the matching result indicates that the reference decoding capability and the expected decoding capability are unsuccessfully matched, selecting a new coding reference frame from the processed game frames of the target cloud game, and using the selected new coding reference frame as an immediate refresh frame of the to-be-processed game frames of the target cloud game so as to perform coding processing on the to-be-processed game frames based on the immediate refresh frame.

14. An encoding processing apparatus characterized by comprising:

the determining module is used for taking the coding strategy corresponding to the abnormal decoding condition as a recommended coding strategy according to the corresponding relation between different abnormal decoding conditions and the coding strategy when the abnormal decoding condition existing in the terminal equipment is detected; the coding strategies comprise at least two types, and the recommended coding strategies corresponding to different decoding abnormal conditions are different;

an obtaining module, configured to obtain a reference decoding capability that the terminal device can provide when decoding the encoded data generated by the recommended encoding policy, and obtain an expected decoding capability that the terminal device needs to have when the terminal device performs normal decoding on the encoded data generated by the recommended encoding policy;

the matching module is used for matching the reference decoding capacity and the expected decoding capacity to obtain a matching result;

and the coding module is used for adopting the recommended coding strategy to code the game frame to be processed in the target cloud game when the matching result indicates that the reference decoding capability is successfully matched with the expected decoding capability.

15. A server, comprising: a processor, memory, and a network interface;

the processor is connected with the memory and the network interface, wherein the network interface is used for providing a network communication function, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the encoding processing method of any one of claims 1-13.

16. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, perform the encoding processing method of any one of claims 1-13.