CN113286146A

CN113286146A - Media data processing method, device, equipment and storage medium

Info

Publication number: CN113286146A
Application number: CN202110845902.5A
Authority: CN
Inventors: 李志成
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2021-08-20
Anticipated expiration: 2041-07-26
Also published as: CN113286146B

Abstract

The embodiment of the application provides a media data processing method, a device, equipment and a storage medium, which are applicable to the fields of cloud games, artificial intelligence, computer technology, block chains, cloud computing and the like. The method comprises the following steps: responding to an access request of a display application, coding media data based on a first coding parameter to obtain first coded data, and sending the first coded data to the display application based on a first transmission code rate; in the process of sending the coded data, determining a first type of display content applied at a first moment, and determining a second transmission code rate and a second coding parameter based on the first type; and coding the media data after the first moment based on the second coding parameter to obtain second coded data, and sending the second coded data to the display application based on the second transmission code rate. By adopting the embodiment of the application, the transmission code rate and the coding parameters can be adjusted in real time based on the type of the display content, and the display effect of the display content is improved.

Description

Media data processing method, device, equipment and storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to a method, an apparatus, a device, and a storage medium for processing media data.

Background

With the continuous development of cloud technology, in order to enable applications with relatively limited graphics processing and data computing capabilities to have high-quality display effects, data such as audio and video are often processed based on cloud services, and finally processed encoded data are sent to display applications, so that the display applications can analyze the encoded data and can directly play the audio and video.

Taking a cloud game as an example, the encoding parameters and the transmission code rate of media data in the existing cloud game solution are always configured according to the resolution and the refresh rate, for example, the transmission code rate of 5Mbps and a certain encoding parameter are configured according to the resolution 720P and the refresh rate 30FPS, on one hand, the bandwidth resources of a server are wasted under the condition of low requirements on a game picture, and on the other hand, the presentation of a high-quality game picture cannot be ensured under the conditions that the game picture has high requirements, the transmission code rate is low and the encoding parameters do not meet the picture quality requirements.

Disclosure of Invention

The embodiment of the application provides a media data processing method, a device, equipment and a storage medium, which can adjust a transmission code rate and a coding parameter in real time, further improve a display effect of display application in real time, save bandwidth resources and have high applicability.

In one aspect, an embodiment of the present application provides a media data processing method, where the method includes:

responding to an access request of a display application, coding media data based on a first coding parameter to obtain first coded data, and sending the first coded data to the display application based on a first transmission code rate;

in the process of sending coded data, determining a first type of display content of the display application at a first moment, and determining a second transmission code rate and a second coding parameter based on the first type;

and encoding the media data after the first time based on the second encoding parameter to obtain second encoded data, and transmitting the second encoded data to the display application based on the second transmission code rate.

On the other hand, an embodiment of the present application provides a media data processing apparatus, where the media data processing apparatus includes:

the data coding module is used for responding to an access request of a display application, coding media data based on a first coding parameter to obtain first coded data, and sending the first coded data to the display application based on a first transmission code rate;

the parameter determining module is used for determining a first type of the display content of the display application at a first moment in the process of sending the coded data, and determining a second transmission code rate and a second coding parameter based on the first type;

the data encoding module is configured to encode the media data after the first time based on the second encoding parameter to obtain second encoded data, and send the second encoded data to the display application based on the second transmission code rate.

In another aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the processor and the memory are connected to each other;

the memory is used for storing computer programs;

the processor is configured to execute the media data processing method provided by the embodiment of the application when the computer program is called.

In another aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and the computer program is executed by a processor to implement the media data processing method provided by the embodiment of the present application.

In another aspect, embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the media data processing method provided by the embodiment of the application.

In the embodiment of the application, the transmission code rate and the coding parameters are adjusted by determining the type of the display content of the display application in the process of sending the coded data, so that the transmission code rate and the coding parameters which accord with the display content are determined, the display effect of the display application is improved in real time, the bandwidth resource is saved, and the adaptability is high.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic network structure diagram of a media data processing method according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a media data processing method according to an embodiment of the present application;

FIG. 3 is a schematic view of a scene for determining a sequence of image features according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a scenario for determining a type of display content provided by an embodiment of the present application;

fig. 5 is another schematic flow chart of a media data processing method provided in an embodiment of the present application;

fig. 6 is a scene schematic diagram of a cloud game data processing method provided in an embodiment of the present application;

fig. 7 is a schematic structural diagram of a media data processing apparatus according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The media data processing method provided by the embodiment of the application can be suitable for the fields of Cloud games (Cloud Gaming), image processing, Artificial Intelligence (AI), computer technology, block chains, Cloud Computing (Cloud Computing) and the like, the corresponding transmission code rate and the corresponding coding parameters can be adjusted in real time based on the type change of display contents of display application, and the related data processing processes can be realized based on the Cloud Computing, so that the visual experience of a user is improved, and the utilization rate of data processing resources and bandwidth resources is improved.

The cloud game may also be called a game on demand (gaming), which is an online game technology based on a cloud computing technology. Cloud game technology enables light-end devices (thin clients) with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud game scene, a game is not operated in a player game terminal but in a cloud server, and the cloud server renders the game scene into a video and audio stream which is transmitted to the player game terminal through a network. The player game terminal does not need to have strong graphic operation and data processing capacity, and only needs to have basic media playing capacity and the capacity of acquiring player input instructions and sending the instructions to the cloud server.

The cloud game server instance maps the received operation instructions into corresponding game mouse keys, and then sends the corresponding game mouse keys to a real cloud game server through the keyboard and mouse drive to complete the whole game service experience.

Referring to fig. 1, fig. 1 is a schematic network structure diagram of a media data processing method according to an embodiment of the present application. As shown in fig. 1, the terminal 10 is a display terminal corresponding to a display application, and includes, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart television, and the like. The display application comprises a cloud game application, a rendering application and other applications capable of displaying display content.

The server 20 is a server corresponding to a display application run by the terminal 10, such as a cloud server corresponding to a cloud game application, a rendering processor corresponding to a rendering application, and the like, and the server 20 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services, which is not limited herein. The display application is directly or indirectly connected through wired or wireless communication between the terminal 10 and the server 20, and is not limited in the embodiment of the present application.

In particular implementations, the server 20 may be responsive to an access request by a display application executed by the terminal 10 to establish a communication connection with the display application. Further, the server 20 may respond to the media data acquisition request of the display application and acquire the corresponding media data according to the media data acquisition request. Alternatively, the server 20 may acquire the media data required for the default configuration information in response to an access request of the display application based on the default configuration information. And if the cloud game server responds to the access request of the cloud game application, the media data of the game picture corresponding to the cloud game application is obtained.

Further, after acquiring the to-be-media data, the server 20 may encode the media data based on the first encoding parameter to obtain first encoded data, and send the first encoded data to the display application through the terminal 10 based on the first transmission code rate. After the display application receives the first encoded data through the terminal 10, the decoded data can be obtained by decoding the first encoded data, so that the corresponding display content is displayed based on the decoded data. After the cloud game server obtains the picture data of the game picture, the picture data is coded based on the first coding parameter to obtain first coding data corresponding to the game picture, and the first coding data is sent to the cloud game application based on the first transmission code rate (for example, 5 Mbps). After receiving the first encoded data, the cloud game application may decode the first encoded data to obtain decoded data, and then display a corresponding game picture based on the decoded data.

In the process that the server 10 sends the encoded data to the display application through the terminal 10, the first type of the display content of the display application at the first time can be determined, so that the encoding parameter of the media data after the first time and the corresponding transmission code rate of the encoded data are adjusted according to the first type of the display content of the display application at the first time. That is, after determining a first type of display content to display the application at the first time, determining a second transmission code rate and a second encoding parameter based on the first type. And then, the media data after the first time is encoded based on the second encoding parameter to obtain second encoded data, and the second encoded data is sent to the display application through the terminal 10 based on the second transmission code rate. Likewise, the display application decodes the second encoded data after the first time and displays the corresponding display content after the first time based on the decoded data.

Based on this, the server 20 may adjust the encoding parameters and the transmission code rate in real time based on the type of the display content of the display application at any time, thereby achieving timely adjustment of the encoding parameters of the media data and the transmission code rate for transmitting the encoded data.

Referring to fig. 2, fig. 2 is a schematic flow chart of a media data processing method according to an embodiment of the present application. As shown in fig. 2, a media processing method provided in an embodiment of the present application may include the following steps:

step S21, in response to the access request of the display application, encoding the media data based on the first encoding parameter to obtain first encoded data, and sending the first encoded data to the display application based on the first transmission code rate.

The display application in the embodiment of the application can display the display contents with different image quality requirements, such as the display contents with different resolutions and different refresh rates. The higher the resolution of the display content is, the clearer the display content displayed by the display application is, the higher the refresh rate is, the smoother the display content displayed by the display application is, and the higher the stability is.

As an example, the display application in the embodiment of the present application is a cloud game application, and if the cloud game application includes a plurality of cloud games, the display application in the embodiment of the present application may display game screens of different game scenes of the various cloud games.

In some possible embodiments, in response to an access request of the display application, the corresponding media data may be acquired, so as to send the encoded data of the media data to the display application, so that the display application displays the corresponding display content of the media data based on the encoded data.

The access request of the display application may be a request for establishing a communication connection with a corresponding server after the display application is started, may also be a request for requesting the server to send media data to the display application, and may also be a request for triggering the server to send media data to the display application in any scene, which may be specifically determined based on the requirements of an actual application scene, and is not limited herein.

As an example, in response to an access request for establishing a communication connection, preset media data is obtained, and encoded data corresponding to the preset media data is sent to a display application. The preset media data includes, but is not limited to, media data corresponding to an application start screen, media data corresponding to an initial game scene of the cloud game, and the like.

As an example, in response to an access request requesting to send media data, the media data requested to be sent by the access request is obtained, and encoded data corresponding to the media data requested to be sent by the access request is sent to a display application. For example, in response to an access request for a first video sent by a video playing application, media data corresponding to the first video is acquired, and encoded data of the media data corresponding to the first video is sent to a display application. For another example, in response to an access request for a first game scene sent by the cloud game application, scene picture data of the first game scene is acquired, and encoded data corresponding to the scene picture data of the first game scene is sent to the cloud game application.

In some possible embodiments, after the access request of the display application is responded and the media data is acquired, an initial encoding parameter for encoding the media data and an initial transmission code rate for transmitting encoded data obtained by encoding the media data based on the initial encoding parameter can be determined. For convenience of description, the initial encoding parameter and the initial transmission code rate are hereinafter referred to as a first encoding parameter and a first transmission code rate, respectively.

After the media data are acquired, the media data can be coded in real time, and meanwhile coded data obtained through coding are sent to the display application in real time. The first encoding parameter is an encoding parameter used when media data is encoded at an initial time, and the first transmission code rate is a transmission code rate used when encoded data obtained based on the first encoding parameter is transmitted to a display application.

The encoding parameters in the embodiment of the present application are parameters of various encoding modes used when encoding media data, including but not limited to parameters such as ref, keyint, aq-mode, me, sub, aq-strength, qp, vbv-maxrate, ipratio, pbratio, and deblock, and may be determined specifically based on requirements of an actual application scene, and are not limited herein.

Wherein the ref parameter is used to indicate the number of reference frames of a forward predictive coded frame (P-frame) of media data in the encoding process; the keyint parameter is used for indicating the interval between key frames of the media data in the encoding process, and the compression rate is positively correlated with the value of the keyint parameter; the aq-mode parameter is used to indicate the adaptive quantization mode in the encoding process; the aq-strength parameter is used for indicating the quantization strength in the encoding process; the me parameter is used for indicating a full-pixel motion prediction mode in the encoding process; the sub-me parameter is used for indicating the sub-pixel estimation complexity in the encoding process; the qp parameter is used for indicating a code rate control mode; the VBV-maxrate parameter is used for indicating the maximum buffer space of a Video Buffering Verifier (VBV) in the encoding process; the deblocking parameter is used for indicating the intensity of a color block generated in the process of removing the coding; the ipratio parameter and pbratio parameter are used to determine the quantization parameters for intra-coded frames (I-frames) and bi-directionally predictive coded frames (B-frames) during the encoding process.

It should be particularly noted that the above-mentioned encoding parameters are only examples, and may be specifically determined based on the actual application scene requirements and the actual encoding mode, and are not limited herein. The media data can be coded in different coding modes based on coding parameters obtained by different combinations of various configuration parameters, so that the final coding parameters can correspond to display contents with different resolutions, refresh rates and other display effects.

The transmission code rate in the embodiment of the application is a data transmission code rate adopted when the coded data is sent to the display application, namely the number of data bits transmitted in unit time in the process of sending the coded data to the display application, and the transmission code rate can be generally measured by Kbps, Mbps and the like. Such as sending the encoded data to the display application at a transmission rate of 8 Mbps.

In some possible embodiments, when determining the first encoding parameter and the first transmission rate, the initial display configuration information of the display application may be determined, and then the first transmission rate and the first encoding parameter may be determined based on the initial display configuration information of the display application. The initial display configuration information of the display application is used to describe display requirements of the display application when displaying the display content at the initial time, including but not limited to picture resolution and/or picture refresh rate, which may be determined based on actual application scene requirements, and is not limited herein.

Specifically, the initial display configuration information of the display application may be preset display configuration information, and after receiving an access request of the display application, the first encoding parameter and the first transmission code rate may be determined based on the preset display configuration information.

The preset display configuration information may be display configuration information of the lowest display requirement that can be achieved by the display application, may also be display configuration information of the highest display requirement that can be achieved by the display application, may also be default display configuration information of a server corresponding to the display application, and the like, and may be specifically determined based on actual application scene requirements, which is not limited herein.

Optionally, the initial display configuration information may be obtained from the display application upon determining the first encoding parameter and the first transmission code rate. For example, after receiving an access request of a display application, initial display configuration information carried by the access request may be acquired, and then a first encoding parameter and a first transmission code rate may be determined based on the initial display configuration information. Or after receiving an access request of the display application, sending a configuration confirmation request to the display application, and further determining initial display configuration information based on response information of the display application to the configuration confirmation request, so as to determine the first encoding parameter and the first transmission code rate based on the initial display configuration information.

Optionally, different media data may correspond to different initial display configuration information, and after the media data is acquired, the initial display configuration information corresponding to the media data may be determined to determine the first encoding parameter and the first transmission code rate based on the initial display configuration information.

As an example, the access request is an access request of a video playing application for a second video, and after media data corresponding to the second video is acquired, initial display configuration information corresponding to the media data of the first video may be acquired.

As an example, in response to an access request of a cloud game application for a first cloud game, media data of a game screen corresponding to the first cloud game is acquired, and initial display configuration information corresponding to the media data, that is, initial display configuration information corresponding to the first cloud game, is determined.

In some possible embodiments, after the initial display configuration information is determined, the first encoding parameter and the first transmission code rate may be determined based on the initial display configuration information and the corresponding relationship between the encoding parameter and the transmission code rate.

Specifically, a data processing parameter set may be obtained, where the data processing parameter set includes a correspondence between display configuration information and coding parameters and transmission code rates, and specific coding parameters and transmission code rates corresponding to different display configuration information. After determining the initial display configuration information, a first encoding parameter and a first transmission code rate may be determined from the set of data processing parameters based on a correspondence of the initial display configuration information to the encoding parameter and the transmission code rate.

As an example, the initial display configuration information includes a resolution, a first encoding parameter is determined from encoding parameters corresponding to the resolution included in the initial display configuration information from the data processing parameter set based on a corresponding relationship between the resolution and the encoding parameter, and a first transmission rate is determined from transmission rates corresponding to the resolution included in the initial display configuration information from the data processing parameter set based on a corresponding relationship between the resolution and the transmission rate.

The corresponding relationship between the resolution and the coding parameter and the corresponding relationship between the resolution and the transmission code rate are independent, that is, each resolution may correspond to at least one coding parameter or at least one transmission code rate. If the resolution included in the initial display configuration information is 720p, the first encoding parameter may be determined from at least one encoding parameter corresponding to the 720p resolution, and the first transmission code rate may be determined from at least one transmission code rate corresponding to the 720p resolution.

Alternatively, each resolution may correspond to a combination of at least one coding parameter and a transmission code rate, in which case, the combination of at least one coding parameter and a transmission code rate corresponding to the resolution included in the initial display configuration information may be determined from the data processing parameter set, and a set of coding parameters and a set of transmission code rates may be determined from the combination of at least one coding parameter and a transmission code rate corresponding to the resolution included in the initial display configuration information. And further determining the coding parameter in the combination as a first coding parameter, and determining the transmission code rate in the combination as a first transmission code rate. If the resolution included in the initial display configuration information is 1080p, at least one combination of the encoding parameters and the transmission rate corresponding to the 1080p resolution can be determined, and if the 1080p resolution can be determined to correspond to the two combinations of the encoding parameters 1 and 5Mbps and the encoding parameters 2 and 7Mbps, a set of the encoding parameters and the transmission rate (e.g., the encoding parameters 2 and 7 Mbps) can be determined. Further, the coding parameter 2 in the combination is determined as a first coding parameter, and 7Mbps in the combination is determined as a first transmission code rate.

As an example, the initial display configuration information includes a refresh rate, a first encoding parameter is determined from encoding parameters corresponding to the refresh rate included in the initial display configuration information from the data processing parameter set based on a corresponding relationship between the refresh rate and the encoding parameter, and a first transmission code rate is determined from transmission code rates corresponding to the refresh rate included in the initial display configuration information based on a corresponding relationship between the refresh rate and the transmission code rate.

The corresponding relationship between the refresh rate and the coding parameters and the corresponding relationship between the refresh rate and the transmission code rate are independent, that is, each refresh rate may correspond to at least one coding parameter or at least one transmission code rate. If the refresh rate included in the initial display configuration information is 120FPS, a first encoding parameter can be determined from at least one encoding parameter corresponding to the refresh rate of 120FPS, and a first transmission code rate can be determined from at least one transmission code rate corresponding to the refresh rate of 120 FPS.

Or, each refresh rate and the combination of at least one coding parameter and transmission code rate simultaneously correspond to, in this case, the combination of at least one coding parameter and transmission code rate corresponding to the refresh rate included in the initial display configuration information may be determined from the data processing parameter set, and a set of coding parameters and transmission code rate may be determined from the combination of at least one coding parameter and transmission code rate corresponding to the refresh rate included in the initial display configuration information. And further determining the coding parameter in the combination as a first coding parameter, and determining the transmission code rate in the combination as a first transmission code rate. If the refresh rate included in the initial display configuration information is 60FPS, at least one combination of the encoding parameter and the transmission code rate corresponding to the 60FPS refresh rate can be determined, and if the 60FPS refresh rate can be determined to correspond to two combinations of the encoding parameters 1 and 5Mbps and the encoding parameters 2 and 7Mbps, a set of the encoding parameters and the transmission code rate (e.g., the encoding parameters 1 and 5 Mbps) can be determined from the refresh rate. Further, the coding parameter 1 in the combination is determined as a first coding parameter, and 5Mbps in the combination is determined as a first transmission code rate.

As an example, the initial display information includes a resolution and a refresh rate, the first encoding parameter is determined from the encoding parameters corresponding to a combination of the resolution and the refresh rate based on the corresponding relationship between the combination and the encoding parameters, and the first transmission code rate is determined from the transmission code rate corresponding to the combination based on the corresponding relationship between the combination of the resolution and the refresh rate and the transmission code rate.

The combination of the resolution and the refresh rate and the corresponding relationship of the coding parameters, and the combination of the resolution and the refresh rate and the corresponding relationship of the transmission code rate are independent corresponding relationships, that is, each combination of the resolution and the refresh rate can correspond to at least one coding parameter, and also can correspond to at least one transmission code rate. If the resolution included in the initial display configuration information is 1080p and the refresh rate is 120FPS, a first encoding parameter can be determined from at least one encoding parameter corresponding to a combination of the 1080p resolution and the 120FPS refresh rate, and a first transmission rate can be determined from at least one transmission rate corresponding to a combination of the 1080p resolution and the 120FPS refresh rate.

Alternatively, each combination of resolution and refresh rate may correspond to at least one combination of coding parameter and transmission code rate, in which case a combination of at least one coding parameter and transmission code rate corresponding to the combination of resolution and refresh rate in the initial display information may be determined from the set of data processing parameters, and a set of coding parameters and transmission code rate may be determined therefrom, and further a coding parameter in the combination of coding parameter and transmission code rate may be determined as a first coding parameter, and a transmission code rate in the combination may be determined as a first transmission code rate. If the initial display configuration information includes a resolution of 720p and a refresh rate of 60FPS, at least one combination of encoding parameters and transmission rate corresponding to the 720p resolution and 60FPS refresh rate combination can be determined, and if the 720p resolution and 60FPS refresh rate combination can be determined to correspond to two combinations of encoding parameters 3 and 10Mbps and encoding parameters 4 and 20Mbps, a set of encoding parameters and transmission rate (e.g., encoding parameters 4 and 20 Mbps) can be determined. Further, the coding parameter 4 in the combination is determined as the first coding parameter, and 20Mbps in the combination is determined as the first transmission code rate.

Optionally, after determining the initial display configuration information of the display application, the first encoding parameter may be determined based on the initial display configuration information and by the above implementation. And determining the network environment information at the same time, and determining a first transmission code rate from the transmission code rates supported by the network environment information. The network environment information includes, but is not limited to, a maximum transmission code rate supported by the display application, a maximum transmission code rate supported by a server corresponding to the display application, and network bandwidth information, and may be specifically determined based on requirements of an actual application scenario, which is not limited herein.

Optionally, after the initial display configuration information is determined, at least one coding parameter and at least one transmission code rate that support a display requirement of the initial display configuration information when displaying content at an initial time may be directly determined based on the initial display configuration information, and then any one of the determined at least one coding parameter is determined as a first coding parameter, and any one of the determined at least one transmission code rate is determined as a first transmission code rate. If the initial display configuration information comprises the resolution and the refresh rate of the display content, the encoding parameters and the transmission code rate which meet the resolution requirement and the refresh rate requirement indicated by the initial display configuration information can be determined, the first encoding parameters are determined from the encoding parameters which meet the resolution requirement and the refresh rate requirement, and the first transmission code rate is determined from the transmission code rate which meets the resolution requirement and the refresh rate requirement.

In some possible embodiments, when determining the first encoding parameter and the first transmission rate based on the initial display configuration information, a type of the initial display content of the display application (hereinafter, referred to as a second type for convenience of description) may also be determined, and then the first encoding parameter and the first transmission rate are determined from the data processing parameter set based on the second type of the initial display content and the initial display configuration information.

The data processing parameter set comprises the type of the display content, configuration information, the corresponding relation between the coding parameters and the transmission code rate, and each specific coding parameter and transmission code rate.

Specifically, the second type of the display content may be determined based on an application type of the display application, and if the application type of the display application may include a 2D application type, a 3D application type, a Virtual Reality (VR) type, an Augmented Reality (AR) type, and the like, the second type of the corresponding display content may be a 2D type, a 3D type, a VR type, an AR type, and the like.

Further, the application type of the display application may be determined based on the function requirement of the display application, and if the display application is a cloud Game application, the application type of the cloud Game application may be further divided into a Multiplayer Online tactical Game (MOBA), a Role-playing Game (RPG), an Action (Action, ACT) Game, a Time Adventure Game (AVG), an Action Adventure Game (ACT), an AAG), a Strategy Game (Game play, SG), a Simulation-type Role-playing Game (Game play, SPRG), a Real-Time Strategy Game (Real-Time Strategy Game, RTS), a Real-Time tactical Game (RTT-one-person games, a combat Game (combat, Game), a Shooting Game (Game play, gun), a Shooting Game (First Shooting view), FPS), Third person perspective Shooting Game (TPS), Puzzle (PZL) Game, Sports Game (Sports Game, SPG), Racing Game (RCG), Card Game (CAG), Table Game (TAB) and the like. Based on this, an application type of the cloud game application may be determined as a type of display content of the display application.

Further, in the case that the display application is a cloud game application, further division may be performed based on a division manner of an application type of the cloud game application. For example, motion games can be classified into 2D motion games (ACT 2D) and 3D motion games (ACT 3D), or further classified into low-level 2D motion games, middle-level 2D motion games, and high-level 2D motion games, for example, for 2D motion games, screen texture complexity, motion variation width, and the like of display contents.

It should be particularly noted that the division of the application types of the display application is only an example, and may be determined based on the actual application scenario requirements, and is not limited herein.

Specifically, for the application type of the display application, application information of the display application may be acquired after receiving an access request of the display application, and the second type of the initial display content may be determined based on the application information of the display application.

The application information of the display application may include, but is not limited to, application identification information, game identification information, identification information of a terminal running the display application, and the like. An application type of the cloud game application may be determined based on the game scene identification, which may in turn determine a second type of the initial display content based on the application type of the cloud game application, a corresponding application type may be determined based on the application identification, such as an application type of the rendering application, a game type of the cloud game application, etc., which may in turn determine the second type of the initial display content based on the application type. It should be noted that the initial display content is the content that the display application originally displays after sending the access request, and the second type of the initial display content may directly correspond to the application type of the display application.

In particular, for display content, the same type of display content may correspond to different encoding parameters and transmission code rates when corresponding to different display configuration information. In other words, for display configuration information, the same display configuration information may correspond to different combinations of encoding parameters and transmission code rates, and each combination of encoding parameters and transmission code rates corresponds to a type of display content.

The corresponding relation between the combination of the coding parameters and the transmission code rates and the type of the display content, and the corresponding relation between the combination of the coding parameters and the transmission code rates and the display configuration information can be stored in the data processing parameter set. Based on this, after the initial display configuration information and the second type of the initial display content are determined, the data processing parameter set may be determined first, and the encoding parameter and the transmission code rate corresponding to the initial display configuration information are determined based on the data processing parameter set, so that the encoding parameter and the transmission code rate corresponding to the second type in the encoding parameter and the transmission code rate corresponding to the initial display configuration information are determined as the first encoding parameter and the first transmission code rate, respectively.

Or, the encoding parameter and the transmission code rate corresponding to the second type may be determined from the data processing parameter set, and then the encoding parameter and the transmission code rate corresponding to the initial display configuration information in the encoding parameter and the transmission code rate corresponding to the second type are determined as the first encoding parameter and the first transmission code rate.

As an example, table 1 shows a correspondence relationship between the type of display content, the display configuration information, the encoding parameter, and the transmission code rate in the data processing parameter set.

TABLE 1

In table 1, different types of display contents correspond to different transmission code rates when corresponding to different combinations of resolutions and refresh rates, and different types of display contents correspond to different encoding parameters. If the initial display configuration information of the display application includes a resolution of 1080P, a refresh rate of 30FPS, and the second type of the initial display frame of the display application is a 2D action game (ACT [2D ]) type, it is determined that the first transmission rate is 5.0Mbps and the first encoding parameter is parameter configuration 1 based on the data processing parameter set.

In some possible embodiments, after determining the first encoding parameter and the first transmission code rate, the media data may start to be encoded based on the first encoding parameter to obtain encoded data (hereinafter, referred to as first encoded data for convenience of description), and then send the first encoded data to the display application based on the first transmission code rate, that is, send the first encoded data to the display application at the first transmission code rate.

In some feasible embodiments, the data processing parameter set may be stored in a cloud server corresponding to the display application, or a cloud storage space, a Database Management System (DBMS), a block chain, and the like corresponding to the display application, and may be specifically determined based on an actual application scenario requirement, which is not limited herein.

The cloud storage space can use a large number of storage devices of various types as a storage space of a data processing parameter set through functions of cluster application, grid technology, distributed storage file system and the like. In short, the database may pre-store each parameter in the data processing parameter set in the database system. A blockchain is a distributed data store, essentially a decentralized database, a chain of cryptographically related generated data blocks, each of which may be used to store a set of data processing parameters.

Step S22, in the process of sending the coded data, determining a first type of the display content applied at a first time, and determining a second transmission code rate and a second coding parameter based on the first type.

In some possible embodiments, in the process of transmitting the encoded data of the media data to the display application, the first encoding parameter and the first transmission code rate before change may not be suitable for encoding the current media data and transmitting the encoded data due to a change in display configuration information of the display application or a change in a type of display content of the display application. If the display content of the cloud game application is changed from the 2D type to the 3D type, if the first encoding parameter is continuously used to encode the media data corresponding to the 3D type display content, an error may be caused in the display content, and since the data amount of the encoded data obtained by encoding the media data corresponding to the 3D type display content is increased, if the first transmission code rate is continuously used to transmit the encoded data, problems such as untimely transmission of the encoded data may be caused. Based on this, at any time (hereinafter referred to as a first time for convenience of description) in the process of sending the encoded data to the display application, a second encoding parameter and a second transmission code rate corresponding to the first time may be determined, the media data after the first time is encoded based on the first encoding parameter to obtain encoded data (hereinafter referred to as a second encoded data for convenience of description), and the second encoded data is sent to the display application based on the second transmission code rate.

Specifically, display configuration information (hereinafter, referred to as first display configuration information for convenience of description) of the display application at the first time may be determined, and then the second encoding parameter and the second transmission code rate may be determined based on the first display configuration information. For a specific implementation manner of determining the second encoding parameter and the second transmission code rate based on the first display configuration information, refer to the implementation manner of determining the first transmission code rate and the first encoding parameter based on the initial display configuration information in step S21 in fig. 1, which is not described herein again.

As an example, if the refresh rate of the display application at the first time is changed from 60FPS to 120PFS, and the resolution at the first time is changed from 720p to 1080p, the second transmission rate and the second encoding parameter need to be determined based on the refresh rate of 60PFS and the resolution of 1080 p.

In some possible embodiments, at a first time during the transmission of the encoded data, a type of display content (hereinafter, referred to as a first type for convenience of description) of the display application at the first time may be determined, and then the second encoding parameter and the second transmission code rate may be determined based on the first type of display content of the display application at the first time.

The first type of the display content displayed at the first time may indicate that the display content displayed after the first time is the first type, and therefore the second encoding parameter and the second transmission code rate determined based on the first type may be continuously used after the first time.

Specifically, at least one frame of display image corresponding to the display application at the first time may be acquired, and the first type of display content of the display application at the first time may be determined based on the at least one frame of display image corresponding to the display application at the first time. Specifically, the type of the display content corresponding to the at least one frame of display image corresponding to the display application at the first time may be determined as the first type of the display content applied at the first time.

The display content corresponding to the display content of the at least one frame of display image applied at the first time is determined as a first type of display content applied at the first time.

Alternatively, the displaying of the at least one frame of display image corresponding to the application at the first time may be displaying of at least one frame of display image that is consecutive after the application at the first time, and the type of the display content corresponding to the displaying of the at least one frame of display image that is consecutive after the application at the first time is determined as the first type of the display content that is displayed at the first time.

Or, the displaying of the at least one frame of display image corresponding to the application at the first time may include at least one frame of display image that is continuous and includes the display image corresponding to the first time, and the display content corresponding to the at least one frame of display image that includes the display image corresponding to the first time is determined as the first type of display content that is applied at the first time. It should be particularly noted that the at least one frame of display image corresponding to the first time may be at least one frame of display image corresponding to any other display content representing the display content at the first time, and may be specifically determined based on the actual application scene requirement, which is not limited herein.

In some possible embodiments, the type of the display content corresponding to the at least one frame of display image is determined based on the at least one frame of display image, and may be specifically determined based on Machine Learning (ML) in the field of artificial intelligence. Artificial intelligence based capabilities can be provided to an intelligent machine to determine a type of corresponding display content based on at least one frame of a display image.

For at least one frame of the acquired display image, the processing may be performed based on one or more Neural Networks such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Long Short-Term Memory Network (LSTM), and the type of the display content corresponding to the at least one frame of the display image is finally obtained.

Specifically, at least one frame of display image may be convolved, filtered, pooled, and the like based on the convolutional neural network, so as to obtain image features corresponding to each display image. For example, for each frame of display image, the display image can be processed through a graph convolution neural network to obtain an initial image feature of the display image, and further, the initial image feature is subjected to dimension reduction processing through a pooling layer to obtain an image feature corresponding to the display image.

In order to better capture the time and space feature information in each display image, three-dimensional convolution can be performed on continuous image features according to a specific time dimension, namely, convolution is performed on feature values of each position of the image features of continuous frames, so that convolution features are obtained, and the convolution features are determined to be the image features of the corresponding display images. Or further processing the convolution characteristics such as maximum pooling, minimum pooling and average pooling to obtain final image characteristics.

Further, an image feature sequence is determined based on the image features corresponding to the display images, and the image feature sequence is used as the input features of the classification neural network. After the image feature sequence is processed by the classification neural network, a classification result corresponding to the image feature sequence can be determined, and then the type of display content corresponding to at least one frame of display image (for aspect description, hereinafter referred to as third Lexi) is determined based on the classification result.

The image feature sequence can be determined by adopting a longitudinal splicing or transverse splicing mode based on the image features corresponding to the display images, or feature fusion can be performed on the image features based on a feature fusion mode, and the fusion features obtained by feature fusion are determined as the image feature sequence.

Referring to fig. 3, fig. 3 is a schematic view of a scene for determining an image feature sequence according to an embodiment of the present application. As shown in fig. 3, a display image sequence is formed by several consecutive frames of display images before the first time, and for any display image in the display image sequence, feature extraction can be performed on the display image based on a convolutional neural network to obtain an image feature corresponding to the display image. Based on this, the image features corresponding to the respective display images can be obtained, and further, the image feature sequence as the input feature of the classification neural network can be obtained based on the image features corresponding to the respective display images.

Referring to fig. 4, fig. 4 is a schematic view of a scenario for determining a type of display content according to an embodiment of the present application. As shown in fig. 4, after the image feature sequence is obtained based on the convolutional neural network, the image feature sequence may be used as an input feature of the classification neural network model, and then the classification neural network performs feature processing on the image feature sequence, and finally a third type of display content corresponding to the continuously displayed image before the first time is obtained through the classifier.

The classification Neural Network includes, but is not limited to, a Recurrent Neural Network (RNN), a Multilayer Perceptron (MLP), a classification model (such as a BERT model) constructed based on the Neural Network, and the like, and may be determined based on actual application scene requirements, which is not limited herein.

In some possible embodiments, since the first encoding parameter and the first transmission code rate are determined based on the initial display configuration information, it may not exactly match the specific display requirements of the display content at the first time. Thus, at a first moment in the transmission of the encoded data, rating information for the display application may also be obtained, based on which a first type of display content for the display application at the first moment is determined.

Specifically, an evaluation information set can be obtained, the evaluation information set includes a plurality of sample evaluation information, each sample evaluation information is labeled with a type label, and the sample label represents an application type of the display application corresponding to the corresponding sample evaluation information. The similarity between the evaluation information of the display application and each sample evaluation information can be further determined, the application type represented by the type label of the sample evaluation information with the highest similarity is determined as the application type of the display application, and then the first type of the display content of the display application at the first moment is determined based on the application type of the display application.

Optionally, the type of the display application may be predicted based on the evaluation information of the display application through a long-term and short-term memory network, a collaborative filtering algorithm, a type prediction model, and the like, and then the first type of the display content of the display application at the first time may be determined based on the predicted type of the display application.

As an example, the evaluation information of the display application may be converted into a feature vector, and the feature vector corresponding to the evaluation information is input into a type prediction model based on the long-term and short-term memory network, so as to obtain a prediction result. And then obtaining the type of the display application based on the prediction result of the type prediction model, and determining the first type of the display content of the display application at the first moment based on the type of the display application. If it is determined based on the prediction result that the type of the display application is the shooting-type game application, the first type of the display content of the display application at the first time is the shooting type.

After determining a first type for displaying the display content applied at the first time based on the above implementation, a second transmission code rate and a second encoding parameter may be determined based on the first type. For example, as shown in step S21 in fig. 1, the implementation manner of determining the first transmission code rate and the first encoding parameter based on the second type of the display content of the display application may be to obtain the display configuration information of the display application at the first time, determine the second transmission code rate and the second encoding parameter based on the display configuration information of the display application at the first time and the first type of the display content at the first time, and determine the second transmission code rate and the second encoding parameter corresponding to the first type and the first display configuration information from the data processing parameter set based on the first display configuration information of the display application at the first time and the first type of the display content at the first time, which is not described herein again.

When the first display configuration information of the display application at the first time is obtained, the related request carrying the first display configuration information sent by the display application at the first time can be obtained, and the specific implementation manner is not repeated herein.

Wherein the first display configuration information of the display application at the first time also includes at least one of a resolution or a refresh rate.

Optionally, when the second encoding parameter and the second transmission rate are determined based on the first type of the display content displayed and applied at the first time, the first type and the second type may also be determined to determine a difference between the display content displayed and applied at the first time and the initial display content, so that the first encoding parameter and the second transmission rate are adjusted based on the difference to obtain the second encoding parameter and the second transmission rate. Such as adjusting the keyint parameter in the encoding parameters to change the interval between key frames of the media data during the encoding process, thereby changing the compression rate of the encoded data.

For example, if the first type of display content displaying the application at the first time is a shooting game type and the second type of display content before that is a strategy game type, a degree of difference between the display content displaying the application at the first time and the previous display content may be determined based on the first type and the second type, and higher refresh rate and resolution may be required for display content such as a shooting type. Based on this, at least one of the first encoding parameters may be adjusted to obtain a second encoding parameter more suitable for high refresh rate and high resolution. In addition, since the display content of the shooting type requires a higher refresh rate and resolution, the data amount of the encoded data after encoding the corresponding media data is larger than before, and therefore, the first transmission rate can be adjusted to obtain the second transmission rate with a larger transmission rate.

For example, if the display application is a game application including a plurality of cloud games, if the initial display content of the display application is a game scene of a shooting game, and the display content of the display application at the first time is a game scene of a card game, due to different display requirements of the game scenes, the display configuration information corresponding to different games is different, so that the corresponding second encoding parameter and the second transmission code rate can be re-determined based on the display configuration information of the display application at the first time.

In some possible embodiments, at least one frame of the display image of the display application may be periodically acquired during the transmission of the encoded data to the display application. If each frame of display image in the preset time interval is obtained at every preset time interval, the coding parameters and the transmission code rate corresponding to the moment are determined through the implementation mode based on each frame of display image, and therefore the coding parameters and the transmission code rate are adjusted in real time.

Step S23, encoding the media data after the first time based on the second encoding parameter to obtain second encoded data, and sending the second encoded data to the display application based on the second transmission code rate.

In some possible embodiments, after determining the second encoding parameter and the second transmission code rate corresponding to the display application at the first time, the media data after the first time may be encoded based on the second encoding parameter to obtain second encoded data, and then the second encoded data is sent to the display application based on the second transmission code rate.

Further, in the process of sending the encoded data to the display application, the type of the display content of the display application at the corresponding time may be determined at preset time intervals, and then a new encoding parameter and a new transmission code rate are determined based on the type of the display content at the time, so as to encode the media data after the time based on the new encoding parameter, and send the encoded data after the time to the display application based on the new transmission code rate. Under the condition that the preset time interval is small, the real-time adjustment of the encoding parameters and the transmission code rate of the media data can be realized, the real-time adjustment of the display content of the display application is further realized, and the display effect and the user experience of the display content are improved.

In some possible embodiments, in order to further make the transmission code rate more meet the actual data transmission requirement, in the process of sending the encoded data to the display application, the transmission code rate may be further adjusted, so as to send the encoded data to the display application based on the adjusted transmission code rate.

A specific adjustment manner for adjusting the transmission code rate during the coded data transmission process can be seen in fig. 5. Fig. 5 is another schematic flow chart of the media data processing method according to the embodiment of the present application, and as shown in fig. 5, the media data processing method according to the embodiment of the present application further includes the following steps:

and S51, acquiring the network environment information of the display application at the second moment in the coded data sending process.

In some possible embodiments, the network environment information of the display application at any time (hereinafter referred to as a second time for convenience of description) may be obtained, that is, the network environment information of the display application may be obtained in real time during the process of transmitting the encoded data to the display application.

The network environment information of the display application may be obtained based on communication interaction information with the display application, such as interaction information based on a Transmission Control Protocol (TCP), report information of the display application, and the like, and may be specifically determined based on a requirement of an actual application scene, which is not limited herein.

Specifically, the network environment information of the display application includes at least one of downlink network information, a maximum reception code RATE, a network delay, or a LOSS RATE (LOSS RATE).

The downlink network information may be used to indicate Bandwidth Estimation (BWE) of the display application, and the network status of the downlink network where the display application receives data may be determined based on the Bandwidth Estimation, the Maximum receiving Bitrate may be used to indicate a Maximum Bitrate (REMB) Estimated by a Receiver of a terminal where the display application is located, based on which the Maximum receiving Bitrate that the display application can match with the fat may be determined, and the packet loss rate may be used to indicate a rate at which the display application loses packets during receiving the encoded data.

And S52, adjusting the transmission code rate corresponding to the second time based on the network environment information.

In some possible embodiments, the corresponding transmission code rate may be adjusted based on the network environment information of the display application at the second time. For example, if the bandwidth of the downlink network to which the received data is applied is shown to be high, the transmission code rate corresponding to the second time may be adaptively increased. If the maximum receiving code rate of the display application is lower, in order to ensure that the display application can normally receive other communication data while receiving the encoded data, the transmission code rate corresponding to the second moment can be adaptively reduced. If the packet loss rate of the display application is higher, the display effect of the display content of the display application is improved for relieving the packet loss rate, and the transmission code rate corresponding to the second moment can be reduced adaptively.

It should be particularly noted that the implementation manner of adjusting the transmission code rate corresponding to the second time based on the network environment information is only an example, and the specific adjustment strategy may be determined based on the actual application scenario requirement, which is not limited herein.

And S53, encoding the media data after the second moment based on the encoding parameters corresponding to the second moment, and sending the encoded data to the display application based on the adjusted transmission code rate.

In some possible embodiments, after the transmission code rate corresponding to the second time of the display application is adjusted, the media data after the second time may be encoded based on the encoding parameter corresponding to the second time of the display application, and the encoded data may be sent to the display application based on the adjusted transmission code rate. That is, at any time after the first encoded data is sent to the display application, the transmission code rate may be adjusted based on the network environment information of the display application, so that the encoded data of the media data is sent to the display application based on the dynamic transmission code rate.

It should be noted that, at any time when the encoded data is sent to the display application, the transmission code rate corresponding to the time does not exceed the preset code rate corresponding to the time. The preset code rate at this moment is the maximum transmission code rate which can be provided by the server corresponding to the display application and used for sending the encoded data to the display application, and may be specifically determined based on the actual application scene requirements, which is not limited herein.

The media data processing method provided by the embodiment of the present application is further described below with reference to fig. 6. Referring to fig. 6, fig. 6 is a scene schematic diagram of a cloud game data processing method provided in the embodiment of the present application. As shown in fig. 6, after the cloud game application initiates an access request including application information to the cloud game server, the cloud game server may encode game picture data corresponding to the cloud game based on the first encoding parameter to obtain first encoded data, and send the first encoded data to the cloud game application through the first transmission code rate.

In the process, the cloud game server performs data acquisition through a Frame Buffer, acquires a display image sequence in each 500ms to 1s interval, determines an image feature sequence corresponding to the display image sequence based on a convolutional neural network, and processes the image feature sequence based on a classification neural network so as to perform type detection on the type of display content corresponding to the display image sequence in each time interval. If the type of the display content is detected to be changed at any moment, a new transmission code rate and a new coding parameter can be determined based on the type of the display content corresponding to the moment, the game picture data after the moment is coded based on the new coding parameter, and the coded data after the moment is sent to the cloud game application based on the new transmission code rate.

In the process that the cloud game server sends the coded data corresponding to the game picture data to the cloud game, the cloud game server can also obtain network environment information reported by the cloud game application in real time, such as information of bandwidth estimation, maximum bit rate estimated by the receiver, packet loss rate and the like, and further adjust the transmission code rate in real time based on the network environment information.

In the embodiment of the present application, the timing for adjusting the transmission rate and/or the encoding parameters may be determined according to the granularity of the basic unit gop (group of pictures) of image encoding and decoding access. The encoder encodes a plurality of display images to produce a section of continuous pictures (GOP), and the decoder reads encoded data corresponding to the section of GOP for decoding and then reads the pictures for rendering display when displaying. Therefore, all GOPs corresponding to the multimedia data can be determined, the type of the display content is determined at the starting time or the ending time of each GOP, and the encoding parameters and the transmission code rate are adjusted based on the type of the display content corresponding to each time.

In the embodiment of the application, the transmission code rate and the coding parameters are adjusted by determining the type of the display content of the display application in the process of sending the coded data, so that the transmission code rate and the coding parameters which accord with the display content are determined, the display effect of the display application is improved in real time, and the bandwidth resource is saved. Meanwhile, the transmission code rate can be further adjusted in real time according to the network environment information of the display application in the sending process of the coded data, the receiving efficiency of the coded data is improved, and the applicability is high.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a media data processing apparatus according to an embodiment of the present application. The media data processing apparatus 700 provided in the embodiment of the present application includes:

a data encoding module 701, configured to respond to an access request of a display application, encode media data based on a first encoding parameter to obtain first encoded data, and send the first encoded data to the display application based on a first transmission code rate;

a parameter determining module 702, configured to determine a first type of the display content of the display application at a first time during a process of sending encoded data, and determine a second transmission code rate and a second encoding parameter based on the first type;

the data encoding module 701 is configured to encode the media data after the first time based on the second encoding parameter to obtain second encoded data, and send the second encoded data to the display application based on the second transmission code rate.

In some possible embodiments, the parameter determining module 702 is further configured to:

determining initial display configuration information of the display application;

and determining a first transmission code rate and a first coding parameter based on the initial display configuration information.

In some possible embodiments, the parameter determining module 702 is configured to:

acquiring application information of the display application, and determining a second type of initial display content of the display application based on the application information;

and determining a data processing parameter set, and determining a first transmission code rate and a first coding parameter corresponding to the second type and the initial display configuration information from the data processing parameter set.

acquiring at least one frame of display image corresponding to the display application at a first moment, and determining a first type of display content of the display application at the first moment based on the at least one frame of display image;

obtaining evaluation information of the display application, and determining a first type of display content of the display application at the first time based on the evaluation information.

processing the at least one frame of display image based on a convolutional neural network to obtain image characteristics corresponding to each display image;

processing each image feature based on a classification neural network to obtain a third type of display content corresponding to the at least one frame of display image;

and determining the third type as the first type of the display content applied to the first time.

In some possible embodiments, the classification neural network includes any one of a recurrent neural network or a multi-layered perceptron.

respectively adjusting the first transmission code rate and the first coding parameter based on the first type to obtain a second transmission code rate and a second coding parameter;

determining first display configuration information of the display application at the first time, and determining a second transmission code rate and a second coding parameter corresponding to the first type and the first display configuration information from the set of data processing parameters.

In some possible embodiments, the media data processing apparatus 700 further includes a parameter adjusting module 703, and the parameter adjusting module 703 is further configured to:

acquiring the network environment information of the display application at a second moment in the process of sending the coded data;

adjusting the transmission code rate corresponding to the second moment based on the network environment information;

and encoding the media data after the second moment based on the encoding parameter corresponding to the second moment, and transmitting the encoded data to the display application based on the adjusted transmission code rate.

In some possible embodiments, the network environment information includes at least one of downlink network information, a maximum reception code rate, or a packet loss rate of the display application at the second time.

In some possible embodiments, the initial display configuration information or the first display configuration information includes at least one of a resolution or a refresh rate.

In some possible embodiments, at any time during the coded data transmission, the transmission code rate corresponding to the time does not exceed the preset code rate corresponding to the time.

In a specific implementation, the media data processing apparatus 700 may execute, through each built-in functional module thereof, the implementation manners provided in each step in fig. 2 and/or fig. 5, which may be referred to specifically for the implementation manners provided in each step, and are not described herein again.

The media data processing means may be a computer program (including program code) running on a computer device, for example, the media data processing means is an application software; the apparatus may be used to perform the corresponding steps in the methods provided by the embodiments of the present application.

In some possible embodiments, the media data Processing Device provided in this embodiment may be implemented by combining hardware and software, and by way of example, the media data Processing Device provided in this embodiment may be a Processor in the form of a hardware decoding Processor, which is programmed to execute the media data Processing method provided in this embodiment, for example, the Processor in the form of a hardware decoding Processor may be implemented by one or more Application Specific Integrated Circuits (ASICs), (Digital Signal processors, DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), Central Processing Units (CPUs), or other electronic components.

In some possible implementations, the media data processing apparatus provided in the embodiments of the present application may be implemented in software, such as a media data processing apparatus 700 stored in a memory, which may be software in the form of programs and plug-ins, and includes a series of modules, including a data encoding module 701, a parameter determining module 702, and a parameter adjusting module 703. The data encoding module 701, the parameter determining module 702, and the parameter adjusting module 703 are used to implement the media data processing method provided in the embodiment of the present invention.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in fig. 8, the electronic device 800 in this embodiment may be a server corresponding to a display application, and specifically may include: the processor 801, the network interface 804 and the memory 805, the electronic device 800 may further include: a user interface 803, and at least one communication bus 802. Wherein a communication bus 802 is used to enable connective communication between these components. The user interface 803 may include a Display (Display) and a Keyboard (Keyboard), and the optional user interface 803 may also include a standard wired interface and a standard wireless interface. The network interface 804 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 804 may be a high-speed RAM memory or a non-volatile memory (e.g., at least one disk memory). The memory 805 may optionally be at least one memory device located remotely from the processor 801 as previously described. As shown in fig. 8, the memory 805, which is a kind of computer-readable storage medium, may include therein an operating system, a network communication module, a user interface module, and a device control application program.

In the electronic device 800 shown in fig. 8, the network interface 804 may provide network communication functions; and the user interface 803 is primarily an interface for providing input to a user; and the processor 801 may be configured to call a device control application stored in the memory 805 to implement the media data processing method provided by the embodiment of the present application.

It should be appreciated that in some possible implementations, the processor 801 may be a central processing unit, and the processor may also be other general purpose processors, digital signal processors, application specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may include both read-only memory and random access memory, and provides instructions and data to the processor. The portion of memory may also include non-volatile random access memory. For example, the memory may also store device type information.

In a specific implementation, the electronic device 800 may execute the implementation manners provided in the steps in fig. 2 through the built-in functional modules, which may specifically refer to the implementation manners provided in the steps, and are not described herein again.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and is executed by a processor to implement the method provided in each step in fig. 2 and/or fig. 5, which may specifically refer to the implementation manner provided in each step, and is not described herein again.

The computer readable storage medium may be the aforementioned media data processing module or an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The computer readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash card (flash card), and the like, which are provided on the electronic device. The computer readable storage medium may further include a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), and the like. Further, the computer readable storage medium may also include both an internal storage unit and an external storage device of the electronic device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the electronic device. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided by the steps of fig. 2 and/or fig. 5.

The terms "first", "second", "third", "fourth", and the like in the claims and in the description and drawings of the present application are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus. Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

The above disclosure is only for the purpose of illustrating the preferred embodiments of the present application and is not intended to limit the scope of the present application, which is defined by the appended claims.

Claims

1. A method of media data processing, the method comprising:

and coding the media data after the first moment based on the second coding parameter to obtain second coded data, and sending the second coded data to the display application based on the second transmission code rate.

2. The method of claim 1, wherein prior to encoding the media data based on the first encoding parameter to obtain the first encoded data, the method further comprises:

a first transmission code rate and a first coding parameter are determined based on the initial display configuration information.

3. The method of claim 2, wherein determining the first transmission code rate and the first coding parameter based on the initial display configuration information comprises:

determining a data processing parameter set, and determining a first transmission code rate and a first coding parameter corresponding to the second type and the initial display configuration information from the data processing parameter set.

4. The method of claim 1, wherein determining the first type of display content for the display application at the first time comprises at least one of:

obtaining evaluation information of the display application, and determining a first type of display content of the display application at the first moment based on the evaluation information.

5. The method of claim 4, wherein determining the first type of display content for the display application at the first time based on the at least one frame of display image comprises:

determining the third type as the first type of display content of the display application at the first time.

6. The method of claim 5, wherein the classification neural network comprises any one of a recurrent neural network or a multi-layered perceptron.

7. The method of claim 3, wherein determining a second transmission code rate and a second coding parameter based on the first type comprises at least one of:

8. The method of claim 1, further comprising:

acquiring network environment information of the display application at a second moment in the process of sending the coded data;

and coding the media data after the second moment based on the coding parameters corresponding to the second moment, and sending the coded data to the display application based on the adjusted transmission code rate.

9. The method of claim 8, wherein the network environment information comprises at least one of downlink network information, a maximum receiving code rate, or a packet loss rate of the display application at the second time.

10. The method of claim 7, wherein the initial display configuration information or the first display configuration information comprises at least one of a resolution or a refresh rate.

11. The method according to any one of claims 1 to 10, wherein at any time during the transmission of the encoded data, the transmission code rate at that time does not exceed the predetermined code rate at that time.

12. An apparatus for media data processing, the apparatus comprising:

the parameter determining module is used for determining a first type of display content of the display application at a first moment in the process of sending the coded data, and determining a second transmission code rate and a second coding parameter based on the first type;

13. An electronic device comprising a processor and a memory, the processor and the memory being interconnected;

the memory is used for storing a computer program;

the processor is configured to perform the method of any of claims 1 to 11 when the computer program is invoked.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1 to 11.