CN114157919A

CN114157919A - Data processing method and system, cloud terminal, server and computing equipment

Info

Publication number: CN114157919A
Application number: CN202010990303.8A
Authority: CN
Inventors: 龙长春; 王坤铭; 吴子侠; 李海跃
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2020-09-07
Filing date: 2020-09-18
Publication date: 2022-03-08
Also published as: WO2022048255A1

Abstract

The method, the system, the cloud terminal, the server, the computing equipment and the computer program product for processing data can solve the problems of insufficient CPU resources and software compatibility of the cloud terminal. The method comprises the following steps: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is to encode the data or decode the data; and the server sends the processed data to the cloud terminal.

Description

Data processing method and system, cloud terminal, server and computing equipment

The present application claims priority from chinese patent application No. 202010928653.1 entitled "method, apparatus, server, and storage medium for encoding/decoding" filed on 7/9/2020, which is incorporated herein by reference in its entirety.

Technical Field

The invention relates to the technical field of cloud, and mainly relates to a data processing method and system, a cloud terminal, a server, computing equipment and a computer program product.

Background

The Cloud Phone (Cloud Phone) is a novel service, has a very good extending and supplementing effect on a physical Phone, and starts to be completely open in scenes such as Cloud Phone games, Cloud Phone live broadcast, mobile office and the like. In the above scenario, the encoding and decoding of the video are all involved, for example: (1) the cloud mobile phone game is characterized in that after video coding is carried out on a mobile phone screen picture (or a game picture), the mobile phone screen picture (or the game picture) is transmitted to a physical mobile phone through a network; (2) in a live broadcast scene, video decoding is required to be carried out on received camera picture data of a physical mobile phone and the camera picture data are displayed on a cloud mobile phone, and when a live broadcast application is started on the cloud mobile phone, the cloud mobile phone carries out video coding on the camera picture data and sends the camera picture data to a live broadcast platform; (3) when workers in industries with higher information security requirements such as mobile office scenes, governments, finance and the like carry out video call through instant messaging application, video coding is required to be carried out on a camera picture on a cloud mobile phone, and when a video conference shares and plays videos, video decoding is required to be carried out.

However, since the video encoding and decoding has a high requirement on the performance of the cloud mobile phone CPU, in order to meet the requirements of live broadcast and mobile office of the cloud mobile phone, the CPU specification of a single cloud mobile phone must be improved, which will cause two disadvantages: (1) the cost of a single cloud mobile phone is increased, and the competitive advantage of the cloud mobile phone in the fields of live broadcast and mobile office is not favorably constructed; (2) in addition, the service time of the cloud mobile phone has a certain period of time, for example, live broadcast and mobile office are respectively concentrated on working in the daytime and leisure time out of work at night, which causes waste of the CPU resource of the cloud mobile phone.

Disclosure of Invention

The method and system for processing data, the cloud terminal, the server, the computing device and the computer program product provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings, so that the occupation of the CPU resources of the cloud mobile phone can be reduced.

In a first aspect of the disclosure, a data processing method is provided and applied between a cloud terminal and a server. The method comprises the following steps: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is to encode the data or decode the data; and the server sends the processed data to the cloud terminal.

According to the data processing method, the encoding and decoding tasks of the cloud terminal are shared on the server, so that processor resources of the cloud terminal are saved, and the cost and the specification of a single cloud terminal are reduced. The server has powerful computing resources, and the efficiency of encoding and decoding tasks on the cloud terminal is further improved.

In one possible design of the first aspect, the server encoding the data includes: the data is hardware-coded or software-coded.

The encoder module of the server is responsible for this encoding task. Wherein, the software coding task runs on the CPU; the hardware coding task runs on a hardware coding and decoding chip. When the encoding task is performed at the server, the server can provide higher encoding efficiency.

In one possible design of the first aspect, the decoding, by the server, the data includes: the data is hardware decoded or software decoded.

The decoding task is taken care of by the decoder module of the server. The software decoding task runs on the CPU, and the hardware decoding task runs on the hardware coding and decoding chip. When the decoding task is performed at the server, the server can provide higher decoding efficiency.

In one possible design of the first aspect, the data processing method further includes: the cloud terminal sends a processing rule to the server; the processing rule is used to determine whether to hardware encode or software decode the data, or the processing rule is used to determine whether to software encode or software decode the data.

The cloud terminal sends an instruction and data to the server, wherein the instruction comprises a processing rule, and the processing rule comprises an encoding mode or a decoding mode: the encoding mode comprises hardware encoding or software encoding, and the decoding mode comprises hardware decoding or software decoding. The server calls different encoding modules and decoding modules to perform operations respectively according to the difference between the encoding mode and the decoding mode, which can refer to the above description.

In one possible design of the first aspect, the data processing method further includes: before the cloud terminal sends data, the cloud terminal acquires a request for processing the data by the terminal.

Before the cloud terminal sends data to the server, the cloud terminal needs to acquire a data processing request from the terminal. This request comes from a physical terminal and is triggered by the user of the terminal.

In one possible design of the first aspect, the data processing method further includes: before the cloud terminal sends data, the cloud terminal obtains a request for processing the data, wherein the request is triggered by an application on the cloud terminal.

Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, and the request is related to the encoding or decoding requirements of the application layer. An encoding request is issued if the application requires encoding and a decoding request is issued if the application requires decoding.

In a third possible implementation manner of the first aspect of the present invention, the processing rule in the request sent by the application is not identical to the processing rule sent by the cloud terminal to the server, that is, the cloud terminal may modify the processing rule and send the modified processing rule to the server. The modification method has four ways: the software coding is changed into hardware coding, the hardware coding is changed into software coding, the software decoding is changed into hardware decoding, and the hardware decoding is changed into software decoding. The method has the advantages that the software coding is changed into the hardware coding, the software decoding is changed into the hardware decoding, the coding and decoding efficiency can be improved, the efficiency of CPU resources of the cloud terminal can be reduced, and the problem that part of the method is incompatible when the method is applied to the cloud terminal to call the hardware coding or the hardware decoding can be solved.

In one possible design of the first aspect, the data processing method further includes: and the cloud terminal sends the virtual desktop obtained according to the data to the terminal.

The application of the cloud terminal obtains a virtual desktop based on the processed data and issues the virtual desktop to the terminal, so that the terminal can display the virtual desktop to a user.

In a second aspect, the present invention provides a system for processing data, the system comprising:

the cloud terminal is used for sending data to the server;

and the server is used for receiving the data, processing the data and then sending the processed data to the cloud terminal, wherein the processing is to encode the data or decode the data.

According to the data processing method, the encoding and decoding tasks of the cloud terminal are shared on the server, so that processor resources of the cloud terminal are saved, and the cost and the specification of a single cloud terminal are reduced. The server has powerful computing resources, and the efficiency of encoding and decoding tasks of the cloud terminal is further improved.

In one possible design of the second aspect, the server further includes: for hardware or software coding of the data.

In one possible design of the second aspect, the server further includes: for hardware or software decoding of the data.

The decoding task is taken care of by the decoder module of the server. Wherein, the software decoding task runs on the CPU; the hardware decoding task runs on a hardware coding and decoding chip. When the decoding task is performed at the server, the server can provide higher decoding efficiency.

In one possible design of the second aspect, the data processing system further includes:

and the cloud terminal is used for sending a processing rule to the server, wherein the processing rule is used for determining to perform hardware coding or software decoding on the data, or the processing rule is used for determining to perform software coding or software decoding on the data.

and the cloud terminal is used for acquiring a request of the terminal for processing the data.

Before the cloud terminal sends data to the server, the cloud terminal acquires a data processing request from the terminal, wherein the request is from a physical terminal and is triggered by a user of the terminal.

In one possible design of the second aspect, the data processing method further includes:

and the cloud terminal is used for acquiring a request for processing the data, wherein the request is triggered by the application on the cloud terminal.

In a third possible implementation manner of the second aspect of the present invention, the processing rule in the request sent by the application is not identical to the processing rule sent by the cloud terminal to the server, that is, the cloud terminal may modify the processing rule and send the modified processing rule to the server. The modification method has four ways: the software coding is changed into hardware coding, the hardware coding is changed into software coding, the software decoding is changed into hardware decoding, and the hardware decoding is changed into software decoding. The method has the advantages that the software coding is changed into the hardware coding, the software decoding is changed into the hardware decoding, the coding and decoding efficiency can be improved, the efficiency of CPU resources of the cloud terminal can be reduced, and the problem that part of the method is incompatible when the method is applied to the cloud terminal to call the hardware coding or decoding can be solved.

and the cloud terminal is used for sending the virtual desktop obtained according to the data to the terminal.

In one possible design of the second aspect, the data processing system further includes: the cloud terminal is a cloud mobile phone.

In a third aspect, the present application provides a cloud terminal, where the cloud terminal is provided in the second aspect or any one of the possible designs of the second aspect.

In a fourth aspect, the present application provides a server according to the second aspect of the present application or any possible design of the second aspect of the present application.

In a fifth aspect, the present application provides a computing device comprising a processor, a memory. The processor executes the instructions in the memory to cause the computing device to implement a cloud terminal or server as provided by the second aspect of the present application or any possible design of the second aspect.

In a sixth aspect, the present application provides a computer program product comprising instructions for implementing the steps performed by the cloud terminal in the method according to the first aspect or any of the possible designs of the first aspect, and/or for implementing the steps performed by the server in the method according to the first aspect or any of the possible designs of the first aspect.

The present application can further combine to provide more implementations on the basis of the implementations provided by the above aspects.

Drawings

Fig. 1 is an exemplary schematic diagram of a system structure of data processing provided in an embodiment of the present application.

Fig. 2 is an exemplary schematic diagram of a system structure of data processing provided in an embodiment of the present application.

Fig. 3 is an exemplary schematic diagram of a system structure of data processing provided in an embodiment of the present application.

Fig. 4 is an exemplary diagram of a system architecture for data processing as provided by way of example in the present application.

Fig. 5 is a flowchart of a method for processing data according to an embodiment of the present disclosure.

Fig. 6 is an exemplary schematic diagram of a system structure of data processing provided in an embodiment of the present application.

Fig. 7 is an exemplary schematic diagram of a system structure of a data processing method according to an embodiment of the present application

Fig. 8 is a diagram of an example of a method framework of data processing according to an embodiment of the present application.

Fig. 9 is an exemplary schematic diagram of a method related to data processing in a live shopping scene of a cloud mobile phone provided in an embodiment of the present application.

Fig. 10 is an exemplary block diagram of an apparatus 300 according to an embodiment of the present disclosure.

Fig. 11 is an exemplary block diagram of a computing device 600 provided by an embodiment of the application.

Detailed Description

Fig. 1 is a schematic structural diagram of a data processing system according to an embodiment of the present disclosure, and as shown in fig. 1, the system may include a cloud terminal 001 and a server 002.

In one embodiment, the cloud terminal 001 may be a virtual device that provides a virtual desktop, for example, the virtual device may be an instance, and the instance may be a virtual machine or a container. In a possible implementation manner, the cloud terminal 001 is a cloud mobile phone.

The server 002 may be a device having computing capabilities, the server 002 having a memory and a processor. The processor of the server 002 may be a Central Processing Unit (CPU). Optionally, the server 002 may further include one or more of a Graphics Processing Unit (GPU), a Neural Network Processing Unit (NPU), an fpga (field Programmable Gate array), and the memory of the server 002 may be a Random Access Memory (RAM) or a solid-state drive (SSD) or other devices or memory instances with storage capability. A solid state drive may also be referred to as a Solid State Drive (SSD)

In an embodiment of the application, an application may be installed on the cloud terminal 001, and the application may include applications such as live video, video call, video conference, video player, audio player, and mobile game. Optionally, the application on the cloud terminal 001 may issue a request for encoding or decoding, or may issue the request for encoding and decoding at the same time.

In one embodiment, the server 002 may be a physical device deployed in a network, and the cloud terminal 001 and the server 002 communicate based on a communication protocol or a call protocol, so that the cloud terminal 001 may call a service provided by the server 002 or perform data interaction with the server 002. For example, the communication protocol may be a Socket (Socket). For example, the calling protocol may be a Remote Procedure Call (PRC).

In one example implementation scenario, the server 002 may be deployed with a software encoding module and a software decoding module. The software coding module is used for coding data. The software decoding module is used for decoding the data. In one possible implementation, the encoding or decoding operations are executed on the CPU of the server 002. In one possible implementation, the software code modules and software code modules may be deployed directly in the server 002 or in an instance where the server 002 is running.

As an example of an implementation scenario, the server 002 may be a device with hardware encoding capability, and the server 002 is deployed with a hardware encoding module and a hardware decoding module. The hardware coding module is used for coding data. The hardware decoding module is used for decoding the data. In one possible implementation, the encoding and decoding operations are executed on a hardware encoding chip of the server 002, for example, the hardware encoding chip may be a GPU or an NPU or other chip with hardware encoding capability.

Fig. 2 is a schematic structural diagram of a second data processing system provided in the embodiment of the present application, and as shown in fig. 2, the server 002 may also be deployed in one cloud server cluster 003. There are multiple servers in cloud server cluster 003 that are similar to server 002. The servers 002 of the Cloud Server cluster 003 may also be comprised of a number of Elastic Cloud Servers (ECS), i.e. the number of servers 002 in the Cloud Server cluster 003 is elastically scalable.

Fig. 2 is a schematic system structure diagram of a third data processing provided in the embodiment of the present application, and as shown in fig. 3, the server 002 may also be deployed in one data center 004.

In one possible embodiment, one or more servers may be included in cloud server cluster 003 (or data center 004), the servers may provide encoding capability, or the servers may provide decoding capability, or the servers provide both encoding and decoding capability. In one example scenario, the server used for encoding and the server used for decoding belong to different clusters or data centers. Namely, a cluster formed by the servers used for encoding is called an encoding cluster, a cluster formed by the servers used for decoding is called a decoding cluster, a data center formed by the servers used for encoding is called an encoding data center, and a data center formed by the servers used for decoding is called a decoding data center.

Fig. 4 is a schematic diagram of another data processing system architecture provided by way of example. As shown in fig. 4, the system may include a terminal 005 based on the content described in the system configuration diagram of any one of fig. 1 to 3. In some embodiments, the terminal 005 may be a cell phone, tablet, or other physical device with display and communication capabilities. The terminal 005 can communicate with the cloud terminal 001 through a network, the terminal 005 uploads local data and instructions to the cloud terminal 001, the cloud terminal 001 can issue the virtual desktop to the terminal 005, and a user can control the operating system with the virtual desktop, which runs on the cloud terminal 001, on the terminal 005. For an example scenario, the server 002 in fig. 4 may also be the cloud server cluster 003 or the data center 004 in fig. 3 and fig. 4, and please refer to the foregoing description for an implementation manner of the cloud server cluster 003 and the data center 004.

For example, as shown in fig. 4, the terminal 005 may collect video data from a local place and upload the collected video data to software of the cloud terminal 001. The cloud terminal 001 may send a service request (e.g., an encoding request or a decoding request) to the server 002 according to the application requirement, and forward the video data collected locally by the terminal 005 to the server 002; the server 002 returns the processed video data to the application of the cloud terminal 001. The application of the cloud terminal obtains a virtual desktop based on the processed video data, for example, a live interface is generated based on the video data, and the live interface belongs to part or all of the content displayed by the virtual desktop. The cloud terminal 001 issues the virtual desktop to a display of the terminal 005.

Fig. 5 is a flowchart of a method for processing data according to an embodiment of the present application, where the method may be applied to any one of the systems shown in fig. 1 to 4. Fig. 5 is an embodiment of the method applied to the system architecture shown in fig. 1, and the system of fig. 1 includes a cloud terminal and a server. The system architecture of fig. 2-4 is more complex than that of fig. 1, and the steps of the flow chart of the embodiment of the method applied to them will be more numerous than those of fig. 5, but their core steps are still similar to those of the flow chart of the embodiment shown in fig. 5. The following describes the flow of the data processing method of the present application with reference to fig. 1 and 5. The process includes steps 201 to 205.

Step 201, the cloud terminal sends data to a server.

The cloud terminal sends data to the server through the network, and the data is to-be-decoded or to-be-encoded.

In a possible implementation manner, the cloud terminal sends an instruction indicating encoding and an instruction indicating decoding to the server, and the instruction contains a rule of encoding or decoding.

Optionally, the instruction carries data to be encoded or decoded.

Alternatively, the instruction and data (data to be encoded or decoded) are sent independently.

In one possible embodiment, the cloud terminal may continuously send a piece of video stream data, and send an instruction to start encoding to the server while starting sending the video stream, for example, after the video stream stops or the application sends a request to stop, the cloud terminal sends an instruction to close the encoding service to the server.

For example, some applications are installed on the cloud terminal, and the applications can send out encoding or decoding requests according to their own service requirements. The encoding mode in the encoding request may be hardware encoding or software encoding. The decoding manner in the decoding request may be hardware decoding or software decoding.

In one possible implementation, the cloud terminal determines the rule of encoding or decoding according to a request issued by an application.

In one possible embodiment, the manner of encoding or decoding in the determined rule may be different from the rule of encoding or decoding in the request issued by the application. For example, the application sends out a request of software coding, and the determined coding rule is hardware coding, or the application sends out a request of hardware coding, and the determined coding rule is software coding, that is, the cloud terminal may modify the coding mode sent out by the application.

In a possible embodiment, the encoding or decoding manner in the determined rule may also be the same as the encoding or decoding rule in the request issued by the application, that is, the cloud terminal may not modify the encoding manner issued by the application.

Step 202, the server receives data.

For example, after receiving the data sent by the cloud terminal, the server also needs to start an encoding or decoding task according to an instruction sent by the cloud terminal synchronously, and returns a corresponding service response to the cloud terminal. In one possible embodiment, the server may continuously receive a piece of video stream data, and after receiving the start command, the server may start encoding or decoding the continuously received video frames, and return a service response to start encoding or decoding to the cloud terminal until the server receives a command to close the encoding or decoding task, and then return a service response to end encoding or decoding to the cloud terminal.

Step 203, the server performs an encoding or decoding service.

For example, the server encodes or decodes the data according to the received encoding rule or decoding rule, where the encoding rule may include an encoding standard, an encoding parameter, and an encoding manner. The decoding rules may include decoding standards, decoding parameters, and decoding methods.

For example implementation of encoding, a server receives an encoding request sent by a cloud terminal, the encoding request includes encoding standards which may be h.264, h.264 and the like, encoding parameters may include resolution, code rate, frame rate, DTS, PTS and the like, and an encoding mode may be software encoding or hardware encoding. For example, video coding is a process of converting an original video format file into another video format file, and the main purpose is to compress the original video for storage and transmission. The encoded data is a compressed frame, and the amount of the compressed frame is smaller than that of the original frame.

For example, if the encoding mode is hardware encoding and decoding, the server invokes a hardware encoding chip (e.g., a GPU chip) to perform encoding calculation on the original frame. If the coding mode is software coding, the server calls a CPU to perform coding calculation on the original frame.

The encoding mode used by the server and the encoding mode in the received instruction may be the same, but in some embodiments, the encoding mode used by the server may be different from the encoding mode in the request sent by the cloud terminal application, and the cloud terminal performs some processing on the content in the request, where the processing may include modifying the encoding mode, which may refer to the description in step 201.

In an example implementation of decoding, the server receives a decoding request sent by the cloud terminal, where the decoding standard, the decoding parameter, and the decoding manner may refer to the description in the encoding rule. For example, video decoding is the reverse process of video encoding, and converts a file in a compressed video format into a file in an original video format by a decompression technology, and the main purpose is to decompress compressed video frames to support the playing and displaying of pictures. The decoded data is the decoded frame (which may be lossy relative to the original frame), and the decoded frame has a larger amount than the compressed frame data, so the visual effect after visualization is better.

For example, if the encoding mode is hardware codec, the server invokes a hardware decoding chip to perform decoding calculation on the compressed frame, for example, a GPU chip. If the decoding mode is software decoding, the server calls the CPU to perform decoding calculation on the compressed frame.

In some embodiments, the decoding scheme employed by the server and the decoding scheme in the received instruction may be the same.

In some embodiments, the decoding mode adopted by the server may be different from the decoding mode in the request sent by the cloud terminal application, and the cloud terminal performs some processing on the content in the request, where the processing may include modifying the decoding mode. For example, the decoding method indicated by the instruction is software decoding, but the decoding method adopted by the server is hardware decoding.

And step 204, the server sends the processed data.

And the server sends the processed data to the cloud terminal through a network.

According to an application scenario example, a server sends an encoded frame to a cloud terminal, wherein the encoded frame is generated by processing of a module in charge of encoding inside the server.

According to an application scenario example, a server sends a decoded frame to a cloud terminal, wherein the decoded frame is generated by processing of a module in the server, which is responsible for decoding.

In one possible implementation, the server is continuously transmitting a processed video stream. In some possible embodiments, after receiving the codec stop instruction or stopping the video streaming, the server may simultaneously send a response to the cloud terminal after stopping the codec service.

And step 205, the cloud terminal receives the processed data.

The cloud terminal receives data returned by the server from the network, and the data may include, for example, a processed decoded frame or a compressed frame.

An example of an application scenario is that a video playing application sends a decoding request before, a cloud terminal forwards data to the video playing application of an application layer through a framework layer after receiving a decoding stream returned by a server, and the video playing application presents the decoded video stream in a user interface of the application. In one possible implementation, the desktop content of the application user interface is sent to a terminal.

An example of an application scenario is that a video call application sends a coding request before, a cloud terminal receives a coding stream returned by a server, and forwards data to the video playing application of an application layer through a framework layer, and the video call application sends the coded video stream to one or more other terminals through a network or stores the coded video stream in a cloud, for example, a terminal used by an object in video call with a current user receives the coded video stream.

Fig. 6 is a schematic system structure diagram of a data processing method according to an embodiment of the present application, including an internal structure diagram of the cloud terminal 001 and the server 002. The cloud terminal comprises an application layer and a framework layer, wherein the application layer comprises various applications, and the cloud terminal mainly relates to applications 101 and 102 in the embodiment. Application 101 requires a service for video encoding by the architecture layer and application 102 requires a service for video decoding by the architecture layer. The framework layer comprises an encoding and decoding management module 111, which is composed of a media encoding and decoding interface, a software encoding and decoding plug-in or a hardware encoding and decoding plug-in and a corresponding control module, and is responsible for receiving encoding or decoding requests of applications such as live broadcast, video call, video playing and the like, calling the corresponding software or hardware plug-in according to the demand difference of different applications, and returning target video frame data. The communication module 114 of the cloud terminal and the communication module 121 of the server are used for transmitting data and control instructions for the cloud terminal and the transcoding cluster in the network through a transmission protocol. The server transcoding management module 122 is responsible for receiving an encoding or decoding request of the cloud mobile phone encoding or decoding module, issuing an encoding or decoding task to the corresponding external encoding or decoding module, and simultaneously, receiving encoded or decoded data and forwarding the data to the communication module 121. The encoder module 123 is responsible for real video encoding work, receives an original video frame, and returns a compressed frame after being encoded by a hardware encoding chip; the decoder set module 124 is responsible for real video decoding work, receives the original compressed frame, and returns the decoded frame after being decoded by the hardware decoding chip. Optionally, the encoder module 123 and the decoder module 124 may belong to the same functional module, which does not affect the scheme of the embodiment.

In another embodiment, as shown in fig. 7, the system is a schematic structural diagram of another data processing method provided in the present application. Compared with fig. 6, the encoding and decoding services of the cloud terminal are carried by a cloud server cluster or a data center, rather than one physical server. In some possible embodiments, such a deployment may enhance the service efficiency of the outer codec. Referring to fig. 7, one or more management nodes 131 and two sub-clusters, respectively an encoding cluster 132 and a decoding cluster 133, may be included in a cloud server cluster 003 or a data center 004, respectively, and are responsible for the actual encoding and decoding work, respectively. Alternatively, the encoding cluster 132 and the decoding cluster 133 may be combined or distributed separately in one or more service clusters. In some possible embodiments, unlike the codec management module of server 002 in fig. 6, the management node 131 in fig. 7 may include both communication and cluster management functions, and each management node may be composed of a server that may not include an encoding or decoding module.

Fig. 8 is a diagram showing an example of a method framework for data processing according to an embodiment of the present application, which is a further explanation of the system in fig. 7. In some embodiments, the transcoding management module 111 is further divided into a media transcoding interface, a software or hardware codec plug-in and a control unit according to functions, wherein the software or hardware transcoding plug-ins 115 and 116 need to be responsible for creating threads and calling the hardware transcoding control module 113 or the software transcoding module 112 to implement software hardware encoding and decoding. The hardware transcoding module 113 is used to implement specific control of hardware encoding or decoding, and includes starting an encoding or decoding device, sending encoding or decoding parameters, sending encoded or decoded video frames, receiving encoded or decoded video frames, and turning off the encoding or decoding device. The software transcoding module 112 is used for implementing specific control of software encoding or decoding, and includes starting an encoding or decoding device, sending encoding or decoding parameters, sending a video frame to be encoded or decoded, receiving an encoded or decoded video frame, and turning off the encoding or decoding device.

401-421 shown in fig. 8 are some key steps related to this embodiment, where the numerical size of the serial number does not represent an actual execution sequence, and represents a certain action and step only as a mark, and if the occurrence number of a certain mark is greater than 1, it represents that the same or similar action and step are executed at the position of the mark. The flow of the method for processing data of the present application is illustrated with reference to fig. 8. The process includes steps 401 through 420.

Step 401, requesting video encoding or decoding.

The request for video encoding or decoding is typically initiated by an application, and different types of applications may issue requests for video encoding or decoding. For example, a live broadcast application of a mobile phone sends out an encoding request, a video play application can send out a software decoding request, and a cloud mobile phone game and a video call in a mobile office scene can simultaneously send out encoding and decoding requests.

And step 402, calling a software coding and decoding plug-in.

The media codec interface calls a software codec plug-in. In some embodiments, the media codec interface may be an interface native to the cloud terminal operating system. The native media codec interface may be preserved in order not to affect the invocation of the application layer.

In some embodiments, the software codec plug-in may be some existing plug-in, or may be a plug-in developed by the inventor.

And step 403, calling a hardware coding and decoding plug-in.

The media codec interface calls a hardware codec plug-in. The cloud mobile phone in the prior art only supports the step 402, so that part of applications need to call the software functions of hardware encoding and decoding, and the software functions are not available. For example, the inventor can develop the hardware codec plug-in by himself according to the requirement. The embodiment of the method can realize the function of calling the hardware chip to carry out encoding or decoding, and improves the compatibility of the cloud mobile phone application software.

And step 404, calling a corresponding software coding and decoding control unit by the software coding and decoding plug-in, creating a thread, managing a plurality of streams, and realizing the master control of hardware coding or decoding work.

And 405, calling a corresponding hardware coding and decoding control unit by the hardware coding and decoding plug-in unit, creating a thread, managing a plurality of streams, and realizing the total control of hardware coding or decoding work.

Step 406, either the original frame or the compressed frame is sent.

The software codec control unit 112 sends data of an original frame or a compressed frame to be encoded or decoded to a client Socket, optionally, a client RPC.

The hardware codec control unit 113 sends data of an original frame or a compressed frame to be encoded or decoded to a client Socket, optionally, may also be a client RPC.

Step 407, start or shut down the encoding or decoding service.

The software coding and decoding control unit sends a starting or closing request, and the client Socket (or RPC) receives and forwards the request to the server Socket (or RPC) through the network, so as to start or close the coding or decoding service of the remote transcoding cluster.

The hardware decoding control unit sends a starting or closing request, and the client Socket (or RPC) receives and forwards the request to the server Socket (or RPC) through the network, so as to start or close the encoding or decoding service of the remote transcoding cluster.

In some embodiments, the request to start or shut down may be based on a request from an application (or user).

Step 408, either the original frame or the compressed frame is sent.

And the client Socket (or RPC) sends the data of the coded frame or the decoded frame received from the network to the software coding and decoding unit.

The client Socket (or RPC) sends the data of the encoded frame or the decoded frame received from the network to the hardware coding and decoding unit.

Step 409, initiate or shut down the response.

And the client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network and sends the service response to the software coding and decoding unit for starting or closing the response operation of the software coding and decoding unit.

And the client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network and sends the service response to the hardware decoding unit, and the service response is used for starting or closing the response operation of the hardware decoding unit.

Step 410, sending a request instruction.

The instructions received from the network are sent by the server Socket (or RPC) to transcoding cluster manager 131, which in some embodiments may include starting encoding, starting decoding, closing encoding, closing decoding, and so on.

Step 411, the original compressed frame or video frame is sent.

The image original frame or the encoded frame received from the network is sent to the transcoding cluster management unit 122 by the server Socket (or RPC).

Step 412, return the processed decoded frame or compressed frame.

The transcoding cluster management 131 returns the processed decoded frame or compressed frame to the server Socket or RPC.

Step 413, return service response.

And returning a processing response by the transcoding cluster management unit, and forwarding the response by the service side Socket (or RPC).

Step 414, send the command to start the encoder.

Transcoding cluster manager 131 sends an encoder startup instruction to encoding cluster 132, and encoding cluster 132 starts encoding upon receiving the startup instruction.

Step 415, sending the original video frame.

Transcoding cluster management 131 sends the original video frames to encoding cluster 132.

Step 416, send command to close encoder.

Transcoding cluster manager 131 sends an instruction to close the encoder to encoding cluster 132, and encoding cluster 132 receives the close instruction and stops transcoding.

Step 417, return compressed frame.

The processed compressed frames are returned by encoding cluster 132 to transcoding cluster manager 131.

Step 418, an instruction to start the decoder is sent.

Transcoding cluster manager 131 sends 133 a decoder startup command to the decoding cluster, and decoding cluster 133 starts decoding upon receiving the startup command.

Step 419 sends an instruction to shut down the decoder.

The transcoding cluster manager 131 sends a command to turn off the decoder to the decoding cluster 133, and the decoding cluster 133 stops transcoding after receiving the start command.

Step 420, the original compressed frame is sent.

The original compressed frames are sent by transcoding cluster manager 131 to decoding cluster 133.

Step 421, the decoded frame is returned.

The processed decoded frames are returned by the decoding cluster 133 to the transcoding cluster manager 131.

The relevant details of the above 401-421 steps can also refer to the above description.

Fig. 9 is a schematic diagram of a method that may involve cloud terminal data processing in a one-time cloud mobile phone shopping live scene according to an embodiment of the present application. In some embodiments, first, a certain number of cloud phones and cloud server clusters in a corresponding ratio are purchased according to usage scenarios and scales of the cloud phones, the cloud server clusters can provide the encoding and decoding services described above, and live broadcast of the cloud phones is larger than the scale of the cloud server clusters required by a mobile office scenario. Then, configuration of the cloud mobile phone to interface with the external transcoding cluster needs to be completed, for example, IP and port configuration of Socket on the cloud mobile phone. And finally, live broadcasting and mobile office are carried out by using the configured cloud mobile phone and cloud server cluster. In the method schematic diagram shown in fig. 9, a

serial number

501 and 514 represent key steps involved in this embodiment, and describe how, in a process from when a user a starts live broadcast, a live broadcast shopping application sends a software coding request until the user a closes the live broadcast in a live broadcast shopping live broadcast of a cloud mobile phone, an embodiment of the method is to perform a remote hardware video coding service based on the cloud mobile phone.

Step 501, a user A opens shopping live broadcast application software on a cloud mobile phone to start live broadcast. In some possible embodiments, the user may be remotely manipulating an application on the cloud terminal 001 based on the visual interface of the terminal 005 shown in fig. 4. For example, the terminal 005 may be a smart phone, which includes a camera or other shooting device, and is configured to collect a video stream of a user during a live broadcast in real time, and synchronously transmit the video stream to the cloud phone.

Step 502, the shopping live broadcast application calls a MediaCodec interface to start video coding operation. The method comprises the steps that a live broadcast application receives a video stream, sends a software coding request to a MediaCodec interface and requests coding service of a cloud server cluster;

step 503-.

And 505, the encoder control unit generates a relevant instruction, wherein the instruction comprises an encoding rule, and the encoding mode in the encoding rule is modified from software encoding to hardware encoding. And then, the encoder control unit establishes socket connection with the external transcoding cluster through a client socket and sends a request for starting an external hardware encoder to a server socket.

Step 506, after receiving the instruction of the hardware code, the service side socket of the cloud server cluster forwards the instruction to the transcoding cluster management node, and after receiving the instruction of starting the specific external encoder, the transcoding cluster management node creates a thread to start the coding cluster.

Step 507, the cloud mobile phone encoder control unit sends a request for acquiring the encoding header information, and the client socket forwards the request to the client socket of the cloud server cluster.

And step 508, requesting and acquiring a coding head from an encoder module of a specific server in the encoding cluster by the transcoding cluster management node, and forwarding the coding head to the cloud mobile phone through the network.

In step 509, the cloud handset encoder control unit receives a response of the encoding header information.

Step 510, the cloud mobile phone encoder control module sends original video frame data to a transcoding management node of the cloud server cluster, and requests an external encoding cluster to perform hardware encoding.

Step 511, the transcoding cluster management node is responsible for allocating the encoding server to the cloud mobile phone, and the cloud mobile phone sends the original video frame to the encoder of the specific server.

And step 512, the encoder returns the encoded compressed video frame to the transcoding cluster management node and forwards the encoded compressed video frame to the cloud mobile phone through the network.

And 513, receiving the compressed video frame data by the encoder module of the cloud mobile phone through the socket, and returning the video frame data to the live broadcast application for use. The live broadcast application on the cloud mobile phone receives the compressed frame data stream and forwards the compressed frame data stream to the public cloud, more network users can download the compressed frame data stream on own mobile phones or PCs and other terminals, and after decoding is carried out locally, live shopping broadcasts in the field can be watched in real time.

And step 514, closing the shopping live broadcast application on the cloud mobile phone by the user A, and ending the live broadcast.

And step 515, the shopping live broadcast application software calls a MediaCodec interface to end the video coding operation.

Step 516 + 517, the MediaCodec process calls the encoder control unit based on the software coding plug-in, and requests to end coding;

step 518, the encoder control unit releases the remote encoder, sends a request to close the encoder of the server, and closes the socket connection with the cloud server cluster.

Step 519, the transcoding cluster management node closes the coding service of the specific server.

Fig. 10 is an apparatus 300 deployed on a server 002 according to an embodiment of the present application, where the apparatus 300 includes a receiving unit 301, a processing unit 302, and a transmitting unit 303.

A receiving unit 301, configured to receive data from the cloud terminal 001. In some embodiments, the data may be data to be encoded or decoded. In other embodiments, the data may further include instructions for indicating the encoding or decoding task, and the content of the instructions refers to the related description above.

A processing unit 302 for encoding or decoding data. For example, the processing unit encodes data to be encoded, and the encoding manner includes hardware encoding or software encoding, please refer to the foregoing description. For example, the processing unit decodes data to be decoded, and the decoding manner includes hardware decoding or software decoding, please refer to the foregoing description.

A sending unit 303, configured to send data to the cloud terminal 001. The transmitted data is data that has been decoded or encoded.

Optionally, the processing unit 302 is configured to perform hardware encoding on the data or perform software encoding on the data.

Optionally, the processing unit 302 is configured to perform hardware decoding on the data or perform software decoding on the data.

The embodiment of the present application further provides a computing device 600, as shown in fig. 11. The computing device 600 may be a device for deploying the cloud terminal 001, or the computing device 600 may be a server 002.

Fig. 11 provides a schematic diagram of a computing device 600, the computing device 600 including a bus 601, a processor 602, and a memory 603. The processor 602 and the memory 603 communicate with each other via a bus 601.

Bus 601 may be a peripheral component interconnect standard bus or an extended industry standard architecture bus or the like. The processor 602 may be a Central Processing Unit (CPU), and may further include one or more of a Graphics Processing Unit (GPU), a Neural Network Processing Unit (NPU), and an fpga (field Programmable Gate array). The memory 603 may include one or more of Random Access Memory (RAM), read-only memory (ROM), flash memory, a hard drive (HDD), or a Solid State Drive (SSD). The memory 603 has stored therein executable program code that the processor 602 executes to implement the aforementioned functions of the cloud terminal 001 or the functions of the server 002, or to execute the steps performed by the terminal 001 or the steps performed by the server 002 in the methods described in the foregoing embodiments.

The description of the flow or structure corresponding to each of the above drawings has emphasis, and a part not described in detail in a certain flow or structure may refer to the related description of other flows or structures.

Claims

1. A method of processing data, comprising:

the cloud terminal sends data;

the server receives the data;

the server processes the data, wherein the processing is encoding the data or decoding the data;

and the server sends the processed data to the cloud terminal.

2. The method of claim 1, wherein the encoding the data comprises:

hardware encoding the data; or,

software encoding the data.

3. The method of claim 1, wherein the decoding the data comprises:

hardware decoding the data; or,

software decoding the data.

4. A method according to any one of claims 1 to 3, characterized in that the method comprises:

the cloud terminal sends a processing rule to the server;

the processing rule is used for determining to perform hardware coding or software decoding on the data, or the processing rule is used for determining to perform software coding or software decoding on the data.

5. The method according to any one of claims 1 to 4, characterized in that it comprises:

and before the cloud terminal sends the data, the cloud terminal acquires a request for processing the data by the terminal.

6. The method according to any one of claims 1 to 4, characterized in that it comprises:

before the cloud terminal sends the data, the cloud terminal obtains a request for processing the data, wherein the request is triggered by an application on the cloud terminal.

7. The method according to any one of claims 1 to 6, characterized in that it comprises:

and the cloud terminal sends the virtual desktop obtained according to the data to the terminal.

8. A system for processing data, the system comprising:

the cloud terminal is used for sending data to the server;

the server is used for receiving the data, processing the data and sending the processed data to the cloud terminal, wherein the processing is to encode the data or decode the data.

9. The system of claim 8,

and the server is used for carrying out hardware coding on the data or carrying out hardware coding on the data.

10. The system of claim 8,

and the server is used for carrying out software coding on the data or carrying out software coding on the data.

11. The system according to any one of claims 8 to 10,

the cloud terminal is configured to send a processing rule to the server, where the processing rule is used to determine to perform hardware encoding or software decoding on the data, or the processing rule is used to determine to perform software encoding or software decoding on the data.

12. The system according to any one of claims 8 to 11,

13. The system according to any one of claims 8 to 11,

the cloud terminal is used for acquiring a request for processing the data, wherein the request is triggered by an application on the cloud terminal.

14. The system according to any one of claims 8 to 13,

15. The system according to any one of claims 8 to 14,

the cloud terminal is a cloud mobile phone.

16. A cloud terminal, characterized in that the cloud terminal is the cloud terminal of any one of claims 8 to 15.

17. A server, characterized in that the cloud terminal is the server according to any one of claims 8 to 15.

18. A computing device, wherein the computing device comprises a processor and a memory; the processor executes the instructions in the memory, causing the computing device to implement the cloud terminal or server of any of claims 8 to 15.

19. A computer program product comprising instructions for carrying out the steps of the method of any one of claims 1 to 7 carried out by a cloud terminal, or for carrying out the steps of the method of any one of claims 1 to 7 carried out by a server.