WO2022048255A1

WO2022048255A1 - Data processing method and system, cloud terminal, server, and computing device

Info

Publication number: WO2022048255A1
Application number: PCT/CN2021/101691
Authority: WO
Inventors: 龙长春; 王坤铭; 吴子侠; 李海跃
Original assignee: 华为云计算技术有限公司
Priority date: 2020-09-07
Filing date: 2021-06-23
Publication date: 2022-03-10
Also published as: CN114157919A

Abstract

A data processing method and system, a cloud terminal, a server, a computing device, and a computer program product, for use in solving the problems of insufficient CPU resources of a cloud terminal and software compatibility. The method comprises: a cloud terminal sends data; a server receives the data; the server processes the data, wherein the processing is to encode or decode the data; the server sends the processed data to the could terminal.

Description

Data processing method and system, cloud terminal, server and computing device

technical field

The present invention relates to the field of cloud technology, and mainly relates to a data processing method and system, a cloud terminal, a server, a computing device and a computer program product.

Background technique

As a new type of service, Cloud Phone plays a very good role in extending and supplementing physical mobile phones, and has begun to emerge in scenarios such as cloud mobile games, cloud mobile live broadcasts, and mobile office. In the above scenarios, the encoding and decoding of video will be involved, for example: (1) cloud mobile game, the screen (or game screen) of the mobile phone needs to be encoded and transmitted to the physical mobile phone through the network; (2) live broadcast scene In the next step, the received camera screen data of the physical mobile phone needs to be decoded and displayed on the cloud mobile phone. When the live application is started on the cloud mobile phone, the cloud mobile phone will encode the camera screen data and send it to the live broadcast platform; (3) In mobile office scenarios, when employees in industries with high demands on information security, such as government and finance, make video calls through instant messaging applications, the camera images on the cloud phone need to be encoded.

However, due to the high performance requirements of the cloud mobile phone for video encoding and decoding, in order to meet the needs of cloud mobile phone live broadcasting and mobile office, it is necessary to increase the CPU specifications of a single cloud mobile phone, which will cause two drawbacks: (1) Make a single cloud mobile phone The increased cost of Taiyun mobile phone is not conducive to building the competitive advantage of cloud mobile phone in the field of live broadcast and mobile office; (2) In addition, the use time of cloud mobile phone has a certain period of time, for example, live broadcast and mobile office are concentrated in the daytime work, Leisure time after get off work at night will cause a waste of CPU resources of the cloud phone.

SUMMARY OF THE INVENTION

The data processing method and system, cloud terminal, server, computing device and computer program product provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings, which can reduce the occupation of CPU resources of cloud mobile phones.

In a first aspect of the present disclosure, a data processing method is provided, which is applied between a cloud terminal and a server. The method includes: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is to encode or decode the data; and the server sends the processed data to the cloud terminal.

The data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal on the server, so that the processor resources of the cloud terminal are saved, and the cost and specifications of a single cloud terminal are reduced. The powerful computing resources owned by the server further improve the efficiency of encoding and decoding tasks on the cloud terminal.

In a possible design of the first aspect, the encoding of the data by the server includes: hardware encoding or software encoding of the data.

The encoder module of the server is responsible for this encoding task. Among them, the software coding task runs on the CPU; the hardware coding task runs on the hardware coding and decoding chip. When the encoding task is performed on the server, the server can provide higher encoding efficiency.

In a possible design of the first aspect, the server decoding the data includes: performing hardware decoding or software decoding on the data.

The decoder module of the server is responsible for this decoding task. Among them, the software decoding task runs on the CPU, and the hardware decoding task runs on the hardware codec chip. When the decoding task is performed on the server, the server can provide higher decoding efficiency.

In a possible design of the first aspect, the data processing method further includes: the cloud terminal sends a processing rule to the server; the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform hardware encoding or software decoding on the data. The data is software encoded or software decoded.

The cloud terminal sends instructions and data to the server, the instructions include processing rules, and the processing rules include encoding or decoding methods: encoding methods include hardware encoding or software encoding, and decoding methods include hardware decoding or software decoding. According to the different encoding and decoding methods, the server calls different encoding modules and decoding modules to perform operations respectively. Please refer to the description above.

In a possible design of the first aspect, the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request from the terminal to process the data.

Before the cloud terminal sends data to the server, the cloud terminal must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the terminal's user.

In a possible design of the first aspect, the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request for processing the data, and the request is triggered by an application on the cloud terminal.

Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, and the request is related to the encoding or decoding requirements of the application layer. An encoding request is issued if the application requires encoding, and a decoding request is issued if the application requires decoding.

The processing rule in the request sent by the application is not the same as the processing rule sent by the cloud terminal to the server in the third possible implementation manner of the first aspect of the present invention, that is, the cloud terminal can perform the processing rule on the processing rule. Modify and send the modified processing rules to the server. There are four ways to modify: software encoding to hardware encoding, hardware encoding to software encoding, software decoding to hardware decoding, and hardware decoding to software decoding. Among them, changing software encoding to hardware encoding and software decoding to hardware decoding can improve the efficiency of encoding and decoding, reduce the efficiency of cloud terminal CPU resources, and can also solve the problem that some applications cannot use hardware encoding or hardware decoding on cloud terminals. compatibility issues.

In a possible design of the first aspect, the data processing method further includes: the cloud terminal sends the virtual desktop obtained according to the data to the terminal.

The application of the cloud terminal obtains the virtual desktop based on the processed data, and delivers the virtual desktop to the terminal, so that the terminal displays the virtual desktop to the user.

In a second aspect, the present invention provides a system for processing data, the system comprising:

Cloud terminal, used to send data to the server;

The server is configured to receive the data, and after processing the data, send the processed data to the cloud terminal, and the processing is to encode or decode the data.

The data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal on the server, so that the processor resources of the cloud terminal are saved, and the cost and specifications of a single cloud terminal are reduced. The powerful computing resources owned by the server further improve the efficiency of the encoding and decoding tasks of the cloud terminal.

In a possible design of the second aspect, the server further includes: hardware encoding or software encoding for the data.

The encoder module of the server is responsible for this encoding task. Among them, the software coding task runs on the CPU; the hardware coding task runs on the hardware codec chip. When the encoding task is performed on the server, the server can provide higher encoding efficiency.

In a possible design of the second aspect, the server further includes: hardware decoding or software decoding for the data.

The decoder module of the server is responsible for this decoding task. Among them, the software decoding task runs on the CPU; the hardware decoding task runs on the hardware codec chip. When the decoding task is performed on the server, the server can provide higher decoding efficiency.

In a possible design of the second aspect, the data processing system further includes:

The cloud terminal is used to send a processing rule to the server, where the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform software encoding or software decoding on the data.

The cloud terminal is used to obtain a request from the terminal to process the data.

Before the cloud terminal sends data to the server, the cloud terminal must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the user of the terminal.

In a possible design of the second aspect, the data processing method further includes:

The cloud terminal is used to obtain a request for processing the data, where the request is triggered by an application on the cloud terminal.

The processing rule in the request sent by the application is not the same as the processing rule sent by the cloud terminal to the server in the third possible implementation manner of the second aspect of the present invention, that is, the cloud terminal can perform the processing rule on the processing rule. Modify and send the modified processing rules to the server. There are four ways to modify: software encoding to hardware encoding, hardware encoding to software encoding, software decoding to hardware decoding, and hardware decoding to software decoding. Among them, changing software encoding to hardware encoding and software decoding to hardware decoding can improve the efficiency of encoding and decoding, reduce the efficiency of cloud terminal CPU resources, and can also solve the incompatibility of some applications when calling hardware encoding or decoding on cloud terminals The problem.

The cloud terminal is used for sending the virtual desktop obtained according to the data to the terminal.

In a possible design of the second aspect, the data processing system further includes: the cloud terminal is a cloud mobile phone.

In a third aspect, the present application provides a cloud terminal, which is the cloud terminal provided by the second aspect of the present application or any possible design of the second aspect.

In a fourth aspect, the present application provides a server, which is the server provided by the second aspect of the present application or any possible design of the second aspect.

In a fifth aspect, the present application provides a computing device, where the computing device includes a processor and a memory. The processor executes the instructions in the memory, so that the computing device implements the cloud terminal or server provided by the second aspect or any possible design of the second aspect of the present application.

In a sixth aspect, the present application provides a computer program product, wherein the computer program product includes instructions for implementing the method executed by the cloud terminal in the first aspect of the present application or the method provided by any possible design of the first aspect. steps, and/or steps executed by a server for implementing the method provided by the first aspect of the application or any possible design of the first aspect.

On the basis of the implementation manners provided by the above aspects, the present application may further combine to provide more implementation manners.

Description of drawings

FIG. 1 is a schematic diagram of an example of a system structure for data processing provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of an example of a system structure for data processing provided by an embodiment of the present application.

FIG. 3 is an exemplary schematic diagram of a system structure for data processing provided by an embodiment of the present application.

FIG. 4 is an example schematic diagram of a system structure for data processing provided by an example in the present application.

FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application.

FIG. 6 is a schematic diagram illustrating an example of a system structure for data processing provided by an embodiment of the present application.

FIG. 7 is a schematic diagram of an example of a system structure of a data processing method provided by an embodiment of the present application

FIG. 8 is an example diagram of a method framework for data processing provided by an embodiment of the present application.

FIG. 9 is a schematic diagram of an example of a method involving data processing in a cloud mobile phone shopping live broadcast scenario provided by an embodiment of the present application.

FIG. 10 is an example structural diagram of an apparatus 300 provided by an embodiment of the present application.

FIG. 11 is an example structural diagram of a computing device 600 provided by an embodiment of the application.

detailed description

FIG. 1 is a schematic structural diagram of a data processing system provided by an embodiment of the present application. As shown in FIG. 1 , the system may include a cloud terminal 001 and a server 002 .

In one embodiment, the cloud terminal 001 may be a virtual device that provides a virtual desktop, for example, the virtual device may be an instance, and the instance may be a virtual machine or a container. In a possible implementation manner, the cloud terminal 001 is a cloud mobile phone.

Server 002 may be a device having computing capabilities, server 002 having memory and a processor. The processor of the server 002 may be a central processing unit (Central Processing Unit, CPU). Optionally, the server 002 may also have one or more of a graphics processor (graphics processing unit, GPU), a neural network processor (Neural network Processing Unit, NPU), and an FPGA (Field Programmable Gate Array). The server 002 The memory can be random access memory (RAM) or solid-state drive (solid-state drive, SSD) and other devices or memory instances with storage capabilities. A solid-state drive may also be called a solid-state drive (SSD)

In the embodiments of the present application, applications may be installed on the cloud terminal 001, and the applications may include applications such as live video, video calls, video conferences, video players, audio players, and mobile games. Optionally, the application on the cloud terminal 001 may issue a request for encoding or decoding, or may issue a request for encoding and decoding at the same time.

In an embodiment, the server 002 may be a physical device deployed in the network, and the cloud terminal 001 and the server 002 communicate based on a communication protocol or a calling protocol, so that the cloud terminal 001 can call the service provided by the server 002 or communicate with the server 002 for data interaction. For example, the communication protocol may be Socket. For example, the calling protocol may be Remote Procedure Call (PRC).

As an example of an implementation scenario, the server 002 may be deployed with a software encoding module and a software decoding module. The software encoding module is used to encode the data. The software decoding module is used to decode the data. In a possible implementation manner, the encoding or decoding operation is executed on the CPU of the server 002 . In a possible implementation manner, the software coding module and the software coding module can be directly deployed in the server 002, or deployed in an instance where the server 002 runs.

As an example of an implementation scenario, the server 002 may be a device capable of hardware encoding, and the server 002 is deployed with a hardware encoding module and a hardware decoding module. The hardware encoding module is used to encode the data. The hardware decoding module is used to decode the data. In a possible implementation manner, the encoding and decoding operations are executed on a hardware encoding chip of the server 002 , for example, the hardware encoding chip may be a GPU or NPU or other chips with hardware encoding capability.

FIG. 2 is a schematic structural diagram of a second data processing system provided by an embodiment of the present application. As shown in FIG. 2 , the server 002 may also be deployed in a cloud server cluster 003 . There are multiple servers similar to the server 002 in the cloud server cluster 003 . The server 002 of the cloud server cluster 003 may also be composed of a certain number of elastic cloud servers (Elastic Cloud Server, ECS), that is, the number of the servers 002 in the cloud server cluster 003 can be elastically scaled.

FIG. 2 is a schematic structural diagram of a third data processing system provided by an embodiment of the present application. As shown in FIG. 3 , the server 002 may also be deployed in a data center 004 .

In a possible embodiment, the cloud server cluster 003 (or the data center 004) may contain one or more servers, the server may provide encoding capability, or the server may provide decoding capability, or the server may provide encoding and decoding at the same time Ability. An example of a scenario is that the server used for encoding and the server used for decoding belong to different clusters or data centers. That is, the cluster consisting of the servers used for encoding is called the encoding cluster, the cluster consisting of the servers used for decoding is called the decoding cluster, the data center where the servers used for encoding are located is called the encoding data center, and the data center where the servers used for decoding are located is called the decoding cluster. Decode the data center.

FIG. 4 is a schematic structural diagram of another data processing system provided by way of example. As shown in FIG. 4 , based on the content described in any one of the schematic diagrams of the system structure in FIG. 1 to FIG. 3 , the system may include a terminal 005 . In some embodiments, the terminal 005 may be a mobile phone, tablet or other physical device with display and communication functions. The terminal 005 can communicate with the cloud terminal 001 through the network, the terminal 005 uploads local data and instructions to the cloud terminal 001, the cloud terminal 001 can deliver the virtual desktop to the terminal 005, and the user can run the cloud terminal 001 on the terminal 005. is installed with a virtual desktop operating system for control. As an example of a scenario, the server 002 in FIG. 4 may also be the cloud server cluster 003 or the data center 004 in FIG. 3 and FIG.

Exemplarily, as shown in FIG. 4 , the terminal 005 may collect video data locally, and upload the collected video data to the software of the cloud terminal 001 . The cloud terminal 001 can send a service request (such as an encoding request or a decoding request) to the server 002 according to the needs of the application, and forward the video data locally collected by the terminal 005 to the server 002; the server 002 returns the processed video data to the cloud terminal 001 application. The application of the cloud terminal obtains a virtual desktop based on the processed video data, for example, a live broadcast interface is generated based on the video data, and the live broadcast interface belongs to part or all of the content displayed by the virtual desktop. The cloud terminal 001 delivers the virtual desktop to the display of the terminal 005 .

FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application, and the method may be applied to any of the systems shown in FIG. 1 to FIG. 4 . FIG. 5 is an embodiment in which the method is applied to the system architecture shown in FIG. 1 . The system in FIG. 1 includes a cloud terminal and a server. The system architectures of FIGS. 2 to 4 are more complex than those of FIG. 1 , and the flowcharts of the method embodiments applied to them have more steps than those of FIG. 5 , but their core steps are still similar to the flowchart of the embodiment shown in FIG. 5 . The flow of the data processing method of the present application is illustrated below with reference to FIG. 1 and FIG. 5 . The flow steps include steps 201 to 205 .

Step 201: The cloud terminal sends data to the server.

The cloud terminal sends the data to the server through the network, and the data is the data to be decoded or encoded.

In a possible implementation manner, the cloud terminal sends instructions for encoding and decoding to the server, and the instructions include encoding or decoding rules.

Optionally, the instruction carries data to be encoded or decoded.

Optionally, the instructions and data (data to be encoded or decoded) are sent independently.

In a possible embodiment, the cloud terminal may continuously send a piece of video stream data, and send an instruction to start encoding to the server while starting to send the video stream. For example, after the video stream stops or the application sends a stop request, The cloud terminal will send an instruction to the server to close the encoding service.

For example, some applications are installed on the cloud terminal, and the applications can send encoding or decoding requests according to their own business requirements. For example, a live broadcast application can send an encoding request, and a video playback application can send a decoding request. The encoding method in the encoding request can be hardware encoding or software encoding. The decoding method in the decoding request may be hardware decoding or software decoding.

In a possible implementation manner, the cloud terminal determines the encoding or decoding rules according to the request sent by the application.

In a possible implementation manner, the encoding or decoding manner in the determined rule may be different from the encoding or decoding rule in the application sending the request. For example, the application sends a software encoding request, and the determined encoding rule is hardware encoding, or the application sends a hardware encoding request, and the determined encoding rule is software encoding, that is, the cloud terminal can modify the encoding method sent by the application. .

In a possible implementation manner, the encoding or decoding method in the determined rule may also be the same as the encoding or decoding rule in the request issued by the application, that is, the cloud terminal may not modify the encoding method issued by the application.

Step 202, the server receives data.

For example, after the server receives the data sent by the cloud terminal, it also needs to start the encoding or decoding task according to the instruction synchronously sent by the cloud terminal, and at the same time return the corresponding service response to the cloud terminal. In a possible embodiment, the server may continuously receive a video stream data, after receiving the start command, the server will start to encode or decode the continuously received video frames, and return to the cloud terminal to start encoding or decoded service response, until the server receives the command to close the encoding or decoding task, and then returns the service response to end encoding or decoding to the cloud terminal.

Step 203, the server performs encoding or decoding services.

For example, the server encodes or decodes the data according to the received encoding rule or decoding rule, where the encoding rule may include an encoding standard, an encoding parameter, and an encoding manner. The decoding rules may include decoding standards, decoding parameters, and decoding methods.

An example implementation of encoding, the server receives the encoding request sent by the cloud terminal, the encoding request contains the encoding standard can be H.264, H.264 and other standards, the encoding parameters can include resolution, bit rate, frame rate, DTS And PTS, etc., the encoding method can be software encoding or hardware encoding. For example, video encoding is the process of converting an original video format file into another video format file, and the main purpose is to compress the original video for storage and transmission. The encoded data is the compressed frame, and the compressed frame has a smaller amount of data than the original frame.

For example, if the encoding method is hardware encoding and decoding, the server invokes a hardware encoding chip (eg, a GPU chip) to perform encoding calculations on the original frame. If the encoding method is software encoding, the server calls the CPU to perform encoding calculation on the original frame.

The encoding method adopted by the server may be the same as the encoding method in the received instruction, but in some embodiments, the encoding method adopted by the server may be different from the encoding method in the request sent by the cloud terminal application. The content has undergone some processing, the processing may include modifying the encoding mode, and reference may be made to the description in step 201 .

For an example implementation of decoding, the server receives a decoding request sent by the cloud terminal, and the decoding standard, decoding parameter, and decoding method can refer to the description in the preceding coding rules. For example, video decoding is the inverse process of video encoding. Through decompression technology, the process of converting the compressed video format file into the original video format file is mainly used to decompress the compressed video frame to support the playback and display of the picture. The decoded data is the decoded frame (which may be lossy relative to the original frame). The decoded frame has a larger amount of data than the compressed frame, and the visual effect after visualization is better.

For example, if the encoding method is hardware encoding and decoding, the server calls a hardware decoding chip to decode and calculate the compressed frame, such as a GPU chip. If the decoding method is software decoding, the server calls the CPU to decode and calculate the compressed frame.

In some embodiments, the decoding method adopted by the server and the decoding method in the received instruction may be the same.

In some embodiments, the decoding method adopted by the server may be different from the decoding method in the request sent by the cloud terminal application, and the cloud terminal performs some processing on the content in the request, and the processing may include modifying the decoding method. For example, the decoding method indicated by the instruction is software decoding, but the decoding method used by the server is hardware decoding.

Step 204, the server sends the processed data.

The server sends the processed data to the cloud terminal through the network.

An example of an application scenario is that the server sends the encoded encoded frame to the cloud terminal, where the encoded frame is processed and generated by a module responsible for encoding within the server.

An example of an application scenario is that the server sends the decoded decoded frame to the cloud terminal, where the decoded frame is processed and generated by a module responsible for decoding inside the server.

In a possible implementation, the server is continuously sending a processed video stream. In some possible embodiments, after receiving an instruction to stop encoding and decoding or stopping video streaming, the server will simultaneously send a response to the cloud terminal after stopping the encoding and decoding service.

Step 205: The cloud terminal receives the processed data.

The cloud terminal receives the data returned by the server from the network. For example, the data may include processed decoded frames or compressed frames.

An example of an application scenario is that a video playback application has previously sent a decoding request, and after receiving the decoded stream returned by the server, the cloud terminal forwards the data to the video playback application at the application layer through the framework layer, and the video playback application sends the decoded data. The video stream is presented in the user interface of the application. In a possible implementation, the desktop content on which the user interface of the application is located will be delivered to a terminal.

An example of an application scenario: a video call application has previously sent an encoding request, and after receiving the encoded stream returned by the server, the cloud terminal forwards the data to the video playback application at the application layer through the framework layer, and the video call application The video stream is sent to one or more other terminals through the network or stored in the cloud. For example, the terminal used by the object of the video call with the current user will receive the encoded video stream.

FIG. 6 is a schematic diagram of a system structure of a data processing method provided by an embodiment of the present application, including an internal structure diagram of a cloud terminal 001 and a server 002 . The cloud terminal includes an application layer and an architecture layer, wherein the application layer includes various types of applications, and this embodiment mainly involves two types of applications 101 and 102 . Application 101 requires the architecture layer to provide video encoding services, and application 102 requires the architecture layer to provide video decoding services. The architecture layer includes a codec management module 111, which consists of a media codec interface, a software codec plug-in or hardware codec plug-in, and a corresponding control module. It is responsible for receiving codec or decoding requests from applications such as live broadcast, video calls, and video playback. Depending on the requirements of different applications, call the corresponding software or hardware plug-in, and return the target video frame data. The communication module 114 of the cloud terminal and the communication module 121 of the server are used to transmit data and control instructions in the network for the cloud terminal and the transcoding cluster through a transmission protocol. The server transcoding management module 122 is responsible for receiving the encoding or decoding request of the cloud phone encoding or decoding module, and delivering the encoding or decoding task to the corresponding external encoding or decoding module, and is responsible for receiving the encoded or decoded data and forwarding it to the communication module 121. The encoder module 123 is responsible for the real video encoding work, receives the original video frame, encodes it by the hardware encoding chip, and returns the compressed frame; the decoder set module 124 is responsible for the real video decoding work, receives the original compressed frame, and is processed by the hardware decoding chip. After decoding, the decoded frame is returned. Optionally, the encoder module 123 and the decoder module 124 may also belong to the same functional module, which does not affect the solution of this embodiment.

In another embodiment, as shown in FIG. 7 , it is a schematic diagram of the system structure of another data processing method provided by the present application. Compared with Figure 6, the codec service of the cloud terminal is carried by a cloud server cluster or data center, rather than a physical server. In some possible embodiments, such a deployment manner can enhance the service efficiency of the external codec. 7 , the cloud server cluster 003 or the data center 004 may include one or more management nodes 131 and two sub-clusters, namely the encoding cluster 132 and the decoding cluster 133, respectively responsible for the real encoding work and decoding work. Optionally, the encoding cluster 132 and the decoding cluster 133 may also be combined or separately distributed in one or more service clusters. In some possible embodiments, different from the codec management module of the server 002 in FIG. 6 , the management node 131 in FIG. 7 may include two functions of communication and cluster management, and each management node may be composed of a server, which The server may not include encoding or decoding modules.

FIG. 8 is an example diagram of a data processing method framework provided by an embodiment of the present application, which is a further explanation of the system in FIG. 7 . In some embodiments, the transcoding management module 111 is further divided into a media encoding/transcoding interface, a software or hardware encoding/decoding plug-in, and a control unit according to functions, wherein the software or hardware transcoding plug-ins 115 and 116 need to be responsible for creating threads and The hardware transcoding control module 113 or the software transcoding module 112 is invoked to implement software and hardware encoding and decoding. The hardware transcoding module 113 is used to implement specific control of hardware encoding or decoding, including enabling encoding or decoding, sending encoding or decoding parameters, sending encoded or decoded video frames, receiving encoded or decoded video frames, and closing Encoder or decoder, etc. The software transcoding module 112 is used to implement specific control of software encoding or decoding, including enabling encoding or decoding, sending encoding or decoding parameters, sending video frames to be encoded or decoded, receiving encoded or decoded video frames, and closing encoding. or decoder etc.

401-421 shown in FIG. 8 are some key steps involved in this embodiment. The size of the serial number does not represent the actual execution order, but only serves as an identifier to represent a certain action and step. If the number of occurrences of a certain identifier is greater than one, it represents The same or similar actions and steps are performed at the identified location. The following describes the flow of the method for processing data of the present application with reference to FIG. 8 . The flow steps include steps 401 to 420 .

Step 401 , request video encoding or decoding.

The video encoding or decoding request is usually initiated by the application, and different types of applications may issue the video encoding or decoding request. For example, mobile phone live broadcast applications can send encoding requests, video playback applications can send software decoding requests, and cloud mobile games and video calls in mobile office scenarios can send encoding and decoding requests at the same time.

Step 402: Invoke a software codec plug-in.

The media codec interface calls the software codec plug-in. In some embodiments, the media codec interface may be a native interface of the cloud terminal operating system. In order not to affect the invocation of the application layer, the native media codec interface can be retained.

In some embodiments, the software codec plug-in may be some existing plug-ins, or a plug-in developed by the inventor.

Step 403, calling the hardware codec plug-in.

The media codec interface calls the hardware codec plug-in. The cloud phone in the prior art only supports step 402, so that some applications need to call the software function of hardware codec unavailable. For example, the inventor can develop a hardware codec plug-in according to the requirements. The embodiment of the method can realize the function of calling a hardware chip to perform encoding or decoding, thereby improving the compatibility of cloud mobile phone application software.

Step 404 , the software codec plug-in calls the corresponding software codec control unit, creates threads, manages multiple streams, and realizes overall control of hardware encoding or decoding work.

Step 405 , the hardware codec plug-in calls the corresponding hardware codec control unit, creates threads, manages multiple streams, and realizes overall control of the hardware codec or decoding work.

Step 406: Send the original frame or the compressed frame.

The software encoding/decoding control unit 112 sends the data of the original frame or the compressed frame to be encoded or decoded to the client Socket, optionally, the client RPC.

The hardware codec control unit 113 sends the data of the original frame or the compressed frame to be encoded or decoded to the client Socket, optionally, the client RPC.

Step 407 , enable or disable the encoding or decoding service.

The software codec control unit sends a start or stop request, and the client Socket (or RPC) receives it and forwards it to the server Socket (or RPC) through the network, which is used to start or close the encoding or decoding service of the remote transcoding cluster.

The hardware decoding control unit sends a start or stop request, and the client Socket (or RPC) receives it and forwards it to the server Socket (or RPC) through the network, which is used to start or stop the encoding or decoding service of the remote transcoding cluster.

In some embodiments, the request to enable or disable may be determined according to a request sent by an application (or user).

Step 408: Send the original frame or the compressed frame.

The client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the software codec unit.

The client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the hardware codec unit.

Step 409, enable or disable the response.

The client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network, and sends it to the software codec unit, which is used to start or close the software codec unit response operation.

The client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network, and sends it to the hardware decoding unit to enable or disable the response operation of the hardware decoding unit.

Step 410: Send a request instruction.

The instruction received from the network is sent to the transcoding cluster management 131 by the server Socket (or RPC), and in some embodiments, the instruction may include starting encoding, starting decoding, closing encoding, closing decoding, and the like.

Step 411: Send the original compressed frame or video frame.

The server Socket (or RPC) sends the original image frame or encoded frame received from the network to the transcoding cluster management unit 122 .

Step 412 , returning the processed decoded frame or compressed frame.

The transcoding cluster management 131 returns the processed decoded frame or compressed frame to the server Socket or RPC.

Step 413, returning a service response.

The processing response is returned by the transcoding cluster management unit, and the server Socket (or RPC) forwards the response.

Step 414, sending an instruction to start the encoder.

The transcoding cluster management 131 sends an instruction to start the encoder to the encoding cluster 132, and the encoding cluster 132 starts encoding after receiving the startup instruction.

Step 415: Send the original video frame.

Transcoding cluster management 131 sends raw video frames to encoding cluster 132 .

Step 416, sending an instruction to turn off the encoder.

The transcoding cluster management 131 sends the instruction of closing the encoder to the encoding cluster 132, and the encoding cluster 132 stops transcoding after receiving the closing instruction.

Step 417, returning to the compressed frame.

The processed compressed frames are returned by the encoding cluster 132 to the transcoding cluster management 131 .

Step 418: Send an instruction to start the decoder.

The transcoding cluster management 131 sends the instruction of starting the decoder to the decoding cluster 133, and the decoding cluster 133 starts decoding after receiving the starting instruction.

Step 419: Send an instruction to close the decoder.

The transcoding cluster management 131 sends an instruction to close the decoder to the decoding cluster 133, and the decoding cluster 133 stops transcoding after receiving the activation instruction.

Step 420: Send the original compressed frame.

The original compressed frames are sent to the decoding cluster 133 by the transcoding cluster management 131 .

Step 421, returning the decoded frame.

The processed decoded frames are returned to the transcoding cluster management 131 by the decoding cluster 133 .

The relevant details of the above steps 401-421 may also refer to the foregoing description.

FIG. 9 is a schematic diagram of a method that may involve cloud terminal data processing in a cloud mobile phone shopping live broadcast scenario provided by an embodiment of the present application. In some embodiments, first, according to the usage scenarios and scale of cloud mobile phones, a certain number of cloud mobile phones and a corresponding cloud server cluster should be purchased. The cloud server cluster can provide the encoding and decoding services described above. Office scenarios require a larger cloud server cluster. Then, you need to complete the configuration for connecting the cloud phone to the external transcoding cluster, such as the IP and port configuration of the socket on the cloud phone. Finally, use the configured cloud phone and cloud server cluster for live broadcast and mobile office. In the schematic diagram of the method shown in FIG. 9, the serial numbers 501-514 represent the key steps involved in this embodiment, which describe that in a cloud mobile phone shopping live broadcast, from user A starting the live broadcast, the shopping live broadcast application sends a software coding request until the user In the process of closing the live broadcast, one embodiment of this method is how to perform remote hardware video encoding services based on cloud mobile phones.

Step 501: User A opens the shopping live broadcast application software on the cloud mobile phone, and starts the live broadcast. In some possible embodiments, the user may remotely control the application on the cloud terminal 001 based on the visual interface of the terminal 005 shown in FIG. 4 . For example, the terminal 005 may be a smart phone, including a camera or other photographing device, which is used to collect the video stream of the user during the live broadcast in real time, and transmit the video stream to the cloud phone synchronously.

Step 502: The shopping live broadcast application invokes the MediaCodec interface to start the video encoding operation. The live application receives the video stream, sends a software encoding request to the MediaCodec interface, and requests the encoding service of the cloud server cluster;

Steps 503-404: The MediaCodec process invokes the encoder control unit based on the software encoding plug-in to create an encoding thread.

Step 505: The encoder control unit generates a relevant instruction, the instruction includes an encoding rule, and modifies the encoding mode in the encoding rule from software encoding to hardware encoding. Next, the encoder control unit establishes a socket connection with the external transcoding cluster through the client socket, and sends a request to start the external hardware encoder to the server socket.

Step 506: The server socket of the cloud server cluster forwards the received hardware coding instruction to the transcoding cluster management node. After the transcoding cluster management node receives the instruction to start the specific external encoder, it creates a thread to start the coding cluster.

Step 507: The cloud phone encoder control unit sends a request for obtaining the encoding header information, and the client socket forwards the request to the client socket of the cloud server cluster.

Step 508: The transcoding cluster management node requests and obtains the encoding header from the encoder module of a specific server in the encoding cluster, and forwards it to the cloud mobile phone through the network.

Step 509: The cloud phone encoder control unit receives the response of the encoding header information.

Step 510: The cloud phone encoder control module sends the original video frame data to the transcoding management node of the cloud server cluster, and requests the external encoding cluster to perform hardware encoding.

Step 511: The transcoding cluster management node is responsible for allocating the encoding server to the cloud mobile phone, and the cloud mobile phone sends the original video frame to the encoder of the specific server.

Step 512: The encoder returns the encoded compressed video frame to the transcoding cluster management node, and forwards it to the cloud mobile phone through the network.

Step 513: The encoder module of the cloud phone receives the compressed video frame data through the socket, and returns it to the live broadcast application for use. The live application on the cloud mobile phone will receive the compressed frame data stream and forward it to the public cloud. More network users can download the compressed frame data stream on their mobile phones or PCs and other terminals. After decoding locally, they can watch the video in real time. Live shopping.

Step 514: User A closes the shopping live broadcast application on the cloud mobile phone, and ends the live broadcast.

Step 515: The shopping live broadcast application software ends the video encoding operation by calling the MediaCodec interface.

Steps 516-517, the MediaCodec process calls the encoder control unit based on the software encoding plug-in, and requests to end encoding;

Step 518: The encoder control unit releases the remote encoder, sends a request to close the encoder of the server, and closes the socket connection with the cloud server cluster.

Step 519: The transcoding cluster management node closes the encoding service of the specific server.

FIG. 10 is an apparatus 300 deployed on a server 002 according to an embodiment of the present application. The apparatus 300 includes a receiving unit 301 , a processing unit 302 , and a sending unit 303 .

The receiving unit 301 is configured to receive data from the cloud terminal 001 . In some embodiments, the data may be data to be encoded or decoded. In other embodiments, the data may further include an instruction for instructing an encoding or decoding task. For the content of the instruction, please refer to the foregoing related description.

The processing unit 302 is used for encoding or decoding data. As an example of an implementation scenario, the processing unit encodes the data to be encoded, and the encoding method includes hardware encoding or software encoding. Please refer to the foregoing description. As an example of an implementation scenario, the processing unit decodes the data to be decoded, and the decoding method includes hardware decoding or software decoding. Please refer to the foregoing description.

The sending unit 303 is configured to send data to the cloud terminal 001 . The transmitted data is already decoded or encoded data.

Optionally, the processing unit 302 is configured to perform hardware encoding on the data or perform software encoding on the data.

Optionally, the processing unit 302 is configured to perform hardware decoding on the data or perform software decoding on the data.

This embodiment of the present application further provides a computing device 600, as shown in FIG. 11 . The computing device 600 may be a device for deploying the cloud terminal 001 , or the computing device 600 may be the server 002 .

FIG. 11 provides a schematic structural diagram of a computing device 600 . The computing device 600 includes a bus 601 , a processor 602 and a memory 603 . The processor 602 and the memory 603 communicate through the bus 601 .

The bus 601 may be a peripheral component interconnection standard bus or an extended industry standard structure bus or the like. The processor 602 may be a central processing unit (Central Processing Unit, CPU), in addition, there may also be a graphics processing unit (graphics processing unit, GPU), a neural network processor (Neural network Processing Unit, NPU), FPGA (Field Programmable Gate) Array) one or more. The memory 603 may include random access memory (RAM), read-only memory (ROM), flash memory, hard drive (HDD) or solid state drive (SSD) ) one or more of. Executable program codes are stored in the memory 603, and the processor 602 executes the executable program codes to realize the functions of the aforementioned cloud terminal 001 or the functions of the server 002, or to execute the steps performed by the terminal 001 in the methods described in the aforementioned embodiments. Or steps performed by server 002 .

The descriptions of the processes or structures corresponding to each of the above-mentioned drawings have their own emphasis, and for the parts that are not described in detail in a certain process or structure, reference may be made to the related descriptions of other processes or structures.

Claims

A method for processing data, comprising:

The cloud terminal sends data;

the server receives the data;

The server processes the data, and the processing is to encode the data or to decode the data;

The server sends the processed data to the cloud terminal.
The method of claim 1, wherein the encoding the data comprises:

hardware-encode the data; or,

The data is software encoded.
The method of claim 1, wherein the decoding the data comprises:

hardware decoding the data; or,

The data is software decoded.
The method according to any one of claims 1 to 3, wherein the method comprises:

sending, by the cloud terminal, processing rules to the server;

The processing rule is used to determine to perform hardware encoding or software decoding on the data, or the processing rule is used to determine to perform software encoding or software decoding on the data.
The method according to any one of claims 1 to 4, wherein the method comprises:

Before the cloud terminal sends the data, the cloud terminal obtains a request from the terminal to process the data.
The method according to any one of claims 1 to 4, wherein the method comprises:

Before the cloud terminal sends the data, the cloud terminal obtains a request for processing the data, and the request is triggered by an application on the cloud terminal.
The method according to any one of claims 1 to 6, wherein the method comprises:

The cloud terminal sends the virtual desktop obtained according to the data to the terminal.
A system for processing data, wherein the system comprises:

Cloud terminal, used to send data to the server;

The server is configured to receive the data, the server processes the data, and sends the processed data to the cloud terminal, and the processing is to encode or decode the data.
The system of claim 8, wherein:

The server is configured to perform hardware encoding on the data, or perform hardware encoding on the data.
The system of claim 8, wherein:

The server is configured to perform software encoding on the data, or perform software encoding on the data.
The system according to any one of claims 8 to 10, characterized in that:

The cloud terminal is configured to send a processing rule to the server, where the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform software encoding on the data Or software decoding.
The system according to any one of claims 8 to 11, wherein:

The cloud terminal is configured to obtain a request from the terminal to process the data.
The system according to any one of claims 8 to 11, wherein:

The cloud terminal is configured to obtain a request for processing the data, where the request is triggered by an application on the cloud terminal.
The system according to any one of claims 8 to 13, wherein:

The cloud terminal is configured to send the virtual desktop obtained according to the data to the terminal.
The system according to any one of claims 8 to 14, wherein:

The cloud terminal is a cloud mobile phone.
A cloud terminal, wherein the cloud terminal is the cloud terminal according to any one of claims 8 to 15.
A server, wherein the cloud terminal is the server according to any one of claims 8 to 15.
A computing device, characterized in that the computing device includes a processor and a memory; the processor executes instructions in the memory, so that the computing device implements the cloud terminal according to any one of claims 8 to 15 or server.
A computer program product, the computer program product comprising instructions for implementing the steps performed by the cloud terminal in the method of any one of claims 1 to 7, or for implementing any one of claims 1 to 7. A method of the steps performed by a server.