WO2022048255A1 - Procédé et système de traitement des données, terminal en nuage, serveur, et dispositif informatique - Google Patents

Procédé et système de traitement des données, terminal en nuage, serveur, et dispositif informatique Download PDF

Info

Publication number
WO2022048255A1
WO2022048255A1 PCT/CN2021/101691 CN2021101691W WO2022048255A1 WO 2022048255 A1 WO2022048255 A1 WO 2022048255A1 CN 2021101691 W CN2021101691 W CN 2021101691W WO 2022048255 A1 WO2022048255 A1 WO 2022048255A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
server
encoding
decoding
cloud terminal
Prior art date
Application number
PCT/CN2021/101691
Other languages
English (en)
Chinese (zh)
Inventor
龙长春
王坤铭
吴子侠
李海跃
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2022048255A1 publication Critical patent/WO2022048255A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
    • H04N21/64723—Monitoring of network processes or resources, e.g. monitoring of network load
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
    • H04N21/64746—Control signals issued by the network directed to the server or the client
    • H04N21/64761—Control signals issued by the network directed to the server or the client directed to the server

Definitions

  • the present invention relates to the field of cloud technology, and mainly relates to a data processing method and system, a cloud terminal, a server, a computing device and a computer program product.
  • Cloud Phone plays a very good role in extending and supplementing physical mobile phones, and has begun to emerge in scenarios such as cloud mobile games, cloud mobile live broadcasts, and mobile office.
  • the encoding and decoding of video will be involved, for example: (1) cloud mobile game, the screen (or game screen) of the mobile phone needs to be encoded and transmitted to the physical mobile phone through the network; (2) live broadcast scene In the next step, the received camera screen data of the physical mobile phone needs to be decoded and displayed on the cloud mobile phone.
  • the cloud mobile phone When the live application is started on the cloud mobile phone, the cloud mobile phone will encode the camera screen data and send it to the live broadcast platform; (3) In mobile office scenarios, when employees in industries with high demands on information security, such as government and finance, make video calls through instant messaging applications, the camera images on the cloud phone need to be encoded.
  • a data processing method is provided, which is applied between a cloud terminal and a server.
  • the method includes: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is to encode or decode the data; and the server sends the processed data to the cloud terminal.
  • the data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal on the server, so that the processor resources of the cloud terminal are saved, and the cost and specifications of a single cloud terminal are reduced.
  • the powerful computing resources owned by the server further improve the efficiency of encoding and decoding tasks on the cloud terminal.
  • the encoding of the data by the server includes: hardware encoding or software encoding of the data.
  • the encoder module of the server is responsible for this encoding task.
  • the software coding task runs on the CPU;
  • the hardware coding task runs on the hardware coding and decoding chip.
  • the server can provide higher encoding efficiency.
  • the server decoding the data includes: performing hardware decoding or software decoding on the data.
  • the decoder module of the server is responsible for this decoding task.
  • the software decoding task runs on the CPU
  • the hardware decoding task runs on the hardware codec chip.
  • the server can provide higher decoding efficiency.
  • the data processing method further includes: the cloud terminal sends a processing rule to the server; the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform hardware encoding or software decoding on the data.
  • the data is software encoded or software decoded.
  • the cloud terminal sends instructions and data to the server, the instructions include processing rules, and the processing rules include encoding or decoding methods: encoding methods include hardware encoding or software encoding, and decoding methods include hardware decoding or software decoding. According to the different encoding and decoding methods, the server calls different encoding modules and decoding modules to perform operations respectively. Please refer to the description above.
  • the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request from the terminal to process the data.
  • the cloud terminal Before the cloud terminal sends data to the server, the cloud terminal must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the terminal's user.
  • the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request for processing the data, and the request is triggered by an application on the cloud terminal.
  • the cloud terminal Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, and the request is related to the encoding or decoding requirements of the application layer.
  • An encoding request is issued if the application requires encoding, and a decoding request is issued if the application requires decoding.
  • the processing rule in the request sent by the application is not the same as the processing rule sent by the cloud terminal to the server in the third possible implementation manner of the first aspect of the present invention, that is, the cloud terminal can perform the processing rule on the processing rule. Modify and send the modified processing rules to the server.
  • the data processing method further includes: the cloud terminal sends the virtual desktop obtained according to the data to the terminal.
  • the application of the cloud terminal obtains the virtual desktop based on the processed data, and delivers the virtual desktop to the terminal, so that the terminal displays the virtual desktop to the user.
  • the present invention provides a system for processing data, the system comprising:
  • Cloud terminal used to send data to the server
  • the server is configured to receive the data, and after processing the data, send the processed data to the cloud terminal, and the processing is to encode or decode the data.
  • the data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal on the server, so that the processor resources of the cloud terminal are saved, and the cost and specifications of a single cloud terminal are reduced.
  • the powerful computing resources owned by the server further improve the efficiency of the encoding and decoding tasks of the cloud terminal.
  • the server further includes: hardware encoding or software encoding for the data.
  • the encoder module of the server is responsible for this encoding task.
  • the software coding task runs on the CPU;
  • the hardware coding task runs on the hardware codec chip.
  • the server can provide higher encoding efficiency.
  • the server further includes: hardware decoding or software decoding for the data.
  • the decoder module of the server is responsible for this decoding task.
  • the software decoding task runs on the CPU;
  • the hardware decoding task runs on the hardware codec chip.
  • the server can provide higher decoding efficiency.
  • the data processing system further includes:
  • the cloud terminal is used to send a processing rule to the server, where the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform software encoding or software decoding on the data.
  • the cloud terminal sends instructions and data to the server, the instructions include processing rules, and the processing rules include encoding or decoding methods: encoding methods include hardware encoding or software encoding, and decoding methods include hardware decoding or software decoding. According to the different encoding and decoding methods, the server calls different encoding modules and decoding modules to perform operations respectively. Please refer to the description above.
  • the data processing system further includes:
  • the cloud terminal is used to obtain a request from the terminal to process the data.
  • the cloud terminal Before the cloud terminal sends data to the server, the cloud terminal must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the user of the terminal.
  • the data processing method further includes:
  • the cloud terminal is used to obtain a request for processing the data, where the request is triggered by an application on the cloud terminal.
  • the cloud terminal Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, and the request is related to the encoding or decoding requirements of the application layer.
  • An encoding request is issued if the application requires encoding, and a decoding request is issued if the application requires decoding.
  • the processing rule in the request sent by the application is not the same as the processing rule sent by the cloud terminal to the server in the third possible implementation manner of the second aspect of the present invention, that is, the cloud terminal can perform the processing rule on the processing rule. Modify and send the modified processing rules to the server.
  • the data processing system further includes:
  • the cloud terminal is used for sending the virtual desktop obtained according to the data to the terminal.
  • the application of the cloud terminal obtains the virtual desktop based on the processed data, and delivers the virtual desktop to the terminal, so that the terminal displays the virtual desktop to the user.
  • the data processing system further includes: the cloud terminal is a cloud mobile phone.
  • the present application provides a cloud terminal, which is the cloud terminal provided by the second aspect of the present application or any possible design of the second aspect.
  • the present application provides a server, which is the server provided by the second aspect of the present application or any possible design of the second aspect.
  • the present application provides a computing device, where the computing device includes a processor and a memory.
  • the processor executes the instructions in the memory, so that the computing device implements the cloud terminal or server provided by the second aspect or any possible design of the second aspect of the present application.
  • the present application provides a computer program product, wherein the computer program product includes instructions for implementing the method executed by the cloud terminal in the first aspect of the present application or the method provided by any possible design of the first aspect. steps, and/or steps executed by a server for implementing the method provided by the first aspect of the application or any possible design of the first aspect.
  • the present application may further combine to provide more implementation manners.
  • FIG. 1 is a schematic diagram of an example of a system structure for data processing provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of an example of a system structure for data processing provided by an embodiment of the present application.
  • FIG. 3 is an exemplary schematic diagram of a system structure for data processing provided by an embodiment of the present application.
  • FIG. 4 is an example schematic diagram of a system structure for data processing provided by an example in the present application.
  • FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram illustrating an example of a system structure for data processing provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of an example of a system structure of a data processing method provided by an embodiment of the present application
  • FIG. 8 is an example diagram of a method framework for data processing provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of an example of a method involving data processing in a cloud mobile phone shopping live broadcast scenario provided by an embodiment of the present application.
  • FIG. 10 is an example structural diagram of an apparatus 300 provided by an embodiment of the present application.
  • FIG. 11 is an example structural diagram of a computing device 600 provided by an embodiment of the application.
  • FIG. 1 is a schematic structural diagram of a data processing system provided by an embodiment of the present application. As shown in FIG. 1 , the system may include a cloud terminal 001 and a server 002 .
  • the cloud terminal 001 may be a virtual device that provides a virtual desktop, for example, the virtual device may be an instance, and the instance may be a virtual machine or a container. In a possible implementation manner, the cloud terminal 001 is a cloud mobile phone.
  • Server 002 may be a device having computing capabilities, server 002 having memory and a processor.
  • the processor of the server 002 may be a central processing unit (Central Processing Unit, CPU).
  • the server 002 may also have one or more of a graphics processor (graphics processing unit, GPU), a neural network processor (Neural network Processing Unit, NPU), and an FPGA (Field Programmable Gate Array).
  • the server 002 The memory can be random access memory (RAM) or solid-state drive (solid-state drive, SSD) and other devices or memory instances with storage capabilities.
  • a solid-state drive may also be called a solid-state drive (SSD)
  • applications may be installed on the cloud terminal 001, and the applications may include applications such as live video, video calls, video conferences, video players, audio players, and mobile games.
  • the application on the cloud terminal 001 may issue a request for encoding or decoding, or may issue a request for encoding and decoding at the same time.
  • the server 002 may be a physical device deployed in the network, and the cloud terminal 001 and the server 002 communicate based on a communication protocol or a calling protocol, so that the cloud terminal 001 can call the service provided by the server 002 or communicate with the server 002 for data interaction.
  • the communication protocol may be Socket.
  • the calling protocol may be Remote Procedure Call (PRC).
  • the server 002 may be deployed with a software encoding module and a software decoding module.
  • the software encoding module is used to encode the data.
  • the software decoding module is used to decode the data.
  • the encoding or decoding operation is executed on the CPU of the server 002 .
  • the software coding module and the software coding module can be directly deployed in the server 002, or deployed in an instance where the server 002 runs.
  • the server 002 may be a device capable of hardware encoding, and the server 002 is deployed with a hardware encoding module and a hardware decoding module.
  • the hardware encoding module is used to encode the data.
  • the hardware decoding module is used to decode the data.
  • the encoding and decoding operations are executed on a hardware encoding chip of the server 002 , for example, the hardware encoding chip may be a GPU or NPU or other chips with hardware encoding capability.
  • FIG. 2 is a schematic structural diagram of a second data processing system provided by an embodiment of the present application.
  • the server 002 may also be deployed in a cloud server cluster 003 .
  • the server 002 of the cloud server cluster 003 may also be composed of a certain number of elastic cloud servers (Elastic Cloud Server, ECS), that is, the number of the servers 002 in the cloud server cluster 003 can be elastically scaled.
  • ECS Elastic Cloud Server
  • FIG. 2 is a schematic structural diagram of a third data processing system provided by an embodiment of the present application. As shown in FIG. 3 , the server 002 may also be deployed in a data center 004 .
  • the cloud server cluster 003 may contain one or more servers, the server may provide encoding capability, or the server may provide decoding capability, or the server may provide encoding and decoding at the same time Ability.
  • An example of a scenario is that the server used for encoding and the server used for decoding belong to different clusters or data centers. That is, the cluster consisting of the servers used for encoding is called the encoding cluster, the cluster consisting of the servers used for decoding is called the decoding cluster, the data center where the servers used for encoding are located is called the encoding data center, and the data center where the servers used for decoding are located is called the decoding cluster. Decode the data center.
  • FIG. 4 is a schematic structural diagram of another data processing system provided by way of example.
  • the system may include a terminal 005 .
  • the terminal 005 may be a mobile phone, tablet or other physical device with display and communication functions.
  • the terminal 005 can communicate with the cloud terminal 001 through the network, the terminal 005 uploads local data and instructions to the cloud terminal 001, the cloud terminal 001 can deliver the virtual desktop to the terminal 005, and the user can run the cloud terminal 001 on the terminal 005.
  • the server 002 in FIG. 4 may also be the cloud server cluster 003 or the data center 004 in FIG. 3 and FIG.
  • the terminal 005 may collect video data locally, and upload the collected video data to the software of the cloud terminal 001 .
  • the cloud terminal 001 can send a service request (such as an encoding request or a decoding request) to the server 002 according to the needs of the application, and forward the video data locally collected by the terminal 005 to the server 002; the server 002 returns the processed video data to the cloud terminal 001 application.
  • the application of the cloud terminal obtains a virtual desktop based on the processed video data, for example, a live broadcast interface is generated based on the video data, and the live broadcast interface belongs to part or all of the content displayed by the virtual desktop.
  • the cloud terminal 001 delivers the virtual desktop to the display of the terminal 005 .
  • FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application, and the method may be applied to any of the systems shown in FIG. 1 to FIG. 4 .
  • FIG. 5 is an embodiment in which the method is applied to the system architecture shown in FIG. 1 .
  • the system in FIG. 1 includes a cloud terminal and a server.
  • the system architectures of FIGS. 2 to 4 are more complex than those of FIG. 1 , and the flowcharts of the method embodiments applied to them have more steps than those of FIG. 5 , but their core steps are still similar to the flowchart of the embodiment shown in FIG. 5 .
  • the flow of the data processing method of the present application is illustrated below with reference to FIG. 1 and FIG. 5 .
  • the flow steps include steps 201 to 205 .
  • Step 201 The cloud terminal sends data to the server.
  • the cloud terminal sends the data to the server through the network, and the data is the data to be decoded or encoded.
  • the cloud terminal sends instructions for encoding and decoding to the server, and the instructions include encoding or decoding rules.
  • the instruction carries data to be encoded or decoded.
  • the instructions and data are sent independently.
  • the cloud terminal may continuously send a piece of video stream data, and send an instruction to start encoding to the server while starting to send the video stream. For example, after the video stream stops or the application sends a stop request, The cloud terminal will send an instruction to the server to close the encoding service.
  • some applications are installed on the cloud terminal, and the applications can send encoding or decoding requests according to their own business requirements.
  • a live broadcast application can send an encoding request
  • a video playback application can send a decoding request.
  • the encoding method in the encoding request can be hardware encoding or software encoding.
  • the decoding method in the decoding request may be hardware decoding or software decoding.
  • the cloud terminal determines the encoding or decoding rules according to the request sent by the application.
  • the encoding or decoding manner in the determined rule may be different from the encoding or decoding rule in the application sending the request.
  • the application sends a software encoding request
  • the determined encoding rule is hardware encoding
  • the application sends a hardware encoding request
  • the determined encoding rule is software encoding, that is, the cloud terminal can modify the encoding method sent by the application. .
  • the encoding or decoding method in the determined rule may also be the same as the encoding or decoding rule in the request issued by the application, that is, the cloud terminal may not modify the encoding method issued by the application.
  • Step 202 the server receives data.
  • the server after the server receives the data sent by the cloud terminal, it also needs to start the encoding or decoding task according to the instruction synchronously sent by the cloud terminal, and at the same time return the corresponding service response to the cloud terminal.
  • the server may continuously receive a video stream data, after receiving the start command, the server will start to encode or decode the continuously received video frames, and return to the cloud terminal to start encoding or decoded service response, until the server receives the command to close the encoding or decoding task, and then returns the service response to end encoding or decoding to the cloud terminal.
  • Step 203 the server performs encoding or decoding services.
  • the server encodes or decodes the data according to the received encoding rule or decoding rule, where the encoding rule may include an encoding standard, an encoding parameter, and an encoding manner.
  • the decoding rules may include decoding standards, decoding parameters, and decoding methods.
  • the server receives the encoding request sent by the cloud terminal, the encoding request contains the encoding standard can be H.264, H.264 and other standards, the encoding parameters can include resolution, bit rate, frame rate, DTS And PTS, etc., the encoding method can be software encoding or hardware encoding.
  • video encoding is the process of converting an original video format file into another video format file, and the main purpose is to compress the original video for storage and transmission.
  • the encoded data is the compressed frame, and the compressed frame has a smaller amount of data than the original frame.
  • the server invokes a hardware encoding chip (eg, a GPU chip) to perform encoding calculations on the original frame.
  • a hardware encoding chip eg, a GPU chip
  • the server calls the CPU to perform encoding calculation on the original frame.
  • the encoding method adopted by the server may be the same as the encoding method in the received instruction, but in some embodiments, the encoding method adopted by the server may be different from the encoding method in the request sent by the cloud terminal application.
  • the content has undergone some processing, the processing may include modifying the encoding mode, and reference may be made to the description in step 201 .
  • the server receives a decoding request sent by the cloud terminal, and the decoding standard, decoding parameter, and decoding method can refer to the description in the preceding coding rules.
  • video decoding is the inverse process of video encoding. Through decompression technology, the process of converting the compressed video format file into the original video format file is mainly used to decompress the compressed video frame to support the playback and display of the picture.
  • the decoded data is the decoded frame (which may be lossy relative to the original frame).
  • the decoded frame has a larger amount of data than the compressed frame, and the visual effect after visualization is better.
  • the server calls a hardware decoding chip to decode and calculate the compressed frame, such as a GPU chip.
  • the decoding method is software decoding
  • the server calls the CPU to decode and calculate the compressed frame.
  • the decoding method adopted by the server and the decoding method in the received instruction may be the same.
  • the decoding method adopted by the server may be different from the decoding method in the request sent by the cloud terminal application, and the cloud terminal performs some processing on the content in the request, and the processing may include modifying the decoding method.
  • the decoding method indicated by the instruction is software decoding, but the decoding method used by the server is hardware decoding.
  • Step 204 the server sends the processed data.
  • the server sends the processed data to the cloud terminal through the network.
  • An example of an application scenario is that the server sends the encoded encoded frame to the cloud terminal, where the encoded frame is processed and generated by a module responsible for encoding within the server.
  • An example of an application scenario is that the server sends the decoded decoded frame to the cloud terminal, where the decoded frame is processed and generated by a module responsible for decoding inside the server.
  • the server is continuously sending a processed video stream.
  • the server after receiving an instruction to stop encoding and decoding or stopping video streaming, the server will simultaneously send a response to the cloud terminal after stopping the encoding and decoding service.
  • Step 205 The cloud terminal receives the processed data.
  • the cloud terminal receives the data returned by the server from the network.
  • the data may include processed decoded frames or compressed frames.
  • An example of an application scenario is that a video playback application has previously sent a decoding request, and after receiving the decoded stream returned by the server, the cloud terminal forwards the data to the video playback application at the application layer through the framework layer, and the video playback application sends the decoded data.
  • the video stream is presented in the user interface of the application.
  • the desktop content on which the user interface of the application is located will be delivered to a terminal.
  • a video call application has previously sent an encoding request, and after receiving the encoded stream returned by the server, the cloud terminal forwards the data to the video playback application at the application layer through the framework layer, and the video call application
  • the video stream is sent to one or more other terminals through the network or stored in the cloud. For example, the terminal used by the object of the video call with the current user will receive the encoded video stream.
  • FIG. 6 is a schematic diagram of a system structure of a data processing method provided by an embodiment of the present application, including an internal structure diagram of a cloud terminal 001 and a server 002 .
  • the cloud terminal includes an application layer and an architecture layer, wherein the application layer includes various types of applications, and this embodiment mainly involves two types of applications 101 and 102 .
  • Application 101 requires the architecture layer to provide video encoding services
  • application 102 requires the architecture layer to provide video decoding services.
  • the architecture layer includes a codec management module 111, which consists of a media codec interface, a software codec plug-in or hardware codec plug-in, and a corresponding control module. It is responsible for receiving codec or decoding requests from applications such as live broadcast, video calls, and video playback.
  • the communication module 114 of the cloud terminal and the communication module 121 of the server are used to transmit data and control instructions in the network for the cloud terminal and the transcoding cluster through a transmission protocol.
  • the server transcoding management module 122 is responsible for receiving the encoding or decoding request of the cloud phone encoding or decoding module, and delivering the encoding or decoding task to the corresponding external encoding or decoding module, and is responsible for receiving the encoded or decoded data and forwarding it to the communication module 121.
  • the encoder module 123 is responsible for the real video encoding work, receives the original video frame, encodes it by the hardware encoding chip, and returns the compressed frame;
  • the decoder set module 124 is responsible for the real video decoding work, receives the original compressed frame, and is processed by the hardware decoding chip. After decoding, the decoded frame is returned.
  • the encoder module 123 and the decoder module 124 may also belong to the same functional module, which does not affect the solution of this embodiment.
  • FIG. 7 it is a schematic diagram of the system structure of another data processing method provided by the present application.
  • the codec service of the cloud terminal is carried by a cloud server cluster or data center, rather than a physical server.
  • the cloud server cluster 003 or the data center 004 may include one or more management nodes 131 and two sub-clusters, namely the encoding cluster 132 and the decoding cluster 133, respectively responsible for the real encoding work and decoding work.
  • the encoding cluster 132 and the decoding cluster 133 may also be combined or separately distributed in one or more service clusters.
  • the management node 131 in FIG. 7 may include two functions of communication and cluster management, and each management node may be composed of a server, which The server may not include encoding or decoding modules.
  • FIG. 8 is an example diagram of a data processing method framework provided by an embodiment of the present application, which is a further explanation of the system in FIG. 7 .
  • the transcoding management module 111 is further divided into a media encoding/transcoding interface, a software or hardware encoding/decoding plug-in, and a control unit according to functions, wherein the software or hardware transcoding plug-ins 115 and 116 need to be responsible for creating threads and
  • the hardware transcoding control module 113 or the software transcoding module 112 is invoked to implement software and hardware encoding and decoding.
  • the hardware transcoding module 113 is used to implement specific control of hardware encoding or decoding, including enabling encoding or decoding, sending encoding or decoding parameters, sending encoded or decoded video frames, receiving encoded or decoded video frames, and closing Encoder or decoder, etc.
  • the software transcoding module 112 is used to implement specific control of software encoding or decoding, including enabling encoding or decoding, sending encoding or decoding parameters, sending video frames to be encoded or decoded, receiving encoded or decoded video frames, and closing encoding. or decoder etc.
  • the size of the serial number does not represent the actual execution order, but only serves as an identifier to represent a certain action and step. If the number of occurrences of a certain identifier is greater than one, it represents The same or similar actions and steps are performed at the identified location.
  • the flow steps include steps 401 to 420 .
  • Step 401 request video encoding or decoding.
  • the video encoding or decoding request is usually initiated by the application, and different types of applications may issue the video encoding or decoding request.
  • mobile phone live broadcast applications can send encoding requests
  • video playback applications can send software decoding requests
  • cloud mobile games and video calls in mobile office scenarios can send encoding and decoding requests at the same time.
  • Step 402 Invoke a software codec plug-in.
  • the media codec interface calls the software codec plug-in.
  • the media codec interface may be a native interface of the cloud terminal operating system. In order not to affect the invocation of the application layer, the native media codec interface can be retained.
  • the software codec plug-in may be some existing plug-ins, or a plug-in developed by the inventor.
  • Step 403 calling the hardware codec plug-in.
  • the media codec interface calls the hardware codec plug-in.
  • the cloud phone in the prior art only supports step 402, so that some applications need to call the software function of hardware codec unavailable.
  • the inventor can develop a hardware codec plug-in according to the requirements.
  • the embodiment of the method can realize the function of calling a hardware chip to perform encoding or decoding, thereby improving the compatibility of cloud mobile phone application software.
  • Step 404 the software codec plug-in calls the corresponding software codec control unit, creates threads, manages multiple streams, and realizes overall control of hardware encoding or decoding work.
  • Step 405 the hardware codec plug-in calls the corresponding hardware codec control unit, creates threads, manages multiple streams, and realizes overall control of the hardware codec or decoding work.
  • Step 406 Send the original frame or the compressed frame.
  • the software encoding/decoding control unit 112 sends the data of the original frame or the compressed frame to be encoded or decoded to the client Socket, optionally, the client RPC.
  • the hardware codec control unit 113 sends the data of the original frame or the compressed frame to be encoded or decoded to the client Socket, optionally, the client RPC.
  • Step 407 enable or disable the encoding or decoding service.
  • the software codec control unit sends a start or stop request, and the client Socket (or RPC) receives it and forwards it to the server Socket (or RPC) through the network, which is used to start or close the encoding or decoding service of the remote transcoding cluster.
  • the hardware decoding control unit sends a start or stop request, and the client Socket (or RPC) receives it and forwards it to the server Socket (or RPC) through the network, which is used to start or stop the encoding or decoding service of the remote transcoding cluster.
  • the request to enable or disable may be determined according to a request sent by an application (or user).
  • Step 408 Send the original frame or the compressed frame.
  • the client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the software codec unit.
  • the client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the hardware codec unit.
  • Step 409 enable or disable the response.
  • the client Socket receives the service response of the encoding or decoding cluster from the network, and sends it to the software codec unit, which is used to start or close the software codec unit response operation.
  • the client Socket receives the service response of the encoding or decoding cluster from the network, and sends it to the hardware decoding unit to enable or disable the response operation of the hardware decoding unit.
  • Step 410 Send a request instruction.
  • the instruction received from the network is sent to the transcoding cluster management 131 by the server Socket (or RPC), and in some embodiments, the instruction may include starting encoding, starting decoding, closing encoding, closing decoding, and the like.
  • Step 411 Send the original compressed frame or video frame.
  • the server Socket (or RPC) sends the original image frame or encoded frame received from the network to the transcoding cluster management unit 122 .
  • Step 412 returning the processed decoded frame or compressed frame.
  • the transcoding cluster management 131 returns the processed decoded frame or compressed frame to the server Socket or RPC.
  • Step 413 returning a service response.
  • the processing response is returned by the transcoding cluster management unit, and the server Socket (or RPC) forwards the response.
  • Step 414 sending an instruction to start the encoder.
  • the transcoding cluster management 131 sends an instruction to start the encoder to the encoding cluster 132, and the encoding cluster 132 starts encoding after receiving the startup instruction.
  • Step 415 Send the original video frame.
  • Transcoding cluster management 131 sends raw video frames to encoding cluster 132 .
  • Step 416 sending an instruction to turn off the encoder.
  • the transcoding cluster management 131 sends the instruction of closing the encoder to the encoding cluster 132, and the encoding cluster 132 stops transcoding after receiving the closing instruction.
  • Step 417 returning to the compressed frame.
  • the processed compressed frames are returned by the encoding cluster 132 to the transcoding cluster management 131 .
  • Step 418 Send an instruction to start the decoder.
  • the transcoding cluster management 131 sends the instruction of starting the decoder to the decoding cluster 133, and the decoding cluster 133 starts decoding after receiving the starting instruction.
  • Step 419 Send an instruction to close the decoder.
  • the transcoding cluster management 131 sends an instruction to close the decoder to the decoding cluster 133, and the decoding cluster 133 stops transcoding after receiving the activation instruction.
  • Step 420 Send the original compressed frame.
  • the original compressed frames are sent to the decoding cluster 133 by the transcoding cluster management 131 .
  • Step 421 returning the decoded frame.
  • the processed decoded frames are returned to the transcoding cluster management 131 by the decoding cluster 133 .
  • FIG. 9 is a schematic diagram of a method that may involve cloud terminal data processing in a cloud mobile phone shopping live broadcast scenario provided by an embodiment of the present application.
  • the cloud server cluster can provide the encoding and decoding services described above. Office scenarios require a larger cloud server cluster. Then, you need to complete the configuration for connecting the cloud phone to the external transcoding cluster, such as the IP and port configuration of the socket on the cloud phone. Finally, use the configured cloud phone and cloud server cluster for live broadcast and mobile office.
  • FIG. 9 is a schematic diagram of a method that may involve cloud terminal data processing in a cloud mobile phone shopping live broadcast scenario provided by an embodiment of the present application.
  • serial numbers 501-514 represent the key steps involved in this embodiment, which describe that in a cloud mobile phone shopping live broadcast, from user A starting the live broadcast, the shopping live broadcast application sends a software coding request until the user
  • this method is how to perform remote hardware video encoding services based on cloud mobile phones.
  • Step 501 User A opens the shopping live broadcast application software on the cloud mobile phone, and starts the live broadcast.
  • the user may remotely control the application on the cloud terminal 001 based on the visual interface of the terminal 005 shown in FIG. 4 .
  • the terminal 005 may be a smart phone, including a camera or other photographing device, which is used to collect the video stream of the user during the live broadcast in real time, and transmit the video stream to the cloud phone synchronously.
  • Step 502 The shopping live broadcast application invokes the MediaCodec interface to start the video encoding operation.
  • the live application receives the video stream, sends a software encoding request to the MediaCodec interface, and requests the encoding service of the cloud server cluster;
  • Steps 503-404 The MediaCodec process invokes the encoder control unit based on the software encoding plug-in to create an encoding thread.
  • Step 505 The encoder control unit generates a relevant instruction, the instruction includes an encoding rule, and modifies the encoding mode in the encoding rule from software encoding to hardware encoding.
  • the encoder control unit establishes a socket connection with the external transcoding cluster through the client socket, and sends a request to start the external hardware encoder to the server socket.
  • Step 506 The server socket of the cloud server cluster forwards the received hardware coding instruction to the transcoding cluster management node. After the transcoding cluster management node receives the instruction to start the specific external encoder, it creates a thread to start the coding cluster.
  • Step 507 The cloud phone encoder control unit sends a request for obtaining the encoding header information, and the client socket forwards the request to the client socket of the cloud server cluster.
  • Step 508 The transcoding cluster management node requests and obtains the encoding header from the encoder module of a specific server in the encoding cluster, and forwards it to the cloud mobile phone through the network.
  • Step 509 The cloud phone encoder control unit receives the response of the encoding header information.
  • Step 510 The cloud phone encoder control module sends the original video frame data to the transcoding management node of the cloud server cluster, and requests the external encoding cluster to perform hardware encoding.
  • Step 511 The transcoding cluster management node is responsible for allocating the encoding server to the cloud mobile phone, and the cloud mobile phone sends the original video frame to the encoder of the specific server.
  • Step 512 The encoder returns the encoded compressed video frame to the transcoding cluster management node, and forwards it to the cloud mobile phone through the network.
  • Step 513 The encoder module of the cloud phone receives the compressed video frame data through the socket, and returns it to the live broadcast application for use.
  • the live application on the cloud mobile phone will receive the compressed frame data stream and forward it to the public cloud. More network users can download the compressed frame data stream on their mobile phones or PCs and other terminals. After decoding locally, they can watch the video in real time. Live shopping.
  • Step 514 User A closes the shopping live broadcast application on the cloud mobile phone, and ends the live broadcast.
  • Step 515 The shopping live broadcast application software ends the video encoding operation by calling the MediaCodec interface.
  • Steps 516-517 the MediaCodec process calls the encoder control unit based on the software encoding plug-in, and requests to end encoding;
  • Step 518 The encoder control unit releases the remote encoder, sends a request to close the encoder of the server, and closes the socket connection with the cloud server cluster.
  • Step 519 The transcoding cluster management node closes the encoding service of the specific server.
  • FIG. 10 is an apparatus 300 deployed on a server 002 according to an embodiment of the present application.
  • the apparatus 300 includes a receiving unit 301 , a processing unit 302 , and a sending unit 303 .
  • the receiving unit 301 is configured to receive data from the cloud terminal 001 .
  • the data may be data to be encoded or decoded.
  • the data may further include an instruction for instructing an encoding or decoding task.
  • the content of the instruction please refer to the foregoing related description.
  • the processing unit 302 is used for encoding or decoding data.
  • the processing unit encodes the data to be encoded, and the encoding method includes hardware encoding or software encoding. Please refer to the foregoing description.
  • the processing unit decodes the data to be decoded, and the decoding method includes hardware decoding or software decoding. Please refer to the foregoing description.
  • the sending unit 303 is configured to send data to the cloud terminal 001 .
  • the transmitted data is already decoded or encoded data.
  • the processing unit 302 is configured to perform hardware encoding on the data or perform software encoding on the data.
  • the processing unit 302 is configured to perform hardware decoding on the data or perform software decoding on the data.
  • This embodiment of the present application further provides a computing device 600, as shown in FIG. 11 .
  • the computing device 600 may be a device for deploying the cloud terminal 001 , or the computing device 600 may be the server 002 .
  • FIG. 11 provides a schematic structural diagram of a computing device 600 .
  • the computing device 600 includes a bus 601 , a processor 602 and a memory 603 .
  • the processor 602 and the memory 603 communicate through the bus 601 .
  • the bus 601 may be a peripheral component interconnection standard bus or an extended industry standard structure bus or the like.
  • the processor 602 may be a central processing unit (Central Processing Unit, CPU), in addition, there may also be a graphics processing unit (graphics processing unit, GPU), a neural network processor (Neural network Processing Unit, NPU), FPGA (Field Programmable Gate) Array) one or more.
  • the memory 603 may include random access memory (RAM), read-only memory (ROM), flash memory, hard drive (HDD) or solid state drive (SSD) ) one or more of.
  • Executable program codes are stored in the memory 603, and the processor 602 executes the executable program codes to realize the functions of the aforementioned cloud terminal 001 or the functions of the server 002, or to execute the steps performed by the terminal 001 in the methods described in the aforementioned embodiments. Or steps performed by server 002 .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Multimedia (AREA)
  • Telephonic Communication Services (AREA)

Abstract

Procédé et système de traitement de données, terminal en nuage, serveur, dispositif informatique, et produit-programme informatique, destinés à être utilisés pour résoudre les problèmes de ressources CPU insuffisantes d'un terminal en nuage et d'une compatibilité logicielle. Le procédé comprend les étapes suivantes : un terminal en nuage envoie des données; un serveur reçoit les données; le serveur traite les données, le traitement étant destiné à coder ou décoder les données; le serveur envoie les données traitées au terminal en nuage.
PCT/CN2021/101691 2020-09-07 2021-06-23 Procédé et système de traitement des données, terminal en nuage, serveur, et dispositif informatique WO2022048255A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202010928653 2020-09-07
CN202010928653.1 2020-09-07
CN202010990303.8 2020-09-18
CN202010990303.8A CN114157919A (zh) 2020-09-07 2020-09-18 数据处理的方法和系统、云终端、服务器和计算设备

Publications (1)

Publication Number Publication Date
WO2022048255A1 true WO2022048255A1 (fr) 2022-03-10

Family

ID=80462166

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/101691 WO2022048255A1 (fr) 2020-09-07 2021-06-23 Procédé et système de traitement des données, terminal en nuage, serveur, et dispositif informatique

Country Status (2)

Country Link
CN (1) CN114157919A (fr)
WO (1) WO2022048255A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160198208A1 (en) * 2013-09-13 2016-07-07 Huawei Technologies Co.,Ltd. Desktop-cloud-based media control method and device
CN108449238A (zh) * 2018-06-11 2018-08-24 西安万像电子科技有限公司 通信方法及通信系统
CN108521581A (zh) * 2018-03-05 2018-09-11 西安万像电子科技有限公司 多媒体数据显示方法、终端和系统
CN110636305A (zh) * 2019-09-26 2019-12-31 华为技术有限公司 图像渲染和编码方法及相关装置
CN111273880A (zh) * 2020-01-13 2020-06-12 奇安信科技集团股份有限公司 基于云端智能设备的远程显示方法及装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110225513A1 (en) * 2010-03-09 2011-09-15 Avistar Communications Corporation Scalable high-performance interactive real-time media architectures for virtual desktop environments
CN102377739A (zh) * 2010-08-11 2012-03-14 中兴通讯股份有限公司 移动终端及其实现方法和服务系统、以及云服务器
CN103179431B (zh) * 2013-03-21 2016-06-01 福建升腾资讯有限公司 Vdi环境下音视频重定向转码分离方法
CN103873568A (zh) * 2014-03-04 2014-06-18 赛特斯信息科技股份有限公司 基于云计算实现远程虚拟桌面显示的系统及方法
KR102232900B1 (ko) * 2014-10-02 2021-03-26 에스케이플래닛 주식회사 클라우드 스트리밍 서비스 시스템, 최적 gpu를 이용한 클라우드 스트리밍 서비스 방법 및 이를 위한 장치
CN106797388B (zh) * 2016-12-29 2020-11-10 深圳前海达闼云端智能科技有限公司 跨系统多媒体数据编解码方法、装置、电子设备和计算机程序产品
CN108989811B (zh) * 2017-06-05 2022-06-03 中兴通讯股份有限公司 云桌面系统、及其图像序列压缩编码方法和介质
US10303497B2 (en) * 2017-06-22 2019-05-28 Vmware, Inc. Hybrid software and GPU encoding for UI remoting
CN107911700B (zh) * 2017-10-31 2020-07-03 深信服科技股份有限公司 一种基于虚拟化的硬件解码方法、解码设备及存储介质
CN107809671A (zh) * 2017-11-20 2018-03-16 深圳市青葡萄科技有限公司 一种虚拟化环境下视频重定向到服务器解码的方法和系统
CN109640116B (zh) * 2018-12-24 2020-09-11 广州微算互联信息技术有限公司 云手机网络视频编码方法与服务器

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160198208A1 (en) * 2013-09-13 2016-07-07 Huawei Technologies Co.,Ltd. Desktop-cloud-based media control method and device
CN108521581A (zh) * 2018-03-05 2018-09-11 西安万像电子科技有限公司 多媒体数据显示方法、终端和系统
CN108449238A (zh) * 2018-06-11 2018-08-24 西安万像电子科技有限公司 通信方法及通信系统
CN110636305A (zh) * 2019-09-26 2019-12-31 华为技术有限公司 图像渲染和编码方法及相关装置
CN111273880A (zh) * 2020-01-13 2020-06-12 奇安信科技集团股份有限公司 基于云端智能设备的远程显示方法及装置

Also Published As

Publication number Publication date
CN114157919A (zh) 2022-03-08

Similar Documents

Publication Publication Date Title
US12003743B2 (en) Video stream decoding method and apparatus, terminal device, and storage medium
WO2021047419A1 (fr) Procédé d'interaction de diffusion en direct, système de diffusion en direct, dispositif électronique et support de stockage
US20200107057A1 (en) Video coding method, system and server
US12081782B2 (en) Region of interest (ROI) request and inquiry in a video chain
WO2022257699A1 (fr) Procédé et appareil d'affichage d'image, dispositif, support de stockage et produit de programme
WO2014139269A1 (fr) Procédé et appareil pour lire et traiter des vidéos sur la base d'un bureau virtuel
CN113542757A (zh) 云应用的图像传输方法、装置、服务器及存储介质
CN114363652A (zh) 视频直播方法、系统及计算机存储介质
CN111880865A (zh) 多媒体数据推送方法、装置、电子设备及存储介质
WO2023040825A1 (fr) Procédé de transmission d'informations multimédias, dispositif informatique et support de stockage
KR20230006495A (ko) 몰입형 원격 회의 및 텔레프레즌스를 위한 다중 그룹화
CN114938408A (zh) 一种云手机的数据传输方法、系统、设备及介质
WO2022048255A1 (fr) Procédé et système de traitement des données, terminal en nuage, serveur, et dispositif informatique
CN115209189B (zh) 一种视频流传输方法、系统、服务器及存储介质
US11985181B2 (en) Orchestrating a multidevice video session
WO2016107174A1 (fr) Procédé et système de traitement de lecteur, de client et de données de fichier multimédia
CN116546262A (zh) 一种数据处理方法、装置、设备及存储介质
CN113630575A (zh) 多人在线视频会议图像显示的方法、系统和存储介质
CN111327869A (zh) 视频会议系统
JP6404915B2 (ja) データの自動圧縮
WO2022068674A1 (fr) Procédé d'appel vidéo, dispositif électronique et système
CN112738026B (zh) 一种指挥调度方法
US11800158B2 (en) Video production systems and methods
WO2024032189A1 (fr) Procédé et appareil de transmission de données
CN113938457B (zh) 一种云手机应用远程摄像头的方法、系统和设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21863323

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21863323

Country of ref document: EP

Kind code of ref document: A1