WO2022057718A1 - 编码调度方法、服务器及客户端和获取远程桌面的系统 - Google Patents

编码调度方法、服务器及客户端和获取远程桌面的系统 Download PDF

Info

Publication number
WO2022057718A1
WO2022057718A1 PCT/CN2021/117361 CN2021117361W WO2022057718A1 WO 2022057718 A1 WO2022057718 A1 WO 2022057718A1 CN 2021117361 W CN2021117361 W CN 2021117361W WO 2022057718 A1 WO2022057718 A1 WO 2022057718A1
Authority
WO
WIPO (PCT)
Prior art keywords
encoding
information
hardware
task
resources
Prior art date
Application number
PCT/CN2021/117361
Other languages
English (en)
French (fr)
Inventor
贾香宏
闵洪波
胡祥
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2022057718A1 publication Critical patent/WO2022057718A1/zh
Priority to US18/184,204 priority Critical patent/US20230221969A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/04—Display device controller operating with a plurality of display units
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/06—Use of more than one graphics processor to process data before displaying to one or more screens
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/08—Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/20—Details of the management of multiple sources of image data
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/001—Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/363—Graphics controllers

Definitions

  • This application relates to, but is not limited to, cloud desktop technology, especially a coding scheduling method, a server and a client, and a system for obtaining a remote desktop.
  • Cloud desktop also known as desktop virtualization and cloud computer
  • Cloud desktop is a new model to replace traditional computers.
  • users do not need to purchase a computer host.
  • the CPU, memory, hard disk and other components contained in the computer host can be virtualized in the back-end server; after installing the client, users can access the virtual machine on the back-end server through relevant communication protocols.
  • the computer can realize interactive operation and achieve the same experience effect as the computer.
  • Cloud desktop not only supports the replacement of traditional computers, but also supports other smart devices such as mobile phones and tablets to access on the Internet. It is also the latest solution for mobile office.
  • a mainstream technology in cloud desktop products is that the server encodes the desktop image into a video stream format, and then transmits it to the client for display through the network.
  • the hardware encoder refers to a hardware unit that uses heterogeneous computing chips for video encoding, such as image processing chips ( GPU, Graphics Processing Unit), Field Programmable Gate Array (FPGA, Field Programmable Gate Array), Application Specific Integrated Circuit (ASIC, Application Specific Integrated Circuit), etc.
  • image processing chips GPU, Graphics Processing Unit
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 1 is a schematic diagram of a typical architecture of a cloud desktop in the related art.
  • a typical architecture of a cloud desktop includes: a server, a hardware encoder, and multiple clients.
  • the client interacts with the server, including requesting to create a remote desktop, connecting to a remote desktop, etc.; the server encodes the image data of the desktop into a video stream, and then transmits it to the requesting client through the network.
  • Each client will occupy all the resources of the hardware encoder of the server.
  • the server can support up to 24 concurrent client connections; and, for cloud desktop user scenarios, the frequency of desktop changes is not high, that is to say, although each client occupies one channel of encoding resources, but the load is not large, which results in a low utilization rate of the hardware encoder, resulting in a waste of resources.
  • the present application provides a coding scheduling method, a server and a client, and a system for obtaining a remote desktop, which can improve the utilization rate of hardware coding resources and save resources.
  • An embodiment of the present invention provides a coding scheduling method, including:
  • the server determines the encoding requirement information of the encoding task according to the information reflecting the creation requirement of the remote desktop;
  • the hardware encoding resource is invoked according to the encoding requirement information of the encoding task to realize the encoding of the encoding task.
  • the information reflecting the remote desktop creation requirements includes any one or any combination of the following: remote desktop image quality requirement information, client information that initiates remote desktop creation, and network transmission quality information.
  • the remote desktop image quality requirement information includes any one or any combination of the following:
  • the update area information used to indicate the area where the information update occurs between the image frame transmitted later and the image frame transmitted adjacently before;
  • User interface element information used to represent the text information and picture information included in the image frame.
  • the determining the encoding requirement information of the encoding task includes:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is updated according to the area.
  • Information calls hardware encoding resources
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to reduce hardware encoding resources.
  • the client information for initiating remote desktop creation includes any one or any combination of the following:
  • User event information used to indicate user events of the client.
  • the determining the encoding requirement information of the encoding task includes:
  • the encoding requirement information of the encoding task is an elevated priority level
  • the encoding requirement information of the encoding task is a high-priority level
  • the encoding requirement information of the encoding task is an elevated priority level.
  • the network transmission quality information includes network service quality information
  • Described determining the coding requirement information of the coding task including:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to reduce hardware encoding resources.
  • the determining the encoding requirement information of the encoding task further includes:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to increase hardware encoding resources
  • Described determining the coding requirement information of coding task also includes:
  • the hardware encoding resources are increased and called according to the priority corresponding to the encoding task. level-to-level transmission.
  • the determining the encoding requirement information of the encoding task further includes:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to keep the hardware encoding resource unchanged. Change
  • Described determining the coding requirement information of coding task also includes:
  • the hardware coding resources are increased and called according to the priority level corresponding to the coding task. to transmit.
  • the encoding requirement information includes: priority level information of the encoding task, encoding parameter information, and requirement information of hardware encoding resources.
  • the invoking hardware encoding resources according to the encoding requirement information of the encoding task includes:
  • the encoding tasks are encoded according to the priority level from high to low.
  • the method further includes: transmitting the encoded image and video stream of the remote desktop to the client.
  • An embodiment of the present application further provides a server, including: a demand determination module, a scheduling module, and a hardware encoding module that provides hardware encoding resources; wherein,
  • the requirement determination module is set to determine the encoding requirement information of the encoding task according to the information reflecting the creation requirement of the remote desktop;
  • the scheduling module is configured to call the hardware encoding resources provided by the hardware encoding module according to the encoding requirement information of the encoding task, so as to realize the encoding of the encoding task.
  • the information reflecting the remote desktop creation requirements includes any one or any combination of the following: remote desktop image quality requirement information, client information that initiates remote desktop creation, and network transmission quality information.
  • the information reflecting the remote desktop creation requirements includes any one or any combination of the following:
  • the update area information used to indicate the area where the information update occurs between the image frame transmitted later and the image frame transmitted adjacently before;
  • User interface element information used to represent the text information and picture information included in the image frame
  • User event information for indicating user events of the client
  • the scheduling module is specifically set as:
  • the scheduling module is further configured to:
  • the image and video stream of the remote desktop obtained after the encoding is transmitted to the client.
  • the hardware encoding module is a hardware encoder.
  • the embodiment of the present application also provides a client, which is configured to initiate a remote desktop creation request to the server; receive an image and video stream of the remote desktop encoded and processed according to information reflecting the remote desktop creation requirement from the server, and decode the received image Stream video and display the remote desktop.
  • the embodiment of the present application also provides a system for obtaining a remote desktop, including: a server and more than one client; wherein,
  • the client is set to initiate a request for remote desktop creation to the server; receive the encoded image and video stream of the remote desktop from the server, decode the received image and video stream and display the remote desktop;
  • the server is configured to receive a request for remote creation, and determine the encoding requirement information of the encoding task according to the information reflecting the creation requirement of the remote desktop; and call the hardware encoding resource provided by the hardware encoding module according to the encoding requirement information of the encoding task, so as to realize the encoding task.
  • Encoding transmits the encoded image and video stream of the remote desktop to the client.
  • the server includes the server described in any of the above exemplary embodiments.
  • the hardware encoding resources are scheduled according to the encoding requirement information of the encoding task determined by the information reflecting the remote desktop creation requirement, and the hardware encoding resources are released in time when the hardware encoding resources are unnecessarily occupied, thereby saving the hardware encoding resources; Moreover, the released hardware encoding resources can be used by other encoding tasks, in other words, the number of concurrent paths is increased, and the utilization rate of hardware encoding resources is also improved.
  • FIG. 1 is a schematic diagram of a typical architecture of a cloud desktop in the related art
  • FIG. 2 is a schematic flowchart of a coding scheduling method in an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a composition structure of a server in an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of a system for obtaining a remote desktop according to an embodiment of the present application.
  • a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • Memory may include forms of non-persistent memory, random access memory (RAM) and/or non-volatile memory in computer readable media, such as read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
  • RAM random access memory
  • ROM read only memory
  • flash RAM flash memory
  • Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology.
  • Information may be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
  • computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.
  • FIG. 2 is a schematic flowchart of a coding scheduling method in an embodiment of the present application, as shown in FIG. 2 , including:
  • Step 200 The server determines the encoding requirement information of the encoding task according to the information reflecting the remote desktop creation requirement.
  • the information reflecting the remote desktop creation requirements may include any one or any combination of the following: remote desktop image quality requirement information, client information that initiates remote desktop creation, network transmission quality information, and the like.
  • the remote desktop image quality requirement information may include, for example:
  • Update area information refers to the information of the area where the information of the image frame transmitted later is updated compared with the image frame transmitted adjacently before;
  • User interface element (UI Elements) information such as text (Text) information, picture (Image) information, etc. included in the image frame;
  • the client information that initiates the creation of the remote desktop may include, for example:
  • the application information may include, for example, the name of the application, the type of the application, and the like. Through the application information, it can be determined whether the application is a popular application;
  • User Configuration information such as: display the user's priority, such as a high-priority user;
  • User event information such as client user events such as mouse events, keyboard events, etc.
  • the network transmission quality information may include, for example, network service quality information (Network QoS), such as: network jitter, number of packet losses, and the like.
  • Network QoS network service quality information
  • the encoding requirement information may include: priority level information of encoding tasks, encoding parameter information, and requirement information of hardware encoding resources.
  • the information reflecting the remote desktop creation requirements includes remote desktop image quality requirement information, and step 200 may include:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to call the hardware encoding resource only according to the area update information. , which will reduce the use of hardware encoding resources;
  • the image quality can be reduced, and the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to reduce hardware encoding resources. In one embodiment, it can be reduced by, for example, 50%.
  • the proportion of picture information is greater than the preset first threshold, the picture quality parameter value used in encoding in the encoding parameter information can be halved, thereby reducing hard-coding resource.
  • the use of hardware coding resources is reduced, the unnecessary occupied hardware coding resources are dynamically released, the hardware coding resources are saved, and the released hardware coding resources It can be used for other encoding tasks, in other words, the number of concurrent channels is increased, and the utilization rate of hardware encoding resources is also improved.
  • the number of concurrent channels is increased, and the utilization rate of hardware encoding resources is also improved.
  • the human eye since the human eye is not so sensitive to the image quality requirements, even if the hardware encoding resources are reduced, the user experience will not be affected, and the unnecessary hardware encoding resources are dynamically released. , which saves hardware coding resources, and the released hardware coding resources can be used for other coding tasks, in other words, the number of concurrent paths is increased, and the utilization rate of hardware coding resources is also improved.
  • the information reflecting the remote desktop creation requirement includes client information that initiates the creation of the remote desktop, and step 200 may include:
  • the encoding requirement information of the encoding task is the priority level of promotion; in one embodiment, it is assumed that the user is preset as the first priority; user events are the second priority; Applications are the third priority; other applications such as non-popular applications and ordinary users can be configured as ordinary priorities. Then, the priority level can be raised from the ordinary priority to the third priority here;
  • the encoding requirement information of the encoding task is a high-priority level; in one embodiment, it is assumed that the user is preset as the first priority; the user event is the second priority ; Popular applications are the third priority; others such as non-popular applications and ordinary users can be configured as ordinary priorities. Then, the priority level of the corresponding user can be upgraded from the normal priority to the first priority here;
  • the encoding requirement information of the encoding task is to increase the priority level; in one embodiment, it is assumed that the user is preset as The first priority; user events are the second priority; popular applications are the third priority; others such as non-popular applications, ordinary users, etc. can be configured as ordinary priorities. Well, here the priority level can be raised from the normal priority to the second priority.
  • the information reflecting the remote desktop creation requirement includes network transmission quality information, and step 200 may include:
  • the encoding requirement information of the encoding task is to reduce the output frame rate, This means reducing hardware encoding resources.
  • the server may further include: acquiring the information reflecting the creation requirement of the remote desktop by the server.
  • a weight can be preset for each information, so that the final priority level and/or hardware encoding can be comprehensively calculated according to the weight. resource reduction.
  • the specific implementation is easy to implement for those skilled in the art based on the description of the embodiments of the present application, and details are not repeated here.
  • the coding tasks in the embodiments of the present application will all have priority levels set in advance, and there will also be corresponding demand information for hardware coding resources.
  • the priority level needs to be raised, or the hardware coding resources need to be reduced.
  • Step 201 Invoke hardware encoding resources according to the encoding requirement information of the encoding task to implement encoding of the encoding task.
  • step 201 may include:
  • the coding tasks are coded according to the priority level in the coding requirement information from high to low.
  • hardware encoding resources may be provided by a hardware encoder.
  • hardware coding resources are scheduled according to coding requirement information of coding tasks determined by information reflecting remote desktop creation requirements, resources are released in time when hardware coding resources are unnecessarily occupied, saving The hardware encoding resources are released; and the released hardware encoding resources can be used by other encoding tasks, in other words, the number of concurrent paths is increased, and the utilization rate of hardware encoding resources is also improved.
  • step 200 if the user interface element information indicates that the proportion of text information in the current image frame is greater than the preset fourth threshold, the image quality can be improved.
  • the requirement information of hardware coding resources is to increase hardware coding resources. In one embodiment, it can be increased by, for example, 50%.
  • the image quality parameter value used in encoding in the encoding parameter information can be increased by, for example, 50%. 50% hardcoded resources.
  • step 201 may further include:
  • the hardware encoding resources will be called additionally, according to the priority level corresponding to the encoding task. transmitted after encoding.
  • step 200 if the update area information indicates that the current image frame is compared with the previous image frame, only a part of the area is updated with information, and the image quality of the updated area can also be improved, then, at this time , the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to keep the hardware encoding resource unchanged. Since the unchanged hardware encoding resource is only used to encode the changed area, it is also equivalent to improving the picture quality. parameter, the image quality is improved. In one embodiment, the ratio of the improvement may be the size of the entire area or the size of the changed area.
  • step 201 may further include:
  • the hardware encoding resources will be called additionally, according to the priority level corresponding to the encoding task. transmitted after encoding.
  • the encoding task may be placed in the waiting queue according to the priority level, until the idle hardware encoding resources are released, and then It is encoded and transmitted according to the priority level from high to low.
  • the entire scheduling process is implemented in the server, which can cache more encoding tasks; moreover, when the hardware encoding resources are running at full load, the time-division multiplexing mechanism improves the throughput rate and improves the supported concurrency
  • the number of channels reduces the cost of a single channel to a certain extent.
  • the embodiment of the present application further includes: transmitting the encoded image and video stream of the remote desktop to the client.
  • the present application also provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are used to execute any of the encoding scheduling methods described above.
  • the present application further provides a device for implementing coding scheduling, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: for executing the steps of any of the coding scheduling methods described above.
  • FIG. 3 is a schematic diagram of the composition structure of a server in an embodiment of the application, as shown in FIG. 3 , including: a demand determination module, a scheduling module, and a hardware encoding module for providing hardware encoding resources; wherein,
  • the requirement determination module is set to determine the encoding requirement information of the encoding task according to the information reflecting the creation requirement of the remote desktop;
  • the scheduling module is configured to call the hardware encoding resources provided by the hardware encoding module according to the encoding requirement information of the encoding task, so as to realize the encoding of the encoding task.
  • the information reflecting the remote desktop creation requirement includes remote desktop image quality requirement information
  • the requirement determination module is specifically set as:
  • the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to call the hardware encoding resource only according to the area update information.
  • the image quality can be reduced, and the hardware encoding resource requirement information in the encoding requirement information of the encoding task is to reduce hardware encoding resources.
  • the use of hardware coding resources is reduced, the unnecessary occupied hardware coding resources are dynamically released, the hardware coding resources are saved, and the released hardware coding resources It can be used for other encoding tasks, in other words, the number of concurrent channels is increased, and the utilization rate of hardware encoding resources is also improved.
  • the number of concurrent channels is increased, and the utilization rate of hardware encoding resources is also improved.
  • the human eye since the human eye is not so sensitive to the image quality requirements, even if the hardware encoding resources are reduced, the user experience will not be affected, and the unnecessary hardware encoding resources are dynamically released. , which saves hardware coding resources, and the released hardware coding resources can be used for other coding tasks, in other words, the number of concurrent paths is increased, and the utilization rate of hardware coding resources is also improved.
  • the information reflecting the remote desktop creation requirement includes client information that initiates the creation of the remote desktop, and the requirement determination module is specifically set as:
  • the coding requirement information of the coding task is the priority level of promotion
  • the encoding requirement information of the encoding task is a high-priority level
  • the encoding requirement information of the encoding task is to increase the priority level.
  • the information reflecting the remote desktop creation requirement includes network transmission quality information
  • the requirement determination module is specifically set as:
  • the encoding requirement information of the encoding task is to reduce the output frame rate, This means reducing hardware encoding resources.
  • the hardware encoding module may be a hardware encoder.
  • the scheduling module is specifically configured to: determine demand information of hardware coding resources according to coding parameter information; call hardware coding resources according to the determined demand information of hardware coding resources; Coding tasks to encode.
  • the scheduling module is further configured to: transmit the encoded image and video stream of the remote desktop to the client.
  • the requirement determination module is further configured to: if the user interface element information indicates that the proportion of text information in the current image frame is greater than the preset fourth threshold, the image quality can be improved, and at this time, the encoding requirement of the encoding task
  • the requirement information of hardware coding resources in the information is to increase hardware coding resources.
  • the scheduling module is also set to:
  • the hardware encoding resources will be called additionally, according to the priority level corresponding to the encoding task. transmitted after encoding.
  • the requirement determination module is further configured to: if the update area information indicates that compared with the previous image frame, the current image frame has only a part of the area where the information is updated, and the image quality of the updated area can also be improved, then , at this time, the encoding requirement information of the encoding task is to keep the hardware encoding resources unchanged.
  • the scheduling module is also set to:
  • the hardware encoding resources will be called additionally, according to the priority level corresponding to the encoding task. transmitted after encoding.
  • the scheduling module is further configured to:
  • the encoding tasks can be placed in the waiting queue according to the priority level, until the free hardware encoding resources are released, and then the priority level is from high to low. Transmit after encoding.
  • the entire scheduling process is implemented in the server, which can cache more encoding tasks; moreover, when the hardware encoding resources are running at full load, the time-division multiplexing mechanism improves the throughput rate and improves the supported concurrency
  • the number of channels reduces the cost of a single channel to a certain extent.
  • the embodiment of the present application also provides a client, which is configured to initiate a remote desktop creation request to the server; receive an image and video stream of the remote desktop encoded and processed according to information reflecting the remote desktop creation requirement from the server, and decode the received image Stream video and display the remote desktop.
  • FIG. 4 is a schematic structural diagram of a system for obtaining a remote desktop in an embodiment of the application, as shown in FIG. 4 , including a server and more than one client; wherein,
  • the client is set to initiate a remote desktop creation request to the server; receive the encoded image video stream of the remote desktop from the server, decode the received image video stream and display the remote desktop;
  • the server is configured to receive a remote creation request, and determine the encoding requirement information of the encoding task according to the information reflecting the creation requirement of the remote desktop; call the hardware encoding resources provided by the hardware encoding module according to the encoding requirement information of the encoding task, so as to realize the encoding task. Encoding; transmits the encoded image and video stream of the remote desktop to the client.
  • the server in the system for implementing coding scheduling provided by the embodiment of the present application includes any one of the servers shown in FIG. 3 .
  • the embodiment of the present application will calculate the priority of each channel using hardware encoding resources, encoding parameters such as frame rate, etc. according to the client's request;
  • the accelerator card applies for the allocation of the corresponding frame rate and hardware encoding resources, and passes the encoding parameters to the hardware encoder for encoding to obtain the image and video stream of the remote desktop.
  • the working time or computing time of the hardware acceleration card can be better utilized under full load conditions.
  • the hardware acceleration card can reallocate hardware encoding resources according to the changed frame rate and area size, which increases the number of concurrent channels supported per unit time and improves throughput Rate.
  • the server determines, according to the user configuration information, that the user corresponding to the client that initiates the request is a high-priority user, then the server will determine the user's priority level The priority is upgraded from the normal priority to the first priority, that is to say, the encoding requirement information of the encoding task is the first priority; in this way, the server will preferentially allocate the required hardware encoding resources to the user's request according to the encoding parameter information.
  • the encoding task performs encoding, the encoded image and video stream of the remote desktop is transmitted to the client. For the client, it realizes efficient remote desktop creation and improves user experience.
  • the server learns, according to the user event information, that user events of the client, such as mouse events, continue to occur within a preset period of time, then the server will promote the user If the user event is preset as the second priority, the server will allocate the required hardware coding resources to process the user event according to the second priority. In this way, for the client, It also achieves efficient remote desktop response and improves user experience.
  • the user can set the priority level of the application that the user considers important, or likes, or commonly used, through a preset interface, and notify the server through a request.
  • the server is performing hardware
  • configuration information from the client can be prioritized. In this way, the individual needs of users are also guaranteed.
  • the client may also display to the user information such as image quality information, network transmission information, current user, and application priority level of the currently displayed remote desktop.
  • the user has an intuitive look and feel for the currently displayed remote desktop processing, and also has qualitatively quantified information.
  • the information displayed to the user can be in a simple and easy-to-understand way.
  • the network quality can be divided into: good, good, and poor to tell the user the current network quality; another example: the displayed remote desktop image quality It can be divided into: high quality, general quality, poor quality, etc., so that the user can see at a glance the quality of the remote desktop obtained at present.
  • the user feels that the quality of the current remote desktop is not good, he can re-request and re-create it; at this time, the user can increase the priority level of the application or user to allocate more hardware encoding resources to obtain a better remote desktop .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

一种编码调度方法、服务器及客户端和获取远程桌面的系统，根据反映远程桌面创建需求的信息确定的编码任务的编码需求信息来对硬件编码资源进行调度，在不必要占用的硬件编码资源时及时释放了资源，节省了硬件编码资源；而且释放的硬件编码资源可提供给其他编码任务使用，提高了并发路数，同时也提升了硬件编码资源的利用率。

Description

编码调度方法、服务器及客户端和获取远程桌面的系统
本申请要求2020年09月16日递交的申请号为202010973546.0、发明名称为“编码调度方法、服务器及客户端和获取远程桌面的系统”中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及但不限于云桌面技术，尤指一种编码调度方法、服务器及客户端和获取远程桌面的系统。
背景技术
云桌面又称桌面虚拟化、云电脑，是替代传统电脑的一种新模式。采用云桌面，用户无需再购买电脑主机，电脑主机所包含的CPU、内存、硬盘等组件都可以在后端的服务器中虚拟出来；用户安装客户端后，通过相关通信协议访问后端服务器上的虚拟机来实现交互式操作，达到与电脑一致的体验效果。云桌面不仅支持用于替换传统电脑，还支持替换手机、平板等其他智能设备在互联网上访问，也是移动办公的最新解决方案。当前，云桌面产品中主流的一个技术就是：服务端将桌面的图像编码成视频流的格式，再通过网络传输到客户端显示。为了提升编码效率，避免过多地消耗服务器的CPU，通常会在服务器上引入硬件编码器，硬件编码器指的是利用异构计算芯片来做视频编码的硬件单元，常见的有如图像处理芯片(GPU，Graphics Processing Unit)、现场可编程门阵列(FPGA，Field Programmable Gate Array)、专用集成电路(ASIC，Application Specific Integrated Circuit)等。
图1为相关技术中云桌面的一个典型架构示意图，如图1所示，云桌面的一个典型架构包括：服务端、硬件编码器和多个客户端。通过网络协议，客户端与服务端交互，包括请求创建远程桌面、连接远程桌面等；服务端将桌面的图像数据编码成视频流后，再通过网络传输给提出请求的客户端。每一个客户端都会占用服务端的硬件编码器的一路资源。在这种架构中，由于服务端的硬件编码器支持的路数是有限的，在有限资源的情况下，一台服务器支持的客户端连接数由硬件编码器的规格确定，比如：硬件编码器支持24路，那么，该服务端最多也就可以支持24路客户端并发连接；而且，对于云桌面的用户场景，桌面变化的频率并不高，也就是说，每个客户端虽然占用了一路编码资源，但是负载并不大，这样造成了硬件编码器利用率不高，导致了资源的浪费。
发明内容
本申请提供一种编码调度方法、服务器及客户端和获取远程桌面的系统，能够提升硬件编码资源的利用率，节省资源。
本发明实施例提供了一种编码调度方法，包括：
服务器根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；
根据编码任务的编码需求信息调用硬件编码资源，以实现对编码任务的编码。
在一种示例性实例中，所述反映远程桌面创建需求的信息包括以下任一项或任意组合：远程桌面画质需求信息、发起远程桌面创建的客户端信息、网络传输质量信息。
在一种示例性实例中，所述远程桌面画质需求信息包括以下任一项或任意组合：
用于表示在后传输的图像帧与相邻在前传输的图像帧相比，发生信息更新的区域的更新区域信息；
用于表示图像帧包括的文字信息、图片信息的用户界面元素信息。
在一种示例性实例中，所述确定编码任务的编码需求信息，包括：
如果所述更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的为部分区域，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为根据所述区域更新信息调用硬件编码资源；
如果所述用户界面元素信息表明当前图像帧中图片信息所占比例大于预设第一阈值，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为减少硬件编码资源。
在一种示例性实例中，所述发起远程桌面创建的客户端信息包括以下任一项或任意组合：
用于确定应用是否属于热门应用的应用程序信息；
用于显示用户的优先级级别的用户配置信息；
用于表明所述客户端的用户事件的用户事件信息。
在一种示例性实例中，所述确定编码任务的编码需求信息，包括：
如果所述应用程序信息表明应用属于热门应用，则所述编码任务的编码需求信息为提升优先级级别；
如果所述用户配置信息表明用户为高优先级用户，则所述编码任务的编码需求信息为高优先级级别；
如果所述用户事件信息表明客户端的用户事件在预设时长内持续发生，则所述编码任务的编码需求信息为提升优先级级别。
在一种示例性实例中，所述网络传输质量信息包括网络服务质量信息；
所述确定编码任务的编码需求信息，包括：
如果所述客户端的网络服务质量信息表明网络质量不好，则所述编码任务的编码 需求信息中的硬件编码资源的需求信息为减少硬件编码资源。
在一种示例性实例中，所述确定编码任务的编码需求信息，还包括：
如果所述用户界面元素信息表明当前图像帧中文字信息所占比例大于预设第四阈值，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为增加硬件编码资源；
所述确定编码任务的编码需求信息，还包括：
根据当前硬件编码资源的负载情况和所述编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照所述编码任务对应的优先级级别进行传输。
在一种示例性实例中，所述确定编码任务的编码需求信息，还包括：
如果所述更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的为部分区域，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为保持硬件编码资源不变；
所述确定编码任务的编码需求信息，还包括：
根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照所述编码任务对应的优先级级别进行传输。
在一种示例性实例中，所述编码需求信息包括：所述编码任务的优先级级别信息、编码参数信息和硬件编码资源的需求信息。
在一种示例性实例中，所述根据编码任务的编码需求信息调用硬件编码资源，包括：
按照所述编码参数信息确定硬件编码资源的需求信息；
根据硬件编码资源的需求信息调用所述调用硬件编码资源；
按照所述优先级级别从高到低对所述编码任务进行编码。
在一种示例性实例中，所述方法还包括：将所述编码后得到的远程桌面的图像视频流传输给客户端。
本申请实施例还提供一种服务器，包括：需求确定模块、调度模块、提供硬件编码资源的硬件编码模块；其中，
需求确定模块，设置为根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；
调度模块，设置为根据编码任务的编码需求信息，调用硬件编码模块提供的硬件编码资源，以实现对编码任务的编码。
在一种示例性实例中，所述反映远程桌面创建需求的信息包括以下任一项或任意 组合：远程桌面画质需求信息、发起远程桌面创建的客户端信息、网络传输质量信息。
在一种示例性实例中，所述反映远程桌面创建需求的信息包括以下任一项或任意组合：
用于表示在后传输的图像帧与相邻在前传输的图像帧相比，发生信息更新的区域的更新区域信息；
用于表示图像帧包括的文字信息、图片信息的用户界面元素信息；
用于确定应用是否属于热门应用的应用程序信息；
用于显示用户的优先级级别的用户配置信息；
用于表明所述客户端的用户事件的用户事件信息；
网络服务质量信息。
在一种示例性实例中，所述调度模块具体设置为：
按照所述编码参数信息确定硬件编码资源的需求信息；根据硬件编码资源的需求信息调用所述调用硬件编码资源；按照所述优先级级别从高到低对所述编码任务进行编码。
在一种示例性实例中，所述调度模块还设置为：
将所述编码后得到的远程桌面的图像视频流传输给客户端。
在一种示例性实例中，所述硬件编码模块为硬件编码器。
本申请实施例还提供一种客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的根据反映远程桌面创建需求的信息编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面。
本申请实施例还提供一种获取远程桌面的系统，包括：包括服务器、一个以上客户端；其中，
客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面；
服务器，设置为接收到远程创建的请求，根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；根据编码任务的编码需求信息调用硬件编码模块提供的硬件编码资源，以实现对编码任务的编码；将编码后得到的远程桌面的图像视频流传输给客户端。
在一种示例性实例中，所述服务器包括上述任一项示例性实例所述的服务器。
本申请实施例中，根据反映远程桌面创建需求的信息确定的编码任务的编码需求信息来对硬件编码资源进行调度，在不必要占用的硬件编码资源时及时释放了资源，节省了硬件编码资源；而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。
本发明的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本发明而了解。本发明的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
附图用来提供对本申请技术方案的进一步理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。
图1为相关技术中云桌面的一个典型架构示意图；
图2为本申请实施例中编码调度方法的流程示意图；
图3为本申请实施例中服务器的组成结构示意图；
图4为本申请实施例中获取远程桌面的系统的架构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚明白，下文中将结合附图对本申请的实施例进行详细说明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互任意组合。
在本申请一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括非暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行。并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
图2为本申请实施例中编码调度方法的流程示意图，如图2所示，包括：
步骤200：服务器根据反映远程桌面创建需求的信息确定编码任务的编码需求信息。
在一种示例性实例中，反映远程桌面创建需求的信息可以包括以下任一项或任意组合：远程桌面画质需求信息、发起远程桌面创建的客户端信息、网络传输质量信息等。
在一种示例性实例中，远程桌面画质需求信息可以包括如：
更新区域信息，指在后传输的图像帧与相邻在前传输的图像帧相比，发生信息更新的区域的信息；
用户界面元素(UI Elements)信息，比如图像帧包括的文字(Text)信息、图片(Image)信息等；
在一种示例性实例中，发起远程桌面创建的客户端信息可以包括如：
优先级指的是排在前面
应用程序信息，可以包括如应用的名称、应用的类型等。通过应用程序信息可以确定出该应用是否属于热门应用；
用户配置(User Configure)信息，比如：显示用户的优先级，如为高优先级用户；
用户事件(User Event)信息，比如客户端的用户事件如鼠标事件、键盘事件等。
在一种示例性实例中，网络传输质量信息可以包括如网络服务质量信息(Network QoS)，比如：网络抖动、丢包次数等。
在一种示例性实例中，编码需求信息可以包括：编码任务的优先级级别信息、编码参数信息和硬件编码资源的需求信息。
在一种示例性实例中，反映远程桌面创建需求的信息包括远程桌面画质需求信息，步骤200可以包括：
如果更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的只有部分区域，则编码任务的编码需求信息中的硬件编码资源的需求信息为仅根据区域更新信息调用硬件编码资源，这样会减少对硬件编码资源的使用；
如果用户界面元素信息表明当前图像帧中图片信息所占比例大于预设第一阈值，可以降低画质，则编码任务的编码需求信息中的硬件编码资源的需求信息为减少硬件编码资源。在一种实施例中，可以减少如50％，当图片信息所占比例大于预设第一阀值时，编码参数信息中编码时采用的画质量化参数值可以减半，从而减少硬编编码资源。
对于只有部分区域发生更新的情况，相比于上一帧图像帧，减少了对硬件编码资 源的使用，动态释放了不必要占用的硬件编码资源，节省了硬件编码资源，而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。对于图像帧中图片信息占比较大的情况，由于人眼对图像的画质要求并没有这么敏感，即使降低了硬件编码资源也不会影响用户体验，而且动态释放了不必要占用的硬件编码资源，节省了硬件编码资源，而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。
在一种示例性实例中，反映远程桌面创建需求的信息包括发起远程桌面创建的客户端信息，步骤200可以包括：
如果应用程序信息表明应用属于热门应用，则编码任务的编码需求信息为提升优先级级别；在一种实施例中，假设预先设置了用户为第一优先级；用户事件为第二优先级；热门应用为第三优先级；其他如非热门应用、普通用户等可以配置为普通优先级。那么，这里可以将优先级级别从普通优先级提升为第三优先级；
如果用户配置信息表明用户为高优先级用户，则编码任务的编码需求信息为高优先级级别；在一种实施例中，假设预先设置了用户为第一优先级；用户事件为第二优先级；热门应用为第三优先级；其他如非热门应用、普通用户等可以配置为普通优先级。那么，这里可以将对应用户的优先级级别从普通优先级提升为第一优先级；
如果用户事件信息表明客户端的用户事件如鼠标事件、键盘事件等在预设时长内持续发生，则编码任务的编码需求信息为提升优先级级别；在一种实施例中，假设预先设置了用户为第一优先级；用户事件为第二优先级；热门应用为第三优先级；其他如非热门应用、普通用户等可以配置为普通优先级。那么，这里可以将优先级级别从普通优先级提升为第二优先级。
通过优先级级别的提升，保证了硬件编码资源被优先调度给高优先级的应用，保证了优先级级别高的用户使用场景的交互响应更及时，一定程度上提升了用户体验。而且，保证高优先级的编码任务先占用硬件编码资源，也提升了硬件编码资源的利用率。
在一种示例性实例中，反映远程桌面创建需求的信息包括网络传输质量信息，步骤200可以包括：
如果客户端的网络服务质量信息表明网络质量不好，比如网络抖动超过预设第二阈值、和/或丢包次数超过预设第三阈值等，则编码任务的编码需求信息为降低输出帧率，也就意味着减少硬件编码资源。
在网络质量不好的情况下，即使按照编码任务的需求分配了硬件编码资源，保证了所需的输出帧率，由于传输过程中的丢包等问题，在客户端也是不能创建出满意的 远程桌面的，因此，此时通过将不必要占用的硬件编码资源释放出来，可以为其他网络质量好的客户端提供充足的硬件编码资源，从而提升了硬件编码资源的利用率。
在一种示例性实例中，步骤200之前还可以包括：服务器获取反映远程桌面创建需求的信息。
需要说明的是，如果反映远程桌面创建需求的信息包括上述任意组合的信息时，对应每个信息可以预先设置一个权重，这样，可以根据权重来综合计算最终的优先级级别，和/或硬件编码资源减少情况。具体实现对于本领域技术人员来讲，基于本申请实施例的记载是容易实现的，这里不再赘述。
需要说明的是，本申请实施例中的编码任务都会预先设置优先级级别的，也会有对应的硬件编码资源的需求信息。步骤200具体实现的描述中，只是强调了优先级别需要提升，或者硬件编码资源需要减少。
步骤201：根据编码任务的编码需求信息调用硬件编码资源，以实现对编码任务的编码。
在一种示例性实例中，步骤201可以包括：
按照编码需求信息中的编码参数信息确定硬件编码资源的需求信息；
根据确定出的硬件编码资源的需求信息调用硬件编码资源；
按照编码需求信息中的优先级级别从高到低对编码任务进行编码。
在一种示例性实例中，硬件编码资源可以通过硬件编码器提供。
通过本申请提供的实现编码调度的方法，根据反映远程桌面创建需求的信息确定的编码任务的编码需求信息来对硬件编码资源进行调度，在不必要占用的硬件编码资源时及时释放了资源，节省了硬件编码资源；而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。
在一种示例性实例中，步骤200中，如果用户界面元素信息表明当前图像帧中文字信息所占比例大于预设第四阈值，可以提升画质，此时，编码任务的编码需求信息中的硬件编码资源的需求信息为增加硬件编码资源。在一种实施例中，可以增加如50％，当文字信息所占比例大于预设第四阀值时，编码参数信息中编码时采用的画质量化参数值可以提升如50％，从而增加如50％硬编编码资源。相应地，步骤201还可以包括：
根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照该编码任务对应的优先级级别进行编码后传输。
在一种示例性实例中，步骤200中，如果更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的只有部分区域，也可以提升更新区域的画质，那么，此 时，编码任务的编码需求信息中的硬件编码资源的需求信息为保持硬件编码资源不变，由于此时不变的硬件编码资源只用来编码发生变化的区域，因此也相当于提高了画质量化参数，提升了画质，在一种实施例中，提升的比例可以是全部区域大小或变化区域大小。相应地，步骤201还可以包括：
根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照该编码任务对应的优先级级别进行编码后传输。
在一种示例性实例中，如果确定出硬件编码资源不足以完成编码任务的传输，那么，可以将编码任务按照优先级级别放置在等待队列中，直到有空闲的硬件编码资源释放出来后，在按照优先级级别从高到低进行编码后传输。
本申请实施例中，整个调度过程在服务器中实现，可以缓存更多路数的编码任务；而且，当硬件编码资源满负载运行时，通过时分复用机制提升了吞吐率，提高了支持的并发路数，一定程度上降低了单路的成本。
在一种示例性实例中，本申请实施例还包括：将编码后的远程桌面的图像视频流传输给客户端。
本申请还提供一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令用于执行上述任一项的编码调度方法。
本申请再提供一种实现编码调度的设备，包括存储器和处理器，其中，存储器中存储有以下可被处理器执行的指令：用于执行上述任一项所述的编码调度方法的步骤。
图3为本申请实施例中服务器的组成结构示意图，如图3所示，包括：需求确定模块、调度模块、提供硬件编码资源的硬件编码模块；其中，
需求确定模块，设置为根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；
调度模块，设置为根据编码任务的编码需求信息调用硬件编码模块提供的硬件编码资源，以实现对编码任务的编码。
在一种示例性实例中，反映远程桌面创建需求的信息包括远程桌面画质需求信息，需求确定模块具体设置为：
如果更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的只有部分区域，则编码任务的编码需求信息中的硬件编码资源的需求信息为仅根据区域更新信息调用硬件编码资源；
如果用户界面元素信息表明当前图像帧中图片信息所占比例大于预设第一阈值，可以降低画质，则编码任务的编码需求信息中的硬件编码资源的需求信息为减少硬件 编码资源。
对于只有部分区域发生更新的情况，相比于上一帧图像帧，降低了对硬件编码资源的使用，动态释放了不必要占用的硬件编码资源，节省了硬件编码资源，而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。对于图像帧中图片信息占比较大的情况，由于人眼对图像的画质要求并没有这么敏感，即使降低了硬件编码资源也不会影响用户体验，而且动态释放了不必要占用的硬件编码资源，节省了硬件编码资源，而且释放的硬件编码资源可提供给其他编码任务使用，换句话说也是提高了并发路数，同时也提升了硬件编码资源的利用率。
在一种示例性实例中，反映远程桌面创建需求的信息包括发起远程桌面创建的客户端信息，需求确定模块具体设置为：
如果应用程序信息表明应用属于热门应用，则编码任务的编码需求信息为提升优先级级别；
如果用户配置信息表明用户为高优先级用户，则编码任务的编码需求信息为高优先级级别；
如果用户事件信息表明客户端的用户的鼠标事件、键盘事件等在预设时长内持续发生，则编码任务的编码需求信息为提升优先级级别。
通过优先级级别的提升，保证了硬件编码资源被优先调度给高优先级的应用，保证了优先级级别高的用户使用场景的交互响应更及时，一定程度上提升了用户体验。而且，保证高优先级的编码任务先占用硬件编码资源，也提升了硬件编码资源的利用率。
在一种示例性实例中，反映远程桌面创建需求的信息包括网络传输质量信息，需求确定模块具体设置为：
如果客户端的网络服务质量信息表明网络质量不好，比如网络抖动超过预设第二阈值、和/或丢包次数超过预设第三阈值等，则编码任务的编码需求信息为降低输出帧率，也就意味着减少硬件编码资源。
在一种示例性实例中，硬件编码模块可以为硬件编码器。
在网络质量不好的情况下，即使按照编码任务的需求分配了硬件编码资源，保证了所需的输出帧率，由于传输过程中的丢包等问题，在客户端也是不能创建出满意的远程桌面的，因此，此时通过将不必要占用的硬件编码资源释放出来，可以为其他网络质量好的客户端提供充足的硬件编码资源，从而提升了硬件编码资源的利用率。
在一种示例性实例中，调度模块具体设置为：按照编码参数信息确定硬件编码资源的需求信息；根据确定出的硬件编码资源的需求信息调用硬件编码资源；按照优先 级级别从高到低对编码任务进行编码。
在一种示例性实例中，调度模块还设置为：将编码后得到的远程桌面的图像视频流传输给客户端。
在一种示例性实例中，需求确定模块还设置为：如果用户界面元素信息表明当前图像帧中文字信息所占比例大于预设第四阈值，可以提升画质，此时，编码任务的编码需求信息中的硬件编码资源的需求信息为增加硬件编码资源。相应地，
调度模块还设置为：
根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照该编码任务对应的优先级级别进行编码后传输。
在一种示例性实例中，需求确定模块还设置为：如果更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的只有部分区域，也可以提升更新区域的画质，那么，此时，编码任务的编码需求信息为保持硬件编码资源不变。相应地，
调度模块还设置为：
根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照该编码任务对应的优先级级别进行编码后传输。
在一种示例性实例中，调度模块还设置为：
如果确定出硬件编码资源不足以完成编码任务的传输，那么，可以将编码任务按照优先级级别放置在等待队列中，直到有空闲的硬件编码资源释放出来后，再按照优先级级别从高到低进行编码后传输。
本申请实施例中，整个调度过程在服务器中实现，可以缓存更多路数的编码任务；而且，当硬件编码资源满负载运行时，通过时分复用机制提升了吞吐率，提高了支持的并发路数，一定程度上降低了单路的成本。
本申请实施例还提供一种客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的根据反映远程桌面创建需求的信息编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面。
图4为本申请实施例中获取远程桌面的系统的架构示意图，如图4所示，包括服务器、一个以上客户端；其中，
客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面；
服务器，设置为接收到远程创建的请求，根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；根据编码任务的编码需求信息调用硬件编码模块提供的硬 件编码资源，以实现对编码任务的编码；将编码后得到的远程桌面的图像视频流传输给客户端。
在一种示例性实例中，本申请实施例提供的实现编码调度的系统中的服务器包括图3所示的任一项服务器。
在云桌面和云应用的实际应用场景中，本申请实施例会根据客户端的请求，计算出每一路使用硬件编码资源的优先级、编码参数如帧率等；然后，再根据每一路优先级从硬件加速卡中申请分配对应帧率和硬件编码资源，并将编码参数传递给硬件编码器进行编码后得到远程桌面的图像视频流。这样，使得硬件加速卡的工作时间或计算时间，在满负荷情况下的利用率更好，而且，服务器从收到编码到完成的时间，优先满足了优先级高的编码任务，降低了响应时间，提升了对编码任务的处理效率。当云桌面和云应用的帧率或者更新区域发生变化时，硬件加速卡可以根据变化之后的帧率和区域大小重新分配硬件编码资源，使得单位时间内支持的并发路数增大，提高了吞吐率。
在一种示例性实例中，假设客户端发起远程桌面创建的请求，服务器根据用户配置信息，确定发起请求的客户端对应的用户为高优先级用户，那么，服务器会将该用户的优先级级别从普通优先级提升为第一优先级，也就是说编码任务的编码需求信息为第一优先级；这样，服务器会按照编码参数信息优先给该用户的请求分配所需的硬件编码资源，优先对编码任务进行编码后将编码处理后的远程桌面的图像视频流传输给客户端。对于客户端来讲，则是实现了高效的远程桌面创建，提升了用户体验。
在一种示例性实例中，假设在客户端使用创建好的远程桌面的过程中，服务器根据用户事件信息获知客户端的用户事件如鼠标事件在预设时长内持续发生，那么，服务器会提升该用户的优先级级别，假设预先设置了用户事件为第二优先级，那么，服务器会按照第二优先级分配所需的硬件编码资源对该用户的用户事件进行处理，这样，对于客户端来讲，也是实现了高效的远程桌面响应，提升了用户体验。
在一种示例性实例中，在客户端，用户可以通过预先设置的界面，设置用户自身认为重要，或喜欢，或常用的应用的优先级级别，并通过请求告知服务器，这样，服务器在进行硬件编码资源调度时，可以优先考虑来自客户端的配置信息。这样，也保证了用户的个性化需求。
在一种示例性实例中，在客户端还可以将当前显示的远程桌面的画质信息、网络传输信息、当前用户、应用的优先级级别等信息显示给用户。这样，使得用户对当前显示的远程桌面处理有直观的感观外，也有质量上量化的信息。需要说明的是，显示给用户的信息可以采用简单易懂的方式，比如：网络质量可以分为：好、较好、差来告诉用户当前网络质量的情况；再如：显示的远程桌面画质可以分为如：高质量、一 般质量、质量较差等，这样用户对当前获得的远程桌面质量就一目了然了。如果用户觉得当前远程桌面质量不佳，可以重新提出请求，重新创建；此时，用户可以将应用或用户的优先级级别提升，以便分配到更多的硬件编码资源，从而获得更优的远程桌面。
虽然本申请所揭露的实施方式如上，但所述的内容仅为便于理解本申请而采用的实施方式，并非用以限定本申请。任何本申请所属领域内的技术人员，在不脱离本申请所揭露的精神和范围的前提下，可以在实施的形式及细节上进行任何的修改与变化，但本申请的专利保护范围，仍须以所附的权利要求书所界定的范围为准。

Claims (21)

  1. 一种编码调度方法，包括：
    服务器根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；
    根据编码任务的编码需求信息调用硬件编码资源，以实现对编码任务的编码。
  2. 根据权利要求1所述的编码调度方法，其中，所述反映远程桌面创建需求的信息包括以下任一项或任意组合：远程桌面画质需求信息、发起远程桌面创建的客户端信息、网络传输质量信息。
  3. 根据权利要求2所述的编码调度方法，其中，所述远程桌面画质需求信息包括以下任一项或任意组合：
    用于表示在后传输的图像帧与相邻在前传输的图像帧相比，发生信息更新的区域的更新区域信息；
    用于表示图像帧包括的文字信息、图片信息的用户界面元素信息。
  4. 根据权利要求3所述的编码调度方法，其中，所述确定编码任务的编码需求信息，包括：
    如果所述更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的为部分区域，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为根据所述区域更新信息调用硬件编码资源；
    如果所述用户界面元素信息表明当前图像帧中图片信息所占比例大于预设第一阈值，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为减少硬件编码资源。
  5. 根据权利要求2所述的编码调度方法，其中，所述发起远程桌面创建的客户端信息包括以下任一项或任意组合：
    用于确定应用是否属于热门应用的应用程序信息；
    用于显示用户的优先级级别的用户配置信息；
    用于表明所述客户端的用户事件的用户事件信息。
  6. 根据权利要求5所述的编码调度方法，其中，所述确定编码任务的编码需求信息，包括：
    如果所述应用程序信息表明应用属于热门应用，则所述编码任务的编码需求信息为提升优先级级别；
    如果所述用户配置信息表明用户为高优先级用户，则所述编码任务的编码需求信息为高优先级级别；
    如果所述用户事件信息表明客户端的用户事件在预设时长内持续发生，则所述编码任务的编码需求信息为提升优先级级别。
  7. 根据权利要求2所述的编码调度方法，其中，所述网络传输质量信息包括网 络服务质量信息；
    所述确定编码任务的编码需求信息，包括：
    如果所述客户端的网络服务质量信息表明网络质量不好，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为减少硬件编码资源。
  8. 根据权利要求3所述的编码调度方法，其中，所述确定编码任务的编码需求信息，还包括：
    如果所述用户界面元素信息表明当前图像帧中文字信息所占比例大于预设第四阈值，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为增加硬件编码资源；
    所述确定编码任务的编码需求信息，还包括：
    根据当前硬件编码资源的负载情况和所述编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照所述编码任务对应的优先级级别进行传输。
  9. 根据权利要求3所述的编码调度方法，其中，所述确定编码任务的编码需求信息，还包括：
    如果所述更新区域信息表明当前图像帧与上一帧图像帧相比，发生信息更新的为部分区域，则所述编码任务的编码需求信息中的硬件编码资源的需求信息为保持硬件编码资源不变；
    所述确定编码任务的编码需求信息，还包括：
    根据当前硬件编码资源的负载情况和编码任务的编码需求信息，在当前硬件编码资源的负载情况能满足增加的硬件编码资源需求时，增加调用硬件编码资源，按照所述编码任务对应的优先级级别进行传输。
  10. 根据权利要求1～9任一项所述的编码调度方法，其中，所述编码需求信息包括：所述编码任务的优先级级别信息、编码参数信息和硬件编码资源的需求信息。
  11. 根据权利要求10所述的编码调度方法，其中，所述根据编码任务的编码需求信息调用硬件编码资源，包括：
    按照所述编码参数信息确定硬件编码资源的需求信息；
    根据硬件编码资源的需求信息调用所述调用硬件编码资源；
    按照所述优先级级别从高到低对所述编码任务进行编码。
  12. 根据权利要求1所述的编码调度方法，所述方法还包括：将所述编码后得到的远程桌面的图像视频流传输给客户端。
  13. 一种服务器，包括：需求确定模块、调度模块、提供硬件编码资源的硬件编码模块；其中，
    需求确定模块，设置为根据反映远程桌面创建需求的信息确定编码任务的编码需 求信息；
    调度模块，设置为根据编码任务的编码需求信息，调用硬件编码模块提供的硬件编码资源，以实现对编码任务的编码。
  14. 根据权利要求13所述的服务器，其中，所述反映远程桌面创建需求的信息包括以下任一项或任意组合：远程桌面画质需求信息、发起远程桌面创建的客户端信息、网络传输质量信息。
  15. 根据权利要求14所述的服务器，其中，所述反映远程桌面创建需求的信息包括以下任一项或任意组合：
    用于表示在后传输的图像帧与相邻在前传输的图像帧相比，发生信息更新的区域的更新区域信息；
    用于表示图像帧包括的文字信息、图片信息的用户界面元素信息；
    用于确定应用是否属于热门应用的应用程序信息；
    用于显示用户的优先级级别的用户配置信息；
    用于表明所述客户端的用户事件的用户事件信息；
    网络服务质量信息。
  16. 根据权利要求13所述的服务器，其中，所述调度模块具体设置为：
    按照所述编码任务的编码参数信息确定硬件编码资源的需求信息；根据硬件编码资源的需求信息调用所述调用硬件编码资源；按照编码任务的优先级级别从高到低对所述编码任务进行编码。
  17. 根据权利要求16所述的服务器，所述调度模块还设置为：
    将所述编码后得到的远程桌面的图像视频流传输给客户端。
  18. 根据权利要求13所述的服务器，其中，所述硬件编码模块为硬件编码器。
  19. 一种客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的根据反映远程桌面创建需求的信息编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面。
  20. 一种获取远程桌面的系统，包括：包括服务器、一个以上客户端；其中，
    客户端，设置为向服务器发起远程桌面创建的请求；接收来自服务器的编码处理后的远程桌面的图像视频流，解码接收到的图像视频流并显示远程桌面；
    服务器，设置为接收到远程创建的请求，根据反映远程桌面创建需求的信息确定编码任务的编码需求信息；根据编码任务的编码需求信息调用硬件编码模块提供的硬件编码资源，以实现对编码任务的编码；将编码后得到的远程桌面的图像视频流传输给客户端。
  21. 根据权利要求20所述的系统，其中，所述服务器包括权利要求13～权利要求18任一项所述的服务器。
PCT/CN2021/117361 2020-09-16 2021-09-09 编码调度方法、服务器及客户端和获取远程桌面的系统 WO2022057718A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/184,204 US20230221969A1 (en) 2020-09-16 2023-03-15 Encoding scheduling method, server, client, and system for acquiring remote desktop

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010973546.0A CN114265648B (zh) 2020-09-16 2020-09-16 编码调度方法、服务器及客户端和获取远程桌面的系统
CN202010973546.0 2020-09-16

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/184,204 Continuation US20230221969A1 (en) 2020-09-16 2023-03-15 Encoding scheduling method, server, client, and system for acquiring remote desktop

Publications (1)

Publication Number Publication Date
WO2022057718A1 true WO2022057718A1 (zh) 2022-03-24

Family

ID=80775903

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/117361 WO2022057718A1 (zh) 2020-09-16 2021-09-09 编码调度方法、服务器及客户端和获取远程桌面的系统

Country Status (3)

Country Link
US (1) US20230221969A1 (zh)
CN (1) CN114265648B (zh)
WO (1) WO2022057718A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117544623A (zh) * 2023-11-14 2024-02-09 江苏海辰云科信息技术有限公司 一种基于预测编码算法的桌面云远程显示系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101835043A (zh) * 2010-03-23 2010-09-15 熔点网讯(北京)科技有限公司 基于块编码的带宽自适应桌面共享方法
CN104994407A (zh) * 2015-06-26 2015-10-21 安徽广行通信科技股份有限公司 一种集中自适应视频转码的方法
CN106681679A (zh) * 2017-01-19 2017-05-17 深圳市云点科技有限公司 远程桌面画面的传输方法和装置
US20170142432A1 (en) * 2015-11-12 2017-05-18 Vmware, Inc. Split framebuffer encoding
CN108322781A (zh) * 2018-02-08 2018-07-24 武汉噢易云计算股份有限公司 提升虚拟桌面中播放高清视频效果的方法及系统
CN111246084A (zh) * 2020-01-09 2020-06-05 西安万像电子科技有限公司 控制帧率方法和装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107295340A (zh) * 2016-03-31 2017-10-24 中兴通讯股份有限公司 一种远程桌面视频编码的方法及装置
CN110868599B (zh) * 2019-12-06 2021-11-19 杭州顺网科技股份有限公司 一种远程桌面的视频压缩方法
CN111010582B (zh) * 2019-12-18 2022-01-18 深信服科技股份有限公司 一种云桌面图像处理方法、装置、设备及可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101835043A (zh) * 2010-03-23 2010-09-15 熔点网讯(北京)科技有限公司 基于块编码的带宽自适应桌面共享方法
CN104994407A (zh) * 2015-06-26 2015-10-21 安徽广行通信科技股份有限公司 一种集中自适应视频转码的方法
US20170142432A1 (en) * 2015-11-12 2017-05-18 Vmware, Inc. Split framebuffer encoding
CN106681679A (zh) * 2017-01-19 2017-05-17 深圳市云点科技有限公司 远程桌面画面的传输方法和装置
CN108322781A (zh) * 2018-02-08 2018-07-24 武汉噢易云计算股份有限公司 提升虚拟桌面中播放高清视频效果的方法及系统
CN111246084A (zh) * 2020-01-09 2020-06-05 西安万像电子科技有限公司 控制帧率方法和装置

Also Published As

Publication number Publication date
CN114265648B (zh) 2023-12-26
US20230221969A1 (en) 2023-07-13
CN114265648A (zh) 2022-04-01

Similar Documents

Publication Publication Date Title
US10810045B2 (en) Method and apparatus for allocating central processing unit (CPU) resources in a default resource pool
WO2022028089A1 (zh) 图像处理方法、装置、服务器及介质
US9146884B2 (en) Push pull adaptive capture
US10897428B2 (en) Method, server system and computer program product for managing resources
US9325530B2 (en) Management of virtual desktop infrastructure (VDI) sessions using real-time network conditions
WO2014026613A1 (zh) 网络带宽分配方法及终端
JP7100154B6 (ja) プロセッサコアのスケジューリング方法、装置、端末及び記憶媒体
US20140333633A1 (en) Apparatuses and methods for policy awareness in hardware accelerated video systems
US10489177B2 (en) Resource reconciliation in a virtualized computer system
US20160127382A1 (en) Determining variable wait time in an asynchronous call-back system based on calculated average sub-queue wait time
US20230342207A1 (en) Graphics processing unit resource management method, apparatus, and device, storage medium, and program product
US20140068165A1 (en) Splitting a real-time thread between the user and kernel space
WO2022222721A1 (zh) 计算资源调度方法及装置
WO2024088268A1 (zh) Rdma事件管理方法、设备及存储介质
WO2022057718A1 (zh) 编码调度方法、服务器及客户端和获取远程桌面的系统
US10037225B2 (en) Method and system for scheduling computing
CN112769788B (zh) 计费业务数据处理方法、装置、电子设备及存储介质
WO2020259208A1 (zh) 内存调度方法、装置、设备及存储介质
US9306995B2 (en) Communicating data in flows between first and second computers connected over a network
JP7418569B2 (ja) 異種プラットフォームでのハードウェアアクセラレーションによるタスクのスケジューリング及び負荷分散のための送信及び同期技術
CN116244231A (zh) 一种数据传输方法、装置、系统、电子设备及存储介质
CN114675972A (zh) 基于积分算法的云网络资源弹性调度方法及系统
CN111737176B (zh) 一种基于pcie数据的同步装置及驱动方法
CN103562869B (zh) 虚拟化环境下的音频处理方法和设备
CN114885023A (zh) 一种资源调度方法、装置及平台设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21868543

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21868543

Country of ref document: EP

Kind code of ref document: A1