RU2575679C2 - Entropy coder for image compression - Google Patents

Abstract

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering. A method of processing graphics data for transmission to a remote computing device, the method comprising receiving graphics data representative of a client screen associated with a virtual machine session; receiving information indicative of available bandwidth for said transmission and, based on the information, determining that the available bandwidth meets a predetermined threshold; and entropy encoding coefficients of transformed graphics data using a compact stream of bit tokens which form groups which align to byte boundaries, wherein series of zeroes are encoded in a variable number by a block with a size which is a multiple of a quantum size; literal values are encoded using an information entry in a cache of recently used literal values; and other values are encoded using a minimum number of blocks with a size which is a multiple of the quantum size.

EFFECT: faster encoding and decoding at the potential expense of loss of compressibility.

10 cl, 11 dwg

Description

BACKGROUND

As a form of networking, the increased popularity can mainly be called remote presentation systems, which can use protocols such as Remote Desktop Protocol (RDP) and Independent Computing Architecture (ICA) to share the desktop and other applications running on a server with a remote client. Such computing systems typically transmit keyboard clicks and mouse clicks or selections from the client to the server, relaying screen updates back to the other direction over a network connection (such as the Internet). Essentially, the user has the feeling that his or her machine is operating completely locally, when in reality only screenshots of the desktop or application programs are sent to the client device as they appear on the server side.

In the Remote Desktop operating environment, the data representing the graphics to be transmitted to the client is usually compressed by the server, transferred from the server to the client through the network, and decompressed by the client and displayed on the user's local display. The data encoding process usually requires significant processor cycles to compress and decompress the data. Such processing requirements can have a direct effect on the delay in encoding and decoding from the server to the client and adversely affect the feelings of the remote user.

SUMMARY OF THE INVENTION

One of the problems of remote presentation systems is that such systems seek to facilitate data compression due to processor performance. Many systems assume that bandwidth is very likely to be limited, and thus sacrifice processor speed to achieve higher levels of data compression and thereby reduce the amount of data that needs to be transmitted over a limited bandwidth link. However, many remote presentation clients are currently lower-level devices that can use less fast processors, but which can have access to wide bandwidth. In such cases, the overall performance and user experience can be improved by using a simpler compression device and compression methods that require less computation, even if this means that the compression is reduced.

In various embodiments, methods and systems for a fast entropy encoder / decoder for use in real-time image compression are disclosed. For example, a method of processing graphic data for transmission to a remote computing device may comprise receiving graphic data representing a client screen that needs to be visualized, receiving information showing the available bandwidth for transmission, and based on this information, determining whether the available band is satisfactory passing a predetermined threshold, and entropy coding of graphic data using a coding stream with a fixed bit size, while the series are well s are encoded in a variable number of fixed size blocks in bits, and alphabetic values are encoded using the information recording in the cache of recently used alphabetic values, or a variable number of blocks with a fixed size in bits.

BRIEF DESCRIPTION OF THE DRAWINGS

Systems, methods, and computer-readable media for graphic data for transmission to a remote computing device according to the present description will now be described with reference to the accompanying drawings.

Systems, methods and computer-readable media for changing the perspective view in a virtual operating environment according to the present description are further explained in the description of specific options for its implementation with reference to the accompanying drawings, in which:

1 and 2 depict an illustrative computer system in which aspects of the present disclosure may be implemented,

figure 3 depicts the operating environment for aspects of the practical implementation of the present disclosure,

4 depicts an operating environment for aspects of the practical implementation of the present disclosure,

5 illustrates a computer system including circuits for implementing remote desktop services,

6 illustrates a computer system including circuits for implementing remote services,

7 illustrates an example decoding process,

8 illustrates an example encoding process,

Fig.9 illustrates an example of a sequence of operations for processing graphic data for transmission to a client computer,

10 illustrates an example flowchart for processing graphic data for transmission to a client computer,

11 depicts an illustrative system for processing graphic data for transmission to a client computer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

General computing environments

Some specific details are set forth in the following description and figures in order to provide a thorough understanding of various embodiments of the disclosure. Some well-known details, often associated with computing technology and software, are not set forth in the following description in order to avoid unnecessarily obscuring various embodiments of the disclosure. In addition, it will be understood by those skilled in the art that they may practice other embodiments of the disclosure without one or more of the details described below. Finally, although various methods are described with reference to the steps and sequences in the following disclosure, the description as such is intended to provide a clear implementation of embodiments of the disclosure, and the steps and sequences of steps may not be performed as required for the practical implementation of the present disclosure.

It should be understood that the various techniques described herein may be implemented in connection with hardware or software, or, if necessary, with a combination of both of them. Thus, the methods and devices of the present disclosure or some of its aspects or parts may take the form of program code (i.e., instructions) embodied in tangible storage media such as floppy disks, CD-ROMs, hard drives or any other computer-readable media; moreover, when the program code is loaded into a machine or executed by a machine, such as a computer, the machine becomes a device for the practical implementation of the present disclosure. In the case of executing program code on programmable computers, the computing device typically includes a processor, a medium readable by the processor (including volatile and non-volatile memory devices and / or storage elements), at least one input device and at least one output device. One or more programs may execute or use the processes described in connection with the present disclosure, for example, through the use of an application programming interface (API), reusable controls, or the like. Such programs are preferably implemented in a high-level procedural language or object-oriented programming language for communication with a computer system. However, if desired, the programs can be implemented in assembler or machine language. In any case, the language may be a translated language or an interpreted language, and may be combined with a hardware implementation.

A remote desktop system is a computer system that supports applications that can be executed remotely by client computer systems. Input data is entered on the client computer system and transferred over the network (for example, using protocols based on the T.120 International Communication Union (ITU) protocol family, such as the Remote Desktop Protocol (RDP)) to the application on the terminal server. The application processes the input data as if the input data were entered on a terminal server. The application generates output in response to the received input, and the output is transferred over the network to the client computer system.

Embodiments may be performed on one or more computers. 1 and 2 and the following description are intended to provide a brief, general description of a suitable computing operating environment in which disclosure may be implemented. Those skilled in the art will appreciate that computer systems 200, 300 may have all or some of the components described with respect to computer 100 of FIGS. 1 and 2.

The term “circuits” as used throughout this specification may include hardware components such as hardware interrupt controllers, hard disk drives, network adapters, GPUs, hardware based on video / audio codecs, and firmware ) / software used to operate with such hardware. The term “circuits” may also include microprocessors configured to perform functions by means of firmware or by switches set in some way, or by one or more logical processors, for example, one or more cores of a multicore central processor. The logical processors in this example can be configured through software instructions that embody logic that functions to execute functions that are loaded from memory, such as random access memory (RAM, RAM), read-only memory (ROM, ROM), firmware, and / or virtual memory. In illustrative embodiments, where the circuitry includes a combination of hardware and software, a developer can write source code embodying logic that is essentially translated into machine-readable code that can be executed by a logical processor. Because one skilled in the art can understand that the state of the art has developed to the point where there is a slight difference between hardware, software, or a combination of hardware / software, choosing hardware versus software to perform functions is simply a constructive choice. Thus, since one skilled in the art can understand that a software process can be converted to an equivalent hardware structure and a hardware structure itself can be converted to an equivalent software process, the choice of implementation in hardware versus implementation in software is trivial and remains for the developer.

Figure 1 depicts an illustrative computer system that is configured to implement aspects of the present disclosure. A computer system may include a computer 20 or the like, including a processor 21, a system memory 22, and a system bus 23 that connects various components of the system, including connecting a system memory to a processor 21. The system bus 23 may be any of several types of structures buses, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. System memory includes read-only memory (ROM, ROM) 24 and random access memory (RAM, RAM) 25. Basic input / output system (BIOS) 26, containing basic routines that help transfer information between items inside the computer 20, like for example, during startup, it is stored in ROM 24. The computer may further include a hard disk drive 27 for reading from or writing to a hard disk (not shown), a magnetic disk drive 28 for reading from a removable magnetic disk 29 or for recording onto it and an optical disk drive 30 for reading from or for writing to a removable optical disk 31, such as a CD ROM or other optical medium. In some illustrative embodiments, computer-executable instructions implementing aspects of the present disclosure may be stored in ROM 24, on a hard disk (not shown), in RAM 25, on a removable magnetic disk 29, an optical disk 31, and / or in the cache memory of the processor 21 The hard disk drive 27, the magnetic disk drive 28 and the optical disk drive 30 are connected to the system bus 23 through the hard disk drive interface 32, the magnetic disk drive interface 33 and the optical disk drive interface 34, accordingly. Drives and drives and associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer 20. Although the operating environment described here uses a hard disk, a removable magnetic disk 29, and a removable optical disk 31, it should be understood by those skilled in the art that other types of computer-readable media that can store data that is accessible can also be used in such an operating environment. A computer, such as magnetic cassettes, flash memory cards, digital video disks, cartridges Bernoulli removable magnetic disk, RAM devices, ROM devices, and the like.

Several software modules may be stored on a hard disk, magnetic disk 29, optical disk 31, ROM 24, or RAM 25, including an operating system 35, one or more application programs 36, other program modules 37, and program data 38. The user can enter commands and information into computer 20 through input devices such as a keyboard 40 and pointing device 42. Other input devices (not shown) may include a microphone, joystick, game console, satellite dish, scanner, etc. . These and other input devices are often connected to the processor 21 via a serial port interface 46, which is connected to the system bus, but can be connected by other interfaces, such as a parallel port, a game port, or a universal serial bus (USB). The display 47 and other types of display devices can also be connected to the system bus 23 via some interface, such as the video adapter 48. In addition to the display 47, computers typically include peripheral output devices (not shown), such as speakers and printers. The system of FIG. 1 also includes a host adapter 55, a small computer system interface (SCSI) bus 56, and an external storage device 62 connected to the SCSI bus 56.

Computer 20 may operate in a network environment using logical connections to one or more remote computers, such as remote computer 49. Remote computer 49 may be another computer, server, router, network PC, peer device or other network node, virtual machine, and typically includes many or all of the elements described above with respect to computer 20, although only memory device 50 is illustrated in FIG. 1. Logical connections shown in FIG. 1, may include a local area network (LAN) 51 and a wide area network (WAN) 52. Such network environments typically occur in offices, enterprise-wide computer networks, intranets, and the Internet.

When used in a LAN network environment, computer 20 may be connected to a LAN 51 via a network interface or adapter 53. When used in a WAN network environment, computer 20 typically includes a modem 54 or other means for establishing communication over a wide area network 52, such as The Internet. The modem 54, which may be internal or external, may be connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules shown with respect to computer 20 or parts thereof may be stored in a remote storage device. It should be understood that network connections are examples and that other means can be used to establish a communication link between computers. Moreover, although it is contemplated that numerous embodiments of the present disclosure are particularly well suited for computer systems, nothing in this document is intended to limit the disclosure to these embodiments.

Turning now to FIG. 2, another embodiment of an exemplary computer system 100 is depicted. Computer system 100 may include a logical processor 102, such as an executive core. Although one logical processor 102 is illustrated, in other embodiments, the computer system 100 may have multiple logical processors, for example, multiple execution cores on a single processor substrate and / or multiple processor substrates, each of which may have multiple execution cores. As shown in this figure, various computer readable media 110 may be interconnected by one or more system buses that connect various system components to logic processor 102. The system buses can be any type of several types of bus structures, including a memory bus or memory controller , a peripheral bus, and a local bus using any of a diverse array of bus architectures. In exemplary embodiments, computer-readable media 110 may include, for example, random access memory (RAM) 104, memory 106, for example, an electromechanical drive, a solid state drive, for example, firmware 108, such as flash ROM or RAM, and removable storage devices 118, such as, for example, CD-ROMs, floppy disks, DVDs, flash drives, external storage devices, etc. Those skilled in the art will appreciate that other types of computer-readable media can be used, such as magnetic cassettes, flash memory cards, digital video discs, and Bernoulli removable magnetic disk cartridges.

Computer-readable media provides non-volatile storage of processor-executable instructions 122, data structures, program modules, and other data for computer 100. A basic input / output system (BIOS) 120 that contains basic routines that help transfer information between elements within computer system 100, such as for example, during startup, may be stored in firmware 108. Several programs may be stored in firmware 108, in memory 106, in RAM 104 and / or in removable storage devices 118 and may be executed by a logical processor 102, including an operating system and / or application programs.

Commands and information may be received by computer 100 through input devices 116, which may include, but are not limited to, a keyboard and pointing device. Other input devices may include a microphone, joystick, game console, scanner, and the like. These and other input devices are often connected to the logical processor 102 via an interface with a serial port that is connected to the system bus, but can be connected by other interfaces, such as a parallel port, a game port, or a universal serial bus (USB). A display or other types of display devices can also be connected to the system bus via some interface, such as a video adapter, which can be connected to or be part of the graphics processor 112. In addition to the display, computers typically include peripheral output devices (not shown), such as speakers and printers. The illustrative system also includes a host adapter, a small computer system interface (SCSI) bus, and an external storage device connected to the SCSI bus.

Computer system 100 may operate in a network environment using logical connections to one or more remote computers, such as a remote computer. The remote computer may be another computer, server, router, network PC, peer-to-peer device, or other network node, and typically includes many or all of the elements described above with respect to computer system 100.

When used in a LAN or WAN network environment, computer system 100 may be connected to a LAN or WAN network through a network interface card (NIC) 114. A NIC 114, which may be internal or external, may be connected to a system bus. In a networked environment, program modules shown with respect to a computer system 100 or parts thereof may be stored in a remote storage device. It should be understood that the network connections described herein are examples and that other means can be used to establish a communication link between computers. Moreover, although it is contemplated that numerous embodiments of the present disclosure are particularly well suited for computer systems, nothing in this document is intended to limit the disclosure to these embodiments.

A remote desktop system is a computer system that supports applications that can be executed remotely by client computer systems. Input data is entered on the client computer system and transferred over the network (for example, using protocols based on the T.120 International Communication Union (ITU) protocol family, such as the Remote Desktop Protocol (RDP)) to the application on the terminal server. The application processes the input data as if the input data were entered on a terminal server. The application generates output in response to the received input, and the output is transferred over the network to the client computer system. The client computer system provides output. Thus, on the client computer system, input data is received and output data is issued, while the processing actually takes place on the terminal server. A session can include a program shell and user interface, such as the desktop, subsystems that track the movement of the mouse cursor on the desktop, subsystems that convert a mouse click on an icon into commands that execute a specific stage of the program, etc. P. In another illustrative embodiment, the session may include some application. In this example, although the application is visualized, an environment that implements the desktop concept can still be generated and remain hidden from the user. It should be understood that the foregoing description is illustrative and that the disclosed invention can be implemented in various client / server operating environments and is not limited to a particular terminal services product.

In most, if not all remote environments that implement the desktop concept, input data (input on a client computer system) usually includes mouse or keyboard data representing commands in a certain application, and output data (generated by the application on a terminal server) usually include video data for displaying video data on the output device. Many remote environments that implement the desktop concept also include functionality that extends to the transfer of other data types.

Communication channels can be used to expand the RDP protocol, allowing plug-in cards to transfer data over an RDP connection. Functionality, such as printer redirection, clipboard redirection, port redirection, etc., use the communication channel technology. Thus, in addition to the input and output, there may be many communication channels that must carry the data. Accordingly, unplanned requests for transferring output data and one or more requests for transferring other data competing for available network bandwidth may occur.

Turning now to FIGS. 3 and 4, which depict high-level block diagrams of computer systems configured for implementing virtual machines. As shown in these figures, the computer system 100 may include the elements described in FIGS. 1 and 2, and components operating to implement virtual machines. One such component is hypervisor 202, which in the prior art may also be referred to as a virtual machine monitor. The hypervisor 202 in the illustrated embodiment can be configured to control and arbitrate access to the hardware of the computer system 100. Generally speaking, the hypervisor 202 can generate program execution environments, called partitions, from child section 1 to child section N (where N is an integer greater than or equal to 1). In some embodiments, the child partition can be considered as the base isolation unit supported by the hypervisor 202, i.e., each child partition can be associated with a set of hardware resources, for example, memory, devices, logical processors, etc., that are under control hypervisor 202 and / or the parent partition, and hypervisor 202 can isolate one partition from access resources of other partitions. In embodiments, the hypervisor 202 may be a standalone software product, part of an operating system, motherboard firmware, specialized integrated circuit, or a combination thereof.

In the above example, computer system 100 includes a parent section 204, which may also be considered domain 0 in the open source community. Parent section 204 can be configured to provide resources for guest operating systems running in child partitions 1-N through the use of virtualization service providers (VSPs) 228, which are also known as internal end drivers in the open source community. In this illustrative architecture, parent section 204 may control access to underlying hardware. VSP 228 providers can be used to multiplex interfaces for hardware resources through Virtualization Services (VSC) clients, which are also known as external end drivers in the open source community. Each child partition may include one or more virtual processors, such as virtual processors 230-232, for which guest operating systems 220-222 can manage and schedule logical connections to execute on them. Basically, virtual processors 230-232 are executable instructions and associated state information that provide a representation of a physical processor with a specific architecture. For example, one virtual machine may have a virtual processor having the characteristics of an Intel x86 processor, while another virtual processor may have the characteristics of a PowerPC processor. The virtual processors of the computer system in this example can be mapped to the logical processors of the computer system, so that the instructions that implement the virtual processors will be supported by the logical processors. Thus, in these illustrative embodiments, multiple virtual processors can be executed simultaneously, while another logical processor executes the instructions of the hypervisor. Generally speaking, and as illustrated in the figures, the combination of virtual processors, various VSC clients and memory in some section can be considered as a virtual machine, such as, for example, virtual machine 240 or 242.

Basically, guest operating systems 220-222 can include any operating system, such as, for example, operating systems from Microsoft®, Apple®, open source communities, and the like. Guest operating systems may include user / kernel operating modes and may have kernels that include schedulers, memory managers, etc. Kernel mode may include execution mode in a logical processor that provides access to at least instructions of a privileged processor. Each guest operating system 220-222 may have associated file systems that may have applications stored therein, such as terminal servers, e-commerce servers, email servers, and the like, and guest operating systems as such. Guest operating systems 220-222 may schedule logical connections for execution on virtual processors 230-232, and examples of such applications may be executed.

We now turn to figure 4, which depicts an alternative architecture that can be used to implement virtual machines. FIG. 4 depicts components similar to those of FIG. 3, however, in this illustrative embodiment, the hypervisor 202 may include virtualization service providers 228 and device drivers 224, and the parent section 204 may include configuration utilities 236. In this architecture, the hypervisor 202 may perform the same or similar functions as the hypervisor 202 of FIG. 3. The hypervisor 202 of FIG. 4 may be a standalone software product, part of an operating system, firmware on the motherboard, or part of the hypervisor 202 may be implemented by specialized integrated circuits. In this example, the parent section 204 may have instructions that can be used to configure the hypervisor 202, however, hardware access requests can be processed by the hypervisor 202 instead of being transmitted to the parent section 204.

Turning now to FIG. 5, the computer 100 may include circuits configured to provide remote desktop services. In an exemplary embodiment, the illustrated operating system 400 may operate directly on hardware, or guest operating systems 220 or 222 may be implemented by a virtual machine, such as VM 216 or VM 218. Basic hardware 208, 210, 234, 212, and 214 are dotted. lines to identify which hardware can be virtualized.

Remote services can be provided for at least one client, such as client 401 (although one client is depicted, remote services can be provided for several clients). Exemplary client 401 may include a computer terminal that is implemented with hardware configured to send user input to a remote server session and display user interface information generated by the session. In another embodiment, the client 401 may be implemented by a computer that includes elements similar to those of the computer 100 of FIG. 1b. In this embodiment, the client 401 may include circuits configured for operating systems and circuits configured to emulate the functionality of the terminals, for example, a remote desktop client application that may be executed by one or more logical processors 102. One skilled in the art will understand that circuits configured to execute an operating system may also include circuits configured to emulate a terminal.

Each connecting client can have a session (such as session 404) that allows the client to access data and applications stored in computer 100. Basically, applications and some components of the operating system can be loaded into a specific memory area assigned to a specific session. Thus, in some cases, some components of the operating system (OS, OS) can be generated N times (where N represents the number of current sessions). These various components of the OS of the system may request services from the kernel 418 of the operating system, which may, for example, manipulate memory; provide read / write discs; and configure the logical connections of each session to be executed on the logical processor 102. Some illustrative subsystems that can be loaded into the session space may include subsystems that generate desktop operating environments, subsystems that track the movement of the mouse cursor on the desktop, subsystems, which convert clicks on the icon into commands that implement an instance of the program, etc. Processes that implement these services, for example, tracking the movement of the mouse cursor, are marked with a specific identifier and loaded into a specific area of memory that is allocated for the session.

A session may be generated by session manager 416, for example, a process. For example, session manager 416 can initialize and monitor each remote session by generating a session identifier for a specific session space; by assigning memory to this session space; and by generating system environment variables and subsystem process instances in the memory assigned to the session space. Session Manager 416 may be activated when the operating system 400 receives a request for a remote desktop session.

A connection request may be first processed by transport stack 410, for example, an RDP protocol stack. Transport stack instructions 410 can configure the logical processor 102 to listen for connection messages on a specific port and route them to session manager 416. When sessions are generated, transport stack 410 can create an instance of the Remote Desktop Protocol stack for each session. Stack instance 414 is an illustrative instance that can be generated for session 404. In general, each instance of the Remote Desktop Protocol stack can be configured to route input to an associated client and route client input to operating system subsystem 444 for the corresponding remote session.

As shown in this figure, in one embodiment, an application 448 (although one is shown, others can also be executed) can execute and generate an array of bits. Said array can be processed by a graphical interface 446, which, in turn, can visualize bitmaps, for example, arrays of pixel values that can be stored in memory. As shown in the figure, an instance of the remote display subsystem 420 can be created that can capture visualization calls and send these calls over the network to the client 401 via the stack instance 414 for the session.

In addition to organizing remote work with graphic data and video data, an instance of the 458 Plug and Play redirector can also be created to organize the remote operation of various devices, such as printers, mp3 players, client file systems, CD-ROM drives, etc. A Plug and Play redirector 458 can receive information from a client-side component that identifies peripheral devices connected to the client 401. A Plug and Play redirector 458 can then configure the operating system 400 to download redirected drivers for the peripheral client devices 401. Redirected device drivers can receive calls from the operating system 400 to access peripheral devices and send calls over the network to client 401.

As discussed above, clients can use the protocol to provide remote presentation services, such as the Remote Desktop Protocol (RDP), and to connect to resources that use terminal services. When a remote desktop client connects to a terminal server gateway, the gateway can open a socket connection with the terminal server and redirect client traffic to the remote presentation port or to a port specifically designed for remote access services. The gateway can also perform some gateway-specific data exchanges with a client using the terminal server's gateway protocol, transmitted over secure hypertext transfer protocol (HTTPS).

Turning now to FIG. 6, a computer system 100 is shown including circuitry for implementing remote services and for implementing aspects of the present invention. As shown in the figure, in one embodiment, the computer system 100 may include components similar to those described in FIGS. 2 and 5, and may implement a remote presentation session. In one embodiment of the present disclosure, a remote presentation session may include aspects of a console session, for example, a session generated for a user using a computer system, and a remote session. Similar to as described above, the session manager 416 can initiate and control the remote presentation session by turning on / off the components to implement the remote presentation session.

One set of components that can be loaded into a remote presentation session is console components that provide high-precision remote transmission capabilities, namely, components that take advantage of 3D graphics and 2D graphics rendered by 3D hardware.

3D / 2D graphics rendered by 3D hardware can be accessed using a driver model that includes a user mode driver 522, an application programming interface (API) 520, a graphics system kernel 524, and a kernel mode driver 530. Application 448 (or any other process, such as a user interface that generates 3D graphics) can generate API structures and send them to an application programming interface (API) 520, such as Direct3D from Microsoft®. In turn, API 520 can communicate with a user mode driver 522, which can generate primitives, for example, fundamental geometric shapes used in computer graphics, represented as vertices and constants, which are used as building blocks for other shapes, and stores them in buffers, such as memory pages. In one embodiment, application 448 may declare how it is going to use the buffer, for example, what type of data it is going to store in the buffer. An application, such as a video game, can use a dynamic buffer to store primitives for an avatar and a static buffer to save data that will not change frequently, such as data that represents a building or forest.

Continuing the description of the driver model, the application can fill buffers with primitives and issue execution commands. When the application issues an execution command, the buffer can be attached to the execution list via the kernel mode driver 530 and scheduled by the graphics core scheduler 528. Each graphics source, such as an application or user interface, can have a context and its own execution list. The graphics engine 524 may be configured to schedule various contexts for execution on the graphics processor 112. The graphics processor 528 may be executed by the logic processor 102, and the scheduler 528 may instruct the kernel mode driver 530 to render the contents of the buffer. The stack instance 414 can be configured to receive commands and send the contents of the buffer over the network to the client 401, where the contents of the buffer can be processed by the client GPU.

The following is an example of the operation of a virtual graphics processor, which is used in combination with an application that calls remote presentation services. 6, in one embodiment, a virtual machine session may be generated by computer 100. For example, session manager 416 may be executed by logic processor 102, and a remote session may be initiated that includes certain remote components. In this example, the generated session may include an operating system core 418, a graphics system core 524, a user mode display driver 522, and a kernel mode display driver 530. The user mode display driver 522 may generate graphic primitives that may be stored in memory. For example, API 520 may include interfaces that may be provided to processes, such as a user interface for operating system 400 or application 448. A process may send high-level API commands, such as Point Lists, Line Lists, stripe lines (Line Strips), lists of triangles (Triangle Lists), strips of triangles (Triangle Strips) or fans of triangles (Triangle Fans), in API 420. API 520 can accept these commands for the user mode driver 522, which can then generate vertices and save them in one or n how many buffers. GPU scheduler 528 may execute and decide to render the contents of the buffer. In this example, a command for server GPU 112 may be captured, and buffer contents (primitives) may be sent to client 401 via network interface board 114. In one embodiment, an API may be provided by session manager 416 with which components can interact to determine whether a virtual graphics processor is available.

In one embodiment, a virtual machine instance can be created, such as the virtual machine 240 in FIGS. 3 or 4, and this virtual machine can serve as an execution platform for the operating system 400. In this example, the operating system 400 may be implemented by a guest operating system system 220. A virtual machine instance can be created when a connection request is received over the network. For example, the parent section 204 may include an instance of the transport stack 410 and may be configured to receive connection requests. Parent section 204 can initialize the virtual machine in response to a connection request along with a guest operating system that includes the ability to implement remote sessions. Then, the connection request can be forwarded to the transport stack 410 of the guest operating system 220. In this example, an instance of each remote session can be created in the operating system, which is executed by its own virtual machine.

In one embodiment, a virtual machine instance can be created, and a guest operating system 220 implementing the operating system 400 can be executed. Similar to as described above, a virtual machine instance can be created when a connection request is received over the network. Remote sessions can be generated by the operating system. Session manager 416 may be configured to determine that the request is for a session that supports 3D rendering, and session manager 416 may load a console session. In addition to loading the console session, the session manager 416 can load the stack instance 414 'for the session and configure the system to receive the primitives generated by the user mode display driver 522.

User mode driver 522 can generate graphic primitives that can be received and stored in buffers available for transport stack 410. Kernel mode driver 530 can attach buffers to the execution list for the application, and GPU scheduler 528 can execute and determine when to issue visualization commands for buffers. When the scheduler 528 issues a render command, the command may be intercepted, for example, by the kernel mode driver 530 and sent to client 401 via the stack instance 414.

GPU scheduler 528 may execute a program and decide to issue instructions to visualize the contents of the buffer. In this example, graphic primitives associated with the visualization instruction can be sent to client 401 via network interface board 114.

In one embodiment, at least one kernel mode process may be performed by at least one logical processor 112, and at least one logical processor 112 may synchronize the rendering vertices stored in different buffers. For example, graphics processing scheduler 528, which can act simultaneously with the operating system scheduler, can schedule GPU operations. The GPU scheduler 528 can combine the individual vertex buffers into the correct execution order, so that the client 401 GPU executes the commands in the order that allows them to be correctly rendered.

One or more logical threads of a particular process, such as a video game, can match multiple buffers, and each thread can issue a draw command. Identification information for the vertices, for example, information generated based on each buffer, on each vertex, or on each group of vertices in the buffer, can be sent to the scheduler 528 of the GPU. This information can be stored in a table along with identification information associated with vertices from the same or other processes and used to synchronize the visualization of various buffers.

An application, such as a word processing program, can be executed, and it can declare, for example, two buffers - one for saving vertices for generating 3D menus and the other storing commands for generating letters that will fill the menu. An application can display a buffer and issue draw commands. The GPU scheduler 528 can determine the execution order of the two buffers, so menus are rendered along with the letters in such a way that it is pleasant to look at them. For example, other processes can issue rendering commands at the same or essentially the same time, and if the vertices were not synchronized vertices from different logical flows of different processes, they could be rendered asynchronously on client 401, thereby making the final image displayed looking chaotic and confused.

The bulk data compression tool 450 may be used to compress the graphic primitives before sending the data stream to the client 401. In one embodiment, the bulk data compression tool 450 may be a user mode component (not shown) or a kernel mode component of the stack instance 414 and may be configured to searching for such combinations in the data stream that is sent to client 401. In this embodiment, since the volume data compression means 450 receives a vertex stream, instead of receiving many numbered API structures, from numerous applications, the volumetric data compression tool 450 has a larger set of vertex data for filtering to find compression capabilities. That is, since the vertices for many processes are remote, instead of scattered API calls, there is a greater chance that the bulk data compression tool 450 will be able to find similar combinations in a given stream.

In one embodiment, the graphics processor 112 may be configured to use virtual addressing instead of physical addresses for memory. Thus, the memory pages used as buffers can be turned over to the system RAM or to a disk from video memory. The stack instance 414 'can be configured to receive virtual buffer addresses and to send content from virtual addresses when a visualization command is captured from the graphics core.

Operating system 400 may be configured, for example, various subsystems or drivers may be loaded to capture primitives and send them to a remote computer, such as client 401. Similar to as described above, session manager 416 may be executed by logical processor 102, and a session may be initialized. which includes remote components. In this example, the generated session may include an operating system core 418, a graphics system core 524, a user mode display driver 522, and a kernel mode display driver 530.

The graphics core can schedule GPU operations. The GPU scheduler 528 can combine the individual vertex buffers into the correct execution order, so that the client 401 GPU executes the commands in the order that allows them to be correctly rendered.

Turning now to FIG. 7, a flowchart illustrating a decoding process according to one embodiment of the present disclosure is shown. The coding processor is shown in FIG. The encoded tile can first be run through an RLGR decoder 900 to generate a coefficient of a quantized tile. The indicated operation can be performed on the central processor.

Dequantization 705 may first be performed on the central processor using SSE2 instructions. After dequantization, ten subranges of the three mosaic tile components can be copied to three Direct16D L16 texture buffers, one for each of Y, U, and V. These three textures can be sent to the GPU and can be used in step 710 of the inverse discrete wavelet transform (DWT) ) as input.

All the variations mentioned for implementing the above sections are only illustrative embodiments, and nothing in this document should be interpreted as limiting the disclosure of any particular aspect of virtualization.

Entropy coding

In a virtual desktop or remote presentation session, user graphics and video data can be visualized on the server for each user. The resulting bitmaps can then be sent to the client for display and interaction. To reduce network bandwidth requirements, bitmaps can be compressed before being sent to the client. It is desirable that the compression method be efficient with a low latency.

A system and method for encoding and decoding bitmaps and other graphic data are described herein. The coding system may include an image fragmentation system with an image fragmentation module that initially divides the original image data into mosaic data elements. Then, the frame differentiation module can output only the modified mosaic data elements to various processing units, which convert the changed mosaic data elements into corresponding mosaic components. According to an embodiment, the quantizer may perform a compression procedure with respect to the mosaic components to generate compressed data according to an adjustable quantization parameter. The adaptive entropy encoder selector may then select one of a plurality of entropy encoders to perform the entropy encoding procedure, thereby producing encoded data. An entropy encoder can also use a feedback loop to adjust the quantization parameter in light of the current bandwidth characteristics. The processes for compressing, encoding, and decoding graphic data mentioned herein can mainly use one or more of the methods and systems described in the assigned US patent No. 7460725, entitled "System And Method For Effectively Encoding And Decoding Electronic Information", and in the patent US application No. 12/399302, entitled "Frame Capture, Encoding, And Transmission Management", filed March 6, 2009, which are fully incorporated here by reference.

In the various methods and systems described herein that improve the processing and control of the various processes described above can be used to provide more efficient processing and thereby a more timely and rich user experience. The methods and systems also provide improvements in providing support for such graphics where network and / or system resources provide adequate bandwidth and / or the client device has lower processing speed or resources. The embodiments for visualizing, encoding, and transmitting graphic data described herein may be performed using various combinations of hardware and software processes. In some embodiments, the functions may be performed entirely in software. In other embodiments, functions may be performed using a combination of hardware and software processes. Such processes may additionally be performed using one or more processors and / or one or more specialized processors, such as a graphics processing unit (GPU) or other special purpose graphic rendering devices.

Moreover, although the following description is provided in the context of remote presentation systems, it should be understood that the disclosed embodiments may be implemented in any type of system in which graphic data is encoded and compressed for delivery over a network.

Various embodiments may comprise using a discrete wavelet transform (DWT) function to convert the components of the YUV tile format into the corresponding sub-bands of the YUV tile format. The quantizer function can perform a quantization procedure by using appropriate quantization methods to compress the subbands of the tiles. The quantizer function can produce compressed image data by reducing the bit rate of the tiles according to a specific compression coefficient, which can be set by an adaptive quantization parameter obtained through a feedback loop from an entropy encoder.

In one embodiment, the GPU processor may be provided with a bitmap with modified rectangles that need to be compressed. The bitmap can additionally be split into logical mosaic elements, and only those mosaic elements that change inside the changed rectangle are encoded and compressed. Thus, the process effectively executes the caching scheme in agreement with the client, where the resulting decoded image is maintained and displayed.

To reduce the bandwidth of the display stream to levels that are acceptable for transmission over local networks, wide area networks, and networks with low bandwidth, remote presentation compression algorithms are used. Such algorithms typically sacrifice server-side processor time in favor of lower bandwidth requirements.

Image compression means that can use an entropy encoder can be used. The function of an entropy encoder is to perform an entropy encoding procedure to generate encoded data. In some embodiments, the entropy encoding procedure further reduces the bit rate of the compressed image data by substituting the corresponding codes for the corresponding bit patterns in the compressed image data received from the quantizer.

Entropy encoding performed in a remote presentation system basically balances processor characteristics (e.g., performance) with a compression ratio. Entropy encoders can be configured for good compression with acceptable processor speed. Typical entropy encoders include series length encoders, Huffman codes, arithmetic and variations of the Golomb-Rice encoders. One of the main problems in designing effective entropy encoders for remote presentation applications is that there are usually large variations in the statistics of encoded integer blocks. Studies have shown that in most cases, before quantization, the data had distribution probabilities that can be much more concentrated near zero than with a Gaussian distribution.

The present disclosure relates to the implementation of a simplified entropy encoder that is configured to improve encoding and decoding performance at the potential cost of compressibility loss. However, in many situations, this trade-off may be acceptable and indeed be more desirable in scenarios that are more limited by low processor speed than throughput. The end result is that an encoder / decoder can be provided, which is two to three times faster than modern encoders / decoders with compressibility losses of the order of 10% to 20%.

Such an encoder / decoder can be useful because it provides the possibility of scenarios in which optimizing processor speed has a higher priority than saving every last bit of the bandwidth. For example, low-grade client devices can achieve better performance with a simpler compression tool to enable faster processing. Remote presentation systems are typically optimized to reduce throughput regardless of processor cost and performance. In many modern systems, bandwidth can be high, while client devices can be simpler devices, such as TV tuners or thin clients.

In one embodiment, the entropy encoder may be configured to avoid using a variable bitstream format. Using a variable bitstream unambiguously allows you to more slowly and efficiently encode and decode. In one embodiment, the encoder may be configured to use half-byte codes (also known as four-bit codes) to achieve a standard fixed-size encoding stream. When using such a stream, the stream can be decoded faster and can be decoded safely (with overflow control), with less processor overhead.

In one illustrative scheme, the encoder performs the following types of operations:

1) Series of zeros (which are the usual input to the entropy encoder) - series of zeros are encoded in a variable number of four bits, consistent with the statistical observation that most series are very short.

2) Literal values - literal values are encoded either in the form of the most recently used (LRU) successful result in the table (cache memory) from recently used alphabetic values, or in the form of a variable number of four bits, again advantageous in terms of statistical properties, which less value, the more likely they are found.

In both cases, there are essentially two streams: (a) a stream of four operation codes and (b) a stream of large values. Some opcodes, either for the length of a series or for the length of a literal value, simply show "get the next value in the stream of large values."

The stream of large values can be encoded using the basic multibyte encoding scheme, which uses fewer bytes for small values than for large values, but at the same time it is guaranteed to be valid only on fixed byte boundaries. Using such a coding scheme, both the stream of four codes and the stream of values can be decoded without the need for complex bit shifting schemes or decoding of variable bits, thus providing the possibility for faster operation than more general / complex entropy encoders (for example, various schemes based on the RLGR algorithm or Huffman codes). Such a simplified encoder can be configured so that any number of bits can be encoded, for example, from 1 to 32 bits. With more complex coding, decoding becomes problematic in terms of computation due to variable bits and necessary processing, which usually requires many coding branches and a significant amount of accounting for the use of system resources. Using this simplified scheme allows minimizing such complexity by using constant-size structures (e.g., four bits). In such a scheme, there are no output characters consisting of more than four bits, and the data is a multiple of bytes without offsets or cyclic permutations. A byte can contain two codes, which, if desired, can be processed in parallel. Moreover, if the number of fours is known, then buffer overflows can be avoided. In tests using typical remote presentation scenarios, losses of only 10-20% of compressibility were measured with a performance gain of 2-3 times using currently available processors.

In some embodiments, logic may be provided to switch between a more complex / slower entropy encoder and a simpler encoder, which is described herein. For example, in FIG. 9, a remote presentation system can provide at least two encoders 910 and 920. Encoder 910 can be a complex entropy encoder, such that runs the RLGR algorithm. Encoder 920 may be a simplified entropy encoder according to the present disclosure. Depending on the conditions of the network 930, for encoding data 900, the system may choose one of the encoders 910 or 920. For example, if the network conditions indicate that the network is congested and that the available bandwidth is limited, a complex encoder 910 may be selected for encoding data 900 to minimize the amount of data transmitted over the network 930. Similarly, if the network conditions indicate that the network is not congested, a simplified encoder 920 may be selected for encoding data 900 to provide faster processor operation on the client device ystve.

Appendix A provides an illustrative embodiment of a simplified decoder according to the present disclosure.

Figure 10 depicts an illustrative process for processing graphical data for transmission to a client computer, including operations 1000, 1002, 1004 and 1006. In figure 10, operation 1000 begins the sequence of operations, and operation 1002 illustrates the reception of graphic data representing the client screen associated with the virtual machine session. Operation 1004 illustrates receiving information showing the available bandwidth for transmission, and based on this information, determining whether the available bandwidth satisfies a predetermined threshold. Operation 1006 illustrates the entropy encoding of the coefficients of the transformed graphic data using a compact stream of bit markers aligned along byte boundaries. In one embodiment, a series of zeros is encoded in a variable number of blocks with a size that is a multiple of the quantization size, letter values are encoded using an information record in the cache of recently used letter values, and other values are encoded using a minimum number of blocks with a quantization size. A bit token may be a string of bits that defines a block of data. For example, a nibble-based system uses a four-bit token.

In various embodiments, the quantization size may be a nibble. In some embodiments, the flow of operations may include generating a stream of entropy coding operation codes and a stream of large values. The procedure may further include entropy encoding the stream of large values using a multibyte encoding scheme that uses fewer bytes for small values than large values, and dividing said graphic data into mosaic data elements, processing said mosaic data elements into mosaic components, and performing said entropy coding with respect to said mosaic components. The encoding scheme can be configured to act only on the boundaries of a fixed byte.

11 depicts an illustrative system for processing graphic data for transmission to a client computer, as described above. As shown in FIG. 11, the system 1100 includes a processor 1110 and a memory 1120. The memory 1120 further comprises computer instructions configured to process the graphic data for transmission to a remote computing device. Block 1122 illustrates receiving graphic data representing a client screen associated with a virtual machine session. Block 1124 illustrates the division of said graphic data into mosaic data elements. Block 1126 illustrates the entropy coding of the coefficients of the transformed mosaic data elements using a stream of bit markers aligned on byte boundaries.

Any of the above aspects may be implemented in methods, systems, computer-readable media, or any type of industrial product. For example, a computer-readable medium may store computer-executable instructions for processing graphics data for transmission to a client computer. Such media may contain a first subset of instructions for receiving graphic data representing a client screen associated with a virtual machine session, and a second subset of instructions for entropy encoding the coefficients of the converted graphic data using a compact stream of bit markers aligned along byte boundaries, so that encoded data can decoded using the byte decoding process. Those skilled in the art will understand that additional sets of instructions may be used to cover the various aspects disclosed herein, and that the two subsets of instructions disclosed herein may vary in detail according to the present disclosure.

The foregoing detailed description may specify various embodiments of systems and / or processes through examples and / or flow diagrams. To the extent that such flowcharts and / or examples contain one or more functions and / or operations, those skilled in the art will appreciate that each function and / or operation in such flowcharts or examples may be implemented separately and / or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof.

It should be understood that the various techniques described herein may be implemented with hardware or software, or, if necessary, with a combination of both of them. Thus, the methods and devices of the present disclosure or some of its aspects or parts can take the form of program code (i.e., instructions) embodied on tangible media such as floppy disks, CD-ROMs, hard drives, or any other computer-readable media, wherein when the program code is downloaded to or executed by a machine, such as a computer, the machine becomes a device for practicing the present disclosure. In the case of executing program code on programmable computers, the computing device typically includes a processor, a medium readable by the processor (including volatile and non-volatile storage devices and / or storage elements), at least one input device and at least one output device. One or more programs may execute or use the processes described in connection with the present disclosure, for example, through the use of an application programming interface (API), reusable controls, or the like. Such programs are preferably implemented in a high-level procedural language or object-oriented programming language for communication with a computer system. However, if desired, the programs can be implemented in assembler or machine language. In any case, the language may be a translated language or an interpreted language and may be combined with a hardware implementation.

Although the invention has been specifically shown and described with reference to some of its preferred embodiments, one skilled in the art will appreciate that various changes in form and detail can be made without departing from the spirit and without departing from the scope of the present invention, which is defined by the following claims inventions. In addition, although elements of the present invention may be described or claimed in the singular, the plural is contemplated, unless specifically limited to the singular.

Claims (10)

1. A method of processing graphic data for transmission to a remote computing device, comprising the steps of:
receiving graphic data representing a client screen associated with a virtual machine session;
receiving information showing the available bandwidth for said transmission, and based on this information, determining whether the available bandwidth satisfies a predetermined threshold; and
perform entropy coding of the coefficients of the converted graphic data using a compact stream of bit markers that form groups that are aligned by byte boundaries, while:
a series of zeros are encoded in a variable number of blocks with a size that is a multiple of the quantization size;
alphabetic values are encoded using an information record in the cache of recently used alphabetic values; and
other values are encoded using a minimum number of blocks with a size that is a multiple of the quantization size. 2. The method according to claim 1, further comprising the step of generating codes of entropy coding operations and a stream of large values. 3. The method of claim 2, further comprising entropy encoding the stream of large values using a multibyte encoding scheme that uses fewer bytes for small values than for large values. 4. The method according to claim 3, in which the encoding scheme is configured to act only on fixed byte boundaries. 5. The method according to claim 3, further comprising the steps of dividing said graphic data into mosaic data elements, processing said mosaic data elements into mosaic components, and performing said entropy coding with respect to said mosaic components. 6. A system for processing graphic data for transmission to a remote computing device, comprising:
a computing device comprising at least one processor;
a storage device communicatively connected to said processor during said system operation, said storage device having computer instructions stored therein which, when executed by at least one processor, provide:
receiving graphic data representing a client screen associated with a virtual machine session;
dividing said graphic data into mosaic data elements;
entropy coding of the coefficients of the converted graphic data using a stream of bit markers that form groups that are aligned by byte boundaries, while:
a series of zeros are encoded in a variable number of blocks with a size that is a multiple of the quantization size;
alphabetic values are encoded using an information record in the cache of recently used alphabetic values; and
other values are encoded using a minimum number of blocks with a quantization size. 7. The system of claim 6, further comprising transmitting the encoded coefficients to a computing device configured to process the encoded coefficients based on the quantization size. 8. The system of claim 6, wherein the encoded data can be used for efficient decoding by an entropy decoding process configured to operate on encoded data on a byte basis. 9. The system of claim 6, further comprising entropy encoding the stream of large values using a multibyte encoding scheme that uses fewer bytes for small values than for large values. 10. A computer-readable medium storing computer-executable instructions for processing graphic data for transmission to a client computer, said instructions being intended for:
receiving graphic data representing a client screen associated with a virtual machine session; and
entropy coding of the coefficients of the converted graphic data using a compact stream of bit markers that form groups that line up byte boundaries, so that encoded data can be decoded using a byte-based decoding process, with:
a series of zeros are encoded in a variable number of blocks with a size that is a multiple of a nibble;
alphabetic values are encoded using an information record in the cache of recently used alphabetic values; and
other values are encoded using a minimum number of blocks with a multiple of a nibble.

Images