KR20060110267A - Method and apparatus for generating graphical and media display at a thin client - Google Patents

Abstract

The invention relates to generating at a client a display having a graphical and/or media components. A method for generating a media presentation at a client includes transmitting output from an application program executing on a server to the client, identifying a media stream within an application output, receiving an compressed data set representing at least a portion of the media stream and transmitting the compressed data set to the client. At least one of timing and control information associated with the media stream is captured and transmitted to the client.

Description

Method and Apparatus for Generating Graphical and Media Display at a Thin Client}

The present invention is generally related to distributed processing and more particularly to the creation of a display having graphics and / or media components at a client.

The thin client protocol can be used to display output generated by an application running on a server to a client running on a computer with limited processing power. Two examples of new client protocols are ICA (Independent Computing Architecture from Citrix Systems) and RDP (Microsoft's Remote Desktop Protocol). The client is sometimes called a remote terminal session. A single scene client protocol intercepts, via an application program, commands to the server operating system (OS) for display on a display screen. Intercepted commands are sent to the remote session using, for example, one or more presentation layer packets. When the remote session (eg scene client) receives the command, the remote session sends the received command to a remote session OS. The scene client uses the received commands to display an application program output on a display. In this way, the application program appears to be running on the scene client.

When an application program displays an image on a display screen, the image is typically displayed in bitmap form. The bitmap format of an image is usually a fairly large data set. Thus, the scene client protocol must send over the network together a bitmap representation of the image and applicable commands for how to display the bitmap representation. For low bandwidth networks, there is a significant delay in time before the complete image is received and displayed on the client. This results in inconvenience and discomfort for the user of the client. In addition, if the user pays directly for the bandwidth used to transmit such large bitmap formats-such as in a wireless network-then there is a significant cost for each transmission.

There is a similar problem when an application program renders a media presentation. Typically, video files are rendered as a series of bitmaps, and audio information is rendered using pulse code modulation. Thus, the scene client protocol transmits over the network a pulse code modulated signal representing a series of bitmaps representing a video file and / or audio information. The transmission is inefficient because it requires excessive bandwidth and significant CPU usage. In addition, even if enough bandwidth is available, it will appear as a slow graphical user interface on the client. For example, video playback can often be of poor quality or jerk and may not be well synchronized with the audio signal.

Thus, there is a need for an improved approach to rendering images and media presentations in remote terminal sessions.

The invention may be better understood with reference to the following description in accordance with the accompanying drawings.

1 is a block diagram illustrating one embodiment of a system for generating a graphical display for a remote terminal session in accordance with the present invention;

2 is a flow diagram illustrating one embodiment of a process for generating a graphical display for a remote terminal session in accordance with the present invention;

3 is a block diagram illustrating one embodiment of an apparatus for generating a media presentation at a client side in accordance with the present invention;

4A, 4B and 4C are flow charts illustrating one embodiment of a method for generating a media presentation at a client side in accordance with the present invention;

5A, 5B and 5C are flow diagrams illustrating one embodiment of a method for generating a media presentation at a client side in accordance with the present invention.

According to one advantage, the present invention saves the time and cost of transmission by replacing images and other non-text elements that were originally displayed in large bitmap formats with compressed formats available instead of the bitmap format prior to transmission. Can be. In some systems images and other multimedia content are transmitted in an already compressed format such as JPEG, PNG, GIF, MPEG3 or MPEG4. In such systems, the time and cost of transferring the media will be reduced by intercepting the already compressed format and replacing it with another version of the media stream, which is in the more compressed format. Sending compressed formats can greatly reduce the bandwidth required to transmit the media stream. The client decompresses the received data using the available libraries. The client then replaces the decompressed image with the original bitmap representation using, for example, modified scene client protocol commands with different identifying data.

According to another advantage of the invention, the compressed data set representing at least a portion of the media stream is intercepted on the first computing device before being decompressed. Alternatively, the compressed data set is decompressed on the first computing device and the decompressed data set is recompressed on the first computing device. On networks with limited bandwidth, the cost and time spent transmitting the data set is consequently reduced by transmitting the compressed data set rather than the decompressed data set.

   Reducing the bandwidth required for communication not only saves time and money, but also improves performance for clients and servers using the present invention. When the server sends a compressed data representation of a non-textual element or media stream in accordance with the present invention, the processing burden on the central processing unit (CPU) of the server is reduced. Otherwise, the CPU of the server decompresses the compressed data, transfers large amounts of uncompressed data, and consumes processing power for additional processing of the increased capacity of network communication. One server can interact with multiple clients, and delegating decompression and rendering operations to clients can also improve server performance. The performance of the client is further improved in that the processing burden on the client side associated with the receiver of large amounts of uncompressed data is reduced.

One aspect of the present invention relates to a method for generating a graphical display on the client side. The method includes transmitting output from an application program running on a server to a client, identifying a bitmap representation in the application output, and determining a check value for the bitmap representation. It includes. The method also includes partially retrieving the compressed data format of the bitmap representation using the check value and sending the compressed data format to the client on behalf of the bitmap representation.

Another aspect of the invention relates to a method of generating a graphical display on the client side. The method includes sending output from an application program running on a server to a client and identifying a non-literal element within the application output. The method includes retrieving a compressed data format associated with the non-literal element and transmitting the compressed data format to a client on behalf of the non-literal element.

In an embodiment according to this aspect of the invention, the method includes identifying a textual element in an application output and sending the textual element to a client. In another embodiment, the method includes receiving a compressed data format and optionally a character element at the client side and generating a display at the client side using the compressed data format and optionally the character element. do. In another embodiment, the method includes transmitting the compressed data using at least one presentation layer protocol packet. In another embodiment, the method includes transmitting at least one presentation layer protocol packet using a command to transfer the file in raw form.

In another embodiment, the method further comprises at least one presentation layer protocol packet. Matching the scene client protocol and / or presentation protocol. In another embodiment, the non-literal element is a bitmap representation, and the method includes replacing the bitmap representation with a compressed data format. In another embodiment, the method includes determining a client's ability to render the non-literal element using the compressed data format. The method includes transmitting an image rendering library capable of rendering the non-literal element using the compressed data format in accordance with determining that the client cannot render the non-literal element using the compressed data format.

In another embodiment, the method includes intercepting the application output and examining the intercepted output of the bitmap representation of the non-literal element. In another embodiment, the method includes calculating a first check value for bitmap representation of a non-literal element and an image store for a compressed data format having a check value that matches the first check value. )).

Another aspect of the invention relates to an apparatus for generating a graphical display on the client side. The apparatus includes an output filter module and a server agent. The output filter module is configured to intercept an output produced by an application program, identifies a non-literal element of the output, and retrieves a compressed data format associated with the non-literal element. The server agent is configured to send a compressed data format to the client instead of the non-literal element.

In one embodiment of the above aspects of the invention, the apparatus includes a server node comprising a server agent and an output filter module. In another embodiment, the system includes a client node. The client node includes a client agent and a display. The client agent is configured to receive the compressed data format and to receive a display of the non-literal element using the received compressed data format. In another aspect, the device comprises a network.

Another aspect of the invention relates to a computer readable program medium for generating a graphical display on the client side. The medium includes a computer readable program medium for performing all the above-mentioned methods.

Another aspect of the invention relates to a method for generating a media presentation on the client side. The method includes sending output from an application program running on a server to the client, identifying a media stream within the application output, and displaying at least a portion of the media stream prior to processing by the application program. Intercepting and transmitting the original compressed data set to the client.

Another aspect of the invention relates to a method for generating a media presentation on the client side. The method includes transmitting output from the application program running on a server to the client, identifying a media stream within the application output, and intercepting a first decompressed data set that represents at least a portion of the media stream. And sending the compressed data set to the client on behalf of the first decompressed data set.

Another aspect of the invention relates to a method for generating a media presentation on the client side. The method includes providing information to a server about at least one media format supported by a client agent installed on the client side, receiving a compressed data set of the media stream on the client side, and generating a decompressed data set. Decompressing the compressed data set at a client side and generating a media presentation at the client side using the decompressed data set.

Another aspect of the invention relates to a computer readable program medium for generating a media stream on the client side. The medium may be a computer readable program medium for transmitting output from an application program running on a server to the client, a computer readable program medium for identifying a media stream in the application output, before the application program processes. A computer readable program medium for intercepting an original compressed data set representing at least a portion of a media stream and a computer readable program medium for transmitting the original compressed data set to the client.

Another aspect of the invention relates to a computer readable program medium for generating a media stream on the client side. The medium represents a computer readable program medium for transmitting output from an application program running on a server to the client, a computer readable program medium for identifying a media stream in the application output, and at least a portion of the media stream. A computer readable program medium for intercepting a first decompressed data set, the computer readable program medium for compressing the intercepted first decompressed data set and a first decompressed data set And a computer readable program medium for transmitting the compressed data set to the client.

Another aspect of the invention relates to a computer readable program medium for generating a media stream on the client side. The medium may be a computer readable program medium for providing a server with information in at least one media format supported by a client agent installed on the client, and a computer readable program medium for receiving compressed data of a media stream on a client side. A computer readable program medium for decompressing the compressed data set at the client side to produce the decompressed data set and a computer readable program for generating a media presentation at the client side using the decompressed data set. Possible program media.

In various embodiments of the last six aspects described herein, the method and computer readable program medium may comprise capturing timing information associated with a media stream, transmitting the timing information to a client, at the client side. Receiving a compressed data set and optionally timing information, decompressing the compressed data set at a client side to produce an uncompressed data set and using the decompressed data set and / or the timing information A step for generating a media presentation at the side. In another embodiment of the last six aspects of the invention, the method and computer readable program medium further comprise steps for sending non-media graphical information from the application output to the client. Receiving the non-media graphics information at the client side and generating a media presentation at the client side using the decompressed data set and non-media graphics information.

Another aspect of the invention relates to an apparatus for generating a media presentation on the client side. The apparatus includes an application program and an output filter module. The application program is configured to identify a media stream in the output produced by the application program. The output filter module is configured to intercept an original compressed data set representing at least a portion of a media stream and transmit the original compressed data set to a client before an application program processes it.

Another aspect of the invention relates to an apparatus for generating a media presentation on the client side. The apparatus includes an application program and an output filter module. The application program is configured to identify a media stream in the output produced by the application program. The output filter module is configured to intercept a first decompressed data set representing at least a portion of a media stream, compress the intercepted first decompressed data set of the media stream, and first decompress data. Send the compressed data set to the client on behalf of the set.

Another aspect of the invention relates to an apparatus for generating a media presentation on the client side. The apparatus includes a server and a client in communication with the server. The client includes a client agent configured to provide a server with information regarding at least one media format supported by the client agent, receive a compressed data set of the media stream, and generate a decompressed data set at the client side. Decompress the compressed data set and generate a media presentation using the decompressed data set.

In various embodiments of the last three aspects of the invention, the output filter module of the device is configured to capture timing information associated with the media stream and send the timing information to a client. In various embodiments of the last three aspects of the invention, the system is configured to receive the compressed data set and optional timing information, and decompresses the compressed data set to produce an uncompressed data set. And generate a media presentation using the decompressed data set and the optional timing information. In another embodiment of the last three aspects of the invention, the client agent receives non-media graphics information and uses the decompressed data set and the non-media graphics information to present a media presentation at the client side. It is configured to generate.

Another aspect of the invention relates to an apparatus for generating a media presentation on the client side. The apparatus includes a network, a server in communication with the network, and a client in communication with the network. The server includes an application program and at least one output filter module. The application program is configured to identify a media stream in the output produced by the application program. The output filter module is configured to intercept a compressed data set that represents at least a portion of a media stream before the application program processes and sends the compressed data set to a client. In some embodiments, the output filter module only intercepts a portion of the compressed data set that represents the media stream. For example, the output filter module may intercept every other frame of the video media stream one frame at a time. In another embodiment, the output filter module discards a portion of the data associated with every frame of video data to reduce the bandwidth required for transmission of the video stream. The client includes a client agent. The client agent is configured to provide the server with information about at least one media format supported by the client agent, the compressed data at the client side to receive the compressed data set and generate an uncompressed data set. Decompress the set and use the decompressed data set to create a media presentation on the client side.

Another aspect of the invention relates to a computer readable program medium for generating a media presentation on the client side. The medium includes a computer readable program medium for intercepting an original compressed data set of a media stream and a computer readable program medium for transferring the original compressed data set to a client using a new client protocol such as ICA or RDP. Include.

Another aspect of the invention relates to a computer readable program medium for generating a media presentation on the client side. The medium may comprise a computer readable program medium intercepting the decompressed data set of a media stream, a computer readable program medium for compressing the intercepted decompressed data set and the compressed data set with an ICA or RDP. Computer-readable program media for transmission to clients using the same scene client protocol.

The present invention relates to a method, an apparatus and a computer readable program medium for producing a graphical display. Compressed data format representations associated with bitmap representations of non-literal elements or images are transmitted from the server to the client over the network on behalf of the non-literal elements or bitmap representations for subsequent display.

1 shows a system for generating a display for a remote terminal session that includes a first computing system (client node) 105 in communication with a second computing system (server node 110) over a network 115. 100 is shown. For example, the network 115 may be a local area network (LAN), such as an intranet, or a wide area network (WAN), such as the Internet or the World Wide Web. The user of the client node 105 may be of various kinds, including standard telephone lines, LANs or WANs (eg, T1, T3, 56kb, X.25), broadband connections (ISDN, frame relay, ATM) and wireless connections. It may be connected to the network 115 via the connection means. The server node 110 includes a server transceiver 135 for establishing a communication relationship with the network 115. The connection can be established using various communication protocols such as ICA, RDP, HTTP, TCP / IP, IPX, SPX, NetBIOS, Ethernet, RS232, and direct asynchronous connections.

The server node 110 may be any computer device having the capability to supply the requested services of the client node 115. In particular, the capability includes generating and transmitting to the client node 105 commands and data indicative of the output produced by the application program 140 running on the server 110. The server node 110 includes the server transceiver 135, an application program 140 execution unit, a server agent 150, an output filter module 155, and an image storage unit 160. The server agent 150 includes a module that interfaces between the client agent 175 and other components of the server node 110 to support the operability of the remote display and application program 140. The server agent module 150 and all the above-mentioned modules may be implemented as a software program and / or a hardware device such as an ASIC or FPGA.

For clarity, all of the above components are shown on server node 110. It may be appreciated that the server node 110 may represent a single server or a plurality of servers in communication via the network 115 or other networks not shown. In a multiple server implementation, the functionality of the components may be distributed among the available servers. For example, in one embodiment of multiple servers, the transceiver 135, the application program 140, the server agent 150 and the output filter module 155 are in an application server and the image storage 160 is in a RAID system. On a storage device such as a disk.

The client node 105 may be used to provide a user interface to an application program 140 running on a server node 110, for example, a personal computer, set-top box, wireless mobile phone, handheld device, It can be any computer device such as personal digital equipment, kiosks. The client node 105 includes a client transceiver 130, a display 145, a client agent 175, and a graphics library (also called an image rendering library, 180). The client agent 175 includes a module for receiving commands and data from the server node 110 and the user of the client node 105 (not shown), which module may be a software program and / or an ASIC or FPGA. It is implemented in the form of a hardware device. The client agent 175 uses the received information when interacting with other components of the client node 105, for example when assigning an operating system to output data on the display 145. The client agent 175 also sends requests and data to the server node 110 in response to commands from the server or user actions at the client node 105.

The server node 110 hosts one or more application programs 140 that are accessed by the client node 105. Examples of such applications include word processing programs such as Microsoft Word® and spreadsheet programs such as Microsoft Excel®. Examples include financial reporting programs, customer registration programs, programs that provide technical support information, customer database applications, and application set managers. Another example of an application is Microsoft's Internet Explorer®, which will be used as the exemplary application program 140 in the following description. It will be appreciated that other application programs may be used.

During execution of an application program 140 such as Internet Explorer®, the server node 110 communicates with the client node 105 via a transport mechanism. In one embodiment, the transport mechanism provides multiple virtual channels 185 over the network 115 through which the server agent 150 can communicate with the client agent 175. One of the virtual channels 185 provides a protocol for transferring graphical screen data from server node 110 to client node 105. The server 110 executes a protocol driver generated by the application program 140 and intercepts graphic display interface commands directed to an operating system of the server, which corresponds to an embodiment as part of the server agent 150. do. The protocol driver converts the commands into a protocol packet suitable for transmission via a transport mechanism.

In an example such as Internet Explorer®, the application program 140 retrieves a web page on the server 110. As described, the application program 140 generates graphical display commands for the server operating system as for displaying output at the server node 110. The server agent 150 intercepts the commands and sends them to the client agent 175. The client agent 175 issues the same or similar commands to the client operating system to produce an output for the display 145 of the client node 105. If the server agent 150 or client agent 175 is a Windows® based operating system manufactured by Microsoft, the graphic display commands may be, for example, a DirectShow®, DirectX®, or Windows Graphics Device Interface (GDI) function. Will have a form.

In one embodiment, the web page includes a fused state of text elements such as titles, text, ASCII characters, and non-text elements such as images, photos, icons, and splash screens. The non-literal element is transmitted from a web server (not shown) to an application program 140 such as Internet Explorer® in the form of compressed data, such as a file or data stream, sometimes referred to as the prototype of the non-literal element. Examples of compression formats include JPEG, GIF, and PNG. The non-literal element can be represented, for example, in the form of compressed data of size 20 kilobytes. If the same non-literal element is decompressed into a bitmap representation, for example, it is 300 kilobytes in size.

When generating a display of a web page, the application program 140 retrieves, for example, the JPEG data format of the non-literal element and decompresses the JPEG data format into a bitmap for display. The output filter module 150 determines whether the bitmap representation is from a compression format, as described in detail below, and obtains a corresponding compression format of the non-letter element from the image storage 160. In one embodiment, the image storage unit 160 is a permanent storage device. In another embodiment, the image storage unit 160 is a temporary storage, a cache, a volatile memory and / or a combination of temporary storage and permanent storage.

The server agent 150 replaces the bitmap representation of the non-textual element with the compressed non-literal element that the output filter module (155) retrieved from the image store (160). . The server agent 150 sends the non-letter elements in the compressed format to the client node 105 along with the graphical display interface commands associated with the bitmap representation. In one embodiment, the associated command may apply to the bitmap representation of the non-letter element, but the server agent 150 uses a unique protocol command that identifies the transmission of the non-letter element rather than the bitmap representation. . In other embodiments, other identification techniques may be used. For example, a protocol command may have a modifier comment or command switch. In addition, the command may use a change of context or a combination of a plurality of commands.

When a non-literal element is stored in an uncompressed format (e.g., Windows® bitmap (bit.) Format), to provide a compressed version of the non-literal element, the server agent 150 provides a non-literal element. It is possible to search for the decompression mark of and compress it, for example using 2DRLE compression. Optionally, agent 150 may also review the data in the decompression indication to select a compression algorithm suitable for application to the decompressed indication, apply the selected algorithm to the decompression indication, and display the client agent ( Provide a compression indication. If the non-literal element is compressed intact, the server agent 150 may choose to recompress the image using the same compression technique or another compression technique.

Algorithms applicable to server agent 150 for the compression of decompressed indications include lossless compression algorithms and lossy compression algorithms. For example, lossless compression algorithms, such as 2DRLE compression, reduce the size of the decompressed representation without loss of information contained within the representation. For example, lossy compression algorithms, such as JPEG or JPEG2000 compression, reduce the size of the decompressed representation in such a way that information contained in the decompressed representation is discarded.

The server agent 150 selects a compression algorithm suitable for compressing the decompressed representation using the content—at least a portion of the decompressed representation. In one embodiment, the server agent 150 determines whether the decompressed indication is a continuous tone image, such as a photographic image, and applies a lossy compression algorithm to the decompressed indication. In another embodiment, server agent 150 determines whether the decompressed representation includes a large area of the same color (eg, a computer-generated image) and applies a lossless compression algorithm. In another embodiment, the number of pixels included in the pixels of the decompressed display is counted, and the number of counted colors exceeds a predetermined threshold value (eg, 256 colors). In that case, a lossy compression algorithm is applied to the decompressed indication.

In another embodiment, the server agent 150 compresses the decompressed display using a lossless compression algorithm, and compares the size of the compressed result with a predetermined value. If the size of the compressed result exceeds a predetermined value, the decompressed representation is compressed using a lossy compression algorithm. The lossy compression algorithm is selected if the size of the lossy compression result is less than a predetermined percentage of the size of the lossless compression result.

The client agent 175 receives the transfer of the non-literal element file in a compressed data format, with graphical display interface commands associated with the bitmap representation of the non-literal element. The client agent 175 determines that the non-character element is not a bitmap representation in the compressed data format. In one embodiment, the client agent 175 determines this because the non-letter elements in the compressed format are sent using a single protocol command. In other embodiments, the client agent 175 may fully determine other characteristics for the non-letter element, including the size of the non-letter element data and / or associated graphical display interface commands.

The client agent 175 determines whether the client node 105 includes a library (180) necessary for decompressing the compressed format of the non-letter elements. If the client node 105 has the appropriate graphics library (s) 180 installed to execute the decompression algorithm, the client agent 175 uses the library 180 to bit the compressed format of the non-literal element. Uncompress the map display. The client agent 175 executes the associated graphical display interface commands received for the bitmap representation to generate non-literal elements for the output of the application program 140 on the client display 145.

In one embodiment, the client agent 175 does not include any decompression algorithms to decompress non-element elements from the compressed format into bitmap representations. If the client node 105 does not have an installed graphics library 180 suitable for executing the decompression algorithm, the client agent 175 requests the server node 110 for the required graphics library. In another embodiment, the client agent 175 determines whether a predetermined set of the most widely used graphics library is installed on the client node 105 before receiving any non-elemental elements at the server node 110. If the most widely used graphics library 180 is not installed on the client node 105, the client agent 175 may receive the server node 110 before receiving any non-literal elements from the server node 110. Request a missing library.

In another embodiment, the client agent 175 determines whether the graphical library of the client node 105 includes the library information and transmits it to the server agent 150. In this embodiment, when the server agent 150 receives the compressed data of the bitmap representation from the output filter module 155, the server agent 150 uses the transmitted library information to generate the client node ( It is determined whether 105 can provide a compressed data format. If the server agent 150 determines that the client node 105 has an applicable library, the server agent 150 substitutes the compressed data format for the bitmap representation of the non-literal element. If the server agent 150 determines that the client node 105 is not an applicable library, the server agent 150 does not replace the compressed data format for the bitmap representation of the non-literal element, but instead returns the bitmap representation to the client ( 105).

In order to replace the bitmap representation of the non-literal element in the server agent 150 with the non-literal element in the compressed format, the output filter module 155 uses the compressed format in which the bitmap representation is contained in the image storage 160. Determine if it occurred. The output filter module 155 determines whether the source of the media stream is exposed by any way by the document (e.g., an application program or file type identifier configured to expose this information). can do. In an embodiment where the media stream is a multimedia stream, the output filter module 155 may be provided with information about the source application associated with the media stream by the multimedia subsystem. In another embodiment, the output filter module 155 calculates one or more check values for the bitmap representation. For example, output filter module 155 may compute one check value for the entire bitmap representation and / or output filter module 155 may check four checks in one of each quadrant for the entire bitmap representation. Can compute a value. In another embodiment, the output filter module 155 may compute N check values on one of the N respective lines in the bitmap representation. The check value is the result of an algorithm that produces a substantially unique value for another array of data. The check value is, for example, the result of a chetag tag, a cyclic redundary code ("CRC"), a check sum, or a hashing function. The check value is based on the bitmap representation and not the data arranged in the compressed data format. However, when the compressed data format is stored in the image storage 160, it is stored as a check value attribute corresponding to one or more check values for the bitmap representation of the compressed data when decompressed.

In one embodiment, the check value is a check tag that includes a fixed identifier and a single identifier. Fixed identifiers and unique identifiers are combined together and concealed within the image. Just as a single identifier is used to identify a particular image, a stator identifier is used to use a check tag. A fixed identifier is, for example, a globally unique identifier that is not statistically likely to be found in the image. For example, a fixed identifier is a byte sequence with a large number of zero and one bits that are easily recognizable and balanced during debugging. A single identifier is a uniquely assigned sequential identifier for each image in the cache. Since the sequential single identifier is a masked XOR with another value, an image identifier with a small value (the most likely value) may have an average number of 0 and 1 bits.

Depending on whether the RGB color component is part of an image or part of a color palette, the check tag is encoded as an RGB color component. Specifically, the check tag is processed into 160 bit streams (ie, 20 individual bytes, the least significant bit starting at 0 bits and the most significant bit ending at 7 bits, respectively). The least significant bit of each byte is overwritten with the next bit of the check tag. The other seven bits of each remaining byte remain unchanged.

The check tag is decoded by simply reversing the encoding procedure. After the check tag is decoded, the fixed identifier and the single identifier are searched for in the check tag. The retrieved fixed identifier is checked against a previously stored fixed identifier, such as identifying a check tag. After a match is found, a single identifier is used to search for information associated with the identified image, such as bitmap data associated with the image.

The output filter module 155 searches the image storage 160 for non-literal elements in the compressed data format having the same check value attribute as one or more check values, and the output filter module 155 calculates the bitmap representation. do. The output filter module 155 searches for the compressed format of the non-literal element with the same check value attribute, such as one or more check values, to convert the compressed format of the non-literal element for transmission to the client agent 175 instead of displaying the bitmap. Send to server agent 150.

After the server node 110 stores the compressed format of the non-literal element in the image storage 160, the application program 140 executes a graphics library (not shown) to create a bitmap representation from the compressed format file. Call). As the output filter module 155 computes the check value associated with the bitmap representation, the application program 140 decompresses the compressed format and generates a bitmap representation. As described above, when the bitmap display is complete, when the quadrant of the bitmap display is complete, or when the line of the bitmap display is complete, the output filter module 155 may calculate the check value. The server 110 stores the check value attribute associated with the compressed format file in the image storage 160 and retrieves the compressed format file, and then anytime next, the application program 140 generates the associated non-literal element. .

Server 110 may be a permanent storage that depends at least in part on image storage 160 in temporary storage (e.g., a RAM memory buffer or cache) or design and hardware limitations (e.g., size of persistent storage). The image storage 160 (eg, disk or non-volatile memory buffer) anywhere stores the compressed format file and its associated check value attribute. One preferred criterion used to determine the number of application programs creates non-literal elements. For example, when the application program 140 generates more than a predetermined number of specific non-letter elements, the server 110 may store a compressed format file and check value attributes corresponding to the specific non-letter elements. Permanently stored in. Image storage 160 may apply an eviction algorithm (eg, a least-recently-used) algorithm to remove data from image storage 160. Can be. Ejection allows server agent 150 and client agent 175 to observe the resource or storage limits imposed on image store 160.

In another embodiment, the server 110 stores non-literal elements if the server 110 is static or complex. For example, if the application program 140 always creates a splash screen at an early stage, the server 110 stores the compressed format file corresponding to the splash screen in the permanent area of the image storage 160. do. In other embodiments, if a non-literal element is created complex, static or / and iteratively, it does not have a corresponding compressed format file, and the output filter module 155 may be in a standards-based or owner-based format. Within) creates a compressed format file for the non-literal elements. In any subsequent transfer, server agent 150 sends the generated compressed format file instead of the bitmap representation. If the compression format is an owner-based format, server agent 150 determines whether client mode 105 includes an owner-based graphics library that is applicable to decompress the compressed format file into a bitmap representation. If this is not included in the client node 105, the server agent 150 sends a library applicable to the client node 105 for installation.

Although the illustrated embodiment shows an image storage unit 160 on the server node 110, in other embodiments at least some image storage unit (not shown) is on the client node 1050. In other embodiments, The output filter module 155 computes one or more check values of the bitmap representation and sends one or more check values to the server agent 150. The server agent 150 sends one or more check values to the client agent 175. The client agent 175 retrieves a portion of the image store on the client node 105 for the compressed data format stored with the same one or more check value attributes. Send search results Server-side and client-side image stores can persist after the session between client and server is terminated Subsequently, when a new session is initiated, the client agent 175 communicates information about the contents of the image store 160 with the server agent 150 and compares it with the image store 160 of the server. The server agent 150 may use non-literal elements in the image store—or their strips, described below—to reduce the amount of bandwidth required to generate graphics and media displays at the client.

If a compressed data format for a non-literal element is present on the client node 105, the server agent 150 does not transmit the compressed data format or bitmap representation over the network 115. Server agent 150 sends only graphical display interface commands associated with the bitmap representation of the non-literal element. If the compressed data format for the non-literal element is not present on the client node 105, the output filter module 155 obtains the corresponding compressed format of the non-literal element from the image storage 160. The server agent 150 replaces the bitmap representation of the non-letter elements with the non-letter elements in the compressed data format that the output filter module 155 has searched for in the image storage 160. The server agent 150 sends the non-letter elements in the compressed format to the client node 105 along with the graphical display interface commands associated with the bitmap representation.

As described above, the bandwidth required to generate non-literal elements in the image storage 160 on the server 110 and / or client node 105 to produce graphics and media displays on the client. Reduce According to the invention, this non-literal element can be subdivided into sub-regions, ie strips, " strips ", and image storage 160 can be strip level granularity for non-letter element caching as described above. granularity). For example, a web page may be composed of multiple images, in whole or in part, that are only visible to a portion of the web page. For an image that is only partially shown, it is possible to send the entire image to a client that includes an opaque portion in another window. However, because bandwidth consumption increases, other data is transmitted more than necessary. Some striking images are identified as "strips" and proceed as described below. Image storage 160 may include strips, optionally strip-associated metadata such as image strip heights, image strip widths, and image strip identifiers (eg, cyclic redundancy check checksum). It includes graphic data to form.

In this embodiment, each non-literal element used within the graphic rendering operation is divided into the aforementioned strips. In one embodiment, these strips have the same width as the non-letter elements and the height less than or equal to the height of the non-letter elements. When the client agent 175 is required to display a non-literal element consisting of one or more strips that do not appear in the image store 16, the required strips are stored in the client and client-side image store 160 'for rendering. Sent to storage.

Server agent 150 maintains a database of strips that are previously sent to clients for rendering. If a subsequent rendering operation provides an area contained within the stored strip, the previously stored strip is identified in server-side image store 160 and a suitable identifier associated with the stored strip is retrieved. The message pointing to the client agent 175 to provide the stored strip associated with the identifier is sent to the client agent 175 and requires less bandwidth than the message containing the strip's own content. The client agent 175 searches for the stored strip copy associated with the identifier sent from the image store 160 'and displays the contents of the stored strip. Similarly, when individual strips are shared between multiple non-literal elements, server agent 150 identifies common strips and instructs the client's own image store 160 to display the strips for display. Avoid retransmission of the strip.

If the strip is modified between successive rendering operations, for example, when the source bitmap in the previous rendering operation becomes the destination bitmap in the subsequent rendering operation, the server agent 10 may detect the modification to the strip and Resend the strip for client agent 175 for rendering and / or the strip for combining in client-side image store 160 '.

FIG. 2 illustrates a preferred embodiment process 200 for generating a display for a remote terminal session using the preferred embodiment of FIG. 1. The output filter module 155 monitors the output of the application program 140 by monitoring calls made by the operating system of the server node 110. When the output filter module 155 detects a display command from the application program (step 205), the output filter module 155 determines whether the application program 140 generates a bitmap representation of the non-literal element (step 210). .

If the application program 140 does not produce a bitmap representation of the non-literal element, the output filter module 155 sends a display command to the server agent 150, where the server agent sends the command (step 215). Or a representation command defined in the protocol is stored in the client agent 175. When application program 140 generates a bitmap representation of the non-literal element, output filter module 155 calculates one or more check values corresponding to the bitmap representation of the non-literal image (step 220).

Using one or more computed check values, output filter module 155 searches the image store to determine if there is a compressed data format having the same check value attribute (step 225). If there is a compressed data format in the image store 160 that has the same check value attribute as the one or more check values computed by the output filter module 155, then the output filter module 155 displays a bitmap representation of the non-letter elements. Replace with a compressed data format (step 230). The output filter module 155 sends the compressed data format to the server agent 150 for final transmission to the client agent 175 (step 230). The output filter module 155 also sends all commands associated with the replaced bitmap indication with the compressed data format.

If there is no compressed data format having the same one or more check value attributes in the image storage unit 160, the output filter module 155 outputs the bitmap representation of the non-letter element corresponding to the compressed data format to the permanent storage. It is determined whether it meets a predetermined criterion (e.g., any criteria described above) (step 235). If the output filter module 155 determines that it meets a predetermined criterion, the output module 155 permanently stores the check value attribute corresponding to the compressed data format and the calculated one or more check value attributes. Store to storage (step 240). If it is determined that the output filter module 155 does not meet a predetermined criterion, the output filter module 155 outputs the corresponding check value attribute equal to the compressed data format and one or more check values calculated. Store in a temporary storage (step 245).

When the output filter module 155 stores in the image storage 160 a corresponding check value attribute equal to the compressed data format and the computed one or more check values, the output filter module 155 displays a bitmap of non-letter elements. Is replaced with the compressed data format (step 230). The output filter module 155 ultimately sends the compressed data format to the server agent 150 for transmission to the client agent 175. The output filter module 155 continuously monitors the output generated by the application program until the output filter module 155 detects another display command from the application program 140 (step 205).

In another aspect, the present invention relates to a method and system for producing a media presentation and to a production thereof. In one embodiment, the compressed data set representing at least a portion of the media stream is intercepted in the first computing device and sent to a second computing device that is decompressed and presented to the user on the network. In another embodiment, the decompressed data set, which represents at least a portion of the media stream, is intercepted and compressed on the first computing device as described above and transmitted to a second computer device that is decompressed and presented to the user.

3 illustrates one embodiment of a system 300 for generating a media presentation in accordance with an aspect of the present invention. The system 300 includes a first computing device (eg, server 310) in communication with a second computing device (eg, client: 305) over a network 315. In general, except for the foregoing, the client 305, server 310, and network 315 have the same characteristics as the client 105, server 110, and network 105 described above.

As shown, the client 305 includes a client agent 375, and a presentation interface 345. The client agent 375 may be implemented with a software program and / or a hardware device (eg, ASIC or FPGA, etc.). Client agent 375 generates a presentation with media and non-media components at presentation interface 345 using client transceiver 330 to communicate over network 315.

In one embodiment, server 310 is an application server. As shown, server 310 includes at least a server transceiver 335, an application program 340, a server agent 350, a first output filter module 355A, and a second output filter module 355b. . The server agent 350, the first output filter module 355A, and the second output filter module 355b may be implemented in software programs and / or hardware devices (eg, ASICs or FPGAs). The server agent 350, the first output filter module 355A, and the second output filter module 355b use the server transceiver 335 to communicate with the network 315.

In other embodiments, the aforementioned components 335, 340, 350, 355A, and 355B are distributed and / or redundant through some servers communicating with each other via a network 315 or other network (not shown). (duplicated). For example, this allows for the transmission of media related characteristic information, media streams, and control information over multiple hops and control information for multiple clients. Alternatively, in another embodiment two or more of the aforementioned components 335, 350, 355A, 355B may be combined into one component, and as described below, two or more components 335, 350, 355A, 355B The function performed by) is performed by a single component.

In another embodiment, the aforementioned components 340, 355A, 355B are duplicated in the same server 310. For example, this allows simultaneous transmission of different / multiple applications, and media-related characteristic information, media streams, and control information with respect to other / multiple media streams within each application. The other application 340 may be identified as a major stream context, and other media streams may be identified as minor contexts within a particular major context.

The application program 340 shown in FIG. 3 is any application program 340 that represents a media stream as part of its output. The media stream can be a video stream, an audio stream, or alternatively any combination of multiple instances. Moreover, the application program 340 can output non-media graphics information. In this context, non-media graphics information generally refers to a codec or equivalent thereof (eg, information including, but not limited to, toolbars and drop-down menus for locating static graphics information on a display screen). All graphic information output by the application program 340 without use. The non-media graphics information also includes information for locating the static graphics information on the display screen, for example. The application program 340 may be, for example, a MICROSOFT ENCARTA application program manufactured by Microsoft Corporation of Redmond, Washington, USA.

In one embodiment, the application program uses an external codec, such as, for example, a codec installed in the operating system of the server 310, to decompress a compressed data set that represents at least a portion of the media stream. In another embodiment, the codec used in the application program 340 is embedded in its own application program. The server 310 may include a plurality of execution application programs 340, some application programs 340 use an external codec, and some application programs 340 use an embedded codec.

If the application program 340 uses an external codec and wants to output the media stream, the operating system of the server 310 may attach the external to decompress the compressed data set representing at least one region of the media stream for subsequent display. Ask to use the codec. If the codec used by the application program 34 is embedded in the application 340 itself, the application program 340 is embedded to decompress the compressed data set itself for subsequent displays when it wishes to output a media stream. The codec used. In addition, the application program 340 may generate and send a graphic display command associated with the non-media graphics information to the operating system of the server 310. If the operating system against server agent 150 or client agent 175 is a type of Windows® system of operating system manufactured by Microsoft Corporation of Redmond, Washington, USA, this graphical display command is, for example, This can be in the form of calls to DirectShow®, DirectX®, or GDI functionality.

According to the present invention, application program 340 performs these tasks as if application program 340 had generated a presentation with media and non-media components at server 310. As described below, the first output filter module 355A, the second output filter module 355B, and the server agent 350 pass through the network 315 to a compressed data set, embedded code, which is passed to an external code. Intercepts the decompressed data generated by the < RTI ID = 0.0 > and < / RTI > non-media graphics information, respectively, and sends them (first compressing the decompressed data set generated by the embedded code). The client agent 375 decompresses the received compressed data set according to the following, and then generates a presentation with media and non-media components in the presentation interface 345 of the client 305. Issue the same or similar graphical display command associated with the media graphical information to the operating system of the client 305. In embodiments, for example, in the DirecrShow® embodiment described below, the communication between server agent 350 and client agent 375 may be media-related (eg, buffer and media type performance, media stream priority, etc.). ) Or control information (e.g., play, pause, flush, end-of-stream, and stop commands, parent window assignment, window) Position and clipping commands, scrolling, audio encouragement commands, volume, balancer, etc.).

When an application program 340 using an external codec attempts to call an external codec to output a media stream, the first output filter module 355A and the second output filter module 355B are called. The first output filter module 355A intercepts the original compressed data set that represents at least a portion of the media stream. Unlike the external codec restoring the data set, the first output filter module 355A sends the original compressed data set to the client agent 375 via the network 315. The second output filter module 355B is an OS-level renderer that captures stream-independent (typical) control commands (e.g., play, pause, stop, flush, end of stream). acts as a renderer and transmits this information to the client agent 375 via the network 315. If the media stream comprises a video stream, the second output filter module 355B may also provide information for determining the image of the video stream on the display screen (eg, parent window assignment, placement and clipping). Command) and transmit this information to the client agent 355B via the network 315. If the media stream comprises an audio stream, the second output filter module 355B also captures information about audio adjustment (eg, volume and balance adjustment commands) and sends this information over the network 315 to the client agent. Send to 375.

In another embodiment, when the application program 340 using the built-in codec attempts to call an OS-level renderer to output a media stream, for example, the second output filter module 355B is called. do. The second output filter module 355B intercepts the first reconstruction data set that represents at least a portion of the media stream from the output of the application program 340. The second output filter module 355B then compresses the intercepted first reconstructed data set and transmits the compressed data set to the client agent 375 via the network 315, as described below. If the media stream comprises a video stream, the second output filter module 355B also captures information for determining the image of the video stream on the display screen and sends this information to the client agent 375 via the network 315. send.

When the operating system for the server agent 350 or the client agent 375 is one of the Windows® family manufactured by Microsoft Corporation of Redmond, Washington, the server agent 350 may be configured for other filters installed on the server 310. You can also implement this interception feature by providing a DirectShow® transform filter that surpasses the filter merit level. As is known, the WINDOWS family of operating systems allows for the installation of multiple filters that can handle certain types of media encodings. The merit level assigned to the filter determines the priority given to the filter; That is, a filter with a high merit level is selected to process the media type ahead of a filter with a low merit level. Support for various media types (eg, video, audio, MIDI, text, etc.) can be implemented by combining conversion filters with the desired media type through system registry changes. In addition, through a registry change, the transform filter may be combined with a particular renderer filter. Obviously, this configuration is extensible and also allows to support new types of media programming.

In a Windows® environment, the server agent 350 can configure the communication functionality with the client using the modified DirectShow® transformation and renderer filter. Instead of converting (restore) the received media data, the modified transform filter sends the data to the client agent 375. Instead of rendering the converted (decompressed) media data, the renderer filter is stream-general (eg, play, pause, stop, flush, end-of-stream) and stream-specific (eg, video original). Windows Assign, Place and Clip, Audio Volume and Balance) commands. If the client 305 uses a Windows® operating system, then the modified DirectShow® source filter at the client 305 may operate to receive the transmitted media and stream-general control data. Finally, the unchanged transform and render filter at the client 305 transforms (restores) and renders the media data for viewing and listening by the user. The renderer filter also receives stream-specific control data.

In one embodiment, server agent 350 maintains a separate media queue for each media stream. When the network bandwidth between the server agent 350 and the client agent 375 is insufficient, each media stream may be prioritized, ie while the transmission of still images from the web browser is obviously slowed down or stopped, Transmission of video conferencing information is allowed. Stream prioritization may be performed by providing separate queues for each stream, different type of media having an assigned priority. The network bandwidth may be determined by known methods such as "pinging" to measure the expected target server and response timing. In an embodiment in which the media stream includes embedded timing information, this information may be used to determine whether the media data is not being transmitted fast enough, i.e., whether the bandwidth of the channel cannot support transmission.

In another embodiment, media queues corresponding to the same application program 340 may have different priorities based on the media type (video, audio, MIDI, text, etc.). If the network bandwidth between server agent 350 and client agent 375 is insufficient to accommodate all media streams, samples may be dropped from low priority queues (ie, high priority queues can be serviced without interruption). , Can be deleted). For example, video transmissions are interrupted and appear as "slide-mode" while audio is not interrupted. In another embodiment, the control information is associated with its own control queue, allowing the priority of transmission of control data before any media data.

In this embodiment, the transfer of control information from server agent 350 to client agent 375 allows control of the scheduler of the client rendering filter. Latency in initial playback can be reduced by pre-filling the media queue (not shown) of client agent 375 as soon as possible before playback begins. Similarly, subsequent fluctuations in network latency can be handled by readjusting the media queue of client agents 375 that need to. The client agent 375 detects that one or more of its own media queues falls down or exceeds any resource threshold, and each burst corresponding to the server agent 350 for the desired amount of media queue adjustment. Or send a stop request.

The use of DirectShow® features in the Windows® environment affects the functionality provided by the operating system and reduces or eliminates the reimplementation of redundant features. For example, the graphic or media display provided by the client agent 375 affects the operating system functionality to provide complex clipping and audio control functionality. In addition, the client agent 375 can use the DirectDraw® and DirectSound® capabilities provided by the operating system, reducing the required CPU resources and improving overall performance.

In yet another embodiment, server agent 350 intercepts and sends graphical display commands associated with non-media graphics information that is output from application program 340 to client agent 375 via network 315.

Also, in one embodiment, the first output filter module 355A, the second output filter module 355B, or both (when the application program 340 uses an external codec), or the second output filter module 355B. (When application program 340 uses an embedded codec), captures timing information associated with the media stream and sends this timing information over network 315 to client agent 375. More specifically, the output filter module 355A, 355B captures the presentation timing for each frame of the media stream and sends it to the client agent 375, whereby the client agent 375 makes a video and audio stream. Can be synchronized and a correct frame rate can be maintained.

As shown, server agent 350 is coupled to server transceiver 335 and application program 340. In one embodiment, as described below, server agent 350 receives a list of media formats supported by client agent 375 from client agent 375 via network 315. Upon receipt of the list of supported media formats, server agent 350 registers output filter modules 355A, 355B by adjusting the configuration of server 310. In one embodiment, for example, server agent 350 registers output filter modules 355A, 355B by editing the registry of server 310. Server agent 350 then informs client agent 375 that server 310 can handle all of these media formats. In another embodiment, client agent 375 and server agent 350 negotiate supported formats and capabilities. For example, client agent 375 may postpone notifying server agent 350 of support of JPEG2000 format media until server agent 35 explicitly requests that client agent 375 generate a JPEG2000 display. Can be.

In another embodiment, when the application program 340 loads the first output filter module 355A while attempting to render the media stream, the filter module 355A may assign, media type, media. Requests are in turn sent to the client agent 375 to communicate with the server agent 350 to generate a media stream in providing media stream characteristics such as stream priority. The client agent 375 responds to the server agent 350's request to create a media stream by indicating the success or failure of the request. The success or failure of the request depends on the validity of each filter module capability, such as media stream reconstruction capability, sufficient memory, and so forth at the client 305. When successful, the application program 340 loads the second output filter module 355B and connects it to the first output filter module 355A, thereby providing sufficient streaming of the compressed media. If unsuccessful, the application program 340 uses Microsoft's own or third party-supplied output filter module to restore and render the stream on the server 310 without loading the first output filter module 355A. Load. For example, when the operating system originating at the client 305 is one of the MICROSOFT WINDOWS families that implement DIRECTX and DIRECTSHOW technology, the first output filter module 355A and the second output filter module 355B are server agent ( Communicates directly with 350).

At client 305, client agent 375 is coupled with client transceiver 330 and presentation interface 345. As described below, the client agent 375 initially informs the server agent 350 of the media format supported by the client agent. In addition, the client agent 375 receives any associated timing information with the compressed data set from the output filter module 355B via the network 315. In addition, the client agent 375 may be configured to stream non-media (eg, play, Receive pause, stop, flush, end-of-stream) and stream-specific (eg, video original window assignment, placement and clipping, audio volume and balance adjustment) commands.

The client agent 375, using an external or embedded codec, restores the compressed data set together with graphical display commands associated with non-media graphics information, any information to determine the image of the video stream on the display screen, and any timing information. And generate a media presentation in the presentation interface 345. In one embodiment, the presentation interface 345 has a display screen that renders a graphical display, such as a video presentation. In another embodiment, the presentation interface 345 includes a speaker that enables audio presentation. Client 305 may include any number of presentation interfaces 345.

Information provided within the media format supported by the client agent 375 may determine the mode of operation at the server 310. If the compressed data set is in a format not supported by the client agent 375, the second output filter module 355B may recompress the decompressed data set in a supported format. In another embodiment, when the client agent 375 does not support the stream format, the application program 340 does not load the first output filter module 355A and does not load the stream from the server 310. Load a native or third common-supported output filter module.

4A, 4B, and 4C, one embodiment of a method 400 for generating a media presentation at a client 305 using the preferred embodiment of FIG. 3 is described. When both client 305 and server 310 are connected to network 315, client agent 375 informs server agent 350 of all media formats supported by client agent 375 (step 404). In one embodiment, the list of supported MIDI formats may be formed by enumerating external codecs installed on the client 305. For example, codecs installed within the operating system of client 305 are enumerated by client agent 375 to server agent 350 over a network. In yet another embodiment, the list of supported media formats may be formed by enumerating codecs embedded within the client agent 375. For example, if the client agent 375 is implemented as a software program, the codecs embedded within the software program are enumerated by the client agent 375 via the network 315 to the server agent 350. Optionally, the client agent 375 forms a list of supported media formats and lists the external codecs installed on the client 305 and the codecs embedded in the client agent 375 to list these supported media formats as server agents. Notify at 350. In another embodiment, client agent 375 and server agent 350 negotiate supported formats and capabilities. For example, client agent 375 may postpone notifying server agent 350 of support of JPEG2000 format media until server agent 35 explicitly requests that client agent 375 generate a JPEG2000 display. Can be.

In another embodiment, the stream is negotiated on a per-stream basis when it is formed. The client agent 375 responds to the server agent 350's request for media presentation formation by indicating the success or failure of the request. The success or failure of the request depends on the validity of each codec capability of the media stream reconstruction capability, sufficient memory, etc., at the client 305. If successful, the application program 340 loads the codec, thereby providing an efficient display of the graphic data. If unsuccessful, the application program 340 loads a Microsoft-embedded or third common-support codec that restores and renders data on the server 310.

In one embodiment, the client agent 375 generates globally unique identifiers ("GUID") and associates each GUID with a particular codec. The client agent 375 then sends a list of GUIDs generated to the server agent 350 to inform the server agent of the media format supported by the client agent 375. In another embodiment, the client agent 375 may list the four property codes (each of the four property codes) in the server agent 350 to inform the server agent 350 of the media format supported by the client agent 375. Is associated with a particular codec).

After receiving the list of supported media formats from the client agent 375, the server agent 350 supports the first output filter module 355A and / or the second output filter module 355B on the server 310. Register in association with the media format (step 408). The server agent 350 then reports back to the client agent 375 that the server 310 can process all listed media formats (step 412).

In step 416, the application program 340 starts running on the server 310. When application program 340 identifies in its own output the presence of media content, such as a media stream, first output filter module 355A, second output filter module 355B, or both are invoked (step 420). In step 424, if the application program 340 uses an external codec, as the application program 340 attempts to call the external codec, the first output filter module 355A and the second output filter module 355B It is called (step 428). In step 432, the first output filter module 355A intercepts the original compressed data set that represents at least a portion of the media stream, and in step 436 the original compressed data set in the client agent 375 without restoring the data set. Send it. In step 440, the client agent 375 receives the original compressed data set, and in step 444, restores the original compressed data set to generate the decompressed data set. The client agent 375 uses an external codec installed on the client 305 or a codec embedded in the client agent 375 itself to restore the original compressed data set.

In step 424, if the application program 340 uses a codec embedded in itself, in step 448, an attempt is made to call the OS-level renderer to display the decompressed data set, thereby displaying the second output filter module ( 355B) is called. Next, in step 452, the second output filter module 355B intercepts the first decompressed data set representing at least a portion of the media stream from the output of the application program 340, and in step 456, the intercepted first 1 Compress the decompressed data set. Various compression techniques include both lossy and lossless compression techniques, and in step 456, may be used by the second output filter module 355B to compress the media stream.

If the media stream is a video stream, in step 456, the intercepted first decompressed data set is passed to the second output filter module 355B using a light weight lossy video encoding algorithm, such as MJPEG compression. Can be compressed. In the use of a small lossy video encoding algorithm, the second output filter module 355B may select the desired compression ratio and may use a predetermined compression ratio. The degree of quality loss selected by the second output filter module 355B will generally depend on the possible bandwidth of the network connection. For example, if the user of the client 305 uses a slow modem to connect to the network 315, the second output filter module 355B may choose to use low quality video. On the other hand, if the user of the client 305 uses a LAN link or broadband to connect to the network 315, the second output filter module 355B may choose to use high quality video.

Following compression of the first decompressed data set intercepted in step 456, in step 460, the second output filter module 355B sends the compressed data set to the client agent 375 instead of the first decompressed data set. send. In step 464, the client agent 375 receives the compressed data set, and in step 468 restores the compressed data set to generate a second decompressed data set. Again, the client agent 375 uses an external codec installed on the client 305 or a codec embedded in the client agent itself to recover the compressed data set.

Regardless of whether the application program 340 uses an external or embedded codec, in step 470, if the media stream comprises a video stream, the second output filter module 355B may then, in step 472, view the video stream on the display screen. It captures information to determine the image and sends the captured information to the client agent 375 via the network 315. In step 474, the client agent 375 receives information for determining an image of the video stream on the display screen.

Again, regardless of whether the application program 340 uses an external or embedded codec, and whether or not the media stream includes the video stream in step 470, the server agent 350 in step 476 executes the application program 340. Intercepts graphic display commands associated with the non-media graphics information outputted by < RTI ID = 0.0 >) and sends < / RTI > At step 480, the client agent 375 receives a graphical display command associated with the non-media graphics information.

In step 484, if the output filter module 355A, 355B captures timing information associated with the media stream, in step 488 the output filter module 355A, 355B sends the timing information to the client agent 375. In step 492, the client agent 375 receives the timing information and in step 496 generates a media presentation in the presentation interface 345. To generate a media presentation at the presentation interface 345, the client agent 375 uses graphics display commands associated with timing information, non-media graphics information, and if the media stream includes a video stream, Video stream on the display screen to consistently combine the decompressed data set (or, more specifically, the second decompressed data set when the application program 340 uses the embedded codec) with the non-media graphics information. Use the information to determine the image of.

If the output filter modules 355A, 355B do not capture the timing information associated with the media stream, then in step 496 the client agent 375 may only extract the decompressed data set (or, more specifically, the application program 340). Second decompressed data set when using the embedded codec), graphical display commands associated with non-media graphics information, and information for determining an image of the video stream on the display screen if the media stream comprises a video stream. To generate a media presentation at the presentation interface 345.

5A-5C illustrate yet another embodiment of the present invention implemented on a server using an operating system selected from the operating systems of the MICROSOFT WINDOWS family. The operating system on the client does not have to be the same operating system adopted on the server.

For the main media types supported (e.g., video, audio, MIDI, text), the appropriate first output filter module 355A and second output filter module 355B are registered with the server 310 (step 504). ). As noted above, the media types identified are illustrative only, and with this architecture, future media types can be added to operations consistent with the present invention.

The application program 340 runs on the server (step 508) and the media stream is identified within the output of the application program 340 (step 512). In one embodiment, the media stream is identified when an application makes a system call to the WINDOWS Media Subsystem to determine the appropriate codec to process the media stream. The application program 340 corresponds to the primary media type of the media stream identified in the application output, and has a first output filter module 355A having any merit of high merit registered in the system capable of processing the primary media type. (Step 516). The first output filter module 355A is configured to transmit the transmitted characteristics (assignment (buffer) characteristics, media type (major: video, audio, etc .; sub: MPEG-1, MPEG-2, etc .; format, etc.), on- Communicate with server agent 350 to request that client agent 375 create a media stream specified by media stream priority for demand quality control, etc.).

For each terminal service session, server agent 350 configures the media stream in a different context. Media streams originating from different instances of the application program have different main contexts. Media streams originating from the same instance of an application program have different subcontexts within the same main context. Server agent 350 creates a command queue for control information for each primary context and a media sample queue for each subcontext. The media sample queue helps the server agent 350 to accommodate changes in bandwidth and network latency (e.g., video that is dropped to select data from various streams to drop and to maintain playback speed). Allow certain frames of data). For example, video data may be removed before audio data is removed.

Queues in other major contexts are serviced in a round-robin fashion. Within the main context, control queues generally have the highest priority, while media queues corresponding to different subcontexts may have different priorities based on the characteristics of the stream. In general, the audio stream will have a higher priority than the video stream.

The client agent 375 attempts to create a media stream that matches the media characteristics specified in the transmitted request (step 520). When the operating system of the client agent 374 is an operating system of the MICROSOFT WINDOWS family, the client agent 375 provides a unique or third public offering that has the ability to attempt to load a generic source filter module and to restore and render the media stream. Connect to the output filter module. Client agents 375 using different operating systems may perform different operations to achieve the same result. For example, a client may load and utilize a multimedia application service that has the ability to reconstruct and render a media stream, such as an MPEG-1 video stream.

The client agent 375 sends a response to the server agent 350 indicating whether the request to create a particular media stream is successful (step 524). The server agent 350 provides this information to the application program 340. If the client succeeds in creating the stream, using DIRECTSHOW intelligent connect logic, the application program 340 loads the second output filter module 355B and connects it to the first output filter module 355A. (Step 528) This enables processing of the compressed information and control information sent to the client agent 375.

The first output filter module 355A intercepts timing information and original compressed data sets that represent at least a portion of the media stream, and also detects dynamic changes in the media type, such as format changes (step 532). The second output filter module 355B provides media stream-general commands (eg, play, pause, stop, flush, end-of-stream) and stream-specific commands (eg, video original window assignment, placement and clipping). Audio volume and balance adjustment command) (step 536).

Server agent 350 transmits the original compressed data set (optionally including timing information) to client agent 375 (step 540). If bandwidth is limited, server agent 350 may drop samples from the low priority media queue to make the high priority media queue available bandwidth. Media cues with the same priority are served in round robin fashion. In addition, the rate of transmission of the second output filter module 355B to the server agent 350 and thus to the client agent is as fast as the client is allowed by the network throughput prior to the start of playback to reduce latency in the initial playback. The media cue of the control is prefilled. The media queue of the client agent 375 is then used to accommodate changes in network latency, as described below. Server agent 350 also sends control information to client agent 375 (step 544). In general, control information is never removed despite limitations on the possibility of network bandwidth.

The client agent 375 receives the original compressed data set and optional timing information (step 548). The client agent 375 also receives any notification of dynamic information (eg, format change) in the control information and media type (step 552). Like server agent 350, client agent 375 creates a single command queue for control information for each primary context and a media sample queue for each subcontext. The media sample queue allows the client agent 375 to accept changes in bandwidth and network latency. Control cues in other major contexts are provided for round-robin fashion. In this embodiment, using the DIRECTSHOW technique, media samples in the queue are used by the associated source filter module (s) based on the timing information.

The client agent sends a status notification to the server agent (step 556). This notification includes an indication of the success or failure of the client for the creation of the requested media stream. Changing the bandwidth or network latency can be handled by adjusting the media queue of the client agent 375 as needed. The client agent 375 detects whether one or more media queues fall down or exceed a certain resource threshold, and the client agent 375 bursts corresponding to the server agent 350 respectively for the desired amount of media queue negotiation. Or send a stop request. The client agent 375 may also send status information to account for unexpected errors encountered within the creation of the media presentation. If the flow-adjustment notification of the client is not timely effective, and therefore the media queue is depleted or overflows, the client agent 375 re-buffers the queue. Pause) or drop media samples (step 560).

The native or third common support output filter module (transform filter) restores the received compressed data set to generate an uncompressed data set (step 564). The client agent 375 applies control information, timing information, and dynamic changes in the media type to a collection of filter modules (eg, an instance of a generic source filter module, a linked instance of a unique or third common output filter module, etc.). (Step 568).

If the client failed to create the requested media stream due to lack of appropriate filter module, processing resources, or memory, or for some other reason, then using the DIRECTSHOW intelligent connection logic, the application program running on server 310 340 ) Loads a native or third public-assisted output filter module that restores and renders the data stream at server 310 without loading first output filter module 355A (step 570).

As the native or third common support output filter module is loaded and operated on, an uncompressed data set that represents at least a portion of the media stream is intercepted (step 574), compressed (step 578), and sent to client 375 Is sent (step 582) and received by the client agent 375 (step 586).

Regardless of whether the client succeeded or failed the creation of the media stream, server agent 350 intercepts and sends a graphical display command associated with the non-media graphics information to client agent 375 (step 590). The client agent 375 receives a graphical display command (step 594). The client agent 375 generates a media presentation using a native or third publicly provided output filter module (renderer filter) to render the decompressed data set on the presentation interface 245 (step 598).

The invention may be provided by a computer-readable program embedded on one or more articles of manufacture. The article of manufacture may be a floppy disk, hard disk, CD ROM, flash memory card, PROM, RAM, ROM, or magnetic tape. In general, computer readable programs may be implemented in any programming language. These languages may be languages including C, C ++, JAVA. The software program may be stored in one or more articles of manufacture as object code.

The invention may be embodied in other specific forms without departing from the essential features or intentions of the invention. For example, it should be understood that both the invention of FIGS. 1 and 2 and the invention of FIGS. 3, 4A-4C and 5A-5C can be implemented by a single server. The above embodiments herein are not intended to limit the present invention and should be considered to explain all aspects. The scope of the invention is indicated by the appended claims rather than the foregoing description, and all changes that come within the meaning and scope of the claims are to be embraced therein.

Claims (78)

A method of creating a media presentation on a client, the method comprising (a) sending the output from the application program running on the server to the client, (b) identifying a media stream within the application output; (c) intercepting an original compressed data set that represents at least a portion of a media stream prior to processing by the application program; (d) sending the original compressed data set to a client; (e) capturing at least one timing information and control information associated with the media stream; (f) transmitting the captured information to a client How to include. The method of claim 1, Receiving the original compressed data set and the captured information at the client; Decompressing the original compressed data set at the client to generate a decompressed data set; Generating the media presentation at the client using the decompressed data set and the captured information. Further comprising a method of generating a media presentation on the client. 4. The method of claim 1, further comprising transmitting the original compressed data set to a second client substantially at substantially the same time as the transmission in step (d). The method of claim 1, wherein the transmission to the client occurs through at least one intermediary node. A method of creating a media presentation on a client, the method comprising (a) sending the output from the application program running on the server to the client, (b) identifying a media stream within the application output; (c) intercepting a first set of compressed data representing at least a portion of the media stream; (d) compressing the intercepted first compressed data set; (e) sending the first set of compressed data to the client in place of the first decompressed data set; (f) capturing at least one timing telegram and control information associated with the media stream; (f) transmitting the captured information to a client How to include. The method of claim 5 Receiving the compressed data set and the captured information at the client; Decompressing the compressed data set at the client to generate a second decompressed data set; Generating the media presentation at the client using the second decompressed data set and the captured information. Further comprising a method of generating a media presentation on the client. 6. The media presentation of claim 5 further comprising transmitting the compressed data set to a second client instead of the first decompressed data set substantially simultaneously with the transmission in step (e). How to generate. 6. The method of claim 5, wherein the transmission to the client occurs through at least one intermediate node. A method of creating a media presentation on a client, the method comprising (a) informing the server of at least one media format supported by a client agent installed on the client, (b) receiving a compressed data set in the client indicating at least a portion of a media stream; (c) decompressing the compressed data set at the client to generate a decompressed data set; (d) creating a media presentation at the client using the decompressed data set; (e) receiving at least one timing telegram and control information associated with the media stream; (f) generating the media presentation using the received information How to include. A system for generating a media presentation at a client, the system comprising (a) an application program configured to identify a media stream within the output generated by the application program, (b) an output filter module configured to intercept an original compressed data set representing at least a portion of a media stream and send the original compressed data set to a client prior to processing by the application program, wherein the output filter module is associated with the media stream. Further configured to capture at least one timing information and control information and to transmit the captured information to a client- System comprising. 11. The method of claim 10, wherein the original compressed data set and the captured information are received, the original compressed data set is decompressed to generate a decompressed data set, and the decompressed data set and the captured information are stored. And a client agent configured to generate a media presentation at the client using the client agent. 11. The system of claim 10, wherein the output filter module further comprises transmitting the original compressed data set to a plurality of clients at substantially the same time. 11. The system of claim 10, wherein the output filter module sends original compressed data to the client via at least one intermediate node. 11. The system of claim 10, wherein the filter merit of the output filter module is superior to the merit of other filter modules on the system. A system for generating a media presentation at a client, the system comprising (a) an application program configured to identify a media stream within the output generated by the application program, (b) intercepting a first set of compressed data representing at least a portion of the media stream, compressing the intercepted first set of compressed data, and replacing the first decompressed data set with the first set of compressed data An output filter module configured to send a set to a client, wherein the output filter module is further configured to capture at least one timing information and control information associated with the media stream and to transmit the captured information to a client And a system for generating a media presentation at the client. 16. The method of claim 15, wherein the compressed data set and the captured information are received, the compressed data set is decompressed to generate a second decompressed data set, and the second decompressed data set and the captured information. And a client agent configured to generate a media presentation at the client using information. 16. The system of claim 15, wherein the output filter module further comprises transmitting the compressed data set to a plurality of clients instead of the first decompressed data set at substantially the same time. 16. The system of claim 15, wherein the output filter module sends the compressed data set to a client instead of the first decompressed data set via at least one intermediate node. 16. The system of claim 15, wherein the filter merit of the output filter module is superior to the merit of other filter modules on the system. In a client, a method of generating a media presentation, the method comprising Server, A client in communication with the server, The client, Inform the server of at least one media format supported by the client agent, receive a compressed data set indicative of at least a portion of the media stream, and decompress the compressed data set at the client to decompress the data set And a client agent configured to generate the media presentation using the decompressed data set, The client agent Receive at least one timing telegram and control information associated with the media stream, and further generate the media presentation using the received information. In a method of displaying, on a client, graphical output from an application program running on a server, Identifying a non-textual element in the application output; Recovering the decompressed data set associated with the non-literal element; Identifying a compression technique for use with the decompressed data in response to the contents of the decompressed data set; Compressing the decompressed data set with the identified compression technique to produce a recompressed data set; Transmitting the recompressed data set in place of the decompressed data set. How to include. The method of claim 21, wherein the identified compression technique is a lossless compression technique. 23. The method of claim 22, wherein the non-literal element is an image having a large area of the same color. The method of claim 22, wherein the non-literal element is a computer-generated image. 23. The method of claim 22, wherein the lossless compression technique is 2DRLE compression. 22. The method of claim 21, wherein the identified compression technique is a lossy compression technique. 27. The method of claim 26, wherein the non-literal element is a continuous tone image. 27. The method of claim 26, wherein the non-literal element is a photographic image. 27. The method of claim 26, wherein the lossy compression technique is JPEG compression. 22. The method of claim 21, wherein identifying the compression technique is Compressing the decompressed data set using a lossless decompressed technique to form a first test data set; Comparing the first test data set with a predetermined value; If the size of the first test data set exceeds a predetermined value, compressing the decompressed data set using a lossy decompression technique to form a second test data set; Selecting a lossy compression technique when the size of the second test data set is less than a certain percentage of the size of the first test data set. Comprising a display method. The method of claim 21, wherein identifying the compression technique comprises: Applying an image processing algorithm to the decompressed data set to determine if a letter element is a photograph; Steps to choose a lossy compression technique if the inscription element is a photo Comprising a display method. The method of claim 21, wherein identifying the compression technique comprises: Applying an image processing algorithm to the decompressed data set to determine if a letter element is a continuous tone; Selecting a lossy compression technique if the non-zero element is a continuous tone Comprising a display method. The method of claim 21, wherein identifying the compression technique comprises: Enumerating the number of pixel colors of the decompressed data set; Selecting a lossy compression technique when the number of counted pixel colors exceeds a predetermined value Comprising a display method. In a system for displaying graphical output from an application program running on a server on a client, Application programs, A compression technique for identifying a non-literal element generated by the application program, restoring a decompressed data set associated with the non-literal element, and using the decompressed data in response to the contents of the decompressed data set. An output filter module configured to identify a, compress the decompressed data with the identified compression technique to generate a recompressed data set, and send the recompressed data set to a client instead of the decompressed data set. System comprising. 35. The display system of claim 34, wherein the identified compression technique is a lossless compression technique. 36. The display system of claim 35, wherein the non-literal element is an image having a large area of the same color. 36. The display system of claim 35, wherein the non-literal element is a computer-generated image. 36. The display system of claim 35, wherein the lossless compression technique is 2DRLE compression. 35. The display system of claim 34, wherein the identified compression technique is a lossy compression technique. 40. The display system of claim 39, wherein the non-literal element is a continuous tone image. The display system of claim 39, wherein the non-literal element is a photographic image. 40. The display system of claim 39, wherein the lossy compression technique is JPEG compression. The method of claim 34, wherein the output filter module, Compressing the decompressed data set using a lossless decompression technique to form a first test data set, comparing the first test data set with a predetermined value, and the size of the first test data set exceeds a predetermined value. Compressing the decompressed data set using a lossy decompression technique to form a second test data set, and loss if the size of the second test data set is less than a predetermined percentage of the size of the first test data set. A display system for identifying a compression technique by selecting a compression technique. The method of claim 34, wherein the output filter module, A display system for identifying a compression technique by applying an image processing algorithm to the decompressed data set to determine if a non-literal element is a photograph and selecting a lossy compression technique if the non-literal element is a photograph. The method of claim 34, wherein the output filter module, And applying an image processing algorithm to the decompressed data set to determine if the non-letter element is a continuous tone and selecting a lossy compression technique if the non-letter element is a continuous tone. The method of claim 34, wherein the output filter module, And enumerating the number of pixel colors of the decompressed data set, and identifying a compression technique by selecting a lossy compression technique when the counted number of pixel colors exceeds a predetermined value. A method of creating a media presentation on a client, the method comprising (a) sending the output from the application program running on the server to the client, (b) identifying a plurality of media streams within the application output; (c) associating each media stream with each server media queue having an assigned priority; (d) assigning priorities to server command queues, (e) transmitting the contents of the server queue to the client, (f) if the network bandwidth between the server and the client is insufficient to transmit all of the server queues, drop the sample from the server queue with the lower assigned priority and sample from the server queue with the higher assigned priority. Steps to send How to include. 48. The method of claim 47, wherein the priority of the media queue is based on the media type stored in each queue. The method of claim 47, (g) receiving the transmission at a client having a plurality of client media queues, each client media queue associated with a server media queue; (h) when the client media queue is full, discarding transmissions associated with the full client media queue. Further comprising a method of generating a media presentation on the client. 48. The method of claim 47, wherein the command queue has the highest priority. A method of creating a media presentation on a client, the method comprising (a) sending the output from the application program running on the server to the client, (b) identifying a plurality of media streams within the application output; (c) sending data representing the media stream to a client Wherein the client media queue is filled by sending as quickly as allowed by network throughput before client playback begins. 53. The method of claim 51, further comprising receiving at the server a flow control command from a client issued according to a data level in a corresponding media queue at the client. A method of generating, at a client, a media presentation from an application program running on a server, the method comprising: Identifying a non-textual element in the application output; Recovering the decompressed data set associated with the non-literal element; Apportioning the decompressed data set into at least one strip; Transmitting the at least one strip to a client. How to include. 54. The method of claim 53 wherein the at least one strip is in a second decompressed data set that represents another letter element. 54. The method of claim 53, further comprising storing the at least one strip in a database at a server. 56. The method of claim 55, further comprising height, width and cyclic redundancy check (CRC) information of at least one strip in the server database. 56. The method of claim 55, further comprising storing, at the client, at least one strip in a database. 59. The method of claim 57, further comprising height, width and cyclic redundancy check (CRC) information of at least one strip in the client database. 59. The method of claim 57, wherein the method is Intercepting a second decompressed data set representing another non-literal element; Distributing the second decompressed data set to a second strip; Identifying the second strip previously stored in the server database; Recovering an identifier associated with the second stored second strip; Sending an identifier to the client instead of sending the second strip to the client Further comprising a media presentation generation method. The method of claim 59, Receiving the identifier at a client; Identifying a strip stored in a client database associated with the received identifier and displaying the stored strip associated with the received identifier at a client. 58. The method of claim 57, further comprising retaining database content of the client after the client is disconnected from the server. 62. The method of claim 61, Reconnecting a client from the server; Transmitting the content list of the client database to a server Further comprising a media presentation generation method. 63. The method of claim 62 further comprising height, width and cyclic redundancy check (CRC) information of the strips stored in the client database. 59. The method of claim 57, further comprising removing instructions stored in the client database from the client database by sending instructions from the server to the client. 65. The method of claim 64, wherein the strip selected for removal is the strip most likely to be used in the future. A method of transmitting a plurality of media streams to a client, the method comprising (a) identifying a first media stream and a second media stream in an application program output; (b) associating the first media stream with a first queue having an assigned priority and associating the second media stream with a second queue having an assigned priority lower than the first queue; (c) intercepting a first compressed data set indicating at least a portion of the first media stream, the first compressed data set comprising a first plurality of packets; (d) intercepting a second compressed data set indicative of at least a portion of said second media stream, said second compressed data set comprising a second plurality of packets, and passing said original compressed data set to a client; Transmitting, (e) storing the first plurality of packets of the first queue associated with the first media stream; (f) storing the second plurality of packets of the second queue associated with the second media stream; (g) transmitting the first plurality of packets from the first queue, (h) transmitting at least a portion of the second plurality of packets from the second queue, (i) dropping another packet of a second plurality of packets from the second queue How to include. 67. The method of claim 66 further comprising executing the application program. 67. The method of claim 66, further comprising creating a control queue for storing control commands. 69. The method of claim 68, wherein the control command comprises one stream-generic and stream-specific command. 69. The method of claim 68, wherein the control queue is assigned the highest priority among all queues. 67. The method of claim 66, wherein the priority of the media queue is based on the corresponding media type stored in each queue. 72. The method of claim 71 further comprising detecting a dynamic change in the media type. 72. The method of claim 71 further comprising invoking a first output filter module. 74. The method of claim 73, further comprising calling a second output filter module in communication with the first output filter module. 75. The method of claim 74, wherein the first output filter module intercepts a compressed data set and the second output filter module intercepts control information. 74. The method of claim 73, wherein the first output filter module requests the client to create a media stream. 77. The method of claim 76, (a) receiving a failure indication of a media stream creation from the client; (b) unloading the first output filter module; (c) loading one of the native and third party output filter modules Further comprising a plurality of media streams to the client. 67. The method of claim 66 further comprising intercepting any timing information.