Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
  • G06Q50/10—Services
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4092—Image resolution transcoding, e.g. by using client-server architectures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
  • H04L9/40—Network security protocols

Definitions

• the present invention relates generally to information processing and, more particularly, to an online system providing methodology for improving online access to digital media and related information.
• a variety of digital image, digital video and digital audio products are currently available to consumers. Regardless of how digital media is recorded, at some later point in time, the information must become available to other devices —that is, become available to a larger network of digital devices, so that such information may be displayed on a screen, printed to hard copy, stored, or shared with other people.
• client devices exist with sufficient graphics capabilities for potentially viewing (and/or listening to) this media.
• WAP Wireless Application Protocol
• a Palm handheld device For instance, the example of a Palm handheld device.
• a "true-color" e.g., 32-bit depth 1024 by 768 digital photograph of his or her family on the Web. If the user connects to the Internet using the Palm device, he or she cannot display the photograph because the Palm device may only support four-level or sixteen- level grayscale. Even if the image could be displayed, the transmission time for downloading the image to the Palm device would be impractical. Still further, even if the image could be downloaded, the display for the Palm may be physically too small to render the image in a manner that is acceptable to the user.
• a "true-color" e.g., 32-bit depth
• This problem is not limited to just image data but also applies to other types of digital objects.
• Many Internet sites display multi-colored images, streaming video, streaming audio, and other content that is targeted primarily at users with desktop and laptop computer devices and Web browser software.
• a desktop or laptop computer has significant processing, storage, and display resources and is capable of rendering large images and video files in multiple colors and formats.
• a smaller device such as a cellular phone or a personal digital assistant (“PDA”) typically does not have the necessary capabilities to display colors or to handle media in particular formats.
• PDA personal digital assistant
• a cellular phone which typically has a small screen for display of messages, does not have the software or display capabilities to render a JPEG image.
• a user with a PDA or cellphone is typically unable to access much of the information available on many Internet sites.
• Certain content providers that are focused primarily on cellular telephone and
• PDA users do offer Internet sites that display media appropriate for these users. These sites include images with lower resolution and fewer colors in formats supported by these types of devices.
• content providers focused on cellular telephone and handheld device users have problems supporting the wide number of devices currently in use. Numerous different types of cellular telephones and handheld devices are in use, each having different specifications and capabilities. These devices have different screen sizes, resolution, and color capabilities. Thus, even if a content provider is focused on cellular telephone users, that content provider often must create images and other media offerings geared towards the least capable devices in order to serve the broadest number of users.
• Apache is a public domain Web server developed by a group of volunteer programmers called the Apache Group.
• the initial version of the Apache server was developed in 1995 based on the httpd Web server developed at the National Center for Supercomputing Applications, University of Illinois, Urbana-Champaign. Additional information on the Apache Web server and copies of the source code for this Web server are currently available on the Internet at http://www.apache.org.
• CC/PP is an abbreviation for Composite Capabilities/Preference Profiles, a proposed standard being developed by the World Wide Web Consortium (W3C).
• W3C World Wide Web Consortium
• a CC/PP profile is a description of device capabilities and user preferences that can be used to guide the adaptation of content presented to that device.
• the current proposed specification is described by "Composite Capability/Preference Profiles (CC/PP): A user side framework for content negotiation", W3C Note, 27 July 1999, available from the
• HTML HyperText Markup Language. Every HTML document requires certain standard HTML tags in order to be correctly interpreted by Web browsers. Each document consists of head and body text. The head contains the title, and the body contains the actual text that is made up of paragraphs, lists, and other elements. Browsers expect specific information because they are programmed according to HTML and SGML specifications. Further description of HTML documents is available in the technical and trade literature; see e.g., Ray Duncan, Power Programming: An HTML Primer, PC Magazine, June 13, 1995.
• HTTP stands for HyperText Transfer Protocol, which is the underlying communication protocol used by the World Wide Web on the Internet. HTTP defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. For example, when a user enters a URL in his or her browser, this actually sends an HTTP command to the Web server directing it to fetch and transmit the requested Web page. Further description of HTTP is available in RFC 2616: Hypertext Transfer Protocol - HTTP/1.1. RFC 2616 is available from the World Wide Web Consortium (W3C), and is currently available via the Internet at http://www.w3.org/Protocols/. Additional description of HTTP is available in the technical and trade literature; see e.g., William Stallings, The Backbone of the Web,
• JPEG Full-size digital images are maintained in a Joint Photographic Experts Group or JPEG format. See e.g., Nelson, M. et al., The Data Compression Book, Second Edition, Chapter 11 : Lossy Graphics Compression (particularly at pp. 326-330), M&T Books, 1996. Also see e.g., JPEG-like Image Compression (Parts 1 and 2), Dr. Dobb's Journal,
• SMTP Simple Mail Transfer Protocol, a protocol for sending e-mail messages between servers. Most e-mail systems that send mail over the Internet use SMTP to send messages from one server to another; the messages can then be retrieved with an e-mail client using either POP or IMAP. In addition, SMTP is generally used to send messages from a mail client to a mail server.
• TCP stands for Transmission Control Protocol.
• TCP is one of the main protocols in TCP/IP networks. Whereas the IP protocol deals only with packets, TCP enables two hosts to establish a connection and exchange streams of data. TCP guarantees delivery of data and also guarantees that packets will be delivered in the same order in which they were sent. For an introduction to TCP, see, e.g., RFC 793.
• TCP/IP Stands for Transmission Control Protocol/Internet Protocol, the suite of communications protocols used to connect hosts on the Internet. TCP/IP uses several protocols, the two main ones being TCP and IP. TCP/IP is built into the UNIX operating system and is used by the Internet, making it the de facto standard for transmitting data over networks.
• RFC 1180 A TCP/IP tutorial. A copy of RFC 1180 is currently available at ftp://ftp.isi.edu/in-notes/rfcl 180.txt.
• UAProf UAProf or WAG UAProf refers to the proposed Wireless Access Group User Agent Profile Specification about how devices such as cellular phones should describe their capabilities to servers.
• the current proposed specification is described as "WAG UAProf (Wireless Application Group User Agent Profile Specification), Wireless Application Protocol Forum, Ltd., Proposed Version 30- May-2001, available from the WAP Forum.
• WAG UAProf Wireless Application Group User Agent Profile Specification
• WAP Forum Proposed Version 30- May-2001, available from the WAP Forum.
• a copy of the specification can currently be found on the Internet at http://wwwl . wapforum. org/tech/ documents/WAP -248- UAProf-20010530-p.pdf.
• URL Abbreviation of Uniform Resource Locator, the global address of documents and other resources oh the World Wide Web. The first part of the address indicates what protocol to use, and the second part specifies the IP address or the domain name where the resource is located.
• WAP stands for Wireless Application Protocol, which is a communication protocol, not unlike TCP/IP, that was developed by a consortium of wireless companies, including Motorola, Ericsson, and Nokia, for transmitting data over wireless networks.
• WAP includes various equivalents of existing Internet protocols.
• WML is a WAP version of the HTML protocol.
• Other examples include a WAP browser that is a WAP equivalent of an
• HTML browser and a WAP gateway (on the server side) that is a WAP equivalent of an
• HTTP server At the WAP gateway, HTTP is translated to/from WAP.
• XML Short for Extensible Markup Language, a specification developed by the W3C.
• XML is a pared-down version of SGML, designed especially for Web documents. It allows designers to create their own customized tags, enabling the definition, transmission, validation, and interpretation of data between applications and between organizations.
• XML Extensible Markup Language
• XML Extensible Markup Language
• the specification is also currently available on the Internet at http://www.w3.org/TR/REC-xml.
• the present invention provides an online system incorporating improved methodologies enabling content providers to effectively serve the widest array of clients. It combines on-the-fly media reformatting with advanced client detection capabilities to enable the delivery of the appropriate and best possible incarnation of a provider's media content to every connected client device.
• the online media delivery system of the present invention receives requests from client devices for media documents or objects, determines the media output capabilities of the device making the request, and serves a transformed version of the media object appropriate for the requesting client device.
• the system includes a client capabilities module (CCM) and a media transformation module (MTM). These two modules cooperate to identify a client from an HTTP request, determine its media output capabilities, and reformat the source media according to those capabilities.
• CCM client capabilities module
• MTM media transformation module
• the system also includes a data store containing information on the capabilities of various client devices, an (optional) front-side cache for storing transformed media content, and a backside cache for local storage of original items of content.
• the system provides access to media content for multiple disparate client devices ⁇ that is, to target devices of varying hardware and software capabilities.
• the system enables delivery of appropriate media content to practically any device with connectivity capability.
• the target devices may include both wireless devices (e.g., cellular phone, handheld personal digital assistant (PDA), and pager) as well as wireline devices (e.g., desktop computer, laptop computer, and videophone).
• wireless devices e.g., cellular phone, handheld personal digital assistant (PDA), and pager
• wireline devices e.g., desktop computer, laptop computer, and videophone
• the improved methodology of the system in providing media content appropriate to a particular device can be summarized as follows. Initially, the URLs of particular full-format multimedia objects on an Internet Web site are modified to be served by the system of the present invention. This is accomplished by modifying the HTML pages on the Web site to replace the URLs of these full-format multimedia objects with URLs which point to the server on which the media delivery system is installed and contains the path to the media objects.
• the system acts as an HTTP proxy to those original objects, intercepting requests for the original content and serving a transformed version of the content applicable to the requesting client.
• the media output capabilities are communicated to the system by the device or are surmised by the system's client capabilities module. If the device is communicating its capabilities, it does so during the initiation or during every interaction. Alternatively, the device's capabilities are previously stored in the system's data store based on prior knowledge of the device. Based on the information communicated to or surmised by the system, an information record is created describing the target device's capabilities. This information indicates to the system what transformation (if any) is required for translating the original item of media content into a format suitable for the target device.
• the client capabilities module (optionally) checks the front-side cache to see whether the cache already stores a version of the object that has been translated into a format suitable for this particular target device. If the object (transformed for the target device) already exists in the front-side cache, then the client capabilities module may simply return that object to the client device. However, if an appropriate transformed object is not in the cache, then the client capabilities module proceeds to pass the object identifier and the client device parameters on to the media transformation module.
• the media transformation module receives requests for a particular item of media content in a particular format from the client capabilities module.
• the media transformation module obtains the original media object, transforms the object from its original format into the format that is desired for the target device (based on the specified target device capabilities), and returns the converted object to the client device.
• the media transformation module utilizes a backside cache as an optimization to provide increased efficiency. Media objects are retained in the backside cache to avoid having to retrieve frequently or recently requested items in response to each request. Use of this backside cache reduces the number of calls over the network and expedites conversion and return of media objects to client devices.
• Fig. 1 is a block diagram of a computer system in which software-implemented processes of the present invention may be embodied.
• Fig. 2 is a block diagram of a software system for controlling the operation of the computer system.
• Fig. 3 is a block diagram illustrating an online media delivery system of the present invention.
• Figs. 4A-B comprise a single flowchart illustrating the detailed method steps of the system in determining the capabilities of a target device and transforming and delivering content to such target device in an appropriate format.
• Fig. 5 is a flowchart illustrating the operations of the client capabilities module of the present invention in acting as a proxy for incoming HTTP requests from non- compliant devices.
• Fig. 1 is a very general block diagram of an IBM-compatible system 100.
• system 100 comprises a central processing unit(s) (CPU) or processor(s) 101 coupled to a random-access memory (RAM) 102, a read-only memory (ROM) 103, a keyboard 106, a printer 107, a pointing device 108, a display or video adapter 104 connected to a display device 105, a removable (mass) storage device 115 (e.g., floppy disk, CD-ROM, CD-R, CD-RW, DVD, or the like), a fixed (mass) storage device 116 (e.g., hard disk), a communication (COMM) port(s) or interface(s) 110, a modem 112, and a network interface card (NIC) or controller 111 (e.g., Ethernet).
• CPU 101 comprises a processor of the Intel Pentium® family of microprocessors. However, any other suitable processor may be utilized for implementing the present invention.
• the CPU 101 communicates with other components of the system via a bidirectional system bus (including any necessary input/output (I/O) controller circuitry and other "glue" logic).
• the bus which includes address lines for addressing system memory, provides data transfer between and among the various components. Description of Pentium-class microprocessors and their instruction set, bus architecture, and control lines is available from Intel Corporation of Santa Clara, CA.
• Random-access memory 102 serves as the working memory for the CPU 101. In a typical configuration, RAM of sixty-four megabytes or more is employed. More or less memory may be used without departing from the scope of the present invention.
• the read-only memory (ROM) 103 contains the basic input/output system code (BIOS) - a set of low-level routines in the ROM that application programs and the operating systems can use to interact with the hardware, including reading characters from the keyboard, outputting characters to printers, and so forth.
• Mass storage devices 115, 116 provide persistent storage on fixed and removable media, such as magnetic, optical or magnetic-optical storage systems, flash memory, or any other available mass storage technology.
• the mass storage may be shared on a network, or it may be a dedicated mass storage.
• fixed storage 116 stores a body of program and data for directing operation of the computer system, including an operating system, user application programs, driver and other support files, as well as other data files of all sorts.
• the fixed storage 116 serves as the main hard disk for the system.
• program logic (including that which implements methodology of the present invention described below) is loaded from the removable storage 115 or fixed storage 116 into the main (RAM) memory 102, for execution by the CPU 101.
• the system 100 accepts user input from a keyboard 106 and pointing device 108, as well as speech-based input from a voice recognition system (not shown).
• the keyboard 106 permits selection of application programs, entry of keyboard-based input or data, and selection and manipulation of individual data objects displayed on the screen or display device 105.
• the pointing device 108 such as a mouse, track ball, pen device, or the like, permits selection and manipulation of objects on the display screen. In this manner, these input devices support manual user input for any process running on the system.
• the computer system 100 displays text and/or graphic images and other data on the display device 105.
• the video adapter 104 which is interposed between the display
• the video adapter 104 which includes video memory accessible to the CPU 101, provides circuitry that converts pixel data stored in the video memory to a raster signal suitable for use by a cathode ray tube (CRT) raster or liquid crystal display (LCD) monitor.
• CTR cathode ray tube
• LCD liquid crystal display
• a hard copy of the displayed information, or other information within the system 100, may be obtained from the printer 107, or other output device.
• Printer 107 may include, for instance, an HP LaserJet® printer (available from Hewlett-Packard of Palo Alto, CA), for creating hard copy images of output of the system.
• the system itself commumcates with other devices (e.g., other computers) via the network interface card (NIC) 111 connected to a network (e.g., Ethernet network, Bluetooth wireless network, or the like), and/or modem 112 (e.g., 56K baud, ISDN, DSL, or cable modem), examples of which are available from 3Com of Santa Clara, CA.
• the system 100 may also communicate with local occasionally-connected devices (e.g., serial cable-linked devices) via the communication (COMM) interface 110, which may include a RS-232 serial port, a Universal Serial Bus (USB) interface, or the like.
• Communication communication
• USB Universal Serial Bus
• IBM-compatible personal computers and server computers are available from a variety of vendors. Representative vendors include Dell Computers of Round Rock, TX,
• Compaq Computers of Houston, TX, and IBM of Armonk, NY Other suitable computers include Apple-compatible computers (e.g., Macintosh), which are available from Apple Computer of Cupertino, CA, and Sun Solaris workstations, which are available from Sun Microsystems of Mountain View, CA.
• Apple-compatible computers e.g., Macintosh
• Sun Solaris workstations which are available from Sun Microsystems of Mountain View, CA.
• a computer software system 200 is provided for directing the operation of the computer system 100.
• Software system 200 which is stored in system memory (RAM) 102 and on fixed storage (e.g., hard disk) 116, includes a kernel or operating system (OS) 210.
• the OS 210 manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device ⁇ IO.
• One or more application programs such as client application software or "programs" 201 (e.g., 201a, 201b, 201c, 201d) maybe "loaded” (i.e., transferred from fixed storage 116 into memory 102) for execution by the system 100.
• System 200 includes a graphical user interface (GUI) 215, for receiving user commands and data in a graphical (e.g., "point-and-click") fashion. These inputs, in turn, may be acted upon by the system 100 in accordance with instructions from operating system 210, and/or client application module(s) 201.
• the GUI 215 also serves to display the results of operation from the OS 210 and application(s) 201, whereupon the user may supply additional inputs or terminate the session.
• the OS 210 operates in conjunction with device drivers 220 (e.g., "Winsock” driver ⁇ Windows' implementation of a TCP/IP stack) and the system BIOS microcode 230 (i.e., ROM-based microcode), particularly when interfacing with peripheral devices.
• device drivers 220 e.g., "Winsock” driver ⁇ Windows' implementation of a TCP/IP stack
• BIOS microcode 230 i.e., ROM-based microcode
• OS 210 can be provided by a conventional operating system, such as Microsoft® Windows 9x, Microsoft® Windows NT, Microsoft® Windows 2000, or Microsoft® Windows XP, all available from Microsoft Corporation of Redmond, WA.
• OS 210 can also be an alternative operating system, such as the previously-mentioned operating systems.
• the above-described computer hardware and software are presented for purposes of illustrating the basic underlying desktop and server computer components that may be employed for implementing the present invention. For purposes of discussion, the following description will present examples in which it will be assumed that there exists a "server” (e.g., Web server) that communicates with one or more "clients” (e.g., media display devices).
• the present invention is not limited to any particular environment or device configuration.
• a client/server distinction is not necessary to the invention, but is used to provide a framework for discussion.
• the present invention may be implemented in any type of system architecture or processing environment capable of supporting the methodologies of the present invention presented in detail below.
• this solution requires an Internet content provider to display a large number of copies of the same media object in order to address the various types of devices on the market and their wide range of capabilities.
• the disadvantages of this approach are that it requires the content provider to create, store, and manage multiple pre-rendered versions of the same media in various formats depending on the number of devices to be supported.
• the present invention allows a content provider to develop content in one form and deliver the content in multiple forms based on the capabilities of the client device requesting the content.
• the present invention includes an online system and methodology for providing a client device with media content appropriate for the media output capabilities of the device.
• the system includes a client capabilities module (CCM) to determine the capabilities of connected client devices and a media transformation (or transcoding) module (MTM) that renders the appropriate media on-the-fly and delivers it to the client device in the appropriate format.
• CCM client capabilities module
• MTM media transformation (or transcoding) module
• the operations of the present invention can be illustrated by the following example of rendering a digital image to a particular client device.
• the original item of media content on an Internet site is a 24-bit color JPEG image and the client device requesting this image is a Palm PDA that supports 16-level grayscale.
• the client device connects wirelessly to the Internet site and invokes the URL for this JPEG image.
• the Internet content provider using the present invention has previously revised the Uniform Resource Locator (URL) for this image to refer to the machine on which the client capabilities module of the present invention is installed. As a result, this URL request is routed to the CCM, which identifies the requested image and the client device from this request.
• URL Uniform Resource Locator
• the CCM determines the capabilities of the client device in an intelligent fashion by examining the client request to the server to obtain information about the client device and by comparing this information to known device characteristics and capabilities stored in its data store. In this case, the CCM recognizes this device as a specific type of Palm PDA and looks up the device's capabilities in the system's database. Based upon this information, the CCM determines that the JPEG image should be supplied to this Palm device in a 16-level grayscale format.
• the CCM (optionally) checks a front-side cache to see whether the cache already stores a version of the image in the required format. If an appropriate transformed image is not in the cache, then the CCM requests the image from the media transformation module in a 16-level grayscale format suitable for rendering to this type of Palm PDA device.
• the MTM obtains the appropriate image, converts it to the appropriate format and serves it to the client device.
• the system includes intelligence that allows it to optimally translate the images (from their original format) into a format suitable for use by the particular target device. The overall translation or transformation process is performed in a manner that preserves performance and scalability criteria desired for the system.
• FIG. 3 illustrates an online environment 300 suitable for implementing the present invention.
• the environment 300 includes an online media delivery system 320 connected via the Internet (shown at 310) to one or more client devices 301 and at least one Internet site (server) 330.
• client device 301 represents one of a variety of target devices (or "clients") that are capable of connecting over the Internet and accessing online content.
• client devices may include both wireless devices (e.g., cellular phone, handheld PDA (personal data assistant), and pager) as well as wireline devices (e.g., desktop computer, laptop computer, and videophone).
• wireless devices e.g., cellular phone, handheld PDA (personal data assistant), and pager
• wireline devices e.g., desktop computer, laptop computer, and videophone.
• the Internet server 330 represents a Web server at which items of media content (e.g., audio, video, documents, blob objects, or other items of interest) are stored. During operation, the Internet server 330 typically stores a number of different items of media content that are to be made available to a wide range of client devices. Actual connection between the Internet server 330 and the online media delivery system 320 may occur over the Internet or, optionally, occur via a non-Internet (e.g., WAN) connection as illustrated by the dashed connection line in the figure. In either instance, the Internet server 330 may be housed at the same site as the online media delivery system 320, or may be housed at a remote site, as desired.
• items of media content e.g., audio, video, documents, blob objects, or other items of interest
• the Internet server 330 typically stores a number of different items of media content that are to be made available to a wide range of client devices.
• Actual connection between the Internet server 330 and the online media delivery system 320 may occur
• the online media delivery system 320 functions to detect client device capabilities and, based on that determination, transforms and delivers media content to such devices in appropriate formats to the client device 301.
• the media delivery system 320 includes a client capabilities module (CCM) 322, a media transformation module (MTM) 325, and a device capabilities data store 324.
• the client capabilities module 322 is in direct communication with a front-side cache 321 and a CCM log 323; the media transformation module 325, similarly, is in direct communication with backside cache 327 and MTM log 326.
• items containing and/or referencing media content are encoded with a URL that directs clients requesting such items to the system 320.
• the Internet server 330 may also include the original items of media content, which may be any type of content including digital images, video, audio, documents, "blob" objects, or the like. Alternatively, original items of media content may be stored locally on the system 320 or on another local or remote server to which the system 320 is connected.
• a request e.g., HTTP request
• the request is routed to the client capabilities module 322 of the media delivery system 320.
• the client capabilities module 322 identifies the (client) device and obtains available information about the device's capabilities. Based on this identification, the client capabilities module 322 retrieves additional information about the capabilities of the client device for displaying or outputting media from the data store 324.
• the data store 324 includes media output capabilities of various devices. In the currently preferred embodiment, a corresponding device identifier is employed to index this information.
• the capabilities stored in data store 324 include information regarding screen resolution, screen color depth, whether images should be rotated to fit on the device's screen display, and other such information as described in more detail below.
• the data store 324 is field upgradable so that as new devices are introduced into the market, the profiles of such devices and their capabilities can be added.
• the client capabilities log 323 includes a record of any client devices that could not be identified or for which capabilities are not available. These log records enable any omitted devices to be identified so that information on these devices can be obtained and added to the data store 324.
• client capabilities module 322 looks into the front-side cache 321, which stores previously converted content, to see if the object is available in the appropriate format.
• the front side cache 321 is an (optional) optimization in which previously converted media objects are retained for supply in response to future requests.
• the front-side cache 321 may be implemented using least-recently used (LRU) technique to "age out” (i.e., remove) the least-recently used items. If the client capabilities module 322 determines that the . appropriate object is not available in the front-side cache 321, it requests the media transformation module 325 to perform an on-the-fly transformation that will supply the object.
• LRU least-recently used
• the media transformation module (MTM) 325 receives requests for a particular item of media content in a particular format from client capabilities module 322.
• the media transformation module 325 obtains an original copy of the requested media object, converts it into the requested format, and returns the converted media object to the client device 301.
• the backside cache 327 is an optimization to provide increased efficiency; it may also be implemented using LRU technique.
• Original objects retrieved from the Internet server 330 (or another source) are retained in the backside cache 327 to avoid having to retrieve a copy of each requested item in response to each request. Use of this backside cache reduces the number of calls over the network. It also expedites conversion and return of available objects by the media transformation module 325 by avoiding the retrieval of large objects (such as high quality color images) from a remote server.
• Figs. 4A-B comprise a single flowchart of the detailed method steps of the operations of the system in detecting the capabilities of connected client devices and delivering media objects to such devices in appropriate formats.
• step 401 URLs for multimedia objects in Web pages at an Internet site are modified so that such objects will be served by the media delivery system of the present invention.
• the URLs are prepended with the server name on which the media delivery system is installed. For example, if the subscriber's site contained a logo normally accessed by the URL: http://www.subscriber.com/img/logo.gif, the modified URL would be: http://eswitch.com/www.subscriber.com/img/logo.gif.
• step 402 an HTTP request from a client device is routed to the media delivery system when a Web page containing these rewritten URLs is opened or the client device selects (clicks on) a rewritten URL.
• step 403 the client capabilities module (CCM) reverses the encoding process performed in step 401 and determines the full URL to the source image. In the currently preferred implementation, this consists of removing the "/eswitch.com" from the request URL.
• the CCM retrieves the client capability configuration from the data store using the HTTP User- Agent header as a key.
• the configuration information specifies the playback capabilities of the client device, such as display size, color depth, audio channels, and so forth.
• the configuration information may require examination of additional HTTP request headers to determine the complete capabilities of the client device. Information gathered during this step allows the CCM to understand exactly what capabilities are supported in the target device. In particular, this information indicates to the system what particular transformation operation(s) is required in order to translate the original media object into a format suitable for the target device.
• step 405 the CCM constructs a URL containing commands specific to the media transformation module (MTM). These commands instruct the MTM to transform the source media document or object to conform to the capabilities of the client device that requested the document.
• This URL points to the MTM server specified in the configuration file.
• the MTM module can be on the same server or a different server than the CCM.
• step 406 the CCM consults a front-side cache for an object matching the constructed MTM URL. In other words, it looks to see if the front-side cache already stores a version of the media object that has been translated in a manner suitable for this particular target device.
• the URL strings used internally within the system are encoded to serve as an index for particular object in a particular format.
• an encoded URL string can indicate that a particular document in a particular format is stored at a particular location.
• the CCM proxies the original client request, replacing the URL sent by the client with the reconstructed URL created by the CCM.
• This process is completely transparent to the client: the client device making the request is not informed or aware that the request has been passed on to the MTM. Rather, this transfer is a back-end process in which the CCM forwards the request made by the client device for fulfillment by the MTM.
• the MTM receives the constructed URL and makes an HTTP request through a backside-caching server for the original media object. If the object is present in the backside cache, it is served from local disk. If not, the caching server requests the object from the Internet site identified in the original URL and caches it for future use.
• the task of the MTM is to transform the media object from its original format into the format that is desired for the target device (based on target device capabilities).
• the MTM performs the media transformation as specified in the reconstructed URL that it received.
• the newly-transformed version of the media object is returned to the client device and
• the present invention may also be used to determine the capabilities of client devices and supply this client capabilities information to other systems or devices.
• client devices such as cellular phones should describe their capabilities to servers, see, e.g., WAG UAProf (Wireless Application Group User Agent Profile Specification), Wireless Application Protocol Forum, Ltd., Proposed Version 30- May-2001, available from the WAP Forum.
• WAG UAProf Wireless Application Group User Agent Profile Specification
• Wireless Application Protocol Forum Ltd., Proposed Version 30- May-2001, available from the WAP Forum.
• This problem can be addressed by configuring the system of the present invention to act as a proxy for incoming HTTP requests from non-compliant devices.
• the client capabilities module looks up the required data and attaches this information to the request before forwarding the request on to its eventual destination. This enables the system to act as a bridge between the non-compliant client device and those Internet and WAP sites that require compliance with UAProf, CC/PP, or other comparable standards.
• Fig. 5 is a flowchart illustrating the operations of the client capabilities module of the media delivery system in acting as a proxy for incoming HTTP requests from non- compliant devices.
• step 501 an HTTP request from a non-compliant client device is forwarded from an Internet or WAP site to the system.
• a "non- compliant" client device is one that is not in compliance with UAProf, CC PP, or similar standards requiring such device to identify its capabilities.
• step 502 the system's client capabilities module (CCM) retrieves the client capability configuration from the data store using the HTTP User- Agent header as a key.
• CCM system's client capabilities module
• the configuration information specifies the capabilities of the client device, such as display size, color depth, audio channels, and so forth.
• the configuration information may require examination of additional HTTP request headers to determine the complete capabilities of the client device. Information gathered during this step allows the CCM to understand exactly what capabilities are supported in the target device.
• step 503 the CCM supplements the request made by the particular client device with information regarding the specific capabilities of such client device as illustrated below.
• the CCM returns the supplemented request including details on the client capabilities to the destination specified in such client request.
• the following is an example of how the system can be used to append device capabilities information to a request.
• An example of an incoming request forwarded to the system is as follows:
• the incoming information reports device capabilities including, for example, color support ("0" or none, for the above device) and screen pixels (171 by 108 pixels for the above device).
• the client capabilities module determines the capabilities of the particular client device in the manner described above.
• the CCM then attaches this information to the request and forwards the supplemented request on to its eventual destination.
• a sample request showing the information appended by the CCM is as follows:
• the generated document is populated with information from the device's HTTP request and the data store or knowledgebase maintained by the media delivery system. Because the system maintains a knowledgebase of client device characteristics, it is much more capable of creating a complete UAProf or CC/PP document than a server which simply transcodes the information in the HTTP headers. E. Detailed methods of operation of CCM and MTM modules
• the client capabilities module identifies a client device from an HTTP request.
• the CCM uses information supplied in the request, together with media output capability information stored in the data store, to determine the media output capabilities of a particular client device. This enables the CCM to determine the optimal transmission size and playback format for the particular type of client device requesting the item of interest (e.g., media object).
• an HTTP browser may indicate the browser name (e.g., "Netscape Navigator” or "Microsoft Internet Explorer"), the browser version, and the graphic types it supports. This information is helpful in instances where graphic support is limited, such as a browser running on a set-top box having very limited graphic support (e.g., JPEG support only).
• the target device In the case that the target device is not able to indicate its capabilities, it will at least indicate its device class, such as a Palm handheld device, a set-top box, a phone with a WAP browser, or the like.
• the CCM obtains information about the capabilities of the device from the device capabilities data store.
• the device class may be a Palm handheld device of a particular model.
• the CCM may discern, by looking up the device class in the knowledgebase maintained in the data store, the capabilities of the device, such as display capability (e.g., 16-color), device memory (e.g., 4-8 MB), and display size (e.g., 300 x 500).
• the CCM also includes methods to log unidentified clients in the CCM log and to provide notifications at regular intervals to ensure that the knowledgebase is as up-to-date as possible. As new client devices are introduced, the configuration information in the knowledgebase can be updated to add information on these new client devices.
• the CCM supports either a "push” or a "push/pull” scheme for updating its configuration files.
• the "push” scheme consists of a secure HTTP POST request or SMTP message sent to the data store containing the replacement configuration file.
• the "push/pull” scheme consists of sending an HTTP GET request or SMTP message to the data store which causes the server to schedule an update of the local configuration files.
• the CCM relies primarily on HTTP headers, particularly the User- Agent header, to identify clients.
• the CCM also examines information in the protocol layer below HTTP (for example, the origin of the request's IP packets).
• the CCM consults a hierarchical configuration file (in the data store) which supplies default values for the device's capabilities and specifies additional headers, such as the Accept header, which can also be used to determine the proper output format for media content.
• the CCM constructs a request to the MTM for the specified media document including the proper reformatting information. The CCM then forwards the client connection to the MTM with the reconstructed request.
• the CCM may be implemented as an Apache module with access to the full HTTP request made by the client.
• the information contained in that request is used to identify the client device making the request through a series of queries against a configuration file.
• the HTTP User- Agent header is the primary indicator of the requesting client. While User- Agent is an optional header in both HTTP/1.0 and HTTP/1.1, in practice all Web client software send some identifying information in that header on each request. Many clients also send custom headers describing the physical capabilities of their devices. For example, the UP browser (Unwired Planet browser supplied by Openwave Systems, Inc. of Redwood City, CA), which is a WAP browser used in mobile phones, sends out "non-standard" heads in the request. "Non-standard" in this context means that such headers are not covered by the HTTP specification. An example of one of these headers is as follows:
• the "x-up-devcap-screenpixels" portion of the above header specifically indicates how many pixels in width and height (120 x 50) the client device is capable of displaying.
• the header shown in the above example contains several details about the capabilities of this particular client device. However, in many cases headers do not include all of these details, and thus the CCM must refer to information stored in the data store to obtain device characteristics. However, the CCM uses information in the headers like "x-up- devcap-screenpixels" portion of the above header whenever possible.
• the CCM configuration file (or knowledgebase) in the data store is written in XML to take advantage of that language's hierarchical features.
• the file consists of a series of ⁇ user-agent> entries.
• tag attribute naming patterns indicate that the values provided to those attributes will be treated as regular expressions.
• standard "match remember” syntax parentheses
• the pattern attribute matches the string "120, 50” and indicates that the substring "50” is to be used to set the enclosing capability value.
• the CCM When the CCM receives a request, it runs through the configuration file, checking each ⁇ user-agent> tag in turn by comparing the value of the HTTP header specified by the header attribute to the string specified in the value attribute. In the majority of cases, the ⁇ user-agent> tag is matched against the HTTP User- Agent header, and matching the tag against the HTTP User- Agent header is the default behavior if the header attribute is omitted.
• Each ⁇ user-agent> block is processed in the order it appears in the configuration file, so ⁇ user-agen > tags with more restrictive value attributes appear before ⁇ user- agent> tags with less-restrictive value attributes. For example, a ⁇ user-agent> tag which reads value- UP.
• Browser/3.1' is placed before a ⁇ user-agent> tag which reads value— UP. Browser'.
• the ⁇ device-class> element is used to separate the different clients into arbitrary groupings based on their capabilities. Some examples of ⁇ device-class> values might be "WAP", "i-mode”, “HDML”, and "j-phone. "
• the CCM determines the MIME type of the media document being requested, generally by issuing an HTTP HEAD request to the document source. Once the MIME type is known, the CCM looks up the appropriate ⁇ content-type> block inside the ⁇ user-agent> block. In the example above, any request for MIME types that start with "image/" will match the regular expression defined in the pattern attribute of the first ⁇ content-type> tag.
• An actual configuration file may have separate blocks for each supported media type or subtype.
• Client capabilities are defined inside the ⁇ content-type> tag by one or more ⁇ capability> tags. Each media type may require different capabilities. To properly display images, one needs to know display width, height, and color depth. To supply appropriate audio streams, one needs to know bandwidth and whether the device is capable of playing multiple channels.
• the configuration file provides previously-stored values for each capability through the mandatory default attribute. Lines 4 and 11 of the above example illustrate this case, setting the color depth and output format for all UP.Browser user agents to 8 bits per pixel and image/bmp, respectively. As described above, some user agents provide additional information to the server in the form of non-standard HTTP headers.
• the ⁇ header> tag can be used inside of a ⁇ capability> tag to instruct the CCM to parse these headers and set the device capabilities depending on the header values, h the example, line 6 indicates that the non-standard "x- up-devcap-screenpixels" header should be used, if present, to set the device display width by applying the regular expression provided in the pattern attribute to that header's value. Parentheses are used to isolate a part of the header value to assign to the capability.
• the underlying communication transport may also be inferred from the class or type of device.
• the system may infer that the underlying communication transport is wireless.
• the target device is a pager that is communicating using WAP, the system may infer that the target device uses wireless communication with limited bandwidth (as compared to a cellular phone).
• the system may usually discern whether the communication transport is wireless or wireline.
• very few devices have both wireless and wireline capability. Typical wireline connections include TI, DSL, cable modem and dial-up connections.
• typical connections include 9600-baud circuit-switched data call, 9600-baud packet-switched data call, or the newer 64K baud GPR call.
• the client capabilities module is also responsible for verifying the source of the original content so that the system can only be used to reformat content of authorized participating sites and not of third parties. Security is enforced by only activating the CCM in response to specific URLs containing the name of a designated server for which content is to be transformed.
• the CCM uses the information it derives about the capabilities of a particular client device to construct a request to the media transformation module for the requested media document. This CCM forwards this constructed request, including the proper reformatting information and the client connection to the MTM.
• the client capabilities module (optionally) implements a front-side cache for transformed media documents.
• This front-side cache is consulted for transformed document meeting the criteria of the constructed request before the request is forwarded to the MTM.
• the purpose of this front-side cache is to minimize the load on the MTM module.
• the media transformation (or transcoding) module accepts HTTP requests for media documents, which contain formatting instructions as request parameters, and reformats the media according to those instructions.
• the MTM may specialize in a single media type, such as image or video or may support multiple types of media.
• the media transformation module supports image reformatting by: converting images both to and from the following MIME types: image/jpg, image/bmp, image/gif, image/tiff, image/wbmp, image/iff, image/pcx, and image/png; decreasing image dimensions and increasing image dimensions; supporting rotation of images to conform to the aspect ratio of the client display; and supporting decreasing image color depth and increasing of image color depth.
• the MTM supports audio reformatting by: converting audio files and streams both to and from the following MIME types: audio/aiff, audio/au, audio/mpeg, audio/wav; and decreasing audio bit rate.
• the MTM supports video reformatting by converting video files and streams both to and from the following MIME types: video/mpeg, video/quicktime, video/x-msvideo (ANI), video/x-ms-asf, video/rm, and video/mjpeg.
• the MTM can also support reformatting of additional multimedia content types and streams as defined by RFC 2046,
• MIME Multipurpose Internet Mail Extensions
• Color space e.g., RGB or Grayscale
• Color palette e.g., True color or indexed
• the MTM may output a picture in the specified output format at the specified size.
• some devices may be characterized differently than their native characteristics. For example a device with a 16-color LCD display may prefer to be characterized as a true-color device. It is then the responsibility of the device to convert the true-color mode image transmitted by the MTM to its internal 16-color mode using internal software/hardware.
• Any suitable compression scheme may be employed, including proprietary or non-proprietary schemes. Examples include JPEG, JBIG, GIF, or the like. See, e.g., JPEG-like Image Compression (Parts 1 and 2), Dr. Dobb's Journal, July 1995 and August 1995 respectively (available on CD ROM as Dr. Dobb 's/CD Release 6 from Dr. Dobb's Journal of San Mateo, CA).
• the specific operations of the MTM in translating the above-mentioned JPEG image are as follows.
• the input picture is decompressed to generate a bitmap in the color space that was employed.
• Clikpix uses GUN color space; JPEG supports multiple color space, including YUV and RGB.
• GUV color space is described in commonly-owned application serial number 09/489,511, filed January 21, 2000; a description of industry-standard color spaces may also be found in that application.
• the picture is then converted to a "standard" intermediate format, typically in one of the industry-standard color spaces. Examples include:
• L,a,b 16bits/pixel/channel e.g., used in Adobe Photoshop
• SRGB 8bits/pixel/channel e.g., used by Microsoft, HP, and others
• a monochrome version of the image is generated using standard conversion methods (e.g., using International Telecommunication Union (ITU) recommendations for generating luminance signal Y from R,G,B signals, see, e.g.: ITU-Recommendation BT.601- 1 Encoding parameters of Digital Television for studio).
• ITU International Telecommunication Union
• bits/pixel is fewer than 8 - then dithering techniques (such as error diffusion, blue noise masking, or the like - see, e.g., Recent Progress in Digital Halftoning, 1994, The Society of Imaging Science and Technology, compiled and edited by Reiner Eschbach, ISBN 0-892080-181-3).
• dithering techniques such as error diffusion, blue noise masking, or the like - see, e.g., Recent Progress in Digital Halftoning, 1994, The Society of Imaging Science and Technology, compiled and edited by Reiner Eschbach, ISBN 0-892080-181-3).
• color dithering schemes are used. Such schemes are discussed in some detail in Computer Graphics-Principles and Practice, Second Edition, 1990, Foley, van Dam, et al., Addison-Wesley, Reading, MA, ISBN 0-201-12110-7.
• Compression is optionally performed before data is streamed out.
• the preferred compression scheme is JPEG.
• PNG are the preferred methods.
• the preferred approach is JBIG compression.
• the generated picture is outputted, and is ready for streaming to a target device for ultimate rendering on that device's display.
• the above AutoRotateOp function automatically rotates an image to better fit the available display of a particular device.
• the function receives a pointer to an image as a parameter.
• the display characteristics of the device are obtained.
• the condition on lines 38 to 39 evaluates whether or not the image should be presented in portrait orientation (i.e., to display the image vertically) or landscape orientation (i.e., to display the image horizontally) to better fit the device display.
• a call is made to IMG RotateRight or IMG RotateLeft as shown on lines 41 to 44 and the image is rotated either clockwise or counterclockwise to fit the display of a particular device.
• Source code listings providing further details on the BVIG RotateRight and IMG RotateLeft functions are attached hereto as Appendix A.
• the MTM uses cached versions of the original media objects whenever possible.
• the MTM caches source media objects in the backside cache to reduce the time required to read the source media and to reduce Internet bandwidth consumption by the media delivery system.
• This backside caching can be performed by the MTM or by an intermediate reverse proxy cache.
• the source objects are cached according to the directives specified in their HTTP cache-control headers.
• an Apache HTTP server configured as a reverse proxy cache is deployed "between" the Internet and the MTM module, so any requests for source multimedia documents are first compared to a local disk cache.
• the Apache reverse proxy cache module decouples the caching task from the media transformation task for better scalability and maintainability.
• IMGLIB_API void IMG_RotateLef t (IMG_image *img) if (img- >pixelBytes 1 4 ) DEBUG_IMG_FORMAT; return; ⁇
• IMG_image templmg if ( !IMG_AllocateImage(&tempImg, newWidth, newHeight, img- >imgFormat) ) return; //lMG_StretchBlit (img, &templmg, 0, 0, newWidth, newHeight, true); if (highQuality) VImage vlmg; vlmg. Import (img) ; vlmg. Scale (newWidth, newHeight, CImageServer: :CUBIC) ; vlmg.Export (&templmg) ; else
• IMG_StretchBlit (img, &templmg, 0, 0, newWidth, newHeight, false) ;
• IMG_CopyTags (img, &templmg) ; IMG_FreeImage (img) ;
• IMG_Resize (img, width, height, highQuality);
• Deviceldentifier :DeviceIdentifier ()
• ⁇ config new XMLConfigFile ("/lsurf/uts/conf/devices .xml") ;
• the device capabilities data storage is implemented as an XML file.
• devcap->maximum_file_size getCapabilitylnt (r, base_xpath_query, “maximum-size”)
• devcap->video_frame_rate getCapabilitylnt (r, base_xpath_query, "video-frame-rate”)
• devcap->audio_encoding_rate getCapabilitylnt (r, base_xpath_query, "audio-encoding-rate”)
• devcap->audio_channels getCapabilitylnt (r, base_xpath_query, "audio-channels”); return DI_SUCCESS; // utility functions to print out the DeviceCapabilities object void Deviceldentifier: :DeviceCapabilities: :print (ostream *os) ⁇
• ostream &operator ⁇ (ostream &os, Deviceldentifier : :DeviceCapabilities kdevcap) ⁇ devcap.print (&os) ; return os;

Landscapes

• Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Business, Economics & Management (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Tourism & Hospitality (AREA)
• Computer Security & Cryptography (AREA)
• Signal Processing (AREA)
• Computer Networks & Wireless Communication (AREA)
• Health & Medical Sciences (AREA)
• Economics (AREA)
• General Health & Medical Sciences (AREA)
• Human Resources & Organizations (AREA)
• Marketing (AREA)
• Primary Health Care (AREA)
• Strategic Management (AREA)
• General Business, Economics & Management (AREA)
• Information Transfer Between Computers (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=21746556

Family Applications (1)

Country Status (7)

Cited By (21)

Families Citing this family (170)

Family Cites Families (49)

• 2001
  • 2001-11-08 US US10/010,616 patent/US20030110234A1/en not_active Abandoned
• 2002
  • 2002-11-07 JP JP2003542457A patent/JP2005527881A/ja active Pending
  • 2002-11-07 EP EP02782286A patent/EP1451662A2/en not_active Withdrawn
  • 2002-11-07 CA CA002466179A patent/CA2466179A1/en not_active Abandoned
  • 2002-11-07 WO PCT/US2002/036064 patent/WO2003040893A2/en not_active Ceased
  • 2002-11-07 KR KR1020047007024A patent/KR20050044379A/ko not_active Withdrawn
  • 2002-11-07 AU AU2002348362A patent/AU2002348362A1/en not_active Abandoned

Also Published As

Similar Documents

Legal Events