WO2000042519A1 - Systeme d'acceleration de la livraison d'un contenu sur l'internet - Google Patents

Systeme d'acceleration de la livraison d'un contenu sur l'internet Download PDF

Info

Publication number
WO2000042519A1
WO2000042519A1 PCT/US2000/000587 US0000587W WO0042519A1 WO 2000042519 A1 WO2000042519 A1 WO 2000042519A1 US 0000587 W US0000587 W US 0000587W WO 0042519 A1 WO0042519 A1 WO 0042519A1
Authority
WO
WIPO (PCT)
Prior art keywords
data files
proxy server
local
data
extracted data
Prior art date
Application number
PCT/US2000/000587
Other languages
English (en)
Other versions
WO2000042519A8 (fr
Inventor
Kyle Powell
Mark Hurst
Karl Merkley
Original Assignee
Edgix Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Edgix Corporation filed Critical Edgix Corporation
Priority to AU28477/00A priority Critical patent/AU2847700A/en
Publication of WO2000042519A1 publication Critical patent/WO2000042519A1/fr
Publication of WO2000042519A8 publication Critical patent/WO2000042519A8/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching

Definitions

  • the present invention relates to Internet acceleration systems. More specifically, the present invention relates to Internet Acceleration systems involving local and remote caching to speed up distribution of content on the Internet.
  • the Internet content delivery acceleration system of the present invention has been developed in response to the present state of the art, and in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available Internet content delivery acceleration systems. Accordingly, it is an overall object of the present invention to provide a system that overcomes many or all of the above-discussed shortcomings in the art.
  • an improved Internet content delivery acceleration system is provided.
  • a central proxy server or caching system, gathers high demand Internet data and disseminates that data to local proxy servers.
  • the dissemination of data may be broadcast, multicast, reliable multicast, unicast, and reliable point to point transport on any suitable medium over any suitable medium, including over the electromagnetic waves, fiber optics itself and over Satellite.
  • the transmission comprises multicast distribution, such as IP multicast.
  • local content filling of the local proxy servers is provided by transmitting Internet data to all subscribing local proxy servers at a high rate of speed. Through the use of multicast protocol, only subscribing or interested local proxy servers receive the transmission.
  • the local proxy servers utilize a locally customizable heuristics determination to determine whether to keep or discard the data.
  • the heuristics determination preferably employs global popularity data, and may also comprise customized local standards or information relevant to the particular customers subscribing to the local proxy server.
  • the local proxy servers are preferably provided with software that enables them to receive the data at the high rate of speed and to store the data in attendant local cache databases for quickly servicing future Internet data requests.
  • the software comprises a cache database management module configured to communicate directly with a caching database.
  • the local proxy server software preferably also comprises a local caching module in integral communication with the cache database management module.
  • the integral communication comprises direct communication between programs on a common server or other computer, and may be facilitated by an applications program interface (API). Because of the tight integration between the cache database management module and the local caching module, sophisticated heuristics determinations may be employed.
  • API applications program interface
  • customized hit information for all transmitted data files and even data files that have not been transmitted may also received from the caching databases and transmitted to the central proxy server for analysis and use in determining which data files to obtain and forward to the local proxy servers. Additional determinations may be made regarding characteristics such as the timing of demand for data files, together with the nature of the data file for customized determinations of demand within the individual local proxy servers.
  • the request is first sent to the local cache database management module to determine whether the requested data is present. If the data is present, it is rapidly transmitted to the user from the local cache database, preferably avoiding any transmission over the Internet. If the data is not present, a request is transmitted to the central proxy server which requests a copy from the hosting site.
  • the system may also transmit selected data to subscribing local intranet sites to rapidly and systematically publish private data to local proxy servers connected to the local intranets.
  • Figure 1 is a schematic block diagram illustrating one embodiment of an Internet content delivery acceleration system of the present invention, including a central proxy server and a plurality of remote proxy servers.
  • Figure 2 is a schematic block diagram illustrating a further embodiment of an Internet content delivery acceleration system of the present invention, including private intranet facilities.
  • Figure 3 is a schematic block diagram illustrating functional components of one embodiment of software used with the systems of Figures 1 and 2.
  • Figure 4 is a schematic block diagram illustrating a packet for satellite pointcast to local private intranets.
  • Figure 5 is a schematic block diagram illustrating the functional components of one embodiment of a heuristics procedure of the present invention.
  • Figure 6 is a schematic flow chart diagram illustrating one manner of operation of one embodiment of software for use on the remote proxy server of Figure 1.
  • Figure 7 is a schematic flow chart diagram illustrating one manner of operation of one embodiment of software for use with the central proxy server of Figure 1.
  • Figure 8 is a schematic block diagram illustrating one embodiment of a central proxy server of the present invention.
  • Figure 9 is an illustration of the structure of one embodiment of a communications packet suitable for use in conjunction with the present invention.
  • Figure 10 is a schematic block diagram of one embodiment of a local proxy server of the present invention.
  • the central proxy server 12 comprises a network server 11, such as a personal computer (PC) operating under a network protocol such as the IPX protocol.
  • a network server 11 such as a personal computer (PC) operating under a network protocol such as the IPX protocol.
  • the master cache database 14 stores frequently used data files 15 extracted from the Internet by the central proxy server 12.
  • the data files 15 in the embodiment of Figure 1 comprises Internet web pages 15a.
  • the data files 15 may comprise any form of Internet content, including HTML files, video and music files, and any other digitally represented information that is transmitted across the Internet.
  • a central caching module 13 preferably operates within the central proxy server 12. Also included in the depicted embodiment of the system 10 is a plurality of local proxy servers 22, each having a local caching module 17 disposed therein. Each local proxy server 22 also comprises or is otherwise in communication with a local cache database 24.
  • the local proxy servers 22 in one embodiment are high-performance, high-capacity PC- based servers. In one embodiment, the proxy servers operate under an IPX protocol, such as Novell's NetwareTM. Facilities in which the local proxy servers 22 may be installed include Internet service providers (ISPs), large and medium businesses, and eventually, as economies of scale make the service highly affordable, small businesses and homes.
  • the local proxy servers 22 are each provided with a local cache database 24. Preferably, the local cache databases 24 are provided with high capacity memory. In one embodiment, the cache databases 24 are provided with hard drive memory in excess of 40
  • a caching database management module 29 is provided to interface with the local cache database 24.
  • the caching database management module comprises an Internet caching system (ICS) program 29, such as Novell Corporation's Border ManagerTM and/or ICSTM software.
  • ICS Internet caching system
  • the local proxy servers are interfaced in integral communication with the caching database management modules, as will be described in greater detail below.
  • the local proxy servers 22 preferably provide users at end stations 30 with access to a global communications network, such as the Worldwide Web existing on the Internet 20. This makes them ideally suited for installation at locations such as ISPs, telephone system switching backbones, or points of presence (POPs), and within networks of large companies, such as fortune 500 companies.
  • a global communications network such as the Worldwide Web existing on the Internet 20.
  • ISPs Internet Protocol
  • POPs points of presence
  • an end user at a station 30 may be connected to a local proxy server 22 remotely over a modem, or locally over a network 44.
  • the local proxy servers 22 can be considered to be at the "edge" of the Internet 20, a position allowing the present system 10 to reduce traffic over the conventional transmission routes of the Internet 20.
  • the local proxy server servers 22 receive and store high-demand data files 15 in the attendant local cache databases 24.
  • the network data comprises web pages 15a emanating from a remote Internet site 35.
  • the web pages 15a are initially gathered by the central proxy server 12 from Internet sites 35 over a communication channel 36 and are subsequently transmitted from the central proxy server
  • this dissemination of information may include the traditional notions of broadcast, as well as multicast, reliable multicast, unicast, and reliable point to point transport on any suitable medium including over electromagnetic waves, electrical cables, fiber optics and satellite.
  • the communication medium comprises transmission by a satellite 18 at a high rate of speed enabled by efficient hard drive data receipt and storage methods encompassed within the present invention. In one embodiment, this rate of speed is about 25 Mbps.
  • a preferred manner of transmission is IP multicast.
  • the web pages 15a are preferably concurrently transmitted to each subscribing local proxy server 22 once any of the local proxy servers 22 requests the web pages 15a.
  • Each local proxy server 22 receives the web pages 15a and stores them in its local cache database 24 for a selected amount of time before considering whether to purge the particular web pages 15a.
  • the selected amount of time is scaled to the amount of memory residing in the particular local cache database 22.
  • the local proxy servers 22 in one embodiment utilize proxy caching protocol built into the http Internet language for proxy servers. With this protocol, every time a user at a station 30 requests a web page 15a over the network of which the local proxy server 22 is a part, the request passes through the local proxy server 22. As part of the protocol, the local proxy server 22 initially checks its attendant local cache database 24 to determine whether the web page 15a is present. In one embodiment, this consists of the local caching module 17 checking through the integral communication interface with the cache database management program 29. If the web page 15a is present, having been transmitted by satellite 18 from the central proxy server 12, the local proxy server 22 immediately transmits the web page 15a through the network 44 to the user at a station 30.
  • proxy caching protocol built into the http Internet language for proxy servers.
  • the request is transparent, and to the user at a station 30, it appears as if the web page 15a was transmitted over the Internet 20.
  • the transmission is much more rapid. Additionally, Internet traffic is reduced, and Internet connection costs are potentially much lower.
  • the http caching protocol causes the request to be passed on through the Internet 20 to the Internet site 35 at which the web page 15a is hosted. The Internet site 35 then transmits the web page 15a over the Internet 20 back to the requesting user at a station 30. This Internet transmission occurs over regular Internet communication channels 38, 40, 42.
  • the request is also passed to the central proxy server 12.
  • the central proxy server 12 receives its copy of the request, it also requests a copy of the requested web page 15a from the hosting Internet site 35. Accordingly, the web page 15a is also transmitted over channels 36, 37 to the central proxy server 12.
  • the central proxy server 12 caches the web page 15a within the attendant master cache database 14. It may also transmit the web page 15a to the communicating local proxy servers through the communication medium if the web page 15a is found to have a sufficiently high priority.
  • This transmission in one embodiment is satellite transmission.
  • the web page 15a is transmitted 32 through a satellite uplink transmitter 16 to the satellite 18 at a speed of, for example, 25 Mega bytes per second (Mbps).
  • the satellite 18 then transmits 34 the web page 15a to each of the subscribing local proxy servers 22, at a rate preferably of about 25 Mpbs.
  • the central proxy server 12 may filter the web page 15a before caching it. It may also keep track of the number of requests and transmit web pages 15a (or other data 15) only after a certain minimum number of requests have been logged for that web page 15a.
  • the central proxy server 12 is in a position to be a "Nelson Ratings System" for the Internet 20, having centralized access to web site demand information.
  • the central proxy server 12 preferably receives global popularity information about data files 15 from subscribing local proxy servers 22.
  • This popularity information in a basic embodiment comprises miss data. Additionally, due to the unique configuration of the present invention, more sophisticated information may also be transmitted. This may include, for instance, hit information and timing information, as will be discussed in greater detail below.
  • the central proxy server may also utilize data 15 popularity information, data 15 characteristics, data associations, and other data useful to local proxy servers 22 in deciding which extracted data 15 to continue to store. Such information may be collected from the local proxy servers 22, and in addition, may also be received from companion data extractor servers 35 communicating with the central proxy server 12.
  • the data extractor servers 35 preferably locate information relevant to the heuristics determinations of the local proxy servers 22 and pass that information to the central proxy server for compilation and later transmission to the local proxy servers 22 and optionally, to other requesting entities, possibly for a subscription fee. Manners in which this information may be gleaned include directly contacting web sites that contain hit rates or other popularity information. Additionally, the data extractor servers 35 may themselves traverse or "crawl" over the web to examine web sites and observe traffic. When examining web sites, the data extractor servers may follow links on those web sites to other web sites to similarly examine the other web sites. This process may be continuous, and the information that is gathered is preferably passed to the central proxy server, as stated.
  • Additional information that may be collected by the central proxy server 12 includes the qualitative information regarding the data files 15, rather than just quantitative data.
  • the types of sites or data 15 may be collected, either from the local proxy servers 22 or from the data extractor servers 35.
  • This data may include, for instance, what the subject matter of the data files 15 is, what types of sites are requesting the data files 15, etc.
  • This information may then be used by the local proxy servers 22 in making a customized heuristics determination according to local demand and local user types.
  • a single central proxy server services the entire system
  • central proxy server 12 may be in operation.
  • different geographical locations may be served by different central proxy servers 12.
  • These multiple central proxy servers 12 may be in communication with each other either through suitable mediums and may share information such as hit and miss rate information and other heuristics-type information.
  • Each local proxy server 22 is preferably provided with a communication facility, such as a satellite down-link receiver 26.
  • the down-link receiver 26 receives the satellite multicast 34 of the extracted web pages 15a.
  • each subscribing local proxy server 22 caches the web page 15a for a selected period of time for access by users at the communicating stations 30.
  • each local proxy server 22 preferably utilizes an individualized local policy and a heuristics program to determine whether to keep the transmitted web page 15a or discard the web page 15a.
  • the central proxy server 12 may also service one or more virtual private intranets 50 through selective pointcast satellite transmission 74.
  • the system 10 preferably operates in substantially the same manner as described above. Accordingly, the system 10 is provided with the central proxy server 12 and a series of local proxy servers 22 configured substantially in the manner described.
  • the system 10 may also be provided with the private intranet 50.
  • the private intranet 50 is shown communicating over the Internet 20 through an intranet proxy server 60.
  • the intranet proxy server 60 may also function as a local proxy server 22, as discussed above.
  • the private intranet 50 connects to end users at remote stations 52 through a communication channel 54.
  • the private intranet 50 is connected with the intranet proxy server 60 through a communication channel 68.
  • the Intranet proxy server is in communication with a satellite down-link receiver 62 through a communication channel 66 and with the Internet 20 with a communications channel 64.
  • One or more of the aforementioned channels may maintain security with a firewall 78.
  • the system 10 may also be provided with a private intranet publisher 70 for generating or otherwise providing private data 55 to be communicated to users at stations 52 within the private intranet 50.
  • the private intranet publisher 70 is a dedicated server computer communicating over the Internet 20 through a secure channel 72.
  • the secure channel 72 is shown provided with a firewall 78.
  • the private intranet publisher 50 generates or collects the private data 55, which may be business, financial, or any other type of data, and transmits the private data 55 in web page form.
  • the private data 55 is passed in a secure manner through the Internet 20 to the central proxy server 12.
  • the central proxy server 12 through a satellite uplink transmitter 16, then uplinks 32 the private data 55 to the satellite 18.
  • the satellite 18 then transmits the private data 55 to intranet proxy servers 60 of the subscribing virtual private intranet using a focused multicast 74.
  • the same satellite 18 may also transmit 34 other data such as web pages 15a to the regular local proxy servers 22.
  • All private intranet transmission over the Internet 20 is preferably kept within a firewall 78, preferably through hashing or other encryption of the data being transmitted.
  • the multicast transmission 74 could be transmission by a private frequency, or more preferably, through the same frequency and channels as the normal web site multicasts 34, but with a header placed on each transmitted packet identifying the transmission as private and identifying the designee.
  • encryption may be employed, and the remote intranets intended to receive the data may be provided with encryption keys.
  • a master key is preferably retained by the central proxy server 12.
  • Local proxy servers 22 for which the data 55 is not intended thus ignore the data 55.
  • the header could also particularly specify whether the data is private data 55 or global data 15.
  • the Virtual Private Intranet of the present invention unlike prior art broadcast intranets, is one-way, and the private data 55 to be transmitted is collected or otherwise designated by the central publisher 70.
  • the publisher 70 collects private data 55 as a result of requests from remote intranet sites, possibly over the Internet 20.
  • the private data 55 is, for example, sales and pricing data.
  • Financial advising companies may also use the system and pointcast 74 data 55 in the form of market data or financial analysis in another example. Banks may make financial transactions with the system 10. Business might transmit sales information, company policies, advertising materials, etc.
  • the private data 55 is transparent, once pointcast 74 to the users 52.
  • the data 15 may be viewed using file transfer protocol of the local network, rather than as an Internet URL site.
  • the users 52 though possibly located across the world from each other, may access the private data 55 from the proxy server as if it were a local part of the remote intranet 50.
  • the private data 55 may be transmitted on demand, but it is contemplated that at least a substantial portion of the private data 55 may be transmitted through the discretion of the Internet Publisher 70, or may comprise standard information, periodically updated and transmitted.
  • FIG. 3 shown therein is a schematic block diagram illustrating more specifically the various functional components of one embodiment of software used to control the system 10 of the present invention.
  • the software modules contained in the schematic block diagram of Figure 3 are generally implemented as software instructions, procedures, routines, or other executable software code. Nevertheless, the modules could also be implemented with other types of programmable logic such as programmable logic arrays (PLAs), ASICs, or even logic circuits or other types of hardware.
  • PDAs programmable logic arrays
  • ASICs application specific integrated circuits
  • a system initialization block 80 In order to initialize the system 10 (of Figures 1 and 2) and enable communication links, the depicted components of a system initialization block 80 is employed.
  • the modules of the system enablement block 80 may be contained as software code within the central proxy server 12, or may be distributed amongst the different hardware components of the system 10.
  • the central proxy server 12 is initialized and brought on-line with a central server enablement module 82. This preferably includes enabling Internet communication over communication channel 36 and enabling satellite communication with the satellite uplink transmitter 16.
  • the local proxy servers 22 are enabled and brought on-line with a local proxy server enablement module 84.
  • Internet communications are enabled through communication channels 38, 40, and 42.
  • the various local proxy servers 22 will generally not all be brought on-line at one time.
  • the system 10 of the present invention will be provided commercially as a service to which customers subscribe. As new customers subscribe, they are provided with the local proxy server 22 and satellite receiver hardware 26 for an initial fee. A monthly subscription fee may be charged thereafter.
  • Satellite communications are enabled with a satellite initialization module 86. This may include establishing the communications link between the satellite 18 and the satellite uplink transmitter 16 and between the satellite 18 and the satellite receivers 26, 62. It may also include establishing a proper protocol for satellite communications.
  • satellite communications for the pointcast transmission must be established with one or more Internet proxy servers 60.
  • the intent of the system 10 is to fill the local cache databases 24 of each subscribing local proxy server 22 with the most highly requested data such that the majority of requests for web pages 15a may be serviced directly by the local proxy servers without having to go to the Internet 20. Accordingly, each new central proxy server that is brought on-line must be filled with the most highly requested web pages 15 a.
  • the central proxy server 12 transmits "old" data that has been previously transmitted in between transmitting "fresh" data that has only recently been requested. This transmission may be conducted in bursts to transmit the entire contents of the master cache database systematically, or according to a selected heuristic formula.
  • a user station block 90 of Figure 3 illustrates the basic functional components and operation of a user station 30 of Figure 1.
  • the data 15 is a web page 15a and the data request module comprises a standard web browser 25 which is connected to the Internet through a local proxy server 22.
  • the data 15 is transparently received from the local proxy server 22 through a data reception module 94.
  • the data reception module 94 may comprise the web browser 25 in conjunction with the http proxy caching protocol which searches local cache databases 24 prior to transmitting a request for
  • the data display module 96 may comprise the web browser 25 together with the proxy caching protocol of the http language operating on a PC or other computer.
  • a central server block 100 illustrates the basic functional components operating within the central proxy server 12 of Figure 1. These components are preferably part of a central caching module 13 of Figure 1.
  • a data request reception module 102 allows the central proxy server 12 to receive the request for data made by the user at a station 30 with the data request module 92.
  • An Internet request module 104 passes the request on through the Internet 20.
  • Data reception module 106 receives the data 15 transmitted by the Internet 20 in response.
  • a cache storage module 108 receives the requested data 15 and stores a copy of the data 15 within the master cache database 14.
  • An uplink transmission control module 109 coordinates with the communication network to distribute selected data 15 to the local proxy servers 22.
  • the uplink transmission control module 109 communicates with the satellite transmitter 16 in transmitting data 15 through the uplink 32 to the satellite 18.
  • other types of transmission could be utilized, including over the Internet itself.
  • a satellite operation block 110 illustrates the basic operation of functional modules operating within the satellite 18 of Figures 1 and 2.
  • an uplink data reception module 112 receives the uplinked web page 15a or other data 15.
  • a data broadcast module 114 formats and transmits the data 15 through satellite multicast 34 to all associated local proxy servers 22.
  • a pointcast data module 116 formats and pointcasts the private data 55.
  • An uplink transmitter block 120 of Figure 3 illustrates general functional modules operating within the satellite uplink transmitter 16 of Figures 1 and 2.
  • the data 15 to be uplinked is formatted into a proper form and protocol for satellite transmission with a data formatting module 124. Header information is added to the packets to be uplinked with a header addition module 124.
  • One simplified example of the structure of a packet 190 and header 192 is shown in Figure 4.
  • the uplink transmission is conducted between the satellite uplink transmitter and the satellite 18 with the use of an uplink transmission module 126.
  • An Internet block 130 illustrates one example of the functional modules operating within the Internet 20 of Figures 1 and 2.
  • Requests for data 15 are transmitted over the Internet 20 with a data request module 132. The request is typically the request generated by the Internet request module 104.
  • the Internet site 35 at which the data 15 is hosted is contacted with a site contacting module 134.
  • a site map of the site, including the location and configuration of the web page 15a is transmitted from the web site
  • a local server block 150 illustrates one embodiment of the functional components operating within the local proxy server 22 of Figures 1 and 2. Preferably, these functional components are contained within the local caching module 17 of Figure 1.
  • a data request block 155 includes a data reception module 152.
  • the data reception module 152 receives requests for Internet data 15 from the end users at the stations 30. Generally, this request is generated by the data request module 92.
  • a search module 154 searches the local cache database 24 for the requested data 15. In one embodiment, the search module 154 comprises the proxy cache protocol employed within the http language.
  • a data request module 156 transmits a request for the data 15 to the central proxy server 12. This request is typically routed over one of the communication channels 38, 40, or 42, through the Internet
  • the request may comprise the original request received from the user at a station 30 modified with the Internet URL of the central proxy server 12.
  • the data request module 156 may also request the data 15 directly from the Internet site 35 where the data 15 is hosted.
  • a data reception module receives the requested data 15.
  • the data 15 is provided by the central proxy server 12 in accordance with the procedure discussed for the central server module 100.
  • the data 15 is uplinked 32 to the satellite 18, which multicasts the data 15 to all subscribing local proxy servers 22.
  • the data 15 is received by the satellite receiver 26 and passed on to the local proxy server 22.
  • the local proxy server 22 may also receive the data 15 directly over the Internet.
  • the data 15 from the earliest arriving of the multicast 18 and the direct Internet transmission is then passed on to the user at the station 30.
  • a data management block 160 illustrates the functional components of the local proxy servers 22 which handle the data 15 received from the satellite transmission 34.
  • a data supervisor module 164 operating within each of the local proxy servers 22 attends to the storage of the data 15 within the local cache database 24.
  • the data 15 is automatically stored for a predetermined amount of time.
  • the predetermined amount of time may be, for instance, a period of four hours.
  • a heuristics formula application module 166 employs a heuristic determination to determine whether to continue to store the data 15 or to discard the data. If, after application of the heuristic determination, it is decided that the likelihood of a request for the data 15 is low for that particular local proxy server 22, a push module 168 will discard the data 15, removing it from the local cache database 24.
  • each local proxy server 22 is provided with its own, individualized, heuristics formula.
  • the modules 158, 160 integrally communicate with acache database management module such as Novell's ICSTM software, which in turn attends to the storage of the data 15 onto disk at the high rate of speeds specified, and/or Novell's Border ManagerTM software which acts as a proxy cache manager and provides proxy content filling and firewall functions.
  • acache database management module such as Novell's ICSTM software, which in turn attends to the storage of the data 15 onto disk at the high rate of speeds specified
  • Novell's Border ManagerTM software which acts as a proxy cache manager and provides proxy content filling and firewall functions.
  • a heuristic determination block 170 illustrates the functional components of one embodiment of a heuristic formula of the present invention.
  • a local statistics collection module 172 resident within each local proxy server 22 monitors the frequency of requests for each file of data 15 located within the local cache database 24.
  • a global statistics reception module 174 receives data demand statistics from a central location.
  • the central location is the central proxy server 12, which collects demand statistics during the process of servicing requests for data 15.
  • the central proxy server 12 acts as a sort of "Nielson Ratings Service" for the Internet. This information may be provided either free or for a fee to interested parties.
  • a heuristic determination module 176 utilizes the statistical information gathered by modules 172 and 174 in making the heuristic determination of whether to keep a particular file of data 15. If the determination is to keep the data 15, a least recently used (LRU) module may be used to decide which existing data is to be discarded to make room for the new data 15. The LRU calculation module 178 calculates the amount of use of each file of data 15 and adds a local use factor into the determination. This heuristic determination is applied by the heuristics application module 166 as described above. The heuristics determination module 176 preferably utilizes localized web page demand statistics compiled by the local proxy server 22 as well as globalized web page demand statistics collected by and transmitted periodically from the central proxy server through satellite transmission.
  • LRU least recently used
  • a hit rate reporting module 179 periodically reports the local hit rates for part or all of the data 15 transmitted from the central proxy server 12. This information is used to calculate global demand statistics.
  • each participating local proxy server 22 tabulates every request from every end user station 30.
  • a hit report is then periodically transmitted to the central proxy server 12.
  • the central proxy server 12 in turn tabulates the accumulated hit reports for all web pages requested and periodically sends updated usage information with the hit totals to the local proxy servers 22.
  • every file of data 15 is assigned a globally unique number which can be stored and transmitted compactly and which is used to identify the particular data 15.
  • the heuristic determination module 170 is closely integrated with the data management module 160.
  • the data management module 160 is implemented through tight integration with the cache database management module program 29 and the heuristic determination module is configured to work closely with the cache database management module 29.
  • the data management module 160 keeps track of related files within a web page 15a or files related to a web page.
  • the related files may be uplinked and transmitted through satellite transmission 34 together.
  • the heuristic determination module 170 being integrated with the data management module 160, may be configured to receive a list of the objects and files associated with a page. In this manner, the data heuristic determination module 170 may able to ensure that the related data stays together and is stored together in the local cache database 24. It can also calculate usage information for each of the objects and files separately and keep all associated files or "trim" a portion of the objects and files that are not highly used.
  • the related files may then be transmitted as a group when one or more of the related files is requested by a user at a communicating station 30.
  • the heuristics module 170 may also track "super grouping" of web page information 15. In many web pages, information such as advertisements change or alternate back and forth. These changes may be kept track of and the differing versions transmitted together by the central proxy server 12.
  • each local proxy server 22 may also have its individualized local policy 204.
  • a relative weighting 206 of local and global web page demand is employed by the heuristics formula 200 to determine whether to keep or discard each separate data file 15 that is received, once the selected period of time has passed.
  • the local proxy server policies 204 may also weight subject matter of the web pages 15a.
  • the policy 204 may also include other local value factors, such as the ease of getting the data 15 for the particular local proxy server 22.
  • the policy 204 may also specify the particular interest areas that the heuristics formula is intended to weight. For instance, if the local proxy server 22 services a school, the policy 204 may give high weight to data 15 containing educational issues. If the local proxy server 22 services a research institution, the policy 204 may give high weight to data 15 relating to the research being conducted highly, etc.
  • a global demand monitoring module 182 resident within the central proxy server 12 monitors all of the requests for information from each local proxy server 22. It also monitors all of the hit rate data transmitted by the reporting modules 179 of each local proxy server 22. In response, a global demand compilation module compiles the data collected by the global demand monitoring module 182. Within those statistics may be information relevant to each particular subject matter such that the heuristics procedures may take the categorized demand data into account according to the particular weighting of the subject categories in their local policies 204.
  • a particular subscribing local proxy server 22 might be an educational institution, business, or news provider, and might weight subject matter statistics from relevant categories more heavily within its own local policy 204 than others of the local proxy servers 22.
  • a global demand transmission module 186 transmits the demand data, preferably by satellite transmission 34, in the same manner as the requested data 15 is transmitted.
  • Figure 6 is a schematic flow chart diagram illustrating one manner of operation of a software program 220, a copy of which preferably operates on each subscribing local proxy server 22.
  • the software program 220 comprises the local caching module 17.
  • the program 220 initializes at a start block 222.
  • the program 220 then proceeds on to a block 224 where it awaits receipt of input to be processed. If the input is a terminate command, the program 220 receives the command at a block 226 and then progresses to an end block 228, where the program 220 terminates.
  • FIG. 6 Three representative procedures are shown in Figure 6, corresponding to several different general functions that may be accomplished by the program 220. For instance, if the program 220 receives a user request for data 15, the program receives the request at a block 230. The program 220 then proceeds to a block 232, where the caching protocol of the http language is utilized to consult the local cache database 24 and thereby determine if the requested data 15 is present. At a query block 234, the program 220 branches in one of two directions depending on whether the data 15 is present in the local cache database 24. If the data is present, the program proceeds to a block 236 where it transmits the data 15 to the requesting station 30 for presentation to the user. The program then proceeds to a block 238, which returns control of the program to the start block 222.
  • the program 220 proceeds from the query block 234 to a block 240, where it passes the request on through the Internet 20 to the hosting site 35. At about the same time, the program 220, at a block 242, transmits a copy of the request to the central proxy server 12. The program 220 then waits at a block 244 for the requested data 15 to return by way of the fastest route, whether it be through satellite transmission 34 or through the Internet 20. Once the data 15 is received, the program 220 transmits the data 15 to the requesting user station 30 for presentation to the user. Thereafter, the program 220 moves to a block 247 where the program 220 updates local demand statistics. Subsequently, the program moves to a block 248 which passes control back to the receive input block 224. If the input received by the receive input block 224 is a the transmission of web pages
  • the program moves to a block 250, where the transmitted data 15 is received.
  • the program 220 then proceeds to a block 252 where it stores the received data 15 in the local cache database 24 for a selected amount of time. After the selected amount of time has passed, the program 220 generates heuristic data utilizing the heuristic formula 200 of Figure 5, and makes a determination of whether to retain or discard the data 15.
  • the program 220 progresses to a block 258 which returns control to the start block 222.
  • the input received by the receive input block 225 is global demand statistics from the central proxy server 12
  • the program 220 progresses to a block 260 where the global demand statistics are received.
  • the local record of global demand statistics is updated.
  • the local use statistics generated at block 247 are consulted and obtained.
  • the local use statistics are compiled together with the global demand statistics at a block 266.
  • the heuristic procedure 200 of Figure 5 is then employed at a block
  • the program 220 then proceeds to a block 274 which passes control back to the receive input block 224.
  • FIG 7 is a flow chart diagram illustrating one manner of operation of a software program 300 resident within and operating on the central proxy server 12 of Figures 1 and 2.
  • the program 300 initializes at a start bock 302 and proceeds to a block 304, where it awaits receipt of a request for data 15.
  • the program 300 could have other means of identifying high demand data 15. Web site search engines or other pre-generated statistical data could be consulted, for instance.
  • the central proxy server 12 could also keep a history of data requests for web pages 15 a, which are updated daily or weekly, etc. in order to re- transmit those pages 15a once updated.
  • the program 300 proceeds to a block 305.
  • the program 300 requests a copy of the data 15 from the hosting Internet site 35.
  • the data 15 is received over the Internet 20 from the hosting site 35.
  • the data 15 may then be filtered before being transmitted and/or stored within the master cache database 312.
  • a morality filter could be used to filter out obscene language or pornographic materials.
  • Other types of filters could also be used by content, or filters could be used to filter out data for which it is known that hit rates in the future will be low.
  • a heuristic scheme or some type of request history might be employed for so doing.
  • the data 15 which has successfully passed through filtering is stored in the master cache database 14 as represented at a block 210.
  • the data 15 is then sent to the broadcast uplink transmitter 16 as represented by a block 312.
  • the communication medium may be any suitable medium, but satellite transmission will be discussed herein.
  • the data 15 is then transmitted to the satellite 18.
  • the data 15 is transmitted by satellite transmission 34 to the subscribing local proxy servers 22. This transmission may be coordinated by the program 300.
  • global demand statistics are updated to reflect the request for the data 15.
  • the updated global demand statistics may be transmitted at this point to the local proxy servers, or this information may be periodically transmitted.
  • the program 300 progresses to a block 322 which returns control to the start block 302.
  • the system 10 of the present invention moves high demand web pages to the edge of the Internet, closer to users. The result is a much faster delivery of a majority of web pages requested by end users at a station 30, and a corresponding decrease in Internet traffic, substantially accelerating the Internet. Connection costs will correspondingly decrease, resulting in savings to users employing the system 10.
  • FIG 8 shown therein is one representative example of the operation of the central proxy server 12 of Figure 1.
  • a number of local proxy servers 22 are shown communicating with a central proxy server 12 via a communications/validation manager 332.
  • the communications/validation manager 332 maintains a secure connection from the central proxy server 12 to each of the local proxy servers 22 in the system.
  • Messages from the local proxy servers 22 are communicated through the message switcher 338 to the appropriate module.
  • One of the primary modules for handling messages is the miss/refresh handler 340.
  • miss handler 340 When a local proxy server 22 is missing any requested HTML data 15a or any other requested global communications network data 15, a miss is then passed onto the central proxy server 12 and received by the miss handler 340.
  • the miss handler 340 then goes out to the cache database management module 29, represented herein as an Internet Caching System (ICS), one example of which is the ICS program available from Novell Corp. of Provo Utah, and requests the data associated with that miss.
  • ICS Internet Caching System
  • the cache database management module of the central proxy server will either have that data 15 locally in the local cache database 22 or it will go out to the Internet 20 and request that data 15.
  • the central proxy server 12 preferably also includes a control sub-system 350 which handles all of the start-up, shutdown, initialization and processing.
  • An attention sub-system 352 is also preferably provided and handles all user interface types of interaction.
  • the central proxy server 12 is preferably in communication with a database 334 which may employ a schema for handling historical logging of information transmitted back from the local proxy servers 22 to the database 334 upon a message that is sent by the central proxy server 12.
  • the web data 15 which may be handled and forwarded through the system include web objects or groups of web objects that form a web page 15a. Web objects are also referred to as HTML objects or HTML data.
  • the priority of that data is altered somewhat continuously as the central proxy server 12 receives indication from the local proxy servers 22 that one or more end users have made a cache hit on that object or that a local proxy server 22 reported another miss.
  • This popularity information keeps accumulating even after the initial insertion into the priority queue and the highest priority data files 15 are considered for transmission.
  • the transmission queue 346 draws the highest priority data files 15 off the priority queue and then attempts to gather together the web objects, container objects, and their component objects and form that data 15 into logical messages.
  • the input side of the transmission queue 346 receives the data 15.
  • the output of the transmission queue 346 is a list or multiple lists of logical messages that represent pieces of that data 15.
  • the transmission queue 346 processes the data 15 and formulates logical messages, all of which are preferably small, such as about 8 Kbytes. These logical messages are placed onto an output queue, or multiple output queues, and that data serves as input to the packetizer module 348.
  • the responsibility of the packetizer module 348 is to take those logical messages, whatever size they are, and make them fit into IP multicast packets in such a way that there is no wasted space. In other words, if there are several small logical messages, they would get packed one end-to-end inside of those IP multicast packets.
  • the maximum packet size is preferably selected to stay within the maximum data payload of an Internet frame, about 1500 bytes. But more importantly, the length of the data in each of the packets is preferably optimized to be approximately 1442 bytes. That number is selected in one embodiment to work with the subsequent transmission of that packet over a transmission medium such as a satellite system which incorporates DVB packets and the multi-protocol encapsulation or MPE encodings. The size is chosen to be optimal for data efficiency, so that there is no wasted bandwidth on the satellite signal. Thus, there are no fragmentary DVB (Digital Video Broadcast) packets, in one embodiment.
  • DVB Digital Video Broadcast
  • the packetizer module 348 Once the packetizer module 348 has assembled the logical messages and filled up a packet, it will forward that packet to the satellite uplink facility 16, which in this embodiment comprises an IP encapsulator.
  • the IP encapsulator takes packets using the IP encapsulation and translates them into bits and pieces that match the DVB packet sizes, which are much smaller.
  • each DVB packet may hold only approximately 188 bytes of data.
  • the total size of the packet being 204. Some of the packets may be occupied by header information and error correction information.
  • the IP encapsulator outputs those DVB packets ready to go, and they then pass through standardized hardware to for radio frequency conversions and the like, as is commercially available and well known.
  • the packets in this embodiment are then sent from the transmission dish up to geosynchronous orbit, hit the satellite, bounce off it, come back down, and are received at each of the subscribing local proxy servers 22.
  • a satellite receiver at each site receives and recombines the DVB packets into the original IP multicast packets that were fed into the IP encapsulator at the uplink facility.
  • IP multicast is a standard protocol known in the industry.
  • the packet structure of IP multicast is the same as any IP packet used with TCP/IP or UDB/IP transport protocols. The only difference is the addressing of where it is going to, i.e. the target addressing uses special range IP addresses which indicate lots of people can receive the same thing at once.
  • a one to many transmission originates at the central proxy server and is delivered to the local proxy servers 22.
  • the satellite receiver or other transmission medium receiver reconstructs and outputs the original IP multicast packet onto a local Ethernet network, which is preferably a private network, and may be a single cable between the satellite receiver and the user net in the local proxy cache machine.
  • the local caching module 17 receives a payload from those packets and is processed as illustrated in Figure 10.
  • FIG. 9 shown therein is a block diagram illustrating one embodiment of the configuration of packets constructed in the transmitting of data 15.
  • Each data file 15 is preferably broken up into multiple object messages 360 from 1 to N.
  • Each object message 360 consists of an object message header 362 and a payload 364.
  • Each object message 360 is then broken up into a satellite packet 366 by the packetizer module 348.
  • the Satellite packets as stated, are 1442 bytes in size or may be another suitable size selected to give the optimal satellite throughput.
  • the satellite packet similarly consists of a packet header 368 and a payload 370. There are 1 to M packets for every given object message.
  • the satellite packets are then converted into IP multicast format via industry standard operations.
  • the IP multicast is then converted into a digital video broadcast (DVB) stream, which is once again the industry standard.
  • the DVB packets can be constructed using a hardware and software solution. An example of this is a DVB packetizer product which is provided by Skystream International.
  • the DVB packets are received by a satellite modem, which then rebuild those packets into IP multicast format.
  • the thusly formatted packets are then received by the local proxy server 22, reassembled, and eventually the entire web object 15 is reconstructed using the inverse of the process previously described.
  • FIG. 10 shown therein is a block diagram illustrating the operation of one embodiment of a local caching module 17 of a local proxy server 22 of Figure 1.
  • a satellite receiver 372 Emanating from the satellite receiver 372 are transmission signals 374 and 376 indicating network transmissions using multicast packets for Internet data files 15.
  • the data files 15 are shown being received by a receive assembler module 378.
  • Each depicted module represents a software subsystem or component of the local caching module 17.
  • a receive assembler 378 receives the IP multicast packets and as it receives one packet and then the next packet and so on, as fast as it is able to, it reconstructs the object messages, which represent portions of data files 15.
  • the data files 15 are transmitted via an message dispatcher 380 , which is basically a switching mechanism, to an injector module 392.
  • the injector module 392 accumulates the object messages, one or more object messages per web object, reassembles those messages into the web object, and injects that web object data 15 into the local cache database management module 29.
  • the injector module 392 preferably not wait until all of the object messages for a data file 15 have been received. As long as they are coming in order and it has the very first message it will proceed to create that object as far as the cache database management module is concerned and will continue to append the additional data as it arrives until the last has arrived. If anything is out of order, it waits until it receives the missing portions.
  • a satellite health manager 386 queries the satellite receiver 372 about every fifteen seconds sending SNMP (Simple Network Management Protocol) queries by using the UDP/IP transport and the satellite receiver, if all is well, responds affirmatively.
  • SNMP Simple Network Management Protocol
  • satellite health manager 386 sends a message via the message dispatcher 380 to a central proxy server session manager 382.
  • the session manager communicates with the central proxy server 12 over a TCP/IP connection 384. That connection is preferably maintained over the standard Internet and the report of satellite receiver misbehavior can thus reach the central caching management module 13 and trigger an alarm or notification such that operations personnel can try to identify the problem and resolve it.
  • the satellite health manager 386 logs an error in the systems error log on the local caching module 17.
  • An attention subsystem 402 is also preferably provided. Therein is shown a GUI information module 404 an error log status module 406 and a statistics module 408.
  • the attention subsystem 402 is responsible for bringing attention to things by displaying information through the graphical user interface to the customer or to any remote administrator that is hooked into the cache database management module and accessing the specific management panels, as well as logging those errors in a local file.
  • a remote console 388 is provided for diagnostic purposes.
  • the present invention allows a remote connection over the regular Internet to the local caching module 17 with a simple utility which implements what appears to be just a simple command line such as a
  • DOS command line The system may enter commands of its own through the local caching module 17 that will give back responses providing information about internal statistics and performance and other diagnostic information and those commands may also be used to set or change internal parameters or characteristics of the local caching module 17 for purposes of optimizing its performance.
  • a client browser 25 may be browsing the Internet through that particular local cache database management module box. If that client requests a web object which has not been previously cached by the local cache database management module, then the local cache database management module will call up the miss manager subsystem 400 and identify the URL (universal resource locator) of that web object on the Internet and give the local caching module 17 an opportunity to take steps to find and gather that web object data and deliver it to the local cache database management module 29 by means of the present system, which is what has been described up to this point, where the data is received over the satellite by the local caching module 17 from the Central proxy server and injected or discarded back into the local cache database management module. So the local cache database management module is given first right of refusal and will try to satisfy the end user request for a particular web option.
  • the URL universal resource locator
  • the local cache database management module 29 is preferably configured to report cache misses to the miss manager 400 of the local caching module 17.
  • the miss manager 400 preferably then transmits a message immediately to the central proxy server 12 via the message dispatcher 380 and the central proxy server session manager 382 to instruct the central proxy server 12 that the local proxy server 22 does not contain a copy of the web object 15 the end user is looking for.
  • the central proxy server 12 then preferably uses the miss information in calculating global miss rate information. That is, the central proxy server 12 uses the miss reports that it receives from the local proxy servers 22 in the overall system 10 to compute the relative priority of web objects 15 in the priority queue 342.
  • the hit manager module 396 is similar to the miss manager 400 in that when a client requests a web object 15 from a local cache database management module 29 and the cache database management module 29 has that object already stored in its cache, a hit is registered.
  • the local proxy server 17 accumulates hit counts on an object-by-object basis as they are reported. It then reports that information at least every few seconds to the central proxy server 12 advisor via the message dispatcher and the central proxy server session manager.
  • the local caching module 17 preferably makes those hit reports under several different circumstances.
  • the central proxy server 12 may only have room to track hits on so many objects at once, and that determines somewhat the size of the report messages that it is going to send to the central proxy server 12. If the message contents are full, then it sends the message allowing it to empty or zero out all of those accumulating buffers, counters, accumulators.
  • Each of the web objects that the system 10 has delivered is preferably assigned a globally unique identifier and the hit count report utilizes that identifier to be very efficient in forwarding those hits to the central proxy server 12. In so doing, it may simply send out an array of structures.
  • the structure is an array comprised of two elements. One element is the global ID which represents the web object that the system 10 is tracking. The other element is a count of the number of hits that has just been recorded with a very recent task.
  • the current message is sent as soon as all of the array structures are in use or, if not all of them are in use, the partial messages are transmitted no greater than about five seconds apart to make sure that information stays timely until it gets back to the central proxy server 12.
  • That array of information or partial array is also transmitted immediately if one or more of the objects which are being hit has experienced a very large number of hits in a very short period of time. A type of frequency measurement is observed, as well as just accumulations of total hits.
  • miss manager of a local caching module 17 needs to send a miss report, it will prepare a message and a header will identify the target as "central proxy server subsystem miss handler.”
  • central proxy server 12 transmits object messages, the header of those messages will target to the local caching module 17's injector subsystem and it is the responsibility of the message dispatcher box to switch those messages back and forth between the appropriate subsystems on the local or master.
  • the present invention thus reports on hits that have been registered on objects that are being tracked by the system 10 and attempts to report those hit counts as quickly and efficiently as possible so that they have relevance.
  • hit report reaches the central proxy server 12
  • those hit counts are matched up with the appropriate web object that is being tracked using the global ID.
  • the hit counts are then added into the prior known hit counts for that object and a new priority for that object can be calculated using miscounts and hit counts to date and some other information.
  • Relative priorities of any particular object are preferably transmitted under two circumstances.
  • the object data itself is transmitted, the priority of that object is transmitted with it. That information is provided to the local caching module 17 and to the local cache database management module.
  • the local cache database management module at that point preferably incorporates that global hit data that has been provided into its own internal calculations of how wonderful this object is and how long it wants to keep it around in its local cache.
  • the second time that n object's priority is typically transmitted is when that priority has changed noticeably, somewhat substantially. But the object data itself would not change. There would be no need to resend the subject data but it is useful to send the very brief, very small administrator message saying the object identified by global ID such and such has a new global priority, so please consider it.
  • the hit manager In addition to recording hit counts on a moment to moment basis for various objects, the hit manager preferably also keeps the longer term total of those hits. In fact, it preferably keeps a total for a time period of, for example, fifteen minutes. Every object that has had any hits on it in 15 minutes, at the end of that 15 minutes is then evaluated relative to all the others. The objects are sorted out and the record of the ordered top 100, 200 or some selected number of objects are logged to a file in the local cache box and that file will be available for subsequent retrieval by the central caching management module 13 and the historical database agent The DB agent goes out periodically and queries each of these local caching modules
  • One advantage of the system 10 of the present invention is that there exists a closely integrated relationship between the local caching module 17 and the cache database management module 29. This allows the system 10 to communicate information such as the hit reports that is not available in prior art caching systems. That information may be gathered even after the object has unloaded into that cache and it can be monitored subsequently.
  • hit counts are accumulated within hit reports at the local cache. Many hit counts may be accumulated before transmission due to time constraints. Thus, a number of hits on a selected object may be transmitted in a single report.
  • a number of a ⁇ ows are shown directed into and out of an cache database management module 29. These arrows in essence represent the functional interface for an application programming interface (API).
  • API application programming interface
  • the tight integration between the cache database management module 29 (one example of which is the cache database management module of Figure 10) and the local caching module 17 is conducted through the use of an API.
  • API as understood in the industry is a collection of programming interfaces that allow one set of software to access services or information from another set of software in a standardized manner which separates and protects each side from the other, through establishing a common method of communication.
  • the two software modules operate on a common server and their communications are transmitted directly between each other.
  • the API can be considered a type of dynamic binding between two applications operating on a common server or other processor.
  • integrated communication includes some sort of binding between two separate applications on a common processor.
  • the binding is preferably a dynamic binding, one example of which is conducted through an API.
  • An API defines all of the interface information that must be implemented by a vendor in a product.
  • the use of an API-type interface in one embodiment allows the present invention to benefit from close, substantially instantaneous communication between the local caching module 17 and the cache database management module 29.
  • arrows pass from the injector module 393 into and out of the cache database management module 29.
  • four functions are provided for the injector to call from within the cache database management module 29. Each of those functions is called with a short command, and if necessary, accompanying parameters. Those commands are responded to with the performance of the requested function, which may, for instance, include the depositing of data in a selected buffer or memory location.
  • the injector module 392 receives web objects or other files 15 and needs to transmit those objects to the cache database management module 29, it uses commands from selected API instruction sets and passes those commands to the cache database management module 29 to achieve that desired function.
  • API functions may include an instruction or command which requests the cache database management module 29 to locate an object 15 within the local cache database 24.
  • Another API instruction may notify the cache database management module 29 that a particular data file 15 is being sent and is to be saved together with another stored data file 15. This data may not actually be part of the original object data 15, but may be related information, such as a component of a web page 15 a.
  • the cache database management module 29 may respond to the instruction asynchronously, that is, at varying times. The responses may be immediate, or may be delayed, to acknowledge the results of an initial request, for instance.
  • Other functions may comprise cache database management module one-way communication to the local caching module 17 to provide notification of certain events such as a hit report or a miss report.
  • an API interface is very highly integrated between the local caching module 17 and the cache database management module 29. This provides the advantages that the cache database management module 29 can be communicated with internally and unique types of information can be received back from the cache database management module, including statistics such as hit reports, almost instantaneously.
  • Prior art caching system typically receive new objects in whatever order they are sent to that caching system. The caching system then just throws out the least recently used data to make way for the new data coming in a very unintelligent manner.
  • Prior art systems may provide limited feedback on misses, due to the need to request data that is not present in the cache. In such systems, there is no sense of global priority. That is, each remote caching system is unaware of what is popular elsewhere, other than a possibly a rudimentary calculation based upon misses. The prior art caching box responds to transmitted data by makings to make room for this new arrival, and use it's own best judgement.
  • one embodiment of the present invention is based directly on having an integrated API relationship between the cache advisor and the cache software. This allows the system 10 to have access in real time to prepare information about what is going on inside the cache database management module 29, and also what is going on relative to particular web objects 15 that are being tracked. Given that information, the local caching module 17 and the central proxy server 12 collaborate. Each of the local caching modules 17, that are online and subscribed to the system 10, report their popularity information to the central caching management module 13, which compiles that information and sends it out with extracted data files 15. The local caching modules 17 then utilize that global popularity data in determining what data 15 to keep and what to discard, optionally in combination with unique local determination factors.
  • Local proxy servers 22 thus act as the eyes and ears across the world. Due to the ability to communicate closely with the cache database management module, the local proxy servers 22 are able to extract at least hits, and optionally, many other types of information from the local cache databases 24. Examples of information that can be extracted from the local caching module 17 include the acquisition time - the time that it takes to fill an object over the Internet; information such as what is being deleted out of the local cache, which may indicate local priorities; time data ⁇ data that is hot in a particular area or time zone; hit time - - the time it took the local cache database management module to retrieve an object from the Internet; and other particularized information about what objects are being kept or thrown out.
  • the system 10 of the present invention allows a web page to be transmitted in its entirety, rather than transmitting a listing of the contents of the page and going back separately requesting those contents as is conducted with most Internet accesses.
  • One feature preferably provided within the cache database management module 29 of the present invention is the ability to examine an HTML web object (such as a web page) 15 and determine its component parts.
  • the HTML object preferably includes a container that describes the complete layout of a particular page 15a that will be seen on the browser window 25 of a user 30.
  • the container or other such directories typically contain most of the readable text and contain specific instructions, using the hypertext mark up language syntax, to identify other web objects, images, Java applets, or other types of web object that are part of the page.
  • the cache database management module also preferably looks at a potential container that refers to all of the other images and elements that comprise the total Web page experience.
  • the cache database management module 29 will, upon receiving the HTML object 15, scan through it immediately, parsing out the particular references to other objects that are components of that page. As the cache database management module 29 finds each of those, it will initiate a request.
  • the system 10 first checks to see if the image is already present in the local cache database 24. If it is not, the system 10 initiates a request over the Internet 20 to retrieve that image and continues to do that for every object that was indicated as a component of that HTML container object.
  • the cache database management module 29 has hopefully retrieved some, if not all, of the component objects that complete a web page by the time that the end user browser 25 has received the original HTML content.
  • the cache database management module 29 preferably begins to analyze and parse the requested object 15 and goes back to the cache database management module 29 requests all component objects of the web page 15. The entire contents of a web page 15a are then streamed to the web browser 25, without waiting for the web browser 25 to request them individually.
  • This "read-ahead” feature is preferably also provided within the central caching management module 13. Accordingly, when a cache miss request arrives from a local caching module 17, and the central caching management module 13 is requested to retrieve this web object 15, that web object will generally be the initial reference to a particular web page 15. Thus, the object 15 is preferably retrieved by the cache database management module of the central proxy server 12 and handed off to the central caching management module 13 along with some identifying information that the received object is an HTML container that may have certain components. The cache database management module 29 of the central proxy server 12 then preferably automatically pre-fetches or pre-locates those components, whether in its local cache or out on the Internet and notifies the central caching management module 13 as each one of those are identified and found.
  • the HTML object, together with a list of component objects that accompany the web page 15a are stored together in the cache database management module 29, both of the local proxy servers 22 and the central proxy server 12.
  • the system 10 knows about the HTML object and when it transmits the content of that web page 15, it not only sends the HTML object, but it preferably also sends, together therewith, data for all of the component objects that it knows about.
  • That information arrives, it is passed through to the injector subsystem. Part of those messages will be the container, the information that identifies that particular HTML object and represents the whole page.
  • the container preferably also identifies which of the objects that are being received are component objects of the page 15a.
  • the injector subsystem can begin to inject all of those web objects into the local cache, and when it has completed putting all of those objects in, it can then inform the local cache 24 of that relationship.
  • the grouping information may come first or it may come last, but the concept is that the local caching module 17 has the opportunity to inform the local cache that these objects all go together and form a logical entity called a web page.
  • This information is preferably used by the local cache to optimize its storage of those objects on the hard disk to keep them in close proximity for later retrieval if need be, so that retrieval time is cut to a minimum.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)
  • Computer And Data Communications (AREA)

Abstract

Ce système et ce procédé accélèrent la livraison d'un contenu sur un réseau de télécommunications global, tel que l'Internet (20). Un serveur central mandataire (12) transmet des fichiers de données extraites, sur un support de télécommunications, afin d'assurer le remplissage en contenus de serveurs locaux mandataires (17). Le support de télécommunications peut être un satellite de transmissions et il peut être géré à l'aide d'une diffusion sélective IP. Les serveurs locaux mandataires reçoivent simultanément les fichiers de données à partir du support de communication, à un débit élevé, et mémorisent ces fichiers de données dans des bases de données locales (29), à antémémoire, d'opérateurs. Les serveurs locaux mandataires mettent en oeuvre un schéma heuristique localisé pour déterminer s'il faut garder ou rejeter les données. Lorsqu'un utilisateur demande des données par l'Internet, sa demande est d'abord reçue par le serveur local mandataire. Si les données demandées existent parmi les fichiers de données extraits, elles sont transmises rapidement et directement à l'utilisateur, à partir du serveur local mandataire, si ces données n'existent pas, une demande est transmise au serveur central mandataire.
PCT/US2000/000587 1999-01-11 2000-01-11 Systeme d'acceleration de la livraison d'un contenu sur l'internet WO2000042519A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU28477/00A AU2847700A (en) 1999-01-11 2000-01-11 Internet content delivery acceleration system

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US11545699P 1999-01-11 1999-01-11
US60/115,456 1999-01-11
US16997199P 1999-12-08 1999-12-08
US60/169,971 1999-12-08
US48028500A 2000-01-10 2000-01-10
US09/480,285 2000-01-10

Publications (2)

Publication Number Publication Date
WO2000042519A1 true WO2000042519A1 (fr) 2000-07-20
WO2000042519A8 WO2000042519A8 (fr) 2000-10-26

Family

ID=27381671

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/000587 WO2000042519A1 (fr) 1999-01-11 2000-01-11 Systeme d'acceleration de la livraison d'un contenu sur l'internet

Country Status (2)

Country Link
AU (1) AU2847700A (fr)
WO (1) WO2000042519A1 (fr)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002013037A1 (fr) * 2000-08-08 2002-02-14 Fineground Networks Procede et systeme pour accelerer l'expedition des contenus dans un environnement reseau
US6434609B1 (en) 1998-03-16 2002-08-13 Cidera, Inc. Comprehensive global information network broadcasting system and methods of distributing information
WO2002089000A1 (fr) * 2001-04-26 2002-11-07 Iinet Limited Systeme de mise en antememoire de donnees au cours d'un transfert de donnees entre homologues
WO2003014942A1 (fr) * 2001-08-03 2003-02-20 Nokia Corporation Procede, systeme et terminal pour reseaux de donnees avec caches repartis
SG108885A1 (en) * 2001-11-30 2005-02-28 Ntt Docomo Inc Content distribution system, description data distribution apparatus, content location management apparatus, data conversion apparatus, reception terminal apparatus, and content distribution method
US7159014B2 (en) 2001-06-04 2007-01-02 Fineground Networks Method and system for efficient and automated version management of embedded objects in web documents
US7310687B2 (en) 2001-03-23 2007-12-18 Cisco Technology, Inc. Methods and systems for managing class-based condensation
US8281029B2 (en) * 2000-02-15 2012-10-02 Gilat Satellite Networks Ltd. System and method for acceleration of a secure transmission over satellite
US8468229B2 (en) 2004-03-31 2013-06-18 Telecom Italia S.P.A. Method and system for controlling content distribution, related network and computer program product therefor
US8799764B2 (en) 2000-08-08 2014-08-05 Cisco Technology, Inc. Method and system for parameterized web documents
US9094090B2 (en) 2011-09-23 2015-07-28 Gilat Satellite Networks Ltd. Decentralized caching system
EP3054652A1 (fr) 2015-02-06 2016-08-10 Thales Ajustement dynamique du mode de transmission dans un systeme de communication satellite
US10423336B2 (en) 2017-11-28 2019-09-24 International Business Machines Corporation Fast locate using imitation reads on tape drives

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6003082A (en) * 1998-04-22 1999-12-14 International Business Machines Corporation Selective internet request caching and execution system
US6012085A (en) * 1995-11-30 2000-01-04 Stampede Technolgies, Inc. Apparatus and method for increased data access in a network file object oriented caching system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6012085A (en) * 1995-11-30 2000-01-04 Stampede Technolgies, Inc. Apparatus and method for increased data access in a network file object oriented caching system
US6003082A (en) * 1998-04-22 1999-12-14 International Business Machines Corporation Selective internet request caching and execution system

Cited By (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6434609B1 (en) 1998-03-16 2002-08-13 Cidera, Inc. Comprehensive global information network broadcasting system and methods of distributing information
US9723055B2 (en) 2000-02-15 2017-08-01 Gilat Satellite Networks Ltd. System and method for acceleration of a secure transmission over satellite
US8281029B2 (en) * 2000-02-15 2012-10-02 Gilat Satellite Networks Ltd. System and method for acceleration of a secure transmission over satellite
US8762478B2 (en) 2000-02-15 2014-06-24 Gilat Satellite Networks Ltd. System and method for acceleration of a secure transmission over satellite
US20170195041A1 (en) * 2000-02-15 2017-07-06 Gilat Satellite Networks Ltd. System and Method for Acceleration of a Secure Transmission Over Satellite
US20120324036A1 (en) * 2000-02-15 2012-12-20 Gilat Satellite Networks Ltd. System And Method For Acceleration Of A Secure Transmission Over Satellite
WO2002013037A1 (fr) * 2000-08-08 2002-02-14 Fineground Networks Procede et systeme pour accelerer l'expedition des contenus dans un environnement reseau
US7047281B1 (en) 2000-08-08 2006-05-16 Fineground Networks Method and system for accelerating the delivery of content in a networked environment
US7139976B2 (en) 2000-08-08 2006-11-21 Fineground Networks Method and system for parameterized web documents
US8799764B2 (en) 2000-08-08 2014-08-05 Cisco Technology, Inc. Method and system for parameterized web documents
US7603483B2 (en) 2001-03-23 2009-10-13 Cisco Technology, Inc. Method and system for class-based management of dynamic content in a networked environment
US7802014B2 (en) 2001-03-23 2010-09-21 Cisco Technology, Inc. Method and system for class-based management of dynamic content in a networked environment
US7310687B2 (en) 2001-03-23 2007-12-18 Cisco Technology, Inc. Methods and systems for managing class-based condensation
WO2002089000A1 (fr) * 2001-04-26 2002-11-07 Iinet Limited Systeme de mise en antememoire de donnees au cours d'un transfert de donnees entre homologues
US7159014B2 (en) 2001-06-04 2007-01-02 Fineground Networks Method and system for efficient and automated version management of embedded objects in web documents
CN1316375C (zh) * 2001-08-03 2007-05-16 诺基亚有限公司 具有分布式高速缓存的数据网络所用的方法、系统和终端
WO2003014942A1 (fr) * 2001-08-03 2003-02-20 Nokia Corporation Procede, systeme et terminal pour reseaux de donnees avec caches repartis
US6952712B2 (en) 2001-11-30 2005-10-04 Ntt Docomo, Inc. Method and apparatus for distributing content data over a network
SG108885A1 (en) * 2001-11-30 2005-02-28 Ntt Docomo Inc Content distribution system, description data distribution apparatus, content location management apparatus, data conversion apparatus, reception terminal apparatus, and content distribution method
US9054993B2 (en) 2004-03-31 2015-06-09 Telecom Italia S.P.A. Method and system for controlling content distribution, related network and computer program product therefor
US8468229B2 (en) 2004-03-31 2013-06-18 Telecom Italia S.P.A. Method and system for controlling content distribution, related network and computer program product therefor
US9094090B2 (en) 2011-09-23 2015-07-28 Gilat Satellite Networks Ltd. Decentralized caching system
US9564960B2 (en) 2011-09-23 2017-02-07 Gilat Satellite Networks Ltd. Decentralized caching system
EP3054652A1 (fr) 2015-02-06 2016-08-10 Thales Ajustement dynamique du mode de transmission dans un systeme de communication satellite
FR3032580A1 (fr) * 2015-02-06 2016-08-12 Thales Sa Ajustement dynamique du mode de transmission dans un systeme de communication satellite
US9853718B2 (en) 2015-02-06 2017-12-26 Thales Dynamically adjusting the transmission mode in a satellite communication system
US10423336B2 (en) 2017-11-28 2019-09-24 International Business Machines Corporation Fast locate using imitation reads on tape drives
GB2581727A (en) * 2017-11-28 2020-08-26 Ibm Fast locate using imitation reads on tape drives
US11099738B2 (en) 2017-11-28 2021-08-24 International Business Machines Corporation Fast locate using imitation reads on tape drives
US11099737B2 (en) 2017-11-28 2021-08-24 International Business Machines Corporation Fast locate using imitation reads on tape drives
US11106364B2 (en) 2017-11-28 2021-08-31 International Business Machines Corporation Fast locate using imitation reads on tape drives
GB2581727B (en) * 2017-11-28 2022-05-04 Ibm Fast locate using imitation reads on tape drives

Also Published As

Publication number Publication date
WO2000042519A8 (fr) 2000-10-26
AU2847700A (en) 2000-08-01

Similar Documents

Publication Publication Date Title
US20020073167A1 (en) Internet content delivery acceleration system employing a hybrid content selection scheme
CA2303739C (fr) Methode et systeme de gestion du rendement des transferts de donnees pour un systeme de consultation de donnees
US8156216B1 (en) Distributed data collection and aggregation
Padmanabhan et al. Using predictive prefetching to improve world wide web latency
US6058418A (en) Marketing data delivery system
US6351467B1 (en) System and method for multicasting multimedia content
US6298373B1 (en) Local service provider for pull based intelligent caching system
EP3281120B1 (fr) Qualité de service de réseau de distribution de contenu côté serveur
US7647418B2 (en) Real-time streaming media measurement system and method
JP5259412B2 (ja) 偽の情報要求の識別
AU2005232076B2 (en) Systems and methods for tracking web activity
US20040002896A1 (en) Collection of behavior data on a broadcast data network
WO2000042519A1 (fr) Systeme d'acceleration de la livraison d'un contenu sur l'internet
US20030061305A1 (en) System and method for enhancing streaming media delivery and reporting
US20020092026A1 (en) Method and apparatus for broadcast delivery of content to a client-side cache based on user preferences
US20090327079A1 (en) System and method for a delivery network architecture
US20020129116A1 (en) Network broadcasting system and method of distrituting information from a master cache to local caches
WO2001065402A2 (fr) Procede et systeme de distribution de contenu de reseau intelligent
EP2473924A1 (fr) Gestion d'analytique de réseau
WO1998011702A1 (fr) Appareil et procede de capture, d'analyse et de visualisation en direct d'informations de reseaux
JP5798523B2 (ja) 通信制御システム、集約サーバおよび通信制御方法
Padmanabhan et al. Improving world wide web latency
KR100643834B1 (ko) 분산 캐시를 갖는 데이터 네트워크를 위한 방법, 시스템및 단말기
US7222170B2 (en) Tracking hits for network files using transmitted counter instructions
KR20010094678A (ko) 인터넷상에서의 컨텐츠 정보 제공방법 및 장치

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AL AM AT AU AZ BA BB BG BR BY CA CH CN CR CU CZ DE DK DM EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
AK Designated states

Kind code of ref document: C1

Designated state(s): AE AL AM AT AU AZ BA BB BG BR BY CA CH CN CR CU CZ DE DK DM EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: C1

Designated state(s): GH GM KE LS MW SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

CFP Corrected version of a pamphlet front page
CR1 Correction of entry in section i

Free format text: PAT. BUL. 29/2000 UNDER (30) REPLACE "NOT FURNISHED" BY "09/480285"

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase