US20130041882A1 - Technology for web site crawling, including action sequences for selecting non-hypertext-link parameters - Google Patents
Technology for web site crawling, including action sequences for selecting non-hypertext-link parameters Download PDFInfo
- Publication number
- US20130041882A1 US20130041882A1 US13/600,201 US201213600201A US2013041882A1 US 20130041882 A1 US20130041882 A1 US 20130041882A1 US 201213600201 A US201213600201 A US 201213600201A US 2013041882 A1 US2013041882 A1 US 2013041882A1
- Authority
- US
- United States
- Prior art keywords
- server
- address
- page
- web
- web pages
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9532—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9574—Browsing optimisation, e.g. caching or content distillation of access to content, e.g. by caching
Definitions
- this invention concerns crawling a web site on the World Wide Web (“WWW”), and more specifically a web site wherein at least one web page in the web site has a reference for executing by a browser to produce an address for a next page.
- WWW World Wide Web
- the invention also concerns web pages that are dynamically generated responsive to queries, such as queries associated with the crawling. This includes preemptively transforming dynamically generated web pages into static web pages.
- the World Wide Web is an interconnected network of computers and information appliances. Clients use the WWW to send requests to servers, which send back responses.
- “Static data” is server response data that already exists on the server at the time of the request and that is merely served back to the client without change by the server.
- a news article is an example of static data. While the news may change daily, or even minute-by-minute, nevertheless, according to the typical scenario, an article is created in response to a news item, and then the article is put on the server as a static document. That is, the article itself does not subsequently change, even though other, newer articles will sooner or later be placed on the server too. This is in contrast to “dynamic data” that is created by the server in direct response to a client request.
- a web page displaying a bank account balance or stock positions in a trading account is an example of a page that is created by the server and that may change with each interaction between the client and the server.
- Dynamically generating data tends to impose a substantial load on server resources. While this load could be mitigated by precomputing responses in anticipation of all possible requests, such a course of action will require substantial resources for both computing the responses and for storing them.
- Some middle ground may be ideal, where the server stores most likely requested information in static form, and creates other information on demand.
- a web site may thus choose to convert a subset of its content to be delivered statically in order to reduce server resource demands.
- the conversion may be done manually, by employing web page designers.
- An alternative is to automatically generate the static responses using the raw data on the web site.
- a program could be set up to format the extracted data (such as from database queries) and encapsulate it within the appropriate HTML content, thus creating static pages.
- the disadvantage of this approach is that the program must be provided with the parameters with which to query the database as well as the inter-document hierarchy which specifies how the documents will be hyperlinked together. Determining and providing this information requires significant resources. There is thus a need for an automated method that can be used to easily convert subsets of a web site to static content.
- crawler Use of a crawler would be advantageous for automating this conversion of a web site to static content, however there are numerous difficulties that prevent this.
- Conventional search engine crawlers start with a URL and repeatedly de-reference all unexplored URLs in the received responses.
- One reason conventional crawlers are not suitable for the “staticizing” problem concerns actions sequences that must be performed to obtain a particular end data set for a conventional HTML query.
- references from one web page to another may not be straightforward. That is, a reference may not be simply set out on the page as a hyperlink address, but instead may be a script, form, selection menu, or button for example.
- a web server that dynamically generates responses is queried and the resulting responses are saved so that the saved static data can be used on the server to avoid subsequent dynamic content generation.
- a crawler program “crawls the web site,” that is, generates queries to the server, responsive to references from one web page to another in the content provided by web site.
- the queries cause the server to generate dynamic data and return web pages that include those results. These web pages are then processed and saved, so that the web pages can be served in response to future queries. This eliminates the need to generate dynamic data on demand for these particular queries.
- the web pages dynamically generated in response to particular requests may have included in them operations that would call for the server to generate more dynamic data if the received data were used to generate additional requests to the server. That is, if one of the pages as received was used to generate additional requests to the server, such as by a user filling in fields on a form in the page, clicking on a button, selecting an item in a selection menu, etc., this would cause the server to generate more data dynamically.
- the received web pages are converted by removing at least some of these operations and replacing them with references to corresponding ones of the returned web pages.
- crawler conventionally refers to a search engine, going through the WWW to obtain data to build indices.
- crawler is programmable to perform particular actions sequences for generating queries to the web server.
- references from one web page to another may not be straightforward. That is, as stated above, a reference may not be simply set out on the page as a hyperlink address, but instead may be specified by a script, for example, so that the address is produced only when a client browser executes the reference.
- a reference that is generated in this manner is very dependent on the context in which it is produced, that is, the history that led up to it, including the state of the server and the client browser.
- the crawler parses each received web page and sends references to an applet developed for an embodiment of the present invention that runs in the browser.
- JEE JavaScript execution engine
- the client browser is configured to use a certain proxy gateway, and refer to a certain file (referred to herein as the “resolver file”) containing cross-referencing for hostname-to-IP-address-resolution.
- the proxy gateway for the client's browser has a certain IP address.
- the resolver file is configured so that the resolver file indicates the IP address of the proxy gateway as the IP address for the web site server.
- the JavaScript execution engine running on the client appears to the client's browser to be from the same server as the web pages, which permits the JEE to communicate with the crawler and the browser unhindered by conventional limitations. Since the proxy gateway is not really the source of the web pages, it may be referred to herein as a “spoof proxy.”
- the JEE needs to know when a web page being received from the server has been fully loaded.
- the “onload” attribute is a conventional HTML document attribute that is useful for this purpose.
- the onload attribute may not be included in the web pages, or may not be included in a fashion suited for use with the JEE. Therefore, in another aspect, the spoof proxy adds the onload attribute to each received web page which does not already have the attribute, and defines an event handler that sets a certain variable. Then, this variable is polled by the JEE to determine when the page is loaded.
- a computer program product provides a crawler for crawling a web site and reduces dynamic data generation on the web site server.
- the computer product includes instructions for querying a web site server by a crawler program responsive to references from one web page to another in the content provided by web site. The queries are for causing the server to generate data and return web pages that include the data.
- the product also includes instructions for processing and saving versions of web pages received from the server, so that the web pages can be served in response to future queries.
- the computer program product includes instructions for processing the received web pages, which includes removing at least some of these operations and replacing them with references corresponding to the returned web pages.
- the computer program product includes instructions for causing the crawler to parse the received web pages and send references to an applet running in a client browser.
- the instructions also include instructions for causing the browser to determine the address for a next page responsive to the references, so that the browser may receive the next page and any cookie for the next page from the server.
- the instructions also include instructions for causing the applet to return the address and any cookie to the crawler program.
- the computer program product includes instructions for causing the proxy to add an onload attribute to a received web page, and define an event handler that sets a certain variable, and instructions for causing the applet to poll the certain variable to determine when the page is loaded.
- an apparatus in another embodiment, includes a processor connected to a network.
- a storage device is connected to the processor and the network for storing a program for controlling the processor.
- the processor is operative with the program to query a web site server, responsive to references from one web page to another in the content provided by web site.
- the processor is also operative with the program to process and save versions of web pages received from the server, so that the web pages can be served in response to future queries.
- the processor is operative with the program to convert the received web pages, by removing at least some of these operations and replacing them with references to corresponding ones of the returned web pages.
- the processor is operative with the program to parse the received web pages and send references to an applet executing within a client browser. Also, the processor is further operative with the program to cause the applet to cause the browser to determine the address for a next page responsive to such a reference, so that the browser may receive the next page and any cookie for the next page from the server. Also, the processor is further operative with the program to cause the crawler to receive the address and any cookie from the JEE.
- the processor is operative with the program to cause the proxy to add an onload attribute to a received web page, and define an event handler therefor that sets a certain variable, and cause the applet to poll the certain variable to determine when the page is loaded.
- the invention advantageously reduces limitations in the usefulness of crawler programs.
- the improved crawler eliminates the need to generate data dynamically for certain queries by creating static web pages for those queries. This reduces computational load on a web server. It is also advantageous that since the static web pages are produced by interaction of the crawler and the web site server, if the web site changes, then the static web pages can be updated merely by having the crawler program crawl the web site again.
- FIG. 1 illustrates crawling references among web pages, in accordance with an embodiment of the present invention.
- FIG. 2 illustrates receiving, processing and saving versions of the web pages, in accordance with an embodiment of the present invention.
- FIG. 3 illustrates performing an action sequence, in accordance with an embodiment of the present invention.
- FIG. 4 illustrates certain interactions among a crawler, applet for JavaScript execution, browser, spoof proxy and web server, in accordance with an embodiment of the present invention.
- FIG. 5 illustrates a flow chart for method steps, in accordance with an embodiment of the present invention.
- a server 100 for hosting a web site.
- the server has a processor 102 coupled to a bus 106 through a bridge 108 .
- memory 104 Also coupled to the bus through the bridge is memory 104 .
- Disk 126 is coupled to the bus through an adapter 112 for storing web pages to be served up to users, such as client 170 , over the network 141 which is coupled to the bus through adapter 110 .
- Server 100 uses its resources (processor 102 , bus 106 , bridge 108 , memory 104 , adaptor 110 , and disk 126 ) to dynamically generate responses to queries received over the network adaptor 110 .
- Client 170 has a crawler program 171 for generating queries to the server 100 responsive to references from one web page to another in the content delivered by web server 100 .
- first page 140 . 1 is shown having a first reference 142 linking page 140 . 1 to page 140 . 2 .
- the crawler program 171 in client 170 responsive to reference 142 , queries server 100 for the second web page 140 . 2 .
- the client 170 queries server 100 for web page 140 . 3 , responsive to reference 146 .
- the queries could proceed in a different sequence, instead of the depth-first search illustrated in FIG. 1 .
- each reference in the top page 140 . 1 could be crawled to a next page before crawling any references in any of the next pages.
- the client 170 After having received page 140 . 3 , the client 170 queries server 100 for web page 140 . 4 responsive to reference 150 , and queries for web page 140 . 5 responsive to reference 152 . Since page 140 . 5 is a JavaScript source page, the crawler program does not produce any queries responsive to page 140 . 5 . Page 140 . 4 does not have any references to other web pages, so the crawling has reached a dead end with web pages 140 . 5 and 140 . 4 .
- the crawler queries for web page 140 . 6 responsive to reference 148 in page 140 . 2 .
- Page 140 . 7 has one reference 158 , which leads to a web page outside a boundary that has been specified to crawler, so this reference is not crawled.
- crawler 171 queries for page 140 . 8 .
- crawler 171 queries for page 140 . 7 , responsive to reference 160 .
- Crawler 171 determines that reference 160 is a reference to page 140 . 7 , which is in common with reference 154 , so reference 160 is not crawled.
- crawler 171 queries so that client 170 receives page 150 . Since page 150 is an image, this page is not crawled.
- references between two pages could be more than just conventional hyperlinks.
- the references could be script references, selection menus, forms, buttons or other elements.
- client 170 is shown with programs running, including browser 205 and crawler 171 .
- programs run on a processor (not shown) of client 170 , responsive to instructions stored on client 170 storage, such as storage 226 .
- a first web page 140 .X is loaded in the browser 205 .
- the crawler 171 includes a function 220 for parsing information from web pages loaded in the browser 205 , such as 140 .X, and passing the information 230 to the browser 205 for interpreting the information, which concerns references to other pages, such as that from 140 .X to 140 .Y, and generating queries for the other pages.
- a reference may not be simply set out on a web page as an address, but instead may be specified by a script, selection menu, form, button or other element, so that the address needs to be produced by the client browser executing the reference.
- a reference that is generated in this manner is very dependent on the context in which it is produced, that is, the history that led up to it, including the state of the server and the client browser.
- the crawler 171 parses each received web page, such as page 140 .X, and sends references parsed from the page to a JavaScript execution engine 210 that runs in the browser 205 , for generating a query for web page 140 .Y. That is, browser 205 determines from the information 230 that page 140 .Y is referenced in referencing information in page 140 .X.
- the crawler 171 may directly obtain the page 140 .X from the web server 100 and parse it, consulting browser 205 only when encountering references other than simple hyperlinks, such as that linking page 140 .X to page 140 .Y.
- Browser 205 generates an address for the page 140 .Y and sends the address in a query 250 to a proxy host 215 , which passes the query to server 100 .
- Server 100 returns the web page 140 .Y, and proxy host 215 adds an onload handler to the page sends the page 140 .Y to browser 205 .
- web page 140 .X includes the results of data dynamically generated in server 100 for an earlier query 250 that resulted in the page 140 .X, but 140 .X also has operations included in it linking it to page 140 .Y, that call for the server 100 to generate more dynamic data to produce web page 140 .Y. Therefore, web page 140 .X needs to be converted by removing these operations and replacing them with references to a “version of” the returned web page 140 .Y. The reference must be to 140 .Y 1 , a “version of” web page 140 .Y, because 140 .Y may itself have operations included linking it to a next page also calling for the server 100 to generate more dynamic data to produce the next page.
- the crawler also includes a processing function 225 .
- the browser 205 passes information 240 back to the crawler 171 for the crawler to use for processing the web pages and saving versions thereof.
- web pages 140 .X and 140 .Y are processed to generate new versions 140 .X 1 and 140 .Y 1 , which are saved in storage 226 .
- the browser uses the onload handler to determine when the page 140 .Y is fully loaded, and signals the crawler when this occurs.
- Browser 205 also sends the address for page 140 .Y to the crawler 171 for use in processing.
- the browser 205 sends to the crawler 171 any cookie that the server 100 sent in association with the query 250 , so the crawler can use the cookie when requesting web page 140 .Y.
- the crawler is programmable to perform particular actions sequences for generating queries to the web server.
- the crawler is programmable to perform particular actions sequences for generating queries to the web server.
- Web page 140 .X is shown in further detail.
- the page 140 .X has two lists 301 and 302 , for selecting parameters for generating a query.
- First list 301 is for selecting a state.
- Second list 302 is for a profile.
- the user performs the following action sequence 305 :
- the crawler 171 of the present invention is programmed as shown at 310 to perform particular actions sequences for generating information 230 to pass to the JEE 210 for generating queries to the web server 100 .
- the information 230 includes the JavaScript command that invokes script 303 when button 304 is clicked.
- JavaScript execution engine 210 Due to JavaScript execution engine 210 , a further difficulty arises for which there is no obvious solution. For security reasons, according to HTML conventions, an applet running on a browser can only interact with objects in a web page if the web page and the applet are loaded onto the client from the same server. Furthermore, the applet cannot conventionally communicate with the client except in a very limited manner. To overcome these limitations, another inventive aspect arises.
- the client browser is configured to use a certain proxy host 210 , and refer to a certain hostname-to-IP-address-resolution resolver file 405 containing cross-referencing for hostname-to-IP-address-resolution.
- the proxy host 215 for the client's browser 205 has a certain IP address, shown for illustration as “IP Addr. Y.”
- the server 100 has a certain hostname, illustrated as “Hostname X,” and a certain IP address, illustrated as “IP Address X.”
- the resolver file 405 is configured so that it indicates the server 100 hostname, “Hostname X” equals proxy host 215 IP address, “IP Addr.
- the JavaScript execution engine running on the client appears to the client's browser to be from the same server as the web pages. This permits the JEE 210 to communicate with the browser 205 unhindered by certain conventional limitations. Since the proxy host 215 is not really the source of the web pages, it may be referred to as a “spoof proxy.”
- the crawler 171 needs to know when a web page being received from the server has been fully loaded by the browser 205 .
- the “onload” attribute There is a conventional HTML document attribute that is useful for this purpose, the “onload” attribute.
- the proxy host 215 adds the onload attribute to each received web page which does not already have the attribute, and defines an event handler that sets a certain variable. Then, this variable is polled by the JEE to determine when the page is loaded, and the JEE 210 171 signals the crawler accordingly.
- a flow chart is shown for method steps, in accordance with an embodiment of the present invention. (It should be understood that the steps are not necessarily performed strictly in the sequence shown. Some variation is sequence is contemplated according to the invention.) Beginning at step 505 , a first web page is loaded. Next, at step 510 the page is parsed and information 230 is sent to the JEE 210 . The JEE 210 and browser 205 determine an address from the information 230 , in step 515 , and the browser sends a query for the server 100 , in step 520 . Next, in step 525 , a second web page is received by the proxy host 215 .
- step 530 the proxy host inserts an onload handler in the second page.
- step 535 the page is loaded in the browser 205 , and a variable set by the onload handler is polled to determine when the page is loaded.
- step 540 the address and cookie for the page are sent from the browser to the crawler 171 .
- step 545 the first web page is processed. A more detailed variation of step 545 is also shown, wherein a new version of the first web page is created, with a local file name, by replacing operations that would otherwise generate dynamic data, and linking the first page to a new version of the second page, the second page also being assigned a local file name.
- HTML 4.0 and JavaScript 1.1 are supported by both Netscape Navigator 3.x and above, and Internet Explorer 4.x and above, both of which are now fairly the de facto standard browsers.
- Microsoft's implementation of JavaScript is officially known as JScript, and can be considered to be the same as JavaScript for the purposes herein.
- the invention is not limited to these embodiments, and is equally applicable to embodiments with other scripting languages such as Visual Basic or Tcl, or to embodiments with Java or cookies.
- An advantage of implementing the invention with JavaScript is that it is supported by most browsers.
- crawler While “staticising” a web site is an application described herein for the crawler of the present embodiment, it should be understood that other applications are within the scope of use of the crawler of the present invention.
- the crawler may also be used to archive a web site.
- FIG. 1 may vary depending on the implementation.
- other peripheral devices such as optical disk drives and the like, may be used in addition to or in place of the hardware depicted in FIG. 1 .
- the depicted example is not meant to imply architectural limitations with respect to the present invention.
- the processes of the present invention may be applied to multiprocessor data processing systems.
Abstract
A web site page has a reference for providing an address for a next page. The web site is crawled by the crawler program, which parses the reference from one of the web pages and sends the reference to an applet running in the browser. The address for the next page is determined by the browser responsive to the reference and is sent to the crawler.
Description
- 1. Field of the Invention
- In one embodiment, this invention concerns crawling a web site on the World Wide Web (“WWW”), and more specifically a web site wherein at least one web page in the web site has a reference for executing by a browser to produce an address for a next page. In this context, the invention also concerns web pages that are dynamically generated responsive to queries, such as queries associated with the crawling. This includes preemptively transforming dynamically generated web pages into static web pages.
- 2. Description of the Related Art
- The World Wide Web is an interconnected network of computers and information appliances. Clients use the WWW to send requests to servers, which send back responses. “Static data” is server response data that already exists on the server at the time of the request and that is merely served back to the client without change by the server. A news article is an example of static data. While the news may change daily, or even minute-by-minute, nevertheless, according to the typical scenario, an article is created in response to a news item, and then the article is put on the server as a static document. That is, the article itself does not subsequently change, even though other, newer articles will sooner or later be placed on the server too. This is in contrast to “dynamic data” that is created by the server in direct response to a client request. A web page displaying a bank account balance or stock positions in a trading account is an example of a page that is created by the server and that may change with each interaction between the client and the server.
- Dynamically generating data tends to impose a substantial load on server resources. While this load could be mitigated by precomputing responses in anticipation of all possible requests, such a course of action will require substantial resources for both computing the responses and for storing them. Some middle ground may be ideal, where the server stores most likely requested information in static form, and creates other information on demand.
- A web site may thus choose to convert a subset of its content to be delivered statically in order to reduce server resource demands. The conversion may be done manually, by employing web page designers. However, such a scheme is inflexible since any changes to the data or its presentation will require a large number of web pages to be manually recreated. An alternative is to automatically generate the static responses using the raw data on the web site. A program could be set up to format the extracted data (such as from database queries) and encapsulate it within the appropriate HTML content, thus creating static pages. The disadvantage of this approach is that the program must be provided with the parameters with which to query the database as well as the inter-document hierarchy which specifies how the documents will be hyperlinked together. Determining and providing this information requires significant resources. There is thus a need for an automated method that can be used to easily convert subsets of a web site to static content.
- Use of a crawler would be advantageous for automating this conversion of a web site to static content, however there are numerous difficulties that prevent this. Conventional search engine crawlers start with a URL and repeatedly de-reference all unexplored URLs in the received responses. One reason conventional crawlers are not suitable for the “staticizing” problem concerns actions sequences that must be performed to obtain a particular end data set for a conventional HTML query. Furthermore, references from one web page to another may not be straightforward. That is, a reference may not be simply set out on the page as a hyperlink address, but instead may be a script, form, selection menu, or button for example. Thus a need exists for improvements in crawler programs, to overcome their limitations so that they may be used for the staticizing problem as well as other applications.
- The foregoing need is addressed in the present invention. According to the invention, a web server that dynamically generates responses is queried and the resulting responses are saved so that the saved static data can be used on the server to avoid subsequent dynamic content generation. An advantage of this approach is that the existing dynamic content generating mechanisms present in the web site are harnessed to do much of the work of producing the static web pages.
- In one embodiment, a crawler program “crawls the web site,” that is, generates queries to the server, responsive to references from one web page to another in the content provided by web site. The queries cause the server to generate dynamic data and return web pages that include those results. These web pages are then processed and saved, so that the web pages can be served in response to future queries. This eliminates the need to generate dynamic data on demand for these particular queries.
- The web pages dynamically generated in response to particular requests, may have included in them operations that would call for the server to generate more dynamic data if the received data were used to generate additional requests to the server. That is, if one of the pages as received was used to generate additional requests to the server, such as by a user filling in fields on a form in the page, clicking on a button, selecting an item in a selection menu, etc., this would cause the server to generate more data dynamically. In another aspect, the received web pages are converted by removing at least some of these operations and replacing them with references to corresponding ones of the returned web pages.
- It is advantageous that since the web pages are produced by interaction of the crawler and the web site, if the web site changes, then static web pages can be updated merely by having the crawler program crawl the web site again.
- The term “crawler” conventionally refers to a search engine, going through the WWW to obtain data to build indices. As stated above, one reason conventional crawlers are not suitable for the present invention concerns actions sequences that must be performed to obtain a particular end data set for a conventional HTML query. Accordingly, in one aspect of an embodiment of the crawler of the present invention, the crawler is programmable to perform particular actions sequences for generating queries to the web server.
- Furthermore, references from one web page to another may not be straightforward. That is, as stated above, a reference may not be simply set out on the page as a hyperlink address, but instead may be specified by a script, for example, so that the address is produced only when a client browser executes the reference. A reference that is generated in this manner is very dependent on the context in which it is produced, that is, the history that led up to it, including the state of the server and the client browser. To generate references of this sort in connection with generating the requests to the server, another aspect of the invention arises. According to an embodiment, the crawler parses each received web page and sends references to an applet developed for an embodiment of the present invention that runs in the browser. (This applet may be referred to herein as a “JavaScript execution engine” or simply “JEE.”) The browser determines the address for a next page responsive to such a reference, so that the browser may receive the next page and any cookie for the next page from the server, and the JEE returns the address and any cookie to the crawler program.
- Due to this JavaScript execution engine, a further difficulty arises for which there is no obvious solution. For security reasons, according to browser conventions, an applet running on a client browser can only interact with objects in a web page if the web page and the applet are loaded onto the client from the same server. Furthermore, the applet cannot conventionally communicate with the client except in a very limited manner. To overcome these limitations, another aspect arises. According to an embodiment, the client browser is configured to use a certain proxy gateway, and refer to a certain file (referred to herein as the “resolver file”) containing cross-referencing for hostname-to-IP-address-resolution. The proxy gateway for the client's browser has a certain IP address. The resolver file is configured so that the resolver file indicates the IP address of the proxy gateway as the IP address for the web site server. In this manner, the JavaScript execution engine running on the client appears to the client's browser to be from the same server as the web pages, which permits the JEE to communicate with the crawler and the browser unhindered by conventional limitations. Since the proxy gateway is not really the source of the web pages, it may be referred to herein as a “spoof proxy.”
- Also due to the JEE another issue arises. In connection with parsing web pages, the JEE needs to know when a web page being received from the server has been fully loaded. There is a conventional HTML document attribute that is useful for this purpose, the “onload” attribute. However, in a normal case the web pages being processed by the JEE were not produced with the JEE in mind. That is, the onload attribute may not be included in the web pages, or may not be included in a fashion suited for use with the JEE. Therefore, in another aspect, the spoof proxy adds the onload attribute to each received web page which does not already have the attribute, and defines an event handler that sets a certain variable. Then, this variable is polled by the JEE to determine when the page is loaded.
- In another embodiment, a computer program product provides a crawler for crawling a web site and reduces dynamic data generation on the web site server. The computer product includes instructions for querying a web site server by a crawler program responsive to references from one web page to another in the content provided by web site. The queries are for causing the server to generate data and return web pages that include the data. The product also includes instructions for processing and saving versions of web pages received from the server, so that the web pages can be served in response to future queries.
- In a further aspect, the computer program product includes instructions for processing the received web pages, which includes removing at least some of these operations and replacing them with references corresponding to the returned web pages.
- In another aspect, the computer program product includes instructions for causing the crawler to parse the received web pages and send references to an applet running in a client browser. The instructions also include instructions for causing the browser to determine the address for a next page responsive to the references, so that the browser may receive the next page and any cookie for the next page from the server. In this aspect, the instructions also include instructions for causing the applet to return the address and any cookie to the crawler program.
- In another aspect, the computer program product includes instructions for causing the proxy to add an onload attribute to a received web page, and define an event handler that sets a certain variable, and instructions for causing the applet to poll the certain variable to determine when the page is loaded.
- In another embodiment, an apparatus includes a processor connected to a network. A storage device is connected to the processor and the network for storing a program for controlling the processor. The processor is operative with the program to query a web site server, responsive to references from one web page to another in the content provided by web site. The processor is also operative with the program to process and save versions of web pages received from the server, so that the web pages can be served in response to future queries.
- In another aspect, the processor is operative with the program to convert the received web pages, by removing at least some of these operations and replacing them with references to corresponding ones of the returned web pages.
- In a further aspect, the processor is operative with the program to parse the received web pages and send references to an applet executing within a client browser. Also, the processor is further operative with the program to cause the applet to cause the browser to determine the address for a next page responsive to such a reference, so that the browser may receive the next page and any cookie for the next page from the server. Also, the processor is further operative with the program to cause the crawler to receive the address and any cookie from the JEE.
- In a still further aspect, the processor is operative with the program to cause the proxy to add an onload attribute to a received web page, and define an event handler therefor that sets a certain variable, and cause the applet to poll the certain variable to determine when the page is loaded.
- It should be appreciated from the foregoing that the invention advantageously reduces limitations in the usefulness of crawler programs. In one resulting application, the improved crawler eliminates the need to generate data dynamically for certain queries by creating static web pages for those queries. This reduces computational load on a web server. It is also advantageous that since the static web pages are produced by interaction of the crawler and the web site server, if the web site changes, then the static web pages can be updated merely by having the crawler program crawl the web site again. These and other advantages of the invention will be further apparent from the following drawings and detailed description.
- The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:
-
FIG. 1 illustrates crawling references among web pages, in accordance with an embodiment of the present invention. -
FIG. 2 illustrates receiving, processing and saving versions of the web pages, in accordance with an embodiment of the present invention. -
FIG. 3 illustrates performing an action sequence, in accordance with an embodiment of the present invention. -
FIG. 4 illustrates certain interactions among a crawler, applet for JavaScript execution, browser, spoof proxy and web server, in accordance with an embodiment of the present invention. -
FIG. 5 illustrates a flow chart for method steps, in accordance with an embodiment of the present invention. - Referring now to
FIG. 1 , aserver 100 is shown for hosting a web site. The server has aprocessor 102 coupled to abus 106 through abridge 108. Also coupled to the bus through the bridge ismemory 104.Disk 126 is coupled to the bus through anadapter 112 for storing web pages to be served up to users, such asclient 170, over thenetwork 141 which is coupled to the bus throughadapter 110.Server 100 uses its resources (processor 102,bus 106,bridge 108,memory 104,adaptor 110, and disk 126) to dynamically generate responses to queries received over thenetwork adaptor 110. -
Client 170 has acrawler program 171 for generating queries to theserver 100 responsive to references from one web page to another in the content delivered byweb server 100. Specifically, first page 140.1 is shown having afirst reference 142 linking page 140.1 to page 140.2. Thecrawler program 171 inclient 170, responsive toreference 142, queriesserver 100 for the second web page 140.2. Then, upon receiving web page 140.2 theclient 170queries server 100 for web page 140.3, responsive toreference 146. (It should be understood that the queries could proceed in a different sequence, instead of the depth-first search illustrated inFIG. 1 . For example, each reference in the top page 140.1 could be crawled to a next page before crawling any references in any of the next pages.) - After having received page 140.3, the
client 170queries server 100 for web page 140.4 responsive toreference 150, and queries for web page 140.5 responsive toreference 152. Since page 140.5 is a JavaScript source page, the crawler program does not produce any queries responsive to page 140.5. Page 140.4 does not have any references to other web pages, so the crawling has reached a dead end with web pages 140.5 and 140.4. - Returning back up the hierarchy of pages to page 140.2, the crawler queries for web page 140.6 responsive to reference 148 in page 140.2. In response to
reference 154 in web page 140.6, the crawler queries for page 140.7. Page 140.7 has onereference 158, which leads to a web page outside a boundary that has been specified to crawler, so this reference is not crawled. - Returning back up the hierarchy of pages to page 140.1, responsive to reference 144
crawler 171 queries for page 140.8. Upon receiving page 140.8,crawler 171 queries for page 140.7, responsive to reference 160.Crawler 171 determines that reference 160 is a reference to page 140.7, which is in common withreference 154, so reference 160 is not crawled. Responsive to reference 162,crawler 171 queries so thatclient 170 receivespage 150. Sincepage 150 is an image, this page is not crawled. - In the above discussion, it must be noted that the references between two pages could be more than just conventional hyperlinks. The references could be script references, selection menus, forms, buttons or other elements.
- Referring now to
FIG. 2 ,client 170 is shown with programs running, includingbrowser 205 andcrawler 171. (It should be understood that programs run on a processor (not shown) ofclient 170, responsive to instructions stored onclient 170 storage, such asstorage 226.) A first web page 140.X is loaded in thebrowser 205. Thecrawler 171 includes afunction 220 for parsing information from web pages loaded in thebrowser 205, such as 140.X, and passing theinformation 230 to thebrowser 205 for interpreting the information, which concerns references to other pages, such as that from 140.X to 140.Y, and generating queries for the other pages. - To understand the context of these operations, it should be understood that a reference may not be simply set out on a web page as an address, but instead may be specified by a script, selection menu, form, button or other element, so that the address needs to be produced by the client browser executing the reference. A reference that is generated in this manner is very dependent on the context in which it is produced, that is, the history that led up to it, including the state of the server and the client browser. So to generate references of this sort in connection with generating the requests to the server, the
crawler 171 parses each received web page, such as page 140.X, and sends references parsed from the page to aJavaScript execution engine 210 that runs in thebrowser 205, for generating a query for web page 140.Y. That is,browser 205 determines from theinformation 230 that page 140.Y is referenced in referencing information in page 140.X. - It must also be understood that the
crawler 171 may directly obtain the page 140.X from theweb server 100 and parse it, consultingbrowser 205 only when encountering references other than simple hyperlinks, such as that linking page 140.X to page 140.Y. -
Browser 205 generates an address for the page 140.Y and sends the address in aquery 250 to aproxy host 215, which passes the query toserver 100.Server 100 returns the web page 140.Y, andproxy host 215 adds an onload handler to the page sends the page 140.Y tobrowser 205. - It should be understood that web page 140.X includes the results of data dynamically generated in
server 100 for anearlier query 250 that resulted in the page 140.X, but 140.X also has operations included in it linking it to page 140.Y, that call for theserver 100 to generate more dynamic data to produce web page 140.Y. Therefore, web page 140.X needs to be converted by removing these operations and replacing them with references to a “version of” the returned web page 140.Y. The reference must be to 140.Y1, a “version of” web page 140.Y, because 140.Y may itself have operations included linking it to a next page also calling for theserver 100 to generate more dynamic data to produce the next page. - To deal with these issues, the crawler also includes a
processing function 225. Thebrowser 205 passesinformation 240 back to thecrawler 171 for the crawler to use for processing the web pages and saving versions thereof. In the illustration, web pages 140.X and 140.Y are processed to generate new versions 140.X1 and 140.Y1, which are saved instorage 226. The browser uses the onload handler to determine when the page 140.Y is fully loaded, and signals the crawler when this occurs.Browser 205 also sends the address for page 140.Y to thecrawler 171 for use in processing. Also, thebrowser 205 sends to thecrawler 171 any cookie that theserver 100 sent in association with thequery 250, so the crawler can use the cookie when requesting web page 140.Y. - In one aspect of an embodiment of the crawler of the present invention, the crawler is programmable to perform particular actions sequences for generating queries to the web server. To clarify, consider an example of actions performed by a user to obtain a particular end data set using a conventional web page form.
- Referring now to
FIG. 3 , Web page 140.X is shown in further detail. The page 140.X has twolists First list 301 is for selecting a state.Second list 302 is for a profile. In the example, in a conventional web page access, where a user is controllingbrowser 205, if the user wants to obtain income information for the state of Texas from a Community Facts page in the American Fact Finder web site (factfinder.census.gov), the user performs the following action sequence 305: - 1. Select Texas as the State from
list 301.
2. Select Income as the Profile fromlist 302.
3. Click on the createbutton 304, which causes ascript 303 to queryserver 100 with a request that includes the parameters selected from thelists crawler 171 of the present invention, the crawler is programmed as shown at 310 to perform particular actions sequences for generatinginformation 230 to pass to theJEE 210 for generating queries to theweb server 100. In the example, theinformation 230 includes the JavaScript command that invokesscript 303 whenbutton 304 is clicked. Also including in theinformation 230 is a context object, the browser window object, and the document object associated with page 140.X in its context as it exists, loaded inbrowser 205. - Due to
JavaScript execution engine 210, a further difficulty arises for which there is no obvious solution. For security reasons, according to HTML conventions, an applet running on a browser can only interact with objects in a web page if the web page and the applet are loaded onto the client from the same server. Furthermore, the applet cannot conventionally communicate with the client except in a very limited manner. To overcome these limitations, another inventive aspect arises. - Referring now to
FIG. 4 , according to an embodiment, the client browser is configured to use acertain proxy host 210, and refer to a certain hostname-to-IP-address-resolution resolver file 405 containing cross-referencing for hostname-to-IP-address-resolution. Theproxy host 215 for the client'sbrowser 205 has a certain IP address, shown for illustration as “IP Addr. Y.” Likewise, theserver 100 has a certain hostname, illustrated as “Hostname X,” and a certain IP address, illustrated as “IP Address X.” Theresolver file 405 is configured so that it indicates theserver 100 hostname, “Hostname X” equalsproxy host 215 IP address, “IP Addr. “Y.” In this manner, the JavaScript execution engine running on the client appears to the client's browser to be from the same server as the web pages. This permits theJEE 210 to communicate with thebrowser 205 unhindered by certain conventional limitations. Since theproxy host 215 is not really the source of the web pages, it may be referred to as a “spoof proxy.” - Also due to the
JEE 210 another issue arises. In connection with parsing web pages, thecrawler 171 needs to know when a web page being received from the server has been fully loaded by thebrowser 205. There is a conventional HTML document attribute that is useful for this purpose, the “onload” attribute. However, in a normal case the web pages being processed by the JEE were not produced with the JEE in mind. That is, the onload attribute may not be included in the web pages, or may not be included in a fashion suited for use with the JEE. Therefore, in another aspect as shown inFIG. 2 , theproxy host 215 adds the onload attribute to each received web page which does not already have the attribute, and defines an event handler that sets a certain variable. Then, this variable is polled by the JEE to determine when the page is loaded, and theJEE 210 171 signals the crawler accordingly. - Referring now to
FIG. 5 , a flow chart is shown for method steps, in accordance with an embodiment of the present invention. (It should be understood that the steps are not necessarily performed strictly in the sequence shown. Some variation is sequence is contemplated according to the invention.) Beginning atstep 505, a first web page is loaded. Next, atstep 510 the page is parsed andinformation 230 is sent to theJEE 210. TheJEE 210 andbrowser 205 determine an address from theinformation 230, instep 515, and the browser sends a query for theserver 100, instep 520. Next, instep 525, a second web page is received by theproxy host 215. Then, instep 530, the proxy host inserts an onload handler in the second page. Then, instep 535, the page is loaded in thebrowser 205, and a variable set by the onload handler is polled to determine when the page is loaded. Also, instep 540, the address and cookie for the page are sent from the browser to thecrawler 171. Instep 545, the first web page is processed. A more detailed variation ofstep 545 is also shown, wherein a new version of the first web page is created, with a local file name, by replacing operations that would otherwise generate dynamic data, and linking the first page to a new version of the second page, the second page also being assigned a local file name. - It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in the form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media such a floppy disc, a hard disk drive, a RAM, and CD-ROMs and transmission-type media such as digital and analog communications links.
- The description of the present embodiment has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. For example, the invention leverages scripting and event notification mechanisms in HTML 4.0 and JavaScript 1.1 to measure and collect response time experienced by a client. HTML 4.0 and JavaScript 1.1 are supported by both Netscape Navigator 3.x and above, and Internet Explorer 4.x and above, both of which are now fairly the de facto standard browsers. Microsoft's implementation of JavaScript is officially known as JScript, and can be considered to be the same as JavaScript for the purposes herein. The invention is not limited to these embodiments, and is equally applicable to embodiments with other scripting languages such as Visual Basic or Tcl, or to embodiments with Java or cookies. An advantage of implementing the invention with JavaScript, is that it is supported by most browsers.
- While the embodiment described herein uses an applet executing within a browser as a script execution engine, it should be understood that other embodiments of the script execution engine are within the scope of the invention. For example, the Gecko browser engine from Netscape could be used for the execution engine.
- While “staticising” a web site is an application described herein for the crawler of the present embodiment, it should be understood that other applications are within the scope of use of the crawler of the present invention. For example, the crawler may also be used to archive a web site.
- Those of ordinary skill in the art will appreciate that the hardware in
FIG. 1 may vary depending on the implementation. For example, other peripheral devices, such as optical disk drives and the like, may be used in addition to or in place of the hardware depicted inFIG. 1 . The depicted example is not meant to imply architectural limitations with respect to the present invention. For example, the processes of the present invention may be applied to multiprocessor data processing systems. - The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention. Various other embodiments having various modifications may be suited to a particular use contemplated, but may be within the scope of the present invention.
Claims (24)
1. A method for crawling a web site, the method comprising:
querying a web site server by a crawler program, wherein at least one page of the web site has a reference, wherein the reference is specified by a script to produce an address for a next page;
parsing such a reference from one of the web pages by the crawler program and sending the reference to an applet running in a browser; and
determining the address for the next page by the browser executing the reference and sending the address to the crawler, wherein the crawler is programmable to perform particular action sequences for selecting non-hypertext-link parameters from the at least one web page in a particular sequence, so that the queries to the web server include the selected parameters and a context arising from the particular sequence.
2. The method of claim 1 , the browser being configured to use a certain proxy and refer to a resolver file for hostname-to-IP-address-resolution, wherein the web site server has an IP address and the proxy for the browser has a certain IP address, the certain IP address of the proxy being different than the IP address of the web site server, and wherein the resolver file indicates the certain IP address of the proxy as the IP address for the web site server.
3. The method of claim 2 , comprising:
adding an onload attribute to one of the web pages by the proxy;
defining an event handler for the onload attribute by the proxy, wherein the event handler sets a certain variable; and
polling the certain variable by the applet to determine when the page is loaded.
4. (canceled)
5. The method of claim 1 , at least one of the web pages being dynamically generated by the server responsive to corresponding ones of the queries, comprising:
processing the server generated web pages to generate corresponding processed versions of the web pages, so that the processed versions can be served in response to future queries, reducing dynamic generation of web pages by the server.
6. The method of claim 5 , wherein at least a first such server generated web page has included in it an operation that would cause the server to dynamically generate a second web page if the first page were used to generate further requests to the server, and wherein processing the server generated web pages comprises:
removing the operation from the first server generated web page and replacing the operation with a reference to a version of another of the server generated web pages.
7. (canceled)
8. (canceled)
9. A computer program product for crawling a web site, wherein the computer program product resides on a computer readable, recordable-type medium having computer readable program code, the program code comprising:
first instructions for querying a web site server by a crawler program, wherein at least one page of the web site has a reference, wherein the reference is specified by a script for producing an address for a next page;
second instructions for parsing such a reference from one of the web pages by the crawler program and sending the reference to an applet running in a browser; and
third instructions for determining the address for the next page by the browser executing the reference and sending the address to the crawler, wherein the first instructions comprise instructions for causing the crawler to perform particular action sequences for selecting non-hypertext-link parameters from the at least one web page in a particular sequence, so that the queries to the web server include the selected parameters and a context arising from the particular sequence.
10. The computer program product of claim 9 , the browser being configured to use a certain proxy, and refer to a resolver file for hostname-to-IP-address-resolution, wherein the web site server has an IP address and the proxy for the browser has a certain IP address, certain IP address of the proxy being different than the IP address of the web site server, and wherein the resolver file indicates the certain IP address of the proxy as the IP address for the web site server.
11. The computer program product of claim 10 , comprising:
fourth instructions for adding an onload attribute to one of the web pages by the proxy;
fifth instructions for defining an event handler for the onload attribute by the proxy, wherein the event handler sets a certain variable; and
sixth instructions for polling the certain variable by the applet to determine when the page is loaded.
12. (canceled)
13. The computer program product of claim 9 , at least one of the web pages being dynamically generated by the server responsive to corresponding ones of the queries, the computer program product comprising:
instructions for processing the server generated web pages to generate corresponding processed versions of the web pages, so that the processed versions can be served in response to future queries, reducing dynamic generation of web pages by the server.
14. The computer program product of claim 13 , wherein at least a first such server generated web page has included in it an operation that would cause the server to dynamically generate a second web page if the first page were used to generate further requests to the server, and wherein the instructions for processing the server generated web pages to generate corresponding processed versions of the web pages comprise:
instructions for removing the operation from the first server generated web page and replacing the operation with a reference to a version of another of the server generated web pages.
15. (canceled)
16. (canceled)
17. An apparatus for crawling a web site, the apparatus comprising:
a processor connected a network,
a storage device connected to the processor and the network, wherein the storage device is for storing a program for controlling the processor, and wherein the processor is operative with the program to execute a crawler program and a browser program for:
querying a web site server by the crawler, wherein at least one page of the web site has a reference, wherein the reference is specified by a script for producing an address for a next page;
parsing such a reference from one of the web pages and sending the reference to an applet running in a browser; and
determining the address for the next page by the browser executing the reference and sending the address to the crawler.
18. The apparatus of claim 17 , the browser being configured to use a certain proxy and refer to a resolver file for hostname-to-IP-address-resolution, wherein the web site server has an IP address and the proxy for the browser has a certain IP address, the certain IP address of the proxy being different than the IP address of the web site server, and wherein the resolver file indicates the certain IP address of the proxy as the IP address for the web site server.
19. The apparatus of claim 18 , wherein an onload attribute is added to one of the web pages by the proxy, and an event handler is defined for the onload attribute to set a certain variable, and wherein the processor is operative with the program for:
polling the certain variable by the applet to determine when the page is loaded.
20. (canceled)
21. The apparatus of claim 17 , at least one of the web pages being dynamically generated by the server responsive to corresponding ones of the queries, wherein the processor is operative with the program for:
processing the server generated web pages to generate corresponding processed versions of the web pages, so that the processed versions can be served in response to future queries, reducing dynamic generation of web pages by the server.
22. The apparatus of claim 21 , wherein at least a first such server generated web page has included in it an operation that would cause the server to dynamically generate a second web page if the first page were used to generate further requests to the server, and wherein processing the server generated web pages comprises:
removing the operation from the first server generated web page and replacing the operation with a reference to a version of another of the server generated web pages.
23. (canceled)
24. (canceled)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/600,201 US20130041882A1 (en) | 2000-12-14 | 2012-08-30 | Technology for web site crawling, including action sequences for selecting non-hypertext-link parameters |
US14/169,115 US9165077B2 (en) | 2000-12-14 | 2014-01-30 | Technology for web site crawling |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/736,349 US8452850B2 (en) | 2000-12-14 | 2000-12-14 | Method, apparatus and computer program product to crawl a web site |
US13/600,201 US20130041882A1 (en) | 2000-12-14 | 2012-08-30 | Technology for web site crawling, including action sequences for selecting non-hypertext-link parameters |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/736,349 Continuation US8452850B2 (en) | 2000-12-14 | 2000-12-14 | Method, apparatus and computer program product to crawl a web site |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/169,115 Continuation US9165077B2 (en) | 2000-12-14 | 2014-01-30 | Technology for web site crawling |
Publications (1)
Publication Number | Publication Date |
---|---|
US20130041882A1 true US20130041882A1 (en) | 2013-02-14 |
Family
ID=24959532
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/736,349 Expired - Fee Related US8452850B2 (en) | 2000-12-14 | 2000-12-14 | Method, apparatus and computer program product to crawl a web site |
US13/600,201 Abandoned US20130041882A1 (en) | 2000-12-14 | 2012-08-30 | Technology for web site crawling, including action sequences for selecting non-hypertext-link parameters |
US14/169,115 Expired - Fee Related US9165077B2 (en) | 2000-12-14 | 2014-01-30 | Technology for web site crawling |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/736,349 Expired - Fee Related US8452850B2 (en) | 2000-12-14 | 2000-12-14 | Method, apparatus and computer program product to crawl a web site |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/169,115 Expired - Fee Related US9165077B2 (en) | 2000-12-14 | 2014-01-30 | Technology for web site crawling |
Country Status (1)
Country | Link |
---|---|
US (3) | US8452850B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150205470A1 (en) * | 2012-09-14 | 2015-07-23 | Ca, Inc. | Providing a user interface with configurable interface components |
US20150234929A1 (en) * | 2011-06-24 | 2015-08-20 | Verisign, Inc. | Multi-mode barcode resolution system |
CN106021257A (en) * | 2015-12-31 | 2016-10-12 | 广州华多网络科技有限公司 | Method, device, and system for crawler to capture data supporting online programming |
CN106547778A (en) * | 2015-09-21 | 2017-03-29 | 北京国双科技有限公司 | The crawling method and device of webpage |
WO2021226954A1 (en) * | 2020-05-14 | 2021-11-18 | 深圳市欢太科技有限公司 | Information crawling method and apparatus, and electronic device and storage medium |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2000065763A2 (en) | 1999-04-26 | 2000-11-02 | Dodots, Inc. | Apparatus and method for delivering internet content |
US8452850B2 (en) | 2000-12-14 | 2013-05-28 | International Business Machines Corporation | Method, apparatus and computer program product to crawl a web site |
US20030056173A1 (en) * | 2001-01-22 | 2003-03-20 | International Business Machines Corporation | Method, system, and program for dynamically generating input for a test automation facility for verifying web site operation |
US8527495B2 (en) * | 2002-02-19 | 2013-09-03 | International Business Machines Corporation | Plug-in parsers for configuring search engine crawler |
US7496636B2 (en) * | 2002-06-19 | 2009-02-24 | International Business Machines Corporation | Method and system for resolving Universal Resource Locators (URLs) from script code |
US7584216B2 (en) * | 2003-02-21 | 2009-09-01 | Motionpoint Corporation | Dynamic language translation of web site content |
US20050192948A1 (en) * | 2004-02-02 | 2005-09-01 | Miller Joshua J. | Data harvesting method apparatus and system |
US7802262B1 (en) | 2004-03-05 | 2010-09-21 | Adobe Systems Incorporated | System and method for communicating state and title information between a browser and a rich internet application with browser forward and back button support |
US7536389B1 (en) * | 2005-02-22 | 2009-05-19 | Yahoo ! Inc. | Techniques for crawling dynamic web content |
US20070005606A1 (en) * | 2005-06-29 | 2007-01-04 | Shivakumar Ganesan | Approach for requesting web pages from a web server using web-page specific cookie data |
US20070022085A1 (en) * | 2005-07-22 | 2007-01-25 | Parashuram Kulkarni | Techniques for unsupervised web content discovery and automated query generation for crawling the hidden web |
US8001458B1 (en) * | 2005-11-14 | 2011-08-16 | Adobe Systems Incorporated | System and method for communicating state and title information between a browser and a rich Internet application |
US8380855B2 (en) | 2005-12-12 | 2013-02-19 | Answer Financial, Inc. | HTTP header intermediary for enabling session-based dynamic site searches |
US20080120257A1 (en) * | 2006-11-20 | 2008-05-22 | Yahoo! Inc. | Automatic online form filling using semantic inference |
US8819215B2 (en) * | 2007-01-29 | 2014-08-26 | Nokia Corporation | System, methods, apparatuses and computer program products for providing step-ahead computing |
US20080235567A1 (en) * | 2007-03-22 | 2008-09-25 | Binu Raj | Intelligent form filler |
US8572065B2 (en) * | 2007-11-09 | 2013-10-29 | Microsoft Corporation | Link discovery from web scripts |
US7899807B2 (en) * | 2007-12-20 | 2011-03-01 | Yahoo! Inc. | System and method for crawl ordering by search impact |
US9411793B2 (en) | 2010-07-13 | 2016-08-09 | Motionpoint Corporation | Dynamic language translation of web site content |
US20120310913A1 (en) * | 2011-05-30 | 2012-12-06 | Jaimie Sirovich | System and method to access a plurality of document results pages |
WO2013051005A2 (en) * | 2011-07-06 | 2013-04-11 | Kanani Hirenkumar Nathalal | A method of a web based product crawler for products offering |
US8990183B2 (en) * | 2012-06-06 | 2015-03-24 | Microsoft Technology Licensing, Llc | Deep application crawling |
CA2786418C (en) * | 2012-08-16 | 2020-04-14 | Ibm Canada Limited - Ibm Canada Limitee | Identifying equivalent javascript events |
US11048858B2 (en) | 2012-11-21 | 2021-06-29 | Roofoveryourhead Marketing Ltd. | Browser extension for the collection and distribution of data and methods of use thereof |
US10679259B2 (en) | 2012-11-27 | 2020-06-09 | Synqy Corporation | Method and system for dynamic online digital brand assets |
US9069857B2 (en) * | 2012-11-28 | 2015-06-30 | Microsoft Technology Licensing, Llc | Per-document index for semantic searching |
US9355176B2 (en) | 2013-03-15 | 2016-05-31 | The Nielsen Company (Us), Llc | Methods and apparatus to supplement web crawling with cached data from distributed devices |
US9866537B2 (en) * | 2013-08-02 | 2018-01-09 | Salesforce.Com, Inc. | Systems and methods for long universal resource locator compression |
US10558715B2 (en) * | 2015-06-15 | 2020-02-11 | Google Llc | Automatic crawling of applications |
CN105512193A (en) * | 2015-11-26 | 2016-04-20 | 上海携程商务有限公司 | Data acquisition system and method based on browser expansion |
US11157568B2 (en) * | 2017-11-01 | 2021-10-26 | Sap Se | Offline mode for mobile application |
CN111666465A (en) * | 2019-03-06 | 2020-09-15 | 上海晶赞融宣科技有限公司 | Method and device for crawling data, storage medium and terminal |
US11366862B2 (en) * | 2019-11-08 | 2022-06-21 | Gap Intelligence, Inc. | Automated web page accessing |
CN111538883B (en) * | 2020-03-25 | 2023-11-17 | 北京市科学技术情报研究所 | Data crawling method, system and equipment |
CN111931024A (en) * | 2020-07-10 | 2020-11-13 | 北京邮电大学 | Crawling method and device for dynamic webpage and electronic equipment |
CN112528120A (en) * | 2020-12-21 | 2021-03-19 | 北京中安智达科技有限公司 | Method for web data crawler to use browser to divide body and proxy |
Family Cites Families (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6038610A (en) * | 1996-07-17 | 2000-03-14 | Microsoft Corporation | Storage of sitemaps at server sites for holding information regarding content |
US6189019B1 (en) * | 1996-08-14 | 2001-02-13 | Microsoft Corporation | Computer system and computer-implemented process for presenting document connectivity |
US6038601A (en) * | 1997-07-21 | 2000-03-14 | Tibco, Inc. | Method and apparatus for storing and delivering documents on the internet |
US6026413A (en) * | 1997-08-01 | 2000-02-15 | International Business Machines Corporation | Determining how changes to underlying data affect cached objects |
US6484149B1 (en) * | 1997-10-10 | 2002-11-19 | Microsoft Corporation | Systems and methods for viewing product information, and methods for generating web pages |
GB9721502D0 (en) * | 1997-10-11 | 1997-12-10 | Int Computers Ltd | Computer system for supporting a website |
US6182085B1 (en) * | 1998-05-28 | 2001-01-30 | International Business Machines Corporation | Collaborative team crawling:Large scale information gathering over the internet |
US6256623B1 (en) * | 1998-06-22 | 2001-07-03 | Microsoft Corporation | Network search access construct for accessing web-based search services |
US6389462B1 (en) * | 1998-12-16 | 2002-05-14 | Lucent Technologies Inc. | Method and apparatus for transparently directing requests for web objects to proxy caches |
US20020087559A1 (en) * | 1999-01-11 | 2002-07-04 | PRATT John | Method and system for retrieving documents using hyperlinks |
US6219676B1 (en) * | 1999-03-29 | 2001-04-17 | Novell, Inc. | Methodology for cache coherency of web server data |
JP3485253B2 (en) * | 1999-06-18 | 2004-01-13 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Information processing method, information terminal support server, storage medium for storing information processing program |
US6735169B1 (en) * | 1999-07-02 | 2004-05-11 | Cisco Technology, Inc. | Cascading multiple services on a forwarding agent |
US6449636B1 (en) * | 1999-09-08 | 2002-09-10 | Nortel Networks Limited | System and method for creating a dynamic data file from collected and filtered web pages |
US6301614B1 (en) * | 1999-11-02 | 2001-10-09 | Alta Vista Company | System and method for efficient representation of data set addresses in a web crawler |
US6263364B1 (en) * | 1999-11-02 | 2001-07-17 | Alta Vista Company | Web crawler system using plurality of parallel priority level queues having distinct associated download priority levels for prioritizing document downloading and maintaining document freshness |
US6418452B1 (en) * | 1999-11-03 | 2002-07-09 | International Business Machines Corporation | Network repository service directory for efficient web crawling |
US6665658B1 (en) * | 2000-01-13 | 2003-12-16 | International Business Machines Corporation | System and method for automatically gathering dynamic content and resources on the world wide web by stimulating user interaction and managing session information |
US8452850B2 (en) | 2000-12-14 | 2013-05-28 | International Business Machines Corporation | Method, apparatus and computer program product to crawl a web site |
US8312003B2 (en) * | 2004-12-16 | 2012-11-13 | Hewlett-Packard Development Company, L.P. | Computer method and apparatus for collaborative web searches |
US20060253582A1 (en) * | 2005-05-03 | 2006-11-09 | Dixon Christopher J | Indicating website reputations within search results |
US20090204610A1 (en) * | 2008-02-11 | 2009-08-13 | Hellstrom Benjamin J | Deep web miner |
CN101546309B (en) * | 2008-03-26 | 2012-07-04 | 国际商业机器公司 | Method and equipment for constructing indexes to resource content in computer network |
-
2000
- 2000-12-14 US US09/736,349 patent/US8452850B2/en not_active Expired - Fee Related
-
2012
- 2012-08-30 US US13/600,201 patent/US20130041882A1/en not_active Abandoned
-
2014
- 2014-01-30 US US14/169,115 patent/US9165077B2/en not_active Expired - Fee Related
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150234929A1 (en) * | 2011-06-24 | 2015-08-20 | Verisign, Inc. | Multi-mode barcode resolution system |
US9727657B2 (en) * | 2011-06-24 | 2017-08-08 | Verisign, Inc. | Multi-mode barcode resolution system |
US20150205470A1 (en) * | 2012-09-14 | 2015-07-23 | Ca, Inc. | Providing a user interface with configurable interface components |
US10379707B2 (en) * | 2012-09-14 | 2019-08-13 | Ca, Inc. | Providing a user interface with configurable interface components |
US10387003B2 (en) | 2012-09-14 | 2019-08-20 | Ca, Inc. | User interface with runtime selection of views |
CN106547778A (en) * | 2015-09-21 | 2017-03-29 | 北京国双科技有限公司 | The crawling method and device of webpage |
CN106021257A (en) * | 2015-12-31 | 2016-10-12 | 广州华多网络科技有限公司 | Method, device, and system for crawler to capture data supporting online programming |
WO2021226954A1 (en) * | 2020-05-14 | 2021-11-18 | 深圳市欢太科技有限公司 | Information crawling method and apparatus, and electronic device and storage medium |
Also Published As
Publication number | Publication date |
---|---|
US20020078136A1 (en) | 2002-06-20 |
US20140149382A1 (en) | 2014-05-29 |
US9165077B2 (en) | 2015-10-20 |
US8452850B2 (en) | 2013-05-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9165077B2 (en) | Technology for web site crawling | |
JP2963087B2 (en) | Access mechanism, storage medium, data processing system, access method, web page processing method, and method of providing access mechanism | |
US6222634B1 (en) | Apparatus and method for printing related web pages | |
US6052730A (en) | Method for monitoring and/or modifying web browsing sessions | |
US7467391B2 (en) | Allowing client applications to programmatically access web sites | |
KR100288490B1 (en) | Preserving state in stateless network protocols | |
US8126946B2 (en) | Method, apparatus and computer program for key word searching | |
US6882996B2 (en) | System, method, and computer program product for reformatting non-XML data for use with internet based systems | |
US7290262B2 (en) | Method and apparatus for dynamically determining information for deploying a web service | |
US9111003B2 (en) | Scalable derivative services | |
US7747604B2 (en) | Dynamic sitemap creation | |
US20030005159A1 (en) | Method and system for generating and serving multilingual web pages | |
US20110035553A1 (en) | Method and system for cache management | |
WO1998039867A2 (en) | Replicating forms from html templates | |
EP1446734A2 (en) | Method, system, and software for transmission of information | |
US20010047397A1 (en) | Method and system for using pervasive device to access webpages | |
US8539340B2 (en) | Method to serve real-time data in embedded web server | |
US20080172396A1 (en) | Retrieving Dated Content From A Website | |
US20040117349A1 (en) | Intermediary server for facilitating retrieval of mid-point, state-associated web pages | |
US20040267961A1 (en) | In a World Wide Web communications network simplifying the Uniform Resource Locators (URLS) displayed in association with received web documents | |
US20040034637A1 (en) | Accessing a set of local or distant resources | |
JP2000285052A (en) | Url conversion method and device | |
US20030172126A1 (en) | Method and system for displaying condensed web page information in response to a user query | |
KR20040005868A (en) | Method, computer system and computer program product for updating a web page | |
Van Veen | Linking electronic documents and standardisation of URL’s |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |