WO2010114913A1 - Method and system of retrieving ajax web page content - Google Patents
Method and system of retrieving ajax web page content Download PDFInfo
- Publication number
- WO2010114913A1 WO2010114913A1 PCT/US2010/029444 US2010029444W WO2010114913A1 WO 2010114913 A1 WO2010114913 A1 WO 2010114913A1 US 2010029444 W US2010029444 W US 2010029444W WO 2010114913 A1 WO2010114913 A1 WO 2010114913A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- javascript
- web page
- codes
- function
- ajax
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/14—Tree-structured documents
- G06F40/143—Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/972—Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
Definitions
- the present disclosure relates to fields of networking technologies and, more particularly, to methods and systems of retrieving Aj ax web page content.
- Asynchronous JavaScript and XML is a web page development technology for creating interactive web page applications.
- Aj ax is a web page development technology for creating interactive web page applications.
- a user can see the updated content only after the entire web page is refreshed.
- a user can obtain the updated portion of the content without refreshing the entire web page. From the perspective of a network device, this avoids repeatedly transmitting unaltered information and thus saves network bandwidth and reduces server workload. From a user's perspective, waiting times during web page browsing can be effectively reduced, thus improving user experience.
- a search engine is a system that can collect, organize and process Internet information with the capability of providing search services to users.
- the search engine has become one of the indispensible auxiliary tools of Internet surfing experience for users.
- a web crawler being an important component of the search engine, can automatically retrieve web page content and provide real-time, updated data to the search engine.
- conventional web pages can be supported, existing web crawlers fail to effectively retrieve contents that are generated dynamically by
- Aj ax on Aj ax web pages cannot be collected by the search engine, leading to a failure of the search engine to provide a complete content search service to the user.
- the present disclosure provides a method and a system of retrieving Aj ax web page content and to provide a solution for the failure of a search engine to collect dynamic content in an Ajax web page.
- a technical scheme is described as follows.
- the present disclosure provides a method of retrieving Aj ax web page content, which includes: obtaining information of web page codes; retrieving JavaScript information of the web page codes; analyzing the JavaScript information to determine functions that include Ajax calls in the web page codes; and triggering the functions that include the Ajax calls to obtain web page content.
- the present disclosure further provides a system of retrieving Ajax web page content, which includes: a web page code acquisition unit, used for obtaining information of web page codes; a script retrieving unit, used for retrieving JavaScript information of the web page codes; a script analyzing unit, used for analyzing the JavaScript information to determine functions that include Aj ax calls in the web page codes; and a web page content acquisition unit, used for triggering the functions that include the Aj ax calls to obtain web page content.
- a web page code acquisition unit used for obtaining information of web page codes
- a script retrieving unit used for retrieving JavaScript information of the web page codes
- a script analyzing unit used for analyzing the JavaScript information to determine functions that include Aj ax calls in the web page codes
- a web page content acquisition unit used for triggering the functions that include the Aj ax calls to obtain web page content.
- the above technical scheme obtains functions that include Ajax calls in the web page codes and further triggers the functions to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page.
- a search engine can collect more complete web page contents and hence provide a better search service to users.
- FIG. 1 shows a flow chart of a method of retrieving Ajax web page content in accordance with the exemplary embodiments of the present disclosure.
- FIG. 2 shows a flow chart of a method of retrieving JavaScript information of web page codes in accordance with the exemplary embodiments of the present disclosure.
- FIG. 3 shows a schematic structural diagram of a system of retrieving Ajax web page content in accordance with the exemplary embodiments of the present disclosure.
- FIG. 4 shows a schematic structural diagram of a script retrieving unit in accordance with the exemplary embodiments of the present disclosure.
- FIG. 5 shows a schematic structural diagram of a script analyzing unit in accordance with the exemplary embodiments of the present disclosure.
- FIG. 1 shows a flow chart of a method of retrieving Aj ax web page content in accordance with the exemplary embodiments of the present disclosure.
- the method includes the following procedures.
- 5101 obtains information of web page codes.
- This block is similar to the working mechanism of a conventional web crawler.
- URL Uniform Resource Locator
- 5102 retrieves JavaScript information of the web page codes.
- JavaScript information therein is retrieved first.
- HTML i.e., Hyper Text Mark-up Language
- HTML Hyper Text Mark-up Language
- JavaScript codes can be placed between ⁇ script> and ⁇ /script>, a tag pair in the HTML codes.
- a common format is shown as follows:
- ⁇ script type "text/javascript "> //js codes ⁇ /script>
- the portion within the ⁇ script> tag pair in the above format corresponds to JavaScript codes. Multiple JavaScript sentences may be included within the ⁇ script> tag pair.
- the ⁇ script> tag may appear in ⁇ head> part of the HTML codes or ⁇ body> part of the HTML codes.
- the JavaScript codes are placed in an external file. This external file is then referenced through a src property of the ⁇ script> tag in the HTML codes.
- a common format is shown below:
- Values of the src property define a storage path and a filename of a JavaScript file.
- the storage path may be an absolute path or a relative path, "test.js" in the above format refers to the JavaScript filename being referenced while “js/" refers to the storage path of that file.
- a JavaScript file generally uses .js as its extension name, and includes mere JavaScript codes, having no ⁇ script> tag or other HTML tags.
- the JavaScript information in the HTML codes can be retrieved by checking the ⁇ script> tag in the web page codes.
- a method of retrieving JavaScript information of web page codes may include the following procedures.
- S 102a checks a JavaScript tag in the web page codes.
- S 102b determines the type of JavaScript information located after the JavaScript tag. If the type corresponds to JavaScript codes, S 102c is executed. If the type corresponds to JavaScript files, S102d is executed.
- S 102c retrieves the JavaScript codes within the JavaScript tag's pair.
- S102d retrieves storage paths and filenames of the JavaScript files.
- JavaScript codes and/or JavaScript files and respective storage paths are obtained. All the retrieved JavaScript codes may be stored in a temporary .js file (e.g., temp.js). With respect to the JavaScript files, analyzing all the codes of the JavaScript files is not required in the technical scheme of the present disclosure. Only the retrieved JavaScript filenames and respective storage paths may be stored at this block.
- S 103 analyzes the JavaScript information to determine functions that include Aj ax calls in the web page codes.
- JavaScript framework documents This type of file generally has a fixed filename, prototype.js, for example.
- Another type refers to non-JavaScript framework documents.
- functions that include Ajax calls are defined in the non- JavaScript framework document. As such, contents of the JavaScript framework document may not need to be further analyzed. However, property values, corresponding to its framework type, can be found based on its filename.
- functions that include Ajax calls and are defined in the non-
- JavaScript framework document may be found using these property values. Furthermore, functions that include Ajax calls in the web page codes may then be found. Specific procedures are shown as follows.
- S 103 a determines the file type of a JavaScript file based on the filename of the JavaScript file.
- JavaScript file is a framework document or a non-framework document. If determined to be a framework document, corresponding property values are obtained.
- S103b determines functions, which include Ajax calls, and are defined in a non-JavaScript framework document based on the Ajax property values. In a JavaScript file, all functions that include Ajax calls must include a specific piece of code which are called Ajax property values. Therefore, whether a function includes an Ajax call can be determined based on whether the function includes these property values.
- Ajax property values may be categorized into two types.
- S103c further determines functions that include Ajax calls in the web page codes based on a result of S 103b.
- S 104 triggers the functions that include Ajax calls to obtain web page content. Upon determining which function includes Aj ax calls in the web page code, web page content that is generated dynamically by Aj ax can be obtained through triggering these functions.
- a number of functions that include Aj ax calls are triggered by user operations, e.g., by mouse clicking or selection. These operations may be realized by simulating operations of a browser.
- API i.e., Application Programming Interface
- Web automated testing tools may be used to simulate relevant operations performed by a user in a browser. For example, through the above procedures of SlOl - S103, a function getinfo on http://www.test.com may be found to include an Aj ax call with the function being triggered by mouse clicking.
- the function Upon simulating mouse clicking, the function is triggered and the content that is dynamically generated by this function can be obtained correspondingly.
- the above technical scheme obtains functions that include Ajax calls in the web page codes and further triggers the function to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page.
- a search engine can collect more complete web page contents and hence provide a better search service to users.
- the method of retrieving Ajax web page content in accordance with the present disclosure is described below using a specific exemplary embodiment. Details of S201 - S204 can be found in corresponding textual descriptions of SlOl - S 104.
- S201 obtains information of web page codes. Let assume the following code information of a certain web page is obtained based on an URL address:
- S202 retrieves JavaScript information of the web page codes.
- JavaScript files i.e., the two files "prototype.js" and “ajax.js” located within ⁇ head> part, are referenced. Information such as paths and filenames of these two files may be stored.
- S203 analyzes the JavaScript information to determine functions that include
- Aj ax calls in the web page codes.
- S203a may determine that "prototype.js" is a framework document and "ajax.js" is a non-framework document based on the filenames of these two JavaScript files retrieved at S202. Based on the types of the two JavaScript files being referenced, S203b may know that two relevant Aj ax property values possibly exist in the web page if an Aj ax call exists.
- the first type is an Ajax calling method provided in the prototype framework with its property value as: "new Ajax.Request".
- var date new Date()
- var localeString date.toLocaleString()
- return localeString
- S203c may determine that getlnfo() is a function that includes an Ajax call in the web page codes based on the intersection of Collection A (of functions that are used in the web page codes) and Collection B (of functions that include Ajax calls in the referenced JavaScript file).
- Collection A of functions that are used in the web page codes
- Collection B of functions that include Ajax calls in the referenced JavaScript file.
- the sameToBilling() function may not be further processed because the sameToBilling() function was not called in the web page.
- S204 triggers the functions that include the Ajax calls to obtain web page content.
- getlnfoO is triggered by simulating the onclick (i.e., mouse clicking) operation to obtain web page content generated thereby.
- the technical scheme provided in this disclosure determines which function (that is defined in a referenced file) includes an Ajax call by analyzing web page codes and on Ajax property values. Within the scope thereof, functions that are actually called in associated web page are found. Moreover, web page content is obtained by simulating execution of these functions, thus avoiding execution of a function that is unrelated to the web page content. As a result, efficiency of retrieving dynamic web page content is improved.
- the exemplary embodiments of the present disclosure further provide a system of retrieving Aj ax web page content. As shown in FIG.
- the system includes: a web page code acquisition unit 310, used for obtaining information of web page codes; a script retrieving unit 320, used for retrieving JavaScript information of the web page codes; a script analyzing unit 330, used for analyzing the JavaScript information to determine functions that include Aj ax calls in the web page codes; and a web page content acquisition unit 340, used for triggering the functions that include the Aj ax calls to obtain web page content.
- the JavaScript information may include JavaScript files and/or JavaScript codes.
- the script retrieving unit 320 may include: a checking sub-unit 321, which is used for checking a JavaScript tag in the web page codes; a first retrieving sub-unit 322, which is used when the JavaScript information, located after the JavaScript tag, corresponds to JavaScript codes, and retrieving the JavaScript codes and storing the retrieved codes into a JavaScript file; a second retrieving sub-unit 323, which is used when the JavaScript information, located after the JavaScript tag, corresponds to JavaScript file, and retrieving the storage path and the filename of the JavaScript file.
- the script analyzing unit 330 may include: a first determining sub-unit 331, used for determining functions that include Ajax calls and are defined in the JavaScript file based on Ajax property values, where the Ajax property values refer to code segments indicating an existence of an Ajax call in a function; and a second determining sub-unit 332, used for further determining functions that include Ajax calls in the web page codes from among the functions that have been determined by the first determining sub-unit 331.
- the Ajax property values may include: property values corresponding to Ajax calls using a type of JavaScript framework and/or property values corresponding to Ajax calls without using any type of JavaScript framework.
- the first determining sub-unit 331 may be used for determining functions that contain Ajax calls and are defined in a non-JavaScript framework document that is referenced in the web page.
- the web page content acquisition unit 340 may be used for triggering the functions that include the Ajax calls by simulating a user operation to obtain the web page content. Because of its fundamental correspondence with the exemplary method, the exemplary system has been described in a relatively simple manner. Details thereof can be referred to pertinent parts of the exemplary method. Descriptions of the above exemplary system are only meant for illustrative purpose. Units described as separate components therein may or may not be separate in a physical sense. Components illustrated in terms of units may or may not be physical units, i.e., may be located in one place or may be distributed among multiple network units. Based on actual needs, the goal of the exemplary embodiments may be achieved by selecting parts or all of the modules. A person of ordinary skills in the art can understand and implement the disclosed system without any innovative effect. For the sake of description, the above system was divided into various units which were separately described. The functions of various units may be implemented in one or more software and/or hardware when implementing the disclosed system.
- the disclosed method and system may be implemented using software and universal hardware platforms.
- the technical scheme of the present disclosure, or portions contributing to existing technologies may be implemented in the form of software products which are stored in a storage media such as ROM/RAM, disk and drive.
- the software includes instructions for a computing device (e.g., personal computer, server or network device) to execute the method described in various exemplary embodiments or a certain part of the exemplary embodiment of the current disclosure.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure describes a method and system of retrieving Ajax web page content. The method of retrieving Ajax web page content includes: obtaining information of web page codes, retrieving JavaScript information in the information of the web page codes; analyzing the JavaScript information to determine function(s) that include(s) Ajax call(s) in the web page codes, and triggering the determined function(s) that include(s) the Ajax call(s) to obtain web page content generated by the function(s) that include(s) the Ajax call(s). Through analyzing JavaScnpt information in web page codes, the technical scheme obtains function(s) that include(s) Ajax call(s ) in the web page codes and further triggers the function(s) to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page. Using the above technical scheme, a search engine can collect more complete web page contents and provide better search services.
Description
METHOD AND SYSTEM OF RETRIEVING AJAX WEB PAGE
CONTENT
RELATED APPLICATIONS The present application claims priority benefit of Chinese patent application
No. 200910133630.5, filed April 2, 2009, entitled "METHOD AND SYSTEM OF RETRIEVING AJAX WEB PAGE CONTENT", which is hereby incorporated in its entirety by reference.
TECHNICAL FIELD
The present disclosure relates to fields of networking technologies and, more particularly, to methods and systems of retrieving Aj ax web page content.
BACKGROUND Asynchronous JavaScript and XML (Aj ax) is a web page development technology for creating interactive web page applications. For a conventional web page, if a portion of the web page content has been updated, a user can see the updated content only after the entire web page is refreshed. With an Aj ax web page however, a user can obtain the updated portion of the content without refreshing the entire web page. From the perspective of a network device, this avoids repeatedly transmitting unaltered information and thus saves network bandwidth and reduces server workload. From a user's perspective, waiting times during web page browsing can be effectively reduced, thus improving user experience.
A search engine is a system that can collect, organize and process Internet information with the capability of providing search services to users. Currently, the
search engine has become one of the indispensible auxiliary tools of Internet surfing experience for users. A web crawler, being an important component of the search engine, can automatically retrieve web page content and provide real-time, updated data to the search engine. Though conventional web pages can be supported, existing web crawlers fail to effectively retrieve contents that are generated dynamically by
Aj ax on Aj ax web pages. As a result, dynamic contents on Aj ax web pages cannot be collected by the search engine, leading to a failure of the search engine to provide a complete content search service to the user.
SUMMARY OF THE DISCLOSURE
The present disclosure provides a method and a system of retrieving Aj ax web page content and to provide a solution for the failure of a search engine to collect dynamic content in an Ajax web page. A technical scheme is described as follows.
The present disclosure provides a method of retrieving Aj ax web page content, which includes: obtaining information of web page codes; retrieving JavaScript information of the web page codes; analyzing the JavaScript information to determine functions that include Ajax calls in the web page codes; and triggering the functions that include the Ajax calls to obtain web page content.
The present disclosure further provides a system of retrieving Ajax web page content, which includes: a web page code acquisition unit, used for obtaining information of web page codes;
a script retrieving unit, used for retrieving JavaScript information of the web page codes; a script analyzing unit, used for analyzing the JavaScript information to determine functions that include Aj ax calls in the web page codes; and a web page content acquisition unit, used for triggering the functions that include the Aj ax calls to obtain web page content.
Through analyzing JavaScript information in web page codes, the above technical scheme obtains functions that include Ajax calls in the web page codes and further triggers the functions to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page. Using the above technical scheme, a search engine can collect more complete web page contents and hence provide a better search service to users.
DESCRIPTION OF DRAWINGS FIG. 1 shows a flow chart of a method of retrieving Ajax web page content in accordance with the exemplary embodiments of the present disclosure.
FIG. 2 shows a flow chart of a method of retrieving JavaScript information of web page codes in accordance with the exemplary embodiments of the present disclosure. FIG. 3 shows a schematic structural diagram of a system of retrieving Ajax web page content in accordance with the exemplary embodiments of the present disclosure.
FIG. 4 shows a schematic structural diagram of a script retrieving unit in accordance with the exemplary embodiments of the present disclosure.
FIG. 5 shows a schematic structural diagram of a script analyzing unit in accordance with the exemplary embodiments of the present disclosure.
DETAILED DESCRIPTION
In order to allow someone skilled in the art to understand the technical scheme provided in the present disclosure, implementation methods in the present disclosure are described below in further details using accompanying figures. FIG. 1 shows a flow chart of a method of retrieving Aj ax web page content in accordance with the exemplary embodiments of the present disclosure. The method includes the following procedures.
5101 obtains information of web page codes.
This block is similar to the working mechanism of a conventional web crawler. Through an URL (i.e., Uniform Resource Locator) address, information of web page codes corresponding to the address is read from the network.
5102 retrieves JavaScript information of the web page codes.
Upon obtaining the information of web page codes, JavaScript information therein is retrieved first. Use HTML (i.e., Hyper Text Mark-up Language) codes as an example. Generally, the following two approaches may be used to add JavaScript information into HTML codes:
1) directly embedding JavaScript codes in the HTML codes; or
2) placing JavaScript codes in an external file and referencing the external file in the HTML codes. Specifically, with respect to the first approach, the JavaScript codes can be placed between <script> and </script>, a tag pair in the HTML codes. A common format is shown as follows:
<script type= "text/javascript "> //js codes </script>
The portion within the <script> tag pair in the above format corresponds to JavaScript codes. Multiple JavaScript sentences may be included within the <script> tag pair. The <script> tag may appear in <head> part of the HTML codes or <body> part of the HTML codes. With respect to the second approach, the JavaScript codes are placed in an external file. This external file is then referenced through a src property of the <script> tag in the HTML codes. A common format is shown below:
< script type= "text/javascript " src= "js/text.js"> </script>
Values of the src property define a storage path and a filename of a JavaScript file. The storage path may be an absolute path or a relative path, "test.js" in the above format refers to the JavaScript filename being referenced while "js/" refers to the storage path of that file. A JavaScript file generally uses .js as its extension name, and includes mere JavaScript codes, having no <script> tag or other HTML tags.
Based on the above two approaches, the JavaScript information in the HTML codes can be retrieved by checking the <script> tag in the web page codes. As shown in FIG. 2, a method of retrieving JavaScript information of web page codes may include the following procedures.
S 102a checks a JavaScript tag in the web page codes.
S 102b determines the type of JavaScript information located after the JavaScript tag. If the type corresponds to JavaScript codes, S 102c is executed. If the type corresponds to JavaScript files, S102d is executed.
S 102c retrieves the JavaScript codes within the JavaScript tag's pair.
S102d retrieves storage paths and filenames of the JavaScript files.
Upon carrying out the above procedures for all nodes of <script> tags in a HTML file code, a certain number of JavaScript codes and/or JavaScript files and
respective storage paths are obtained. All the retrieved JavaScript codes may be stored in a temporary .js file (e.g., temp.js). With respect to the JavaScript files, analyzing all the codes of the JavaScript files is not required in the technical scheme of the present disclosure. Only the retrieved JavaScript filenames and respective storage paths may be stored at this block.
S 103 analyzes the JavaScript information to determine functions that include Aj ax calls in the web page codes.
Generally, two types of JavaScript files are referenced in web page codes. One type refers to JavaScript framework documents. This type of file generally has a fixed filename, prototype.js, for example. Another type refers to non-JavaScript framework documents. Within web page codes, functions that include Ajax calls are defined in the non- JavaScript framework document. As such, contents of the JavaScript framework document may not need to be further analyzed. However, property values, corresponding to its framework type, can be found based on its filename. Moreover, functions that include Ajax calls, and are defined in the non-
JavaScript framework document, may be found using these property values. Furthermore, functions that include Ajax calls in the web page codes may then be found. Specific procedures are shown as follows.
S 103 a determines the file type of a JavaScript file based on the filename of the JavaScript file.
According to the above method, determination is made as to whether the JavaScript file is a framework document or a non-framework document. If determined to be a framework document, corresponding property values are obtained. S103b determines functions, which include Ajax calls, and are defined in a non-JavaScript framework document based on the Ajax property values.
In a JavaScript file, all functions that include Ajax calls must include a specific piece of code which are called Ajax property values. Therefore, whether a function includes an Ajax call can be determined based on whether the function includes these property values. Ajax property values may be categorized into two types. Apart from the aforementioned property values that correspond to various types of JavaScript framework, other property values exist in a situation when Ajax is called without using any type of JavaScript framework. These latter property values may be stored in advance in a database file. By matching codes of functions in a non- JavaScript framework document with these property values, functions that include Ajax calls can be determined.
S103c further determines functions that include Ajax calls in the web page codes based on a result of S 103b.
An assumption is made that functions used in the web page code constitute collection A and functions that have been determined at S 103b to include Ajax calls constitute collection B. The intersection of A and B corresponds to the required functions that include Ajax calls in the web page codes. A function that is not called in the web page, i.e., a function that belongs to collection B but does not belong to collection A, may not need to undergo further processing. It should be noted that S 103 a may be skipped if the JavaScript information obtained at S 102 includes only JavaScript codes without JavaScript. Furthermore, the temporary file, temp.js, which is used to store the JavaScript codes as described above, may be processed as a non-framework document here.
S 104 triggers the functions that include Ajax calls to obtain web page content.
Upon determining which function includes Aj ax calls in the web page code, web page content that is generated dynamically by Aj ax can be obtained through triggering these functions.
A number of functions that include Aj ax calls are triggered by user operations, e.g., by mouse clicking or selection. These operations may be realized by simulating operations of a browser. Specifically, API (i.e., Application Programming Interface) provided by certain Web automated testing tools may be used to simulate relevant operations performed by a user in a browser. For example, through the above procedures of SlOl - S103, a function getinfo on http://www.test.com may be found to include an Aj ax call with the function being triggered by mouse clicking.
Therefore, the following method may be used to simulate triggering of this function:
IE ie = new IE();
// Simulate an operation of opening a browser ie.start(http://www.test.com); // Simulate entering http://www.test.com into the browser to open this web page
HtmlElement e=ie.htmlElement(attribute("onclick", "getinfo"));
// Found an element node of the JavaScript function, getinfo, that is called upon click e.clickO;
// Simulate an operation of clicking the element node through a mouse
Upon simulating mouse clicking, the function is triggered and the content that is dynamically generated by this function can be obtained correspondingly.
Through analyzing JavaScript information in web page codes, the above technical scheme obtains functions that include Ajax calls in the web page codes and
further triggers the function to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page. Using the above technical scheme, a search engine can collect more complete web page contents and hence provide a better search service to users. The method of retrieving Ajax web page content in accordance with the present disclosure is described below using a specific exemplary embodiment. Details of S201 - S204 can be found in corresponding textual descriptions of SlOl - S 104.
S201 obtains information of web page codes. Let assume the following code information of a certain web page is obtained based on an URL address:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3. org/TR/html4/strict. dtd"> <html lang="en"> <head>
<script type="text/javascript" src="js/prototype.js" /> <script type="text/javascript" src="js/ajax.js" /> <title>AJAX, Web 2.0 example</title> <link rel="stylesheet" type="text/css" href="css/style.css" /> </head>
<body>
<div id="navhover">
<img src="img/hammer.jpg" alt="Builders"> <img src="img/lightbulb.jpg" alt="Electricians"> <img src="img/wrench.jpg" alt="Plumbers">
<img src="img/taxi.jpg" alt="Taxis"> </div> <div id="nav">
<ul> <lixa href="#" onclick="getlnfo(l);" id="navl " title=" Electricians " ><span>Builders</span></a></li> <lixa href="#" onclick="getlnfo(2);" id="nav2" title="Plumbers"><span>Electricians</span></a></li> <lixa href="#" onclick="getlnfo(3);" id="nav3" title="Builders"><span>Plumbers</span></ax/li>
</ul> <ul>
<li><img src="img/taxi.jpg" onclick='getlnfo(4);' id="img" name="img" style="cursor:hand"></li>
</ul> </div>
<div id="results">Search results... </div> </body> </html>
It should be noted that the real codes provided in this exemplary embodiment are only meant for the purpose of illustrating the disclosed technical scheme and should be not construed as a limitation of the technical scheme. S202 retrieves JavaScript information of the web page codes.
According to the description at S 102, the above codes are found to have no JavaScript codes directly embedded. However, JavaScript files, i.e., the two files "prototype.js" and "ajax.js" located within <head> part, are referenced. Information such as paths and filenames of these two files may be stored. S203 analyzes the JavaScript information to determine functions that include
Aj ax calls in the web page codes.
S203a may determine that "prototype.js" is a framework document and "ajax.js" is a non-framework document based on the filenames of these two JavaScript files retrieved at S202. Based on the types of the two JavaScript files being referenced, S203b may know that two relevant Aj ax property values possibly exist in the web page if an Aj ax call exists. The first type is an Ajax calling method provided in the prototype framework with its property value as: "new Ajax.Request".
Another type corresponds to a situation when no JavaScript framework is used. Respective property value is:
"new ActiveObj ect("Microsoft.XMLHTTP")".
The non-framework document, "ajax.js", is then further analyzed. Assume contents of the codes of "ajax.js" be the following:
function getlnfo(id) { var xmlhttp; try{ xmlhttp=new XMLHttpRequest(); }catch(e){ xmlhttp=new ActiveXObject("Microsoft.XMLHTTP"); } xmlhttp . onreadystatechange=function() { if(xmlhttp.readyState == 1 ) { document.getElementById('results').innerHTML = 'Loading...';
} if(xmlhttp.readyState == 4){ var response = xmlhttp.responseText; document.getElementById('results').innerHTML = response;
} } xmlhttp. op en('get', 'data.jsp?company=' + id, true); xmlhttp. send("company="+id); } function getDate()
{ var date = new Date(); var year = date.getYear(); var month = date.getMonth()+l; var day = date.getDay(); return year+""+(month>=10?month:("0"+""+month))+""+(day>=10?day:("0"+""+day)); } function getDateTime()
{ var date = new Date(); var localeString = date.toLocaleString(); return localeString;
} function sameToBilling(obj) { var my = new Ajax.Request(url,
{ method: 'get', parameters: pars, asynchronous: true , onComplete: showResponse
} );
Four functions are found to be defined in "ajax.js" and are getlnfo(), getDate(), getDateTime() and sameToBilling() respectively. Moreover, property values "new ActiveXObject("Microsoft.XMLHTTP")" and "new Ajax.Request" appear in the getlnfo() function and the sameToBilling() function, respectively. As a result, getlnfo() and sameToBilling() can be determined to be functions that include
Aj ax calls.
S203c may determine that getlnfo() is a function that includes an Ajax call in the web page codes based on the intersection of Collection A (of functions that are used in the web page codes) and Collection B (of functions that include Ajax calls in the referenced JavaScript file). On the other hand, the sameToBilling() function may not be further processed because the sameToBilling() function was not called in the web page.
S204 triggers the functions that include the Ajax calls to obtain web page content. getlnfoO is triggered by simulating the onclick (i.e., mouse clicking) operation to obtain web page content generated thereby.
As illustrated in the above exemplary embodiment, the technical scheme provided in this disclosure determines which function (that is defined in a referenced file) includes an Ajax call by analyzing web page codes and on Ajax property values. Within the scope thereof, functions that are actually called in associated web page are found. Moreover, web page content is obtained by simulating execution of these functions, thus avoiding execution of a function that is unrelated to the web page content. As a result, efficiency of retrieving dynamic web page content is improved.
Corresponding to the above exemplary method, the exemplary embodiments of the present disclosure further provide a system of retrieving Aj ax web page content. As shown in FIG. 3, the system includes: a web page code acquisition unit 310, used for obtaining information of web page codes; a script retrieving unit 320, used for retrieving JavaScript information of the web page codes; a script analyzing unit 330, used for analyzing the JavaScript information to determine functions that include Aj ax calls in the web page codes; and a web page content acquisition unit 340, used for triggering the functions that include the Aj ax calls to obtain web page content.
The JavaScript information may include JavaScript files and/or JavaScript codes.
As shown in FIG. 4, the script retrieving unit 320 may include: a checking sub-unit 321, which is used for checking a JavaScript tag in the web page codes; a first retrieving sub-unit 322, which is used when the JavaScript information, located after the JavaScript tag, corresponds to JavaScript codes, and retrieving the JavaScript codes and storing the retrieved codes into a JavaScript file; a second retrieving sub-unit 323, which is used when the JavaScript information, located after the JavaScript tag, corresponds to JavaScript file, and retrieving the storage path and the filename of the JavaScript file.
As shown in FIG. 5, the script analyzing unit 330 may include: a first determining sub-unit 331, used for determining functions that include Ajax calls and are defined in the JavaScript file based on Ajax property values, where
the Ajax property values refer to code segments indicating an existence of an Ajax call in a function; and a second determining sub-unit 332, used for further determining functions that include Ajax calls in the web page codes from among the functions that have been determined by the first determining sub-unit 331.
The Ajax property values may include: property values corresponding to Ajax calls using a type of JavaScript framework and/or property values corresponding to Ajax calls without using any type of JavaScript framework.
The first determining sub-unit 331 may be used for determining functions that contain Ajax calls and are defined in a non-JavaScript framework document that is referenced in the web page.
The web page content acquisition unit 340 may be used for triggering the functions that include the Ajax calls by simulating a user operation to obtain the web page content. Because of its fundamental correspondence with the exemplary method, the exemplary system has been described in a relatively simple manner. Details thereof can be referred to pertinent parts of the exemplary method. Descriptions of the above exemplary system are only meant for illustrative purpose. Units described as separate components therein may or may not be separate in a physical sense. Components illustrated in terms of units may or may not be physical units, i.e., may be located in one place or may be distributed among multiple network units. Based on actual needs, the goal of the exemplary embodiments may be achieved by selecting parts or all of the modules. A person of ordinary skills in the art can understand and implement the disclosed system without any innovative effect.
For the sake of description, the above system was divided into various units which were separately described. The functions of various units may be implemented in one or more software and/or hardware when implementing the disclosed system.
From the exemplary embodiments described above, someone skilled in the art can clearly understand that the disclosed method and system may be implemented using software and universal hardware platforms. Based on this understanding, the technical scheme of the present disclosure, or portions contributing to existing technologies, may be implemented in the form of software products which are stored in a storage media such as ROM/RAM, disk and drive. The software includes instructions for a computing device (e.g., personal computer, server or network device) to execute the method described in various exemplary embodiments or a certain part of the exemplary embodiment of the current disclosure.
Above are only a few exemplary embodiments of the present disclosure. It is understood that a person of ordinary skills in the art can alter or modify the current disclosure in many different ways without departing from the spirit and the scope of this disclosure. These alterations and modifications should be considered to fall within the scope of the claims of the present disclosure and their equivalents.
Claims
1. A method of retrieving asynchronous JavaScript and XML (Aj ax) web page content, the method comprising: obtaining information of web page codes; retrieving JavaScript information from the information of the web page codes; analyzing the JavaScript information to determine at least one function that comprises at least one Ajax call in the web page codes; and triggering the determined at least one function that comprises the at least one
Ajax call to obtain web page content generated by the determined at least one function.
2. The method as recited in claim 1, wherein the JavaScript information comprises one or more JavaScript files, JavaScript codes, or a combination thereof.
3. The method as recited in claim 2, wherein retrieving the JavaScript information from the information of the web page codes comprises: checking for a JavaScript tag in the web page codes; when the JavaScript information retrieved from the information of the web page codes and located after the JavaScript tag comprises JavaScript codes: retrieving the JavaScript codes; and storing the retrieved JavaScript codes in a JavaScript file; and when the JavaScript information retrieved from the information of the web page codes and located after the JavaScript tag comprises a JavaScript file, retrieving a storage path and a filename of the JavaScript file.
4. The method as recited in claim 3, wherein analyzing the JavaScript information to determine at least one function that comprises at least one Aj ax call in the web page codes comprises: determining at least one function comprising at least one Aj ax call and defined in the JavaScript file based on Aj ax property values; and from the determined at least one function comprising at least one Aj ax call and defined in the JavaScript file, determining a function that comprises an Ajax call in the web page codes, the Ajax property values indicative of the function containing a code segment for the Ajax call.
5. The method as recited in claim 4, wherein the Ajax property values comprise: property values corresponding to Ajax calls using a type of JavaScript framework, property values corresponding to Ajax calls without using a type of JavaScript framework, or a combination thereof.
6. The method as recited in claim 4, wherein determining the at least one function that comprises the at least one Ajax call and is defined in the JavaScript file comprises: determining the at least one function that comprises the at least one Ajax call and is defined in a non-JavaScript framework document that is referenced in a web page.
7. The method as recited in claim 1, wherein triggering the determined at least one function that comprises the at least one Aj ax call comprises: simulating a user operation to trigger the determined at least one function that comprises the at least one Aj ax call.
8. A system of retrieving asynchronous JavaScript and XML (Aj ax) web page content, the system comprising: a web page code acquisition unit that obtains information of web page codes; a script retrieving unit that retrieves JavaScript information from the information of the web page codes; a script analyzing unit that analyzes the JavaScript information to determine at least one function that comprises at least one Aj ax call in the web page codes; and a web page content acquisition unit that triggers the determined function that comprises at least one Aj ax call to obtain web page content generated by the determined function.
9. The system as recited in claim 8, wherein the JavaScript information comprises one or more JavaScript files, JavaScript codes, or a combination thereof.
10. The system as recited in claim 9, wherein the script retrieving unit comprises: a checking sub-unit that checks for a JavaScript tag in the web page codes; a first retrieving sub-unit that retrieves the JavaScript codes and stores the retrieved codes in a JavaScript file when the JavaScript information retrieved from the information of the web page codes and located after the JavaScript tag comprises JavaScript codes; a second retrieving sub-unit that retrieves a storage path and a filename of the JavaScript file when the JavaScript information retrieved from the information of the web page codes and located after the JavaScript tag comprises the JavaScript file.
11. The system as recited in claim 10, wherein the script analyzing unit comprises: a first determining sub-unit that determines the function comprising at least one Aj ax call and defined in the JavaScript file based on Aj ax property values, the Ajax property values indicative of the function containing a code segment for the Ajax call; and a second determining sub-unit that determines an Ajax calling function is contained in the web page codes from the function determined by the first determining sub-unit.
12. The system as recited in claim 11, wherein the Ajax property values comprise: property values corresponding to Ajax calls using a type of JavaScript framework, property values corresponding to Ajax calls without using a type of JavaScript framework, or a combination thereof.
13. The system as recited in claim 11, wherein the first determining sub-unit further determines at least one function that comprises at least one Ajax call and is defined in a non- JavaScript framework document that is referenced in a web page.
14. The system as recited in claim 8, wherein the web page content acquisition unit further simulates a user operation to trigger the determined at least one function that comprises the at least one Aj ax call to obtain the web page content generated by the at least one function that comprises the at least one Aj ax call.
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP10759351A EP2414929A4 (en) | 2009-04-02 | 2010-03-31 | Method and system of retrieving ajax web page content |
US12/863,320 US8413044B2 (en) | 2009-04-02 | 2010-03-31 | Method and system of retrieving Ajax web page content |
JP2012503668A JP5695027B2 (en) | 2009-04-02 | 2010-03-31 | Method and system for acquiring AJAX web page content |
US13/756,886 US9767082B2 (en) | 2009-04-02 | 2013-02-01 | Method and system of retrieving ajax web page content |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2009101336305A CN101515300B (en) | 2009-04-02 | 2009-04-02 | Method and system for grabbing Ajax webpage content |
CN200910133630.5 | 2009-04-02 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/863,320 A-371-Of-International US8413044B2 (en) | 2009-04-02 | 2010-03-31 | Method and system of retrieving Ajax web page content |
US13/756,886 Continuation US9767082B2 (en) | 2009-04-02 | 2013-02-01 | Method and system of retrieving ajax web page content |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010114913A1 true WO2010114913A1 (en) | 2010-10-07 |
Family
ID=41039753
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2010/029444 WO2010114913A1 (en) | 2009-04-02 | 2010-03-31 | Method and system of retrieving ajax web page content |
Country Status (6)
Country | Link |
---|---|
US (2) | US8413044B2 (en) |
EP (1) | EP2414929A4 (en) |
JP (2) | JP5695027B2 (en) |
CN (1) | CN101515300B (en) |
HK (1) | HK1136053A1 (en) |
WO (1) | WO2010114913A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102609518A (en) * | 2012-02-09 | 2012-07-25 | 清华大学 | Method and system for acquiring content of multistate AJAX (asynchronous javascript and extensible markup language) webpage |
JP2014504764A (en) * | 2011-01-18 | 2014-02-24 | アップル インコーポレイテッド | System and method for supporting JIT in a secure system with randomly allocated memory ranges |
US9684718B2 (en) | 2011-07-15 | 2017-06-20 | International Business Machines Corporation | System for searching for a web document |
US9870256B2 (en) | 2011-07-29 | 2018-01-16 | International Business Machines Corporation | Hardware acceleration wait time awareness in central processing units with multi-thread architectures |
Families Citing this family (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8539337B2 (en) * | 2008-02-21 | 2013-09-17 | International Business Machines Corporation | Internet services and methods thereof |
US8898623B2 (en) * | 2008-12-30 | 2014-11-25 | The Regents Of The University Of California | Application design and data flow analysis |
CN101515300B (en) | 2009-04-02 | 2011-07-20 | 阿里巴巴集团控股有限公司 | Method and system for grabbing Ajax webpage content |
CN102005130B (en) * | 2009-09-03 | 2014-01-22 | 上海宝信软件股份有限公司 | WebGIS (Web Geographic Information System)-based system for displaying real-time state of intelligent transportation equipment |
WO2011091388A2 (en) * | 2010-01-22 | 2011-07-28 | The Regents Of The University Of California | Web application development framework |
CN102236546B (en) * | 2010-04-30 | 2014-03-12 | 英业达股份有限公司 | System and method for executing corresponding script by searching interactive elements |
CN102262635A (en) * | 2010-05-25 | 2011-11-30 | 北京启明星辰信息技术股份有限公司 | Page crawler system and page crawler method |
EP2413265B1 (en) * | 2010-07-29 | 2017-10-18 | Tata Consultancy Services Ltd. | A system and method for classification of moving object during video surveillance |
CN102479231A (en) * | 2010-11-24 | 2012-05-30 | 财团法人资讯工业策进会 | Webpage climbing method and device thereof |
CN102073728A (en) * | 2011-01-13 | 2011-05-25 | 百度在线网络技术(北京)有限公司 | Method, device and equipment for determining web access requests |
US9805135B2 (en) | 2011-03-30 | 2017-10-31 | Cbs Interactive Inc. | Systems and methods for updating rich internet applications |
US8527862B2 (en) | 2011-06-24 | 2013-09-03 | Usablenet Inc. | Methods for making ajax web applications bookmarkable and crawlable and devices thereof |
CN103020087A (en) * | 2011-09-26 | 2013-04-03 | 百度在线网络技术(北京)有限公司 | Method and device for generating clicking journals, as well as method and device for adjusting search results |
CN102609481A (en) * | 2012-01-20 | 2012-07-25 | 苏州简拔林网络科技有限公司 | Method for updating and gathering comment information in real time |
CN102662737B (en) * | 2012-03-14 | 2014-06-11 | 优视科技有限公司 | Calling method and device of extension program |
CN103365919B (en) * | 2012-04-09 | 2018-07-31 | 北京京东尚科信息技术有限公司 | Web analysis container and method |
CN103577427A (en) * | 2012-07-25 | 2014-02-12 | 中国移动通信集团公司 | Browser kernel based web page crawling method and device and browser containing device |
CN107609135B (en) * | 2012-09-03 | 2020-08-21 | 阿里巴巴集团控股有限公司 | Page element determining method and device, and user behavior path determining method and device |
CN102929599B (en) * | 2012-09-26 | 2015-12-02 | 广州市动景计算机科技有限公司 | The amending method at browser of mobile terminal interface and device, mobile terminal |
CN103853717B (en) * | 2012-11-28 | 2018-10-12 | 北京京东尚科信息技术有限公司 | network crawler system |
US10152552B2 (en) * | 2013-01-29 | 2018-12-11 | Entit Software Llc | Analyzing a structure of a web application to produce actionable tokens |
CN103268361B (en) * | 2013-06-07 | 2019-05-31 | 百度在线网络技术(北京)有限公司 | Extracting method, the device and system of URL are hidden in webpage |
CN104111836A (en) * | 2014-07-14 | 2014-10-22 | 浪潮软件集团有限公司 | Method for collecting and processing asynchronous loading data by network |
US9772829B2 (en) * | 2014-09-09 | 2017-09-26 | Liveperson, Inc. | Dynamic code management |
US11120461B1 (en) | 2014-11-06 | 2021-09-14 | Capital One Services, Llc | Passive user-generated coupon submission |
US11068921B1 (en) | 2014-11-06 | 2021-07-20 | Capital One Services, Llc | Automated testing of multiple on-line coupons |
US10649740B2 (en) * | 2015-01-15 | 2020-05-12 | International Business Machines Corporation | Predicting and using utility of script execution in functional web crawling and other crawling |
US10425693B2 (en) * | 2015-02-13 | 2019-09-24 | Panasonic Intellectual Property Management Co., Ltd. | Content reproduction system, video recording apparatus, terminal apparatus, and content reproduction method |
US11057446B2 (en) | 2015-05-14 | 2021-07-06 | Bright Data Ltd. | System and method for streaming content from multiple servers |
CN106294397B (en) * | 2015-05-20 | 2019-10-25 | 无锡天脉聚源传媒科技有限公司 | A kind of method and device of acquisition task |
CN104965901A (en) * | 2015-06-30 | 2015-10-07 | 北京奇虎科技有限公司 | Method and apparatus for grabbing content of target page |
CN105183453B (en) * | 2015-08-07 | 2019-04-02 | 安一恒通(北京)科技有限公司 | Web-based information acquisition method and device |
CN105243088B (en) * | 2015-09-09 | 2019-10-01 | 深圳Tcl数字技术有限公司 | The method and device of web page contents is obtained in android system |
US10082937B2 (en) | 2015-09-11 | 2018-09-25 | International Business Machines Corporation | Intelligent rendering of webpages |
CN105183886A (en) * | 2015-09-25 | 2015-12-23 | 中国民生银行股份有限公司 | Webpage content extraction method and device |
US10452653B2 (en) * | 2015-10-07 | 2019-10-22 | Capital One Services, Llc | Automated extraction of data from web pages |
CN105740419A (en) * | 2016-01-29 | 2016-07-06 | 广州酷狗计算机科技有限公司 | Method and apparatus for acquiring dynamically loaded content in webpage |
US10223353B1 (en) * | 2016-09-20 | 2019-03-05 | Amazon Technologies | Dynamic semantic analysis on free-text reviews to identify safety concerns |
CN106959995A (en) * | 2016-12-21 | 2017-07-18 | 四川长虹电器股份有限公司 | Compatible two-way automatic web page contents acquisition method |
CN108306918B (en) * | 2017-01-13 | 2021-08-31 | 南京邮电大学盐城大数据研究院有限公司 | Automatic website access information acquisition method based on program dynamic analysis |
CN106991188A (en) * | 2017-04-11 | 2017-07-28 | 焦点科技股份有限公司 | A kind of efficient internet dynamic data automatic screening and grasping means and system |
US11205188B1 (en) | 2017-06-07 | 2021-12-21 | Capital One Services, Llc | Automatically presenting e-commerce offers based on browse history |
CN109150984B (en) * | 2018-07-27 | 2021-11-02 | 平安科技(深圳)有限公司 | Method and device for acquiring data resources |
CN108984801A (en) * | 2018-08-22 | 2018-12-11 | 百卓网络科技有限公司 | A kind of search engine optimization method identifying asynchronous loading content based on html tag |
JP7018202B2 (en) * | 2018-11-27 | 2022-02-10 | 株式会社クリエイト | Information retrieval system |
US11483371B2 (en) | 2020-11-23 | 2022-10-25 | International Business Machines Corporation | User-derived webpage activity control |
CN113076460A (en) * | 2021-05-07 | 2021-07-06 | 北京华云安信息技术有限公司 | Page data crawling method, device, equipment and computer-readable storage medium |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070101258A1 (en) * | 2005-10-14 | 2007-05-03 | Ebay Inc. | Asynchronously loading dynamically generated content across multiple internet domains |
US20080040653A1 (en) * | 2006-08-14 | 2008-02-14 | Christopher Levine | System and methods for managing presentation and behavioral use of web display content |
Family Cites Families (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7178106B2 (en) * | 1999-04-21 | 2007-02-13 | Sonic Solutions, A California Corporation | Presentation of media content from multiple media sources |
US7346920B2 (en) * | 2000-07-07 | 2008-03-18 | Sonic Solutions, A California Corporation | System, method and article of manufacture for a common cross platform framework for development of DVD-Video content integrated with ROM content |
AU2002258769A1 (en) | 2001-04-09 | 2002-10-21 | America Online Incorporated | Server-based browser system |
WO2003009202A1 (en) * | 2001-07-19 | 2003-01-30 | Live Capsule, Inc. | Method for transmitting a transferable information packet |
US7752326B2 (en) * | 2001-08-20 | 2010-07-06 | Masterobjects, Inc. | System and method for utilizing asynchronous client server communication objects |
US20060190561A1 (en) | 2002-06-19 | 2006-08-24 | Watchfire Corporation | Method and system for obtaining script related information for website crawling |
US8032860B2 (en) * | 2003-02-26 | 2011-10-04 | Oracle International Corporation | Methods for type-independent source code editing |
US7454410B2 (en) | 2003-05-09 | 2008-11-18 | International Business Machines Corporation | Method and apparatus for web crawler data collection |
US7685296B2 (en) | 2003-09-25 | 2010-03-23 | Microsoft Corporation | Systems and methods for client-based web crawling |
US7584194B2 (en) * | 2004-11-22 | 2009-09-01 | Truveo, Inc. | Method and apparatus for an application crawler |
WO2006058075A2 (en) | 2004-11-22 | 2006-06-01 | Truveo, Inc. | Method and apparatus for an application crawler |
EP1662405A1 (en) * | 2004-11-30 | 2006-05-31 | Alcatel | Method of displaying data on a client computer |
US7536389B1 (en) | 2005-02-22 | 2009-05-19 | Yahoo ! Inc. | Techniques for crawling dynamic web content |
US7636883B2 (en) * | 2005-05-18 | 2009-12-22 | International Business Machines Corporation | User form based automated and guided data collection |
US7725574B2 (en) * | 2006-01-23 | 2010-05-25 | International Business Machines Corporation | Web browser-based programming language error determination and reporting |
JP2008071048A (en) * | 2006-09-13 | 2008-03-27 | Nippon Telegr & Teleph Corp <Ntt> | System for presenting dynamic content and its program |
JP2008107987A (en) * | 2006-10-24 | 2008-05-08 | Logly Kk | Information providing device and information providing method |
JP2008186160A (en) | 2007-01-29 | 2008-08-14 | Fuji Xerox Co Ltd | Document display apparatus and program |
US7774788B2 (en) | 2007-03-07 | 2010-08-10 | Ianywhere Solutions, Inc. | Selectively updating web pages on a mobile client |
US8065667B2 (en) * | 2007-03-20 | 2011-11-22 | Yahoo! Inc. | Injecting content into third party documents for document processing |
US9594731B2 (en) * | 2007-06-29 | 2017-03-14 | Microsoft Technology Licensing, Llc | WYSIWYG, browser-based XML editor |
US9563718B2 (en) | 2007-06-29 | 2017-02-07 | Intuit Inc. | Using interactive scripts to facilitate web-based aggregation |
US8131591B2 (en) | 2007-09-12 | 2012-03-06 | Microsoft Corporation | Updating contents of asynchronously refreshable webpages |
US7672938B2 (en) | 2007-10-05 | 2010-03-02 | Microsoft Corporation | Creating search enabled web pages |
US8250585B2 (en) * | 2007-11-05 | 2012-08-21 | International Business Machines Corporation | Extensible framework for managing UI state in a composite AJAX application |
US8572065B2 (en) * | 2007-11-09 | 2013-10-29 | Microsoft Corporation | Link discovery from web scripts |
US8527860B1 (en) * | 2007-12-04 | 2013-09-03 | Appcelerator, Inc. | System and method for exposing the dynamic web server-side |
US7958232B1 (en) * | 2007-12-05 | 2011-06-07 | Appcelerator, Inc. | Dashboard for on-the-fly AJAX monitoring |
US8347405B2 (en) * | 2007-12-27 | 2013-01-01 | International Business Machines Corporation | Asynchronous java script and XML (AJAX) form-based authentication using java 2 platform enterprise edition (J2EE) |
CA2712015C (en) * | 2008-01-16 | 2017-09-19 | Bayalink Solutions Corp. | Secured presentation layer virtualization for wireless handheld communication device having endpoint independence |
CN101546309B (en) * | 2008-03-26 | 2012-07-04 | 国际商业机器公司 | Method and equipment for constructing indexes to resource content in computer network |
CN102428461B (en) * | 2009-03-18 | 2015-11-25 | 谷歌公司 | Utilize the web translation that display is replaced |
CN101515300B (en) | 2009-04-02 | 2011-07-20 | 阿里巴巴集团控股有限公司 | Method and system for grabbing Ajax webpage content |
-
2009
- 2009-04-02 CN CN2009101336305A patent/CN101515300B/en active Active
-
2010
- 2010-02-24 HK HK10101951.3A patent/HK1136053A1/en unknown
- 2010-03-31 WO PCT/US2010/029444 patent/WO2010114913A1/en active Application Filing
- 2010-03-31 JP JP2012503668A patent/JP5695027B2/en active Active
- 2010-03-31 EP EP10759351A patent/EP2414929A4/en not_active Ceased
- 2010-03-31 US US12/863,320 patent/US8413044B2/en active Active
-
2013
- 2013-02-01 US US13/756,886 patent/US9767082B2/en active Active
-
2015
- 2015-02-05 JP JP2015021591A patent/JP5990605B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070101258A1 (en) * | 2005-10-14 | 2007-05-03 | Ebay Inc. | Asynchronously loading dynamically generated content across multiple internet domains |
US20080040653A1 (en) * | 2006-08-14 | 2008-02-14 | Christopher Levine | System and methods for managing presentation and behavioral use of web display content |
Non-Patent Citations (1)
Title |
---|
See also references of EP2414929A4 * |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014504764A (en) * | 2011-01-18 | 2014-02-24 | アップル インコーポレイテッド | System and method for supporting JIT in a secure system with randomly allocated memory ranges |
US9684718B2 (en) | 2011-07-15 | 2017-06-20 | International Business Machines Corporation | System for searching for a web document |
US9690855B2 (en) | 2011-07-15 | 2017-06-27 | International Business Machines Corporation | Method and system for searching for a web document |
US9870256B2 (en) | 2011-07-29 | 2018-01-16 | International Business Machines Corporation | Hardware acceleration wait time awareness in central processing units with multi-thread architectures |
CN102609518A (en) * | 2012-02-09 | 2012-07-25 | 清华大学 | Method and system for acquiring content of multistate AJAX (asynchronous javascript and extensible markup language) webpage |
Also Published As
Publication number | Publication date |
---|---|
CN101515300B (en) | 2011-07-20 |
US20120011431A1 (en) | 2012-01-12 |
US8413044B2 (en) | 2013-04-02 |
HK1136053A1 (en) | 2010-06-18 |
US20130145253A1 (en) | 2013-06-06 |
CN101515300A (en) | 2009-08-26 |
US9767082B2 (en) | 2017-09-19 |
JP5990605B2 (en) | 2016-09-14 |
EP2414929A1 (en) | 2012-02-08 |
EP2414929A4 (en) | 2012-11-28 |
JP5695027B2 (en) | 2015-04-01 |
JP2012523047A (en) | 2012-09-27 |
JP2015135680A (en) | 2015-07-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8413044B2 (en) | Method and system of retrieving Ajax web page content | |
US8560964B2 (en) | Method and system for predictive browsing | |
US9330179B2 (en) | Configuring web crawler to extract web page information | |
US9703883B2 (en) | Social bookmarking of resources exposed in web pages | |
US8321396B2 (en) | Automatically extracting by-line information | |
US8424004B2 (en) | High performance script behavior detection through browser shimming | |
CN103268361B (en) | Extracting method, the device and system of URL are hidden in webpage | |
CN103500194A (en) | Method, device and browser for loading webpage | |
CN104063401B (en) | The method and apparatus that a kind of webpage pattern address merges | |
CN104408204A (en) | Method and device for obtaining webpage page link address | |
JP2008529133A (en) | Technology to change the presentation of information displayed to end users of computer systems | |
CN103092936B (en) | A kind of Internet of Things dynamic page real-time information collection method | |
EP3185145A1 (en) | Methods for analyzing web sites using web services and devices thereof | |
KR101287371B1 (en) | Method and Device for Collecting Web Contents and Computer-readable Recording Medium for the same | |
WO2006046323A1 (en) | Internet information collection device, program, and method | |
CN110719344B (en) | Domain name acquisition method and device, electronic equipment and storage medium | |
EP2711838A1 (en) | Documentation parser | |
CN104881453A (en) | Method and device for indentifying type of webpage | |
US20240104106A1 (en) | Data extraction cycles with multiple parsing frames | |
TW201044197A (en) | A method and system for capturing contents of Ajax web pages | |
CN118013146A (en) | Method and device for supporting automatic font collection and compression | |
CN117574010A (en) | Data acquisition method, device, equipment and storage medium | |
JP2013225246A (en) | Web browsing history acquisition device and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 12863320 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10759351 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012503668 Country of ref document: JP Ref document number: 2010759351 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |