TWI456416B - Ajax web content crawling method and system - Google Patents

Claims (16)

A method for capturing webpage content, comprising: acquiring webpage code information; extracting script language information according to the webpage code information, the scripting language information being included in one or more script language files; according to the one The file name of a script file in one or more script language files, determining the type of the one script file, if the one script file is a frame file, obtaining an asynchronous script language feature value, and using the non-synchronized script language feature a value determining at least one function, the at least one function comprising a call associated with an asynchronous web application of the web page, the web page being associated with the web page code, and if the one script file is a non-frame file, according to the corresponding non-synchronized script language An eigenvalue and a code associated with the function defined in the non-framework file, the at least one function is obtained; and the at least one function including the at least one asynchronous scripting language call is triggered to obtain the generated webpage content. The method of claim 1, wherein the extracting comprises: querying a scripting language tag in the webpage code; if the scripting language information extracted from the webpage code information and located after the scripting language tag includes a scripting language code, Extracting the script language code, and saving the extracted script language code in the script language file; if the webpage code information is extracted from the webpage code mark The script language information includes a script language file, and the storage path of the script language file and the file name are extracted. The method of claim 1, wherein the asynchronous script language feature value comprises at least one of: calling a feature value corresponding to the asynchronous script language using a script language framework type, and calling the asynchronous script without using the script language frame type The feature value corresponding to the language. The method of claim 1, wherein the at least one function is defined in a non-JavaScript framework file referenced in the web page. A non-synchronized JavaScript and XML web content crawling system, comprising: a webpage code obtaining unit for acquiring webpage code information; a script extracting unit, configured to extract scripting language information in the webpage code information; and a script parsing unit And: determining, according to a file name of a script file, a type of the script file indicated by the script language information, if the script file is a frame file, acquiring an unsynchronized script language feature value, and using the asynchronous script language feature a value that determines a function that includes a call associated with an unsynchronized web application of the web page, the web page being associated with the web page code, and if the script file is a non-frame file, based on the corresponding non-synchronization a script language feature value and a code associated with the function defined in the non-framework file to obtain the function; and a webpage content obtaining unit for triggering the function including at least one asynchronous scripting language call to obtain webpage content. The system of claim 5, wherein the scripting language information comprises at least one of a scripting language file and a scripting language code. According to the system of claim 6, the script extracting unit includes: a query subunit for querying a script language mark in the webpage code; and a first extracting subunit for extracting information from the webpage code and located in the script When the script language information after the language tag includes the script language code, the script language code is extracted, and the extracted script language code is saved in the script language file; and the second extracting subunit is configured to extract the information from the webpage code. And when the script language information after the script language tag contains the script language file, the storage path and file name of the script language file are extracted. According to the system of claim 7, the script parsing unit comprises: a first determining subunit, configured to determine, according to the non-synchronized script language feature value, the at least one asynchronous script language call defined in the script language file The non-synchronized scripting language feature value is: a code segment that can identify a non-synchronized scripting language call in the function; and a second determining subunit, configured to determine the function in the function determined by the first determining subunit The webpage code contains a non-synchronized scripting language call function. The system according to claim 8, wherein the non-synchronized script language feature value comprises at least one of: calling a feature value corresponding to the asynchronous script language using a script language framework type, and calling the non-synchronized script language frame type instead of calling The feature value corresponding to the synchronization script language. The system of claim 8 wherein the first determining subunit is further for determining at least one function comprising at least one asynchronous scripting language call defined in a non-scripting language framework file referenced in the page. The system of claim 10, wherein the webpage content obtaining unit is further configured to trigger the determined at least one function including at least one asynchronous scripting language call by simulating a user operation to obtain at least one non-contained The content of the web page generated by the at least one function invoked by the synchronization scripting language. A non-transitory storage medium, comprising computer executable instructions, when executed by a computer, configuring the computer to perform an action, the action comprising: extracting script information in a webpage, the script information indicating one or more a script file; determining a type of the script file according to a file name of one of the one or more script files, and if the one script file is a frame file, acquiring one or more asynchronous script languages Characteristic values, and Using the one or more non-synchronized scripting language feature values, determining a function that includes a call associated with an asynchronous web application of the web page, and if the one script file is a non-frame file, based on the corresponding non-synchronized script The language feature value and the code associated with the function defined in the non-framework file acquire the function; and call the function to generate content related to the web page. The storage medium of claim 12, wherein the asynchronous web application is generated using asynchronous scripting language technology. The storage medium of claim 13, wherein the one or more script files include a frame file and a non-frame file, the non-frame file definition function including a function of the asynchronous script language call, and the actions further include: The file name of the framework file, obtaining one or more non-synchronized script language feature values corresponding to the framework file; and determining the function according to the one or more non-synchronized script language feature values and the non-framework file. The storage medium of claim 12, wherein the actions further comprise obtaining the web page in a markup language format. According to the storage medium of claim 12, wherein the script information includes script language information.