TWI402694B - System and method for controlling downloading data of webpage - Google Patents

Description

Web page data download control system and method

The invention relates to a webpage data download control system and method.

With the use of the Internet, it is more and more common to download various materials from the website to the database. Usually, the downloaded data is not displayed on one page, but is divided into many pages. In the actual operation process, each download is from scratch to Executing once, that is, executing from the first page to the last page, repeating the execution of the already downloaded data is inefficient and wastes network resources.

In view of the above, it is necessary to provide a webpage data download control system for controlling webpage flipping by comparing the number of data in the downloaded material list with the number of generated download data.

In addition, it is also necessary to provide a webpage data download control method for controlling webpage flipping by comparing the number of data in the downloaded material list with the number of generated download data.

A webpage data download control system includes an application server and a database connected to the application server, the application server includes: a setting module, configured to set data download control parameters, wherein the data download control parameter includes downloading The website address of the data and the time range for allowing the download; the parsing module is configured to parse the data list of a web page under the set website address; the obtaining module is configured to obtain the parsed data information of the data list, The data information of the data list includes the time when each piece of data is published; the calculation module is configured to calculate the number of all the data in the data list; and the determining module is configured to judge the data list by using the obtained data information Whether each piece of data has been downloaded in the database, and according to the release time of each piece of data, whether the pieces of data in the list of materials are within the time range allowed for downloading; generating a module for generating the data The script language download command in the list is also within the time range allowed for download and there is no downloaded material; download mode For downloading the corresponding data by using the generated script language download command, and saving the downloaded data to the database; the computing module is further configured to calculate the generated script language download command number; the determining module, The method is further configured to determine whether the number of the data in the data list is equal to the number of generated script language download commands, and the page turning module is configured to: when the number of the data in the data list is equal to the number of generated script language download commands, Perform page flipping of the web page.

A webpage data download control method, the method comprising the steps of: setting a data download control parameter, wherein the data download control parameter includes a website address of a data to be downloaded and a time range for allowing downloading; and parsing a set of a website address a data list of the webpage; obtaining the data information of the data list obtained by the parsing, the data information of the data list includes the publishing time of each data; calculating the number of all the data in the data list; and judging the information by the obtained information Whether each piece of data in the data list has been downloaded in the database, and according to the publication time of each piece of data, it is judged whether each piece of data in the item list is within the allowable download time range; if there is no data in the item list Is downloaded, and the publishing time of the undownloaded material is within the allowable download time range, then the script language download command that is not within the time range allowed for downloading and is not downloaded is generated; The script language download command downloads the corresponding data and saves the downloaded data. Go to the database; calculate the number of generated script language download commands; determine whether the number of all the data in the data list is equal to the number of generated script language download commands; if the number of all the data in the data list is equal to the generated script If the number of languages is downloaded, the page flipping action is performed.

Compared with the prior art, the webpage data download control system and method can determine whether or not to turn pages by analyzing whether the webpage has been downloaded, thereby avoiding the program from the first page to the last page each time. The cumbersome steps are performed once, which improves the efficiency of downloading and saves network resources.

FIG. 1 is a hardware structural diagram of a preferred embodiment of a webpage data download control system according to the present invention. The system includes an application server 1, a database 2, a client 3, a firewall 4, and a network 5. The application server 1 is connected to the network 5 through the firewall 4, and is used for downloading the materials published on the website, and saving the downloaded materials to the database 2. The application server 1 can be a personal computer, a web server, or any other suitable data processing device. The firewall 4 is used to control the information security of the network 5. The network 5 can be the Internet or a local area network.

The application server 1 is connected to a database 2 for storing data downloaded by the application server 1. The database 2 can be built in the application server 1 or external to the application server 1.

In addition, the application server 1 is connected to at least one user terminal 3 for providing an interactive interface to the user, so that the user can input the download information to the application server 1.

As shown in FIG. 2, it is a functional module diagram of the application server in FIG. The application server 1 includes a setting module 10, an analysis module 12, a calculation module 14, a determination module 16, an acquisition module 18, a generation module 20, a download module 22, and a page turning module 24.

The setting module 10 is configured to set a data download control parameter, and save the set parameter to the data base 2. The data download control parameter includes a website address that needs to download data, a time range allowed for downloading the data, and a path for saving the downloaded data.

The parsing module 12 is configured to parse the data list of a webpage under the set website address, and the data information of the data list includes the date of each downloading data, the address and the title of each downloaded data.

The calculation module 14 is configured to calculate the number of all the materials in the current webpage data list, and calculate the generated script language download command number. The scripting language in this embodiment refers to the Xquery scripting language.

The determining module 16 is configured to determine whether the next piece of data exists in the current webpage data list.

The obtaining module 18 is configured to obtain the data information of the data list obtained by the parsing.

The determining module 16 is further configured to determine whether each piece of data in the current webpage data list has been downloaded in the database, and determine whether each piece of data in the data list is allowed to be downloaded according to the publishing time of each piece of data. Within the time range.

The generating module 20 is configured to generate a scripting language (Xquery) command in the current page material list that is still within the time range that is allowed to be downloaded and has not been downloaded. In this embodiment, when parsing a piece of material in the material list, if the piece of material is not downloaded and is within the allowed download date range, the generation module 20 generates an Xquery language command. How many pieces of data are generated by the generation module 20, how many pieces of data are downloaded.

The determining module 16 is further configured to determine whether the downloaded data in the current webpage data list is all detected.

The download module 22 is configured to download the corresponding data by using the generated script language download command and save the downloaded data to the application server 1 in the form of an Extensible Markup Language (XML) file.

The determining module 16 is further configured to determine whether the number of pieces of data in the current page material list is greater than the number of generated Xquery commands.

The page turning module 24 is configured to perform a page turning action of the webpage when the number of pieces of material in the current page material list is equal to the number of generated script language download commands.

FIG. 3 is a flow chart of a preferred embodiment of a webpage data download control method according to the present invention. First, in step S10, the setting module 10 sets the data download control parameters, and saves the set parameters to the database 2. The data download control parameter includes: a website address for downloading the data, a time range allowed for downloading the data, and a path for saving the downloaded data. The website address of the downloaded data refers to: a URL link address. Specifically, taking the technology news material on the Sina website as an example, the setting module 10 sets the URL address of the technology news material under the Sina website in the system.

In step S11, the parsing module 12 parses the data list of the current webpage under the website address of the set download data to obtain the data information of the data list. The data list of the current webpage refers to a plurality of materials. The data information of the data list includes the date of publication of each downloaded data in the data list, the address and title of each downloaded data. The information available for download on the website is displayed in the form of a data list and one piece of data stored in the data list on the page, and the parsing module 12 parses each piece of information stored in the data list under the web page.

In step S12, the calculation module 14 calculates the number of all the data in the current webpage data list.

In step S13, the determining module 16 determines whether the next piece of data exists in the current webpage material list.

Step S14: When the next piece of data exists in the current webpage data list, the obtaining module 18 acquires basic information of a piece of data. The basic information includes: a release date, a download address, and a title of the piece of information.

In step S15, the determining module 16 determines whether the piece of data in the current webpage material list has been downloaded in the database. Specifically, the title of the article and the download address (URL) are used to check whether the data is in the database 2; if the data is in the database 2, the data has been downloaded, if the database 2 If there is no such information, the information has not yet been downloaded.

In step S16, when a single piece of data in the current webpage material list has not been downloaded to the database, the judging module 16 judges whether the publishing time of the single piece of data in the current webpage material list is within the set time range for allowing downloading.

In step S17, when the single data publishing time is within the set time range for allowing downloading, the generating module 20 generates an Xquery command for downloading the single piece of data. A piece of data corresponds to an Xquery command.

In step S18, the determining module 16 determines whether the data of the current webpage data list is all detected.

In step S19, when all the data in the current webpage data list is detected, the downloading module 22 downloads the corresponding current webpage data through the generated Xquery command, and downloads the file in the form of an Extensible Markup Language (XML) file. The data is saved to the application server 1 according to the set path of the stored download data.

In step S20, the calculation module 14 calculates the number of Xquery commands of the downloaded data generated in the current webpage material list.

In step S21, the determining module 16 determines whether the number of pieces of data in the current webpage material list is greater than the number of generated Xquery commands.

In step S22, when the number of all the data in the current webpage material list is equal to the number of generated Xquery commands, the page turning module 24 performs the page turning operation of the webpage, and then returns to step S11.

In step S13, if the next piece of data does not exist in the current web page material list, then go to step S20.

In step S15, if a single piece of data in the current web page material list has been downloaded, the process returns to step S13.

In step S16, if the single piece of material publication time is not within the set time period of the permitted download, the process returns to step S13.

In step S18, if there is still no data detected in the current web page material list, the process returns to step S13.

In step S22, if the number of pieces of data in the current web page material list is not equal to the number of generated Xquery commands, the process ends.

Application server. . . 1

database. . . 2

user terminal. . . 3

Firewall. . . 4

network. . . 5

Set the module. . . 10

Analytic module. . . 12

Calculation module. . . 14

Judging the module. . . 16

Get the module. . . 18

Generate modules. . . 20

Download module. . . twenty two

Page turning module. . . twenty four

1 is a hardware frame diagram of a preferred embodiment of a web page data download control system of the present invention.

2 is a functional block diagram of the application server of FIG. 1.

3 is a flow chart of a preferred embodiment of a web page data download control method of the present invention.

parameter settings. . . S10

Parse the data list information in the current page. . . S11

Calculate the current page list data. . . S12

Is there a next item? . . S13

Get basic information about a single piece of information. . . S14

Whether the single piece of information obtained has been downloaded. . . S15

Whether the release time of the single piece of data is within the set time range. . . S16

Generate an Xquery command to download the single piece of information. . . S17

Whether all tests have been completed. . . S18

Download the current page and save it to the database. . . S19

Calculates the number of Xquery commands for the downloaded data generated by the current page. . . S20

Whether the number of list data is equal to the number of generated Xquery commands. . . S21

Turn over the next page. . . S22

Claims (7)

A webpage data download control system includes an application server and a database connected to the application server, wherein the application server comprises: a setting module, configured to set data download control parameters, wherein the data download control parameters include The website address of the data to be downloaded and the time range for allowing the download; the parsing module is used to parse the data list of a web page under the set website address; and the obtaining module is used to obtain the parsed data information of the data list. The data information of the data list includes the time when each piece of data is published; the calculation module is configured to calculate the number of all the data in the data list; and the determining module is configured to judge the data by using the obtained data information Whether each piece of data in the list has been downloaded in the database, and according to the release time of each piece of data, it is judged whether each piece of data in the item list is within the time range allowed for downloading; generating a module for generating the house a script language download command in the list of materials that is still within the time range allowed for download and has no downloaded material; The loading module is configured to download the corresponding data by using the generated script language download command, and save the downloaded data to the database; the computing module is further configured to calculate the generated script language download command number; the determining The module is further configured to determine whether the number of the data in the data list is equal to the number of generated script language download commands; and the page turning module is configured to: when the number of the data in the data list is equal to the generated script language download command When the number is counted, the page flipping action of the web page is performed. The webpage data download control system of claim 1, wherein the data download control parameter further includes a path for saving the downloaded data; and the downloading module downloads the data according to the path described. The data is downloaded to the database. The webpage data download control system according to the first aspect of the patent application, wherein the data information of the data list includes: a download address and a title of each piece of data; and the downloading position of the piece of data by the judging module The address and title determine whether the material has been downloaded in the database. A webpage data download control method, wherein the method comprises the following steps: setting a data download control parameter, wherein the data download control parameter includes a website address of a data to be downloaded and a time range for allowing downloading; and parsing the set website address a data list of a webpage; obtaining information information of the data list obtained by the parsing, the data information of the data list includes the publishing time of each data; calculating the number of all the data in the data list; and obtaining the information information Determining whether each piece of data in the data list has been downloaded in the database, and determining whether each piece of data in the item list is within the allowable download time range according to the publication time of each piece of data; The data is not downloaded, and the publishing time of the undownloaded material is within the allowable download time range, and the script language download command of the material that is still within the time range allowed for downloading and is not downloaded is generated; The generated script language download command downloads the corresponding data and will download the The material is saved in the database; the number of generated script language download commands is calculated; whether the number of all the data in the data list is equal to the number of generated script language download commands; if the number of all the data in the data list is equal to the generated The script language download command number, then perform page flipping action. The webpage data download control method of claim 4, wherein the method further comprises the step of: if the number of all the data in the data list is not equal to the number of generated script language download commands, the process ends. The webpage data download control method according to the fourth aspect of the invention, wherein the data download control parameter further includes a path for saving the downloaded data; in the process of downloading the data, downloading the data according to the path Go to the database. The method for controlling downloading of webpage data according to item 4 of the patent application scope, wherein the information of the data list further includes: a download address and a title of each piece of data in the data list; and a download position of each piece of data The address and title are used to determine if each piece of material has been downloaded in the database.