TWI683225B - Script generation method and device - Google Patents

Abstract

The present application discloses a script generation method and device, which are used to solve the problem of low efficiency in manually writing crawl scripts when using web crawlers to crawl web content in the prior art. The method includes: determining the webpage content selected by the user in the displayed webpage; determining the webpage code corresponding to the displayed webpage content according to the determined webpage content; and generating a crawling script according to the webpage code.

Description

Script generation method and device

This application relates to the field of computer technology, and in particular to a script generation method and device.

In the prior art, since web crawlers can crawl text content in web pages, they are currently widely used in fields such as search and data extraction. Web crawlers can crawl all the content in a webpage, or they can crawl part of the content in a webpage.

At present, to use a web crawler to crawl the target content in the target webpage, the worker needs to first write a script that crawls the target content, so that the web crawler can crawl the target content according to the script.

For example, if you now want to use a web crawler to grab the price information of the products in the web page shown in Figure 1, that is, "price: $149.99". Then, the staff must visit the corresponding webpage through the browser, and then find the webpage code corresponding to "price: $149.99" in the webpage code corresponding to the webpage, that is, the smallest document object model (Document Object) corresponding to "price: $149.99" Model, dom) tree.

Among them, the minimum dom tree corresponding to "price: $149.99" is as follows:

After finding the smallest dom tree corresponding to "Price: $149.99", obtain the attribute value information of HyperText Markup Language (HTML) corresponding to "Price: $149.99", such as id="kfs_family_16", class="kfs -price" etc. Based on the attribute value information, the staff writes a grab script that contains the HTML attribute value information. The written script and the webpage code corresponding to the product are sent to the parsing engine, so that the parsing engine can find the smallest dom tree corresponding to "price: $149.99" according to the id and class in the crawl script, and Extract the "price: $149.99" from the smallest dom tree Price information.

Although the web crawler can crawl the content in the webpage through the above method, it needs to manually write a crawling script, which is inefficient.

Embodiments of the present application provide a script generation method and device, which are used to solve the problem of low efficiency in manually writing crawl scripts when using web crawlers to crawl webpage content in the prior art.

The embodiment of the present application adopts the following technical solution: a script generation method, including: determining a webpage content selected by a user in a displayed webpage; determining a webpage code corresponding to the webpage content according to the determined webpage content; and according to the webpage code To generate a crawl script.

A script generation device includes: a content determination module to determine the webpage content selected by the user in the displayed webpage; a code determination module to determine the webpage code corresponding to the webpage content according to the determined webpage content; a script generation module, According to the webpage code, generate a crawling script.

The above-mentioned at least one technical solution adopted by the embodiment of the present application can achieve the following beneficial effects: Compared with the prior art, when a web crawler is used to crawl webpage content, a crawling script needs to be manually written, and the script generator provided by the embodiment of the present application is used Method, by determining the content of the webpage selected by the user in the webpage, the webpage code corresponding to the webpage content is determined, and a crawling script is generated according to the webpage code, thereby solving the problem of manually crawling webpage content by using web crawlers in the prior art The problem of low efficiency in writing crawl scripts.

21‧‧‧Display web page

22‧‧‧Confirm the content of the webpage selected by the user in the displayed webpage

23‧‧‧According to the determined webpage content, determine the webpage code corresponding to the webpage content

24‧‧‧ Generate crawl script based on the webpage code

31‧‧‧Content determination module

32‧‧‧Code confirmation module

33‧‧‧Script generation module

The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an undue limitation on the present application. In the drawings: FIG. 1 is the content in the target webpage in the prior art; FIG. 2a is a specific flowchart of a script generation method provided by an embodiment of the present application; FIG. 2b is a method for determining HTML attribute values provided by an embodiment of the present application Figure 2c is a page that asks the user which web content to crawl provided by an embodiment of the present application; Figure 2d is a page displayed after the user selects the web content provided by an embodiment of the present application; and Figure 2e provides an embodiment of the present application. Of users display the page after framing the content of the web page twice; FIG. 3 is a schematic diagram of a specific structure of a script generating device provided by an embodiment of the present application.

In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be described clearly and completely in conjunction with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

The technical solutions provided by the embodiments of the present application will be described in detail below in conjunction with the drawings.

In order to solve the problem of low efficiency in manually writing crawling scripts when using web crawlers to crawl webpage content in the prior art, embodiments of the present application provide a script generation method.

The execution subject of the method may be, but not limited to, user terminals such as mobile phones, tablet computers or personal computers (Personal Computer, PC), or applications (Application, APP) running on these user terminals, or may also be servers Other equipment.

For convenience of description, the following takes the execution body of the method as a PC as an example to introduce the implementation manner of the method. It can be understood that the execution body of the method is a PC is only an exemplary description, and should not be understood as a limitation of the method.

The specific flow diagram of this method is shown in Figure 2a, including the following steps:

In step 21, the web page is displayed.

In the embodiment of the present application, when a user wants to use a web crawler to grab a web When the content of the webpage in the page is accessed, the web address can be accessed through a browser installed in the PC or other applications with browser functions, so that the PC can display the webpage for subsequent operations. The following uses the browser as an example for description.

Specifically, the user can enter the web address in the web address input box in the browser and access the web address, and the PC displays the web page corresponding to the web address.

Step 22: Determine the webpage content selected by the user in the displayed webpage.

After the webpage is displayed on the PC, the user can select the webpage content to be crawled in the webpage according to actual needs, so that the browser can determine the webpage content selected by the user in the webpage, and then perform subsequent operations to finally generate a crawl script.

The reason why the user can select the webpage content in the webpage is because the first script exists in the webpage code of the webpage. The first script is used to provide the function of selecting webpage content in the webpage. The first script includes a Cascading Style Sheets (CSS) script. In general, the first script is located at the top or bottom of the webpage code of the webpage. This is because if the first script is embedded in the middle of the webpage code of the webpage, when the browser performs subsequent operations, the first script may be mistakenly used as a part of the webpage code of the webpage, thereby affecting the final crawl script generate. Therefore, the first script is generally embedded at the top or bottom of the web page code.

In actual applications, the first script exists in the webpage code of the webpage, possibly because the user has accessed the URL through the browser, the first script already exists in the webpage code of the webpage returned by the server, or it may be because Before determining the content of the webpage selected by the user in the webpage, the browser embeds the preset first script into the webpage code returned by the server.

If the first script already exists in the webpage code of the webpage returned by the server after accessing the web address through the browser, it may be that the server will preset the webpage code after receiving the webpage code request from the browser The first script is embedded in the webpage code of the webpage, and then the webpage code is sent to the browser; it may also be that the developer of the browser negotiates with the developer of the webpage in advance, and the developer of the webpage is writing the webpage of the webpage When coding, the first script is embedded in the code of the webpage. Therefore, the first script exists in the web page code returned by the server. Among them, the first script does not affect the rendering of the webpage by the browser.

Step 23: Determine the webpage code corresponding to the webpage content according to the determined webpage content.

After performing step 12, the browser may determine the webpage code corresponding to the webpage content according to the webpage content determined by performing step 12.

This is because before the browser determines the webpage code corresponding to the webpage content according to the webpage content determined by performing step 12, the browser embeds the preset second script into the webpage code of the webpage, so that the browser The webpage code corresponding to the target content can be determined through the second script according to the target content. The second script includes Javascript (JS) script.

Or, after the user visits the URL corresponding to the webpage in the browser, the second script exists in the webpage code returned by the server. In this way, the browser can To determine the webpage code corresponding to the webpage content through the second script according to the webpage content. Among them, the reason why the second script exists in the webpage code returned by the server may be that the server embeds the preset second script into the webpage code of the webpage after receiving the webpage code request from the browser to obtain the webpage, Then send the webpage code to the browser; it may be that the developer of the browser negotiates with the developer of the webpage in advance. When the webpage developer writes the webpage code of the webpage, the second script is embedded in the webpage Code. Therefore, the second script exists in the web page code returned by the server. Among them, the second script does not affect the rendering of the web page by the browser.

Then, the specific implementation of the browser to determine the webpage code corresponding to the webpage content through the second script according to the webpage content determined by performing step 12 may be as follows: The browser determines the minimum dom tree corresponding to the webpage content in the webpage code of the webpage through the second script according to the webpage content determined by performing step 12, and then determines the corresponding webpage content in the minimum dom tree HTML attribute value.

When the browser renders the webpage according to the webpage code, it can establish the mapping relationship between the webpage content and the webpage code, or establish the mapping relationship between the coordinate points where the webpage content in the webpage is located and the webpage code. Therefore, the browser may determine the webpage code corresponding to the webpage content, that is, the minimum dom tree corresponding to the webpage content, according to the above mapping relationship and the webpage content determined by performing step 12. If the webpage content determined by performing step 12 corresponds to a smallest dom tree, then in the smallest dom tree, it is determined that the webpage content determined by performing step 12 corresponds to HTML attribute value. Among them, HTML attribute value can be class, can also be id and class. Specifically, before the browser determines the HTML attribute value, the browser may display an inquiry page on the screen of the PC where the browser is located to ask the user to determine the id and class in the smallest dom tree or determine the smallest dom The class in the tree. For example, as shown in the page shown in FIG. 2b, the page includes controls for determining id and class, and controls for determining class. If the user clicks to determine the control items of id and class, the browser determines the id and class. If the user clicks to determine the control items of class, the browser determines the class.

If the webpage content determined by performing step 12 corresponds to at least two smallest dom trees, then in the smallest dom trees, the HTML attribute values corresponding to the webpage content determined by performing step 12 are respectively determined. Before determining the HTML attribute value, the browser may display an inquiry page as shown in FIG. 2b on the screen of the PC where the browser is located, to ask the user to determine the id and class in each smallest dom tree, or to determine Each class in the smallest dom tree. The user can determine which control item to click according to actual needs, so that the browser determines the corresponding HTML attribute value according to the user's selection.

After determining the above HTML attribute value, step 14 can be performed to finally generate a crawling script.

Step 24: Generate a crawling script based on the webpage code.

After step 13 is performed, the browser may add the HTML attribute value in the determined webpage code corresponding to the webpage content to be crawled to the preset script generation template to generate the crawling script. Among them, the grab foot This is used to grab webpage content that matches the HTML attribute value.

Among them, if the HTML attribute values determined by the browser are id and class, then the browser can add the determined id and class in each dom to {id=XXX, class=XXX} In the preset script generation template, a crawl script is generated. If the HTML attribute value determined by the browser is class, then the browser can add the determined class in each dom to the preset script generation template in the form of {class=XXX}, to generate the capture Take the script.

After the browser generates the crawl script, it can save the crawl script locally. In addition, the browser can also save the smallest dom tree corresponding to the webpage content determined by performing step 12 and all webpage codes locally, so that the browser can use the above crawling script and the smallest dom in subsequent operations Tree and webpage code to crawl webpage content.

After generating the crawling script, the browser may pop up a page in the PC where the browser is located to inform the user that the crawling script has been generated and ask the user whether to crawl the webpage content.

For example, the page may be as shown in Figure 2c. This page includes the first crawl control item and the second crawl control item. If the user clicks the first crawl control item, the browser sends the crawl script and the smallest dom tree corresponding to the web page content determined by performing step 12 to the parsing engine. If the crawl script contains id and class, the parsing engine finds the smallest dom tree where the id is located based on the id, and then extracts the smallest dom tree that the user wants to catch based on the class that exists in the same combination with the id Fetched web content. For example, if the determined user wants to crawl the webpage shown in Figure 1, the minimum dom tree corresponding to "Price: $149.99" is:

The scraping script for scraping "price: $149.99" contains id="kfs_family_16", and class="kfs-price" corresponding to "price: $149.99".

Then, when the user clicks the first crawl control item, the browser can send the above minimum dom tree and crawl script to the parsing engine, and the parsing engine can find the smallest dom where the id is located according to id="kfs_family_16" Tree, and then extract the price information "price: $149.99" from the dom tree according to class="kfs-price".

If id is not included in the crawl script, only class is included, the browser will The class can extract the web page content matching the class in all the smallest dom trees sent to the parsing engine.

When the user clicks the second crawl control item, the browser sends the crawl script and the web page code of the web page to the parsing engine. If the crawl script contains id and class, the parsing engine finds the smallest dom tree where the id is located according to the id, and then extracts the smallest dom tree that the user wants to catch based on the class that exists in the same combination with the id Fetched web content.

If the crawl script does not contain id, but only class, the browser can extract the webpage content that matches the class in the webpage code of the webpage according to the class.

It should be noted that the execution bodies of the steps of the method provided in the embodiments of the present application may be the same or different. For example, after the browser performs step 13, the browser may send the determined webpage code and HTML attribute value to the server, so that the server generates a crawling script according to the webpage code. In addition, the above is only an exemplary description taking the browser installed in the PC as an example. The execution subject of the embodiment of the present application may be a browser installed in the PC, or may have other browser functions installed in the PC. The application may also be an APP with a browser function in the mobile terminal, which is not limited in this application.

In the embodiment of the present application, when the browser executes step 12, specifically, in one embodiment, once the user starts frame selection on the webpage, the browser can start to confirm the content of the webpage selected by the user. Or, after the user's frame selection is completed, the framed target content is framed by a rectangular frame, and the web page will display controls for continuing frame selection, submission, and cancellation. its In, the target content framed by a rectangular frame can be displayed in a highlighted manner, or it can be displayed in the initial display mode of the target content when the web page is displayed. This can be set according to user needs. This embodiment of the present application There are no restrictions on this. For example, as shown in FIG. 2d, the page displayed in FIG. 2d is the page displayed after the user selects the price information of a product in a shopping website that he wants to crawl. The price information on this page is framed by a rectangular box, and the control options for continued frame selection, submission, and cancellation are displayed on the right side of the price information, respectively. Among them, the framed price information is not highlighted, but displayed in the initial display state.

After the controls to continue to frame, submit, and cancel appear on the web page, if the user wants to continue to frame other content, they can click to continue to frame the control and continue to frame other content. If the user no longer wants to frame other content, he can click the submit control item, and the browser determines the content of the webpage selected by the user as the target content. If the user wants to cancel the previously selected content and select other content, then click Cancel Control to reselect the content.

In addition, in the script generation method provided by the embodiment of the present application, when the user selects the content of the web page to be crawled in the web page, a rough selection can be made first, and the browser can determine the web page according to the content of the web page selected by the user for the first time. The smallest dom corresponding to the content. Then, the user makes a second selection based on the first-selected webpage content, and the browser can determine the HTML attribute in the webpage code corresponding to the second-selected webpage content in the smallest dom determined value. For example, as shown in FIG. 2e, if the user wants to crawl the price information of “￥175” in the webpage, the user can roughly include the price information of “￥175” when selecting for the first time. When the webpage content is selected together, the browser can determine the smallest dom tree corresponding to the webpage content according to the first selected content. When the user selects for the second time, he can only select "￥175", then the browser can determine the HTML attribute value corresponding to "￥175" in the smallest dom tree. There are two rectangular boxes in FIG. 2e, where all the content in the larger rectangular box is the content selected by the user for the first time, and the content of the web page in the smaller rectangular box is the content selected by the user for the second time.

In the embodiments of the present application, a script generating device may also be used to implement the script generating method provided in the embodiments of the present application.

As shown in FIG. 3, it is a schematic structural diagram of a script generation device provided by an embodiment of the present application, which mainly includes the following device: a content determination module 31, which determines the content of a webpage selected by the user in the displayed webpage.

The code determination module 32 determines the webpage code corresponding to the webpage content according to the determined webpage content.

The script generation module 33 generates a crawling script based on the webpage code.

In one embodiment, the device further includes a first embedding module, which embeds a preset first script into the webpage before the content determination module 31 determines the webpage content selected by the user in the displayed webpage In the page code. Wherein, the first script is used to provide a function of selecting webpage content in the webpage, and the first script includes a cascading style sheet CSS script.

In one embodiment, the device further includes: The second embedding module, before the code determining module 32 determines the webpage code corresponding to the webpage content according to the determined webpage content, embeds the preset second script into the webpage code of the webpage, the second The script includes JS script.

Then, the code determination module 32 determines the webpage code corresponding to the webpage content through the second script according to the determined webpage content.

In one embodiment, the code determination module 32 determines the smallest document object model dom tree corresponding to the webpage content in the webpage code of the webpage; and determines the webpage content in the smallest dom tree Corresponding HTML attribute value of hypertext markup language.

In one embodiment, the script generation module 33 adds the determined HTML attribute value to a preset script generation template to generate a crawl script, and the crawl script is used to capture and HTML attributes Web content with matching values.

In one embodiment, the device further includes a content parsing module, which sends the crawling script and webpage code to a parsing engine, and crawls the corresponding webpage content through the parsing engine.

Compared with the prior art of using web crawlers to crawl webpage content, it is necessary to manually write a crawling script. Using the script generation method provided in this embodiment of the present application, the webpage content is determined by determining the webpage content selected by the user in the webpage Corresponding webpage code, and generating a crawling script according to the webpage code, thereby solving the problem of low efficiency in manually writing crawling scripts when using web crawlers to crawl webpage content in the prior art.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention can take the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code .

The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and/or block in the flowchart and/or block diagram and a combination of the flow and/or block in the flowchart and/or block diagram may be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing equipment to produce a machine that allows instructions executed by the processor of the computer or other programmable data processing equipment Means for generating the functions specified in one block or multiple blocks of the flowchart one flow or multiple flows and/or block diagrams.

These computer program instructions can also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device The instruction device implements the functions specified in one block or multiple blocks in one flow or multiple flows in the flowchart and/or one block in the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing equipment, so that they can be executed on the computer or other programmable equipment Perform a series of operating steps to generate computer-implemented processing so that instructions executed on a computer or other programmable device provide a block or blocks for implementing a flow or a flow in a flowchart or a flow and/or block diagram The steps of the function specified in.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent memory, random access memory (RAM) and/or non-volatile memory in computer-readable media, such as read-only memory (ROM) or flash memory (flash) RAM). Memory is an example of computer-readable media.

Computer-readable media, including permanent and non-permanent, removable and non-removable media, can be stored by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM) , Read-only memory (ROM), electrically erasable and programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disc read-only memory (CD-ROM), digital multifunction Optical discs (DVDs) or other optical storage, magnetic cassette tapes, magnetic tape storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also includes Other elements not explicitly listed, or include elements inherent to this process, method, commodity, or equipment. Without more restrictions, the element defined by the sentence "include one..." does not exclude that there are other identical elements in the process, method, commodity or equipment that includes the element.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may take the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code .

The above are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included in the scope of the patent application of this application.

Claims (10)

A script generation method, characterized in that the method includes: determining a webpage content selected by a user in a displayed webpage; determining a webpage code corresponding to the webpage content according to the determined webpage content; and generating a crawling script according to the webpage code Where determining the webpage code corresponding to the webpage content specifically includes: determining the smallest document object model dom tree corresponding to the webpage content in the webpage code of the webpage; determining the webpage content in the smallest dom tree Corresponding HTML attribute value of hypertext markup language. The method according to item 1 of the patent application scope, wherein, before determining the content of the webpage selected by the user in the displayed webpage, the method further comprises: embedding a preset first script into the webpage code of the webpage; wherein, The first script is used to provide a function of selecting webpage content in the webpage, and the first script includes a cascading style sheet CSS script. The method according to item 1 of the patent application scope, wherein, before determining the webpage code corresponding to the webpage content according to the determined webpage content, the method further includes: embedding a preset second script into the webpage code of the webpage , The second script includes a JS script; According to the determined webpage content, determining the webpage code corresponding to the webpage content specifically includes: determining the webpage code corresponding to the webpage content through the second script according to the determined webpage content. The method according to item 1 of the patent application scope, wherein generating a script according to the web page code specifically includes: adding the determined HTML attribute value to a preset script generation template to generate a crawling script, which captures The fetch script is used to grab the webpage content that matches the HTML attribute value. The method as described in item 4 of the patent application scope, wherein the method further comprises: sending the crawling script and the webpage code to the parsing engine, and crawling the corresponding webpage content through the parsing engine. A script generating device, characterized in that the device includes: a content determination module to determine the webpage content selected by the user in the displayed webpage; a code determination module to determine the webpage code corresponding to the webpage content according to the determined webpage content; The script generation module generates a crawling script based on the webpage code, wherein the code determination module determines the smallest document object model dom tree corresponding to the webpage content in the webpage code of the webpage; In the smallest dom tree, determine the HTML attribute value of the hypertext markup language corresponding to the content of the web page. The device as described in item 6 of the patent application scope, wherein the device further includes: a first embedded module, which presets the first script before the content determination module determines the content of the webpage selected by the user in the displayed webpage Embedded in the webpage code of the webpage; wherein, the first script is used to provide a function of selecting webpage content in the webpage, and the first script includes a cascading style sheet CSS script. The device as described in item 6 of the patent application scope, wherein the device further comprises: a second embedded module, before the code determination module determines the webpage code corresponding to the webpage content according to the determined webpage content, the preset The second script is embedded in the webpage code of the webpage. The second script includes a JS script; then the code determination module determines the webpage code corresponding to the webpage content through the second script based on the determined webpage content. . The device as described in item 6 of the patent application scope, wherein the script generation module adds the determined HTML attribute value to a preset script generation template to generate a crawl script, which is used for crawling With that HTML Web content with matching attribute values. The device as described in item 9 of the patent application scope, wherein the device further includes: a content parsing module, which sends the crawling script and webpage code to the parsing engine, and crawls the corresponding webpage content through the parsing engine.

Images