CN114610970A

CN114610970A - Data acquisition method, device, equipment and medium

Info

Publication number: CN114610970A
Application number: CN202210234928.0A
Authority: CN
Inventors: 刘建华
Original assignee: Ping An Puhui Enterprise Management Co Ltd
Current assignee: Ping An Puhui Enterprise Management Co Ltd
Priority date: 2022-03-09
Filing date: 2022-03-09
Publication date: 2022-06-10

Abstract

The application relates to the technical field of internet application, and particularly discloses a data acquisition method, a device, equipment and a medium, wherein the data acquisition method responds to a target website sent by a user and displays a target page corresponding to the target website; receiving selection operation executed by a user on the data which can be acquired in the target page to obtain a target acquisition data item; acquiring a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein, there are input parameters depending on the page; and generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task. The user can complete the automatic generation of the data acquisition task only by simple selection operation, so that the user can generate and define the data acquisition task without professional network data acquisition knowledge, and the data is acquired from the website through the data acquisition task, thereby improving the efficiency and the universality of data acquisition.

Description

Data acquisition method, device, equipment and medium

Technical Field

The application relates to the technical field of internet application, in particular to a data acquisition method, a device, equipment and a medium.

Background

With the vigorous development of internet technology, various content platforms are in endless, and the mass content is exponentially increased. In the past, a mode of obtaining concerned webpage information through information browsing in a content platform is no longer applicable with the increase of the content platform and the continuous growth of mass content, and how to obtain concerned webpage information efficiently becomes a problem to be solved urgently at present.

The existing implementation of webpage information acquisition is realized by browsing contents in each large content platform one by one, and on the other hand, the search initiation continuously acquires webpage information search results from each large content platform, so that webpage information can be acquired more quickly. However, in any implementation mode, manual intervention is required for searching, browsing and screening, various types of concerned web page information cannot be automatically and quickly acquired, efficient acquisition of numerous types of web page information cannot be realized, web page information acquisition completed manually is relied on, and the method is not universal.

In summary, aiming at a plurality of content platforms and exponentially increasing mass content, the defects of high efficiency and universality caused by dependence on manual work in webpage information acquisition are urgently to be overcome.

Disclosure of Invention

In order to solve the technical problem, embodiments of the present application provide a data acquisition method, apparatus, device, and medium, so that a user can simply and quickly acquire web page information.

In a first aspect, the present application provides a data acquisition method, including: responding to a target website sent by a user, and displaying a target page corresponding to the target website; the target page comprises collectable data; receiving selection operation executed by a user on the data which can be acquired in the target page to obtain a target acquisition data item; acquiring a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein the dependent page has input parameters; and generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task.

According to the preferred embodiment of the present invention, receiving a selection operation performed by a user on data collectable in a target page to obtain a target collection data item, includes: receiving at least two selection operations executed by a user on the data which can be collected in the target page to obtain first selection data and second selection data; respectively acquiring data item identifications of first selected data and second selected data; and if the data item identifications of the first selected data and the second selected data are the same, taking the data item identifications as target acquisition data items.

According to the preferred embodiment of the invention, the generation of the data collection task based on the dependent page comprises the following steps: if the data item identifications of the first selected data and the second selected data are the same, acquiring a first data extraction step of the first selected data and a second data extraction step of the second selected data; a target data extraction step of obtaining a target acquisition data item based on the first data extraction step and the second data extraction step; and generating a data acquisition task based on the target data extraction step and the dependent page.

According to a preferred embodiment of the present invention, the target data extracting step of obtaining the target collected data item based on the first data extracting step and the second data extracting step includes: and a target data extraction step of generating a target collected data item based on the same substeps included in the first data extraction step and the second data extraction step.

According to the preferred embodiment of the present invention, acquiring a dependent page corresponding to a target acquisition data item in a target page based on an input parameter required by the target acquisition data item in the target page includes: acquiring page path information associated with a target page based on input parameters required by a target acquisition data item in the target page, and taking all pages contained in the page path information as candidate pages; receiving parameter configuration operation executed by a user on a target acquisition data item in a target page to obtain a target data acquisition range; and screening the candidate pages based on the target data acquisition range to obtain the dependent pages corresponding to the target acquisition data items in the target pages.

According to the preferred embodiment of the invention, the generation of the data collection task based on the dependent page comprises the following steps: acquiring website information corresponding to the dependent page; acquiring data acquisition request information based on the website information corresponding to the dependent page; and generating a data acquisition task based on the data acquisition request information.

According to the preferred embodiment of the invention, the data acquisition task comprises data acquisition request information and a target data extraction step; data acquisition is carried out according to a data acquisition task, and the data acquisition method comprises the following steps: acquiring a target page to be acquired based on the data acquisition request information; and acquiring target data in a target page to be acquired based on the target data extraction step, and storing the target data.

In a second aspect, the present application provides a data acquisition device comprising: the display module is configured to respond to a target website sent by a user and display a target page corresponding to the target website; the target page comprises collectable data; the target data acquisition module is configured to receive selection operation executed by a user on the data which can be acquired in the target page to obtain a target acquisition data item; the dependent page acquisition module is configured to acquire a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein, the dependent page has input parameters; and the data acquisition module is configured to generate a data acquisition task based on the dependency page and acquire data according to the data acquisition task.

In a third aspect, the present application provides a computer device comprising a memory and a processor; a memory for storing a computer program; a processor for executing the computer program and for implementing the steps of the data acquisition method described above when executing the computer program.

In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to implement the steps of the data acquisition method described above.

According to the data acquisition method, the data acquisition device, the data acquisition equipment and the data acquisition medium, a target page corresponding to a target website is displayed by responding to the target website sent by a user; the target page comprises collectable data; receiving selection operation executed by a user on the data which can be acquired in the target page to obtain a target acquisition data item; acquiring a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein, there are input parameters depending on the page; and generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task. The user can complete the automatic generation of the data acquisition task only by simple selection operation, so that the user can generate and define the data acquisition task without professional network data acquisition knowledge, and the data is acquired from the website through the data acquisition task, thereby improving the efficiency and the universality of data acquisition.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

FIG. 1 is a schematic diagram of an application environment of a data acquisition method provided in an embodiment of the present application;

FIG. 2 is a flow chart of a data collection method provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a target page provided by an embodiment of the present application;

fig. 4 is a flowchart of step S220 in fig. 2 according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of an extraction step of the acquired target data provided in the embodiment of the present application;

FIG. 6 is a schematic diagram of an acquire dependency page provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of another fetch dependency page provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a target data acquisition range provided by an embodiment of the present application;

fig. 9 is a schematic diagram illustrating interaction among a user terminal, a data acquisition server, and a target server according to an embodiment of the present application;

FIG. 10 is a schematic block diagram of a data acquisition device provided by an embodiment of the present application;

FIG. 11 is a schematic block diagram of a computer device provided by an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to the present application. Rather, they are merely examples of the same apparatus and methods as some aspects of the present application, as detailed in the appended claims.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. the functional entities may be implemented in the form of application programs or in one or more hardware modules or integrated circuits or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

In the present application, the term "plurality" means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.

Fig. 1 is a system architecture diagram illustrating an operating environment of an exemplary embodiment of the present application, and referring to fig. 1, the system may include a user terminal 110, a data collection server 120, and at least one target server 130. The user terminal 110 and the data collection server 120 are connected through network communication; the data acquisition server 120 and the target server 130 are connected through network communication; the user terminal 110 and the target server 130 are connected through network communication; the network may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user terminal 110 is configured to display information to a user, receive an operation of the user to obtain data or an instruction corresponding to the operation of the user, and send the data or the instruction to the data collection server 120 or the target server 130 through a network. The user terminal 110 may be hardware or software. When the user terminal 110 is hardware, it may be various electronic devices, including but not limited to an in-vehicle smart terminal, a smart phone, a tablet computer, a smart band, and the like. When the user terminal 110 is software, it can be installed in the electronic devices listed above.

The data collection server 120 is configured to generate and execute a data collection task, and obtain the web page information of the target server 130 according to the data collection task, so as to feed back the obtained web page information to the user terminal 110.

The target server 130 is used to implement the operation of the web address, and provides the web page information in the web address to the machine accessing the target server 130. It should be understood that the target server 130 interacting with the data collection server 120 implements a website, that is, a website sent by the user terminal 110 to the data collection server 120, and therefore, the data collection server 120 is oriented to at least one target server 130, and the data collection server 120 performs data collection based on the exemplary embodiment provided in the present application through interaction with at least one target server 130.

The data collection server 120 and the target server 130 may be independent physical servers, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be cloud servers providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platforms.

It should be understood that the numbers of the user terminals 110, the data collection servers 120 and the target servers 130 in fig. 1 are only illustrative and are only used for understanding the embodiments of the present application, and the numbers of the specific user terminals 110, the data collection servers 120 and the target servers 130 should be flexibly determined according to practical situations.

Referring to fig. 2, fig. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present disclosure. As shown in fig. 2, the method includes steps S210 to S240.

Step S210, responding to a target website sent by a user, and displaying a target page corresponding to the target website; wherein the target page includes collectable data.

The target website refers to a network address corresponding to data that a user needs to collect, and some information is stored on the internet page by page, each page has a corresponding address, so that other users can access to obtain information data, and such an address is called a website.

For example, the user terminal may receive a website input operation of a user, obtain a target website according to the website input operation, send the target website to the data acquisition server, then the data acquisition server sends the target website to the target server, the target server feeds back a target page to the data acquisition server according to the target website, and finally the data acquisition server sends the target page to the user terminal, so that the user terminal displays the target page.

For example, the user terminal sends a data acquisition instruction to the data acquisition server, so that the data acquisition server presents the task creation page to the user through the user terminal according to the data acquisition instruction. The task creation page comprises a website input frame control and a website submission control, website input operation of a user is received through the website input frame control, website information input by the user is obtained, and if the website submission control is detected to be triggered, the website information in the website input frame control is used as a target website and sent to the data acquisition server.

Illustratively, the user terminal may receive a web page access operation of a user, obtain a target website according to the web page access operation, and send the target website to the data acquisition server. For example, the user terminal sends a data acquisition instruction to the data acquisition server, so that the data acquisition server presents the task creation page to the user through the user terminal according to the data acquisition instruction. The task creating page comprises a web browsing interface and a website submitting control, the web browsing interface displays a currently browsed web interface of a user, if the website submitting control is triggered, website information of the currently displayed web interface in the web browsing interface is acquired, and the website information is used as a target website and sent to the data acquisition server.

It can be understood that the method for acquiring the target website can be flexibly selected according to the actual situation, which is not limited in the present application.

Further, after the data acquisition server acquires the target website sent by the user terminal, the data acquisition server sends the target website to the target server, so that the target server feeds back information of the target page to the data acquisition server according to the target website, and then the data acquisition server sends the information of the target page to the user terminal, so that the user terminal displays the target page according to the information of the target page.

For example, the data acquisition server generates a page request message according to the target website, and sends the page request message to the target server. And the target server analyzes the page request message to obtain a target website, inquires according to the target website to obtain the information of the target page, and sends a feedback message generated according to the information of the target page to the data acquisition server. And the data acquisition server sends the feedback message to the user terminal so that the user terminal analyzes the feedback message to display the target page.

It is understood that collectible data is included in the target page. For example, the target page is a page showing commodity information, and the collectable data of the target page includes data such as a commodity name, a commodity price, a commodity image, and the like.

It is to be understood that the data type of the collectable data included in the target page may be text data, image data, and the like, and the exemplary embodiment of the present application is not limited thereto.

And step S220, receiving selection operation executed by the user on the data which can be acquired in the target page to obtain a target acquisition data item.

The target acquisition data item refers to a data item which needs to be acquired by a user, and the target acquisition data item is obtained by monitoring the selection operation of the user in a target page.

Each piece of collectible data displayed in the target page corresponds to one target collection data item.

It can be understood that the selection operation may be a single click operation, a long press operation, a drag operation, and the like, and a specific execution manner of the selection operation may be flexibly selected according to an actual application situation, which is not limited in the present application.

For example, referring to fig. 3, fig. 3 is a schematic diagram of a target page in an exemplary embodiment of the present application, where the target page is a commodity list page and is displayed on a user terminal. As shown in fig. 3, the target page includes the commodity names, commodity images, and commodity prices of the commodities, and when the user terminal detects that the data "commodity a" in the target page is clicked, it indicates that the user performed a selection operation on the data "commodity a", so that the data item name corresponding to the data "commodity a" is obtained, that is, the target collection data item is the "commodity name".

Wherein the selected collectable data may be highlighted, for example, the font of the collectable data is changed in the target page to distinguish from the unselected data; or adding a parameter configuration frame to fig. 3, and displaying the selected collectable data in the parameter configuration frame after the collectable data is selected.

In practical application, only one selected data can be obtained through one selection operation, and when variable data exists in the step of extracting the selected data, the data acquisition is incomplete. For example, the commodity prices of all commodities in the commodity information display list need to be collected, so that the user performs a selection operation on the collectable data "10", and the data item name corresponding to the collectable data "10" is the "commodity price", that is, the target collection data item is the "commodity price". When the data acquisition server acquires data in a message returned by a target server, the step of extracting '10' comprises the following steps: acquiring information of a commodity information display list, acquiring information of a commodity A from the commodity information display list, and acquiring information of commodity price from the commodity A; it can be expressed as: "merchandise information display List- > merchandise A- > 10". Since the commodity names of the commodities in the commodity information display list are different, the step of acquiring the commodity a in the step of extracting the 10 is a variable, and if the commodity prices of all the commodities are acquired according to the step of extracting the 10, the data acquisition is incomplete, and if the price of the commodity B cannot be acquired.

Therefore, in some embodiments, please refer to fig. 4, fig. 4 is a schematic flowchart of step S220 in the exemplary embodiment of the present application, and as shown in fig. 4, step S220 includes step S221 to step S223.

Step S221, at least two selection operations executed by a user on the data which can be collected in the target page are received, and first selection data and second selection data are obtained.

The data which can be collected and selected by the two selection operations are different, so that the first selection data and the second selection data are obtained respectively. It can be understood that if the selection operation is performed three times, the first selection data, the second selection data and the third selection data are obtained respectively; if the selection operation is executed more times, more selection data are obtained, and the specific execution times of the selection operation is not limited in the application.

Step S222, respectively obtaining data item identifiers of the first selected data and the second selected data.

The data item identifier includes, but is not limited to, a data name, a data Identity identification number (ID) and the like of the collectable data corresponding to the selection operation, and the data item identifier may uniquely encode the data type of the collectable data. It can be understood that the data types of the collected data displayed on the target page may be different, the data content of the collected data of the same data type may be different, and the user performs a selection operation on the data content of the collected data to obtain the data item identifier according to the data type corresponding to the selected data content.

Step S223, if the data item identifiers of the first selected data and the second selected data are the same, taking the data item identifier as a target acquisition data item.

For example, the selecting operation corresponding to the first selecting data and the second selecting data may be a continuous selecting operation. For example, when it is detected that a user performs a first selection operation on the collectable data "commodity a", a data item identifier of the first selection data "commodity a" is obtained as a "commodity name"; if the fact that the user executes a second selection operation on the collectable data commodity B is detected again, the data item identification of the obtained second selection data commodity B is also the commodity name, the data item identifications of the first selection data and the second selection data are the same, and the data item identification commodity name is used as a target collection data item; if it is detected again that the user performs the second selection operation on the collectable data price B to obtain the data item identifier of the second selection data price B as the commodity price, the data item identifiers of the first selection data and the second selection data are different, the record of the collectable data commodity A can be cleared, and the price B is taken as the first selection data to continue to judge.

For example, the selecting operation corresponding to the first selecting data and the second selecting data may be a discontinuous selecting operation. For example, when it is detected that the user performs a selection operation on the collectable data "commodity a", the data item identifier of the first selection data "commodity a" is obtained as "commodity name"; if the fact that the user performs selection operation on the collectable data price B is detected again, the data item identification of the second selection data price B is 'commodity price', the data item identifications of the first selection data and the second selection data are different, the commodity A and the price B can be both used as the first selection data, and judgment is continued; until it is detected that the user performs a selection operation on the collectable data commodity B, the data item identifier of the obtained selection data commodity B is also the commodity name, and if it is detected that the recorded first selection data contains the first selection data commodity A with the data item identifier of the commodity name, the data item identifier of the commodity name is used as a target collection data item.

In some embodiments, a user may select multiple targeted acquisition data items in a targeted page to acquire the multiple targeted acquisition data items simultaneously.

Illustratively, the obtaining manner of the target data extracting step may include: if the data item identifications of the first selected data and the second selected data are the same, acquiring a first data extraction step of the first selected data and a second data extraction step of the second selected data; and a target data extraction step of obtaining a target acquisition data item based on the first data extraction step and the second data extraction step.

The first data extraction step and the second data extraction step respectively comprise a plurality of sub-steps, and the sub-steps in the first data extraction step and the second data extraction step are compared with each other to obtain a target data extraction step of a target acquisition data item.

Exemplarily, referring to fig. 5, fig. 5 is a schematic diagram of an extraction step of the acquired target data in an exemplary embodiment of the present application.

As shown in fig. 5, the first data extracting step of obtaining the first selected data includes:

a substep: acquiring information (Items) of a commodity information display list;

and a substep b: acquiring information (Com-A) of the commodity A from the commodity information display list;

and a substep c: information (Price) of Price a is acquired from article a.

The second data extraction step of obtaining the second selected data comprises:

and a substep d: acquiring information (Com-B) of a commodity B from a commodity information display list;

and a substep c: information (Price) of Price B is acquired from article B.

Further, a target data extraction step of generating a target collected data item based on the same sub-step included in the first data extraction step and the second data extraction step.

Therefore, when the sub-step b and the sub-step d are different from each other by comparing the first data extraction step and the second data extraction step, the different sub-steps are used as variable data, and the target data extraction step includes "sub-step a- > sub-step c", where "sub-step" means an arbitrary step by which the commodity prices of all commodities in the commodity information display list can be extracted.

At least two selection operations are executed in the target page to obtain at least two selection data with the same data item identification, so that the steps of extracting the two selection data can be compared, and the variables existing in the step of extracting the two selection data can be eliminated to obtain the target data extraction step of the target acquisition data item. The user can automatically generate the target data extraction step of the target acquisition data item by only performing the data selection operation twice, the user experience is improved on the premise of ensuring the integrity and correctness of data acquisition, the data acquisition efficiency is improved, the target data extraction step of the target acquisition data item does not need to be manually input, the data acquisition process is simplified, and the universality is improved.

Step S230, acquiring a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein there are input parameters depending on the page.

The input parameters required by the target acquisition data item refer to parameters which must be carried in a request message when the user terminal displays the target acquisition data item in a request target page. For example, the target page is a commodity detail page, and when the user terminal requests that the target acquisition data items displayed in the commodity detail page are all comments of the commodity a, the identification of the commodity a needs to be carried in the request message corresponding to the display target acquisition data item.

It can be understood that, if the target collection data item is not on one page, the page jump is required in the data collection process, and a plurality of request messages need to be sent to the target server to obtain different message data.

For example, as shown in fig. 6, the target page is a commodity detail page, the target collection data item is a commodity sales amount in the commodity detail page, and the commodity sales amount is only displayed in the commodity detail page. When a user terminal obtains the commodity sales volume of a commodity from a target server, a skip link of a commodity detail page of the corresponding commodity needs to be clicked on a commodity list page to obtain a commodity identifier of the commodity in the commodity list page, and then a request message containing the commodity identifier of the commodity is sent to the target server by clicking the skip link, so that the target server feeds back information of the commodity detail page of the commodity to the user terminal according to the commodity identifier of the commodity. The skip link may be a commodity name, a commodity picture, or the like in fig. 6, which is not limited in this application.

Further, if the objective of data acquisition is to acquire the sales volume of a plurality of commodities, a request message of a commodity detail page corresponding to the commodities needs to be sent to the target server, and the request message needs to carry a commodity identifier corresponding to the currently viewed commodity, such as a commodity name, a commodity number, and the like. Therefore, the page data of the commodity list page shown in fig. 6 needs to obtain commodity identifiers corresponding to different commodities to generate request messages corresponding to different commodities.

In the related art, when a target acquisition data item is not on one page, a user needs to manually input the relationship between the pages to obtain a page jump logic for acquiring the target acquisition data item in the data acquisition process, so that the generation of a data acquisition task still needs to be manually judged, and the efficiency and the universality are low.

Therefore, the method and the device for acquiring the page jump logic of the target acquisition data item acquire the dependent page required by acquiring the target acquisition data item through the input parameters required by the target acquisition data item in the target page, and then the user can only execute the selection operation of the target acquisition data item, so that the accurate page jump logic for acquiring the target acquisition data item can be acquired. The dependent page refers to a page containing input parameters needed by the target acquisition data item; for example, the input parameter required by the target acquisition data item "sales volume of a product" in the product detail page of fig. 6 is a product identifier, and the product identifier can be acquired in the product list page of fig. 6, so the product list page of fig. 6 is a dependent page of the product detail page.

In some embodiments, acquiring a dependency page corresponding to a target acquisition data item in a target page based on an input parameter required by the target acquisition data item in the target page includes: acquiring page path information associated with a target page based on input parameters required by a target acquisition data item in the target page, and taking all pages contained in the page path information as candidate pages; receiving parameter configuration operation executed by a user on a target acquisition data item in a target page to obtain a target data acquisition range; and screening the candidate pages based on the target data acquisition range to obtain the dependent pages corresponding to the target acquisition data items in the target pages.

The page path information refers to information of a jump relationship between pages, and if a first page contains a jump link of a second page, a page path for the first page to reach the second page exists.

The page path information associated with the target page may be page path information when the target page reaches other pages, or may be page path information when other pages reach the target page. It should be noted that, in the present application, the page path information associated with the target page refers to page path information of other pages reaching the target page.

In the exemplary embodiment, the page path information associated with the target page is obtained according to the input parameters required by the target acquisition data item to obtain the candidate page, and then the target data acquisition range input by the user is obtained to obtain the dependent page corresponding to the target acquisition data item.

For example, as shown in fig. 7, the page includes an item classification page, an item list page, and an item detail page; the commodity classification page is used for displaying commodity classification information, such as food commodities, life commodities, electronic commodities and the like; the commodity list page is used for displaying commodity abstract information of a certain type of commodities, such as commodity pictures, commodity names and the like; the commodity detail page is used for displaying commodity detail information of a certain commodity, such as the price of the commodity, the introduction of the commodity, the comment of the commodity and the like. And when the fact that the commodity comment data in the commodity detail page are selected is detected, the commodity comment is used as a target acquisition data item, and the commodity detail page is used as a target page. Further, by analyzing a request message of a commodity detail page sent to the target server by the user terminal, the acquired input parameters required by the commodity comment include commodity classification information and a commodity name. And then obtaining page path information associated with the commodity detail page as a commodity list page according to the commodity name, obtaining a page path information commodity classification page associated with the commodity detail page according to the commodity classification information, and taking the commodity list page and the commodity classification page as candidate pages.

Then, determining a dependent page corresponding to the target acquisition data item from the candidate pages. Because the dependent pages of the target acquisition data items corresponding to different data acquisition ranges are different, it is necessary to receive the parameter configuration operation executed by the user on the target acquisition data items in the target page to obtain the target data acquisition range.

For example, after the candidate page is obtained, an operation interface of the parameter configuration operation may be displayed to the user, so as to obtain the parameter configuration operation performed by the user on the target acquisition data item in the target page. Referring to fig. 8, fig. 8 is an operation interface for performing parameter configuration operation on a target acquisition data item, which includes a schematic diagram of a target page, a schematic diagram of candidate pages, and a schematic diagram of a page path between each page, where the schematic diagram of each candidate page is correspondingly provided with a selection control. As shown in fig. 7 and 8, the candidate pages include a commodity list page and a commodity classification page, the target page is a commodity detail page, arrows between the pages point to page path information used for representing pages between the pages, and the target acquisition data item is a commodity price; if the selection control corresponding to the commodity list page in fig. 8 is triggered, it indicates that the target data acquisition range is the commodity price of all commodities in the commodity list page; if the selection control corresponding to the commodity classification page in fig. 8 is detected to be triggered, it indicates that the target data acquisition range is the commodity price of all commodities in all commodity classes and all commodities under all commodity classes in the commodity classification page.

Further, after the target data acquisition range is obtained, screening the candidate pages according to the target data acquisition range to obtain the dependent pages corresponding to the target acquisition data items. For example, when the target data acquisition range is the commodity prices of all commodities in the commodity list page, the dependence page corresponding to the commodity prices comprises the commodity list page; when the target data acquisition range is the commodity prices of all the commodity categories in the commodity classification page and all the commodities under all the commodity categories, the dependence page corresponding to the commodity prices comprises a commodity classification page and a commodity list page.

It can be understood that the operation interface for the parameter configuration operation may be a single display page, or may be displayed after being combined with other pages, for example, after being spliced with a target page.

It can be understood that the operation interface of the parameter configuration operation may also perform configuration of other parameters, such as acquisition quantity, acquisition time, acquisition speed, document format of acquired data output, and the like corresponding to the target acquisition data item, and may be flexibly selected according to the actual application situation, which is not limited in this application.

The page path information associated with the target acquisition data item is obtained by analyzing the target acquisition data item, and the dependent page corresponding to the target acquisition data item is obtained according to the target data acquisition range input by the user, so that the page jump logic for acquiring the target acquisition data item in the data acquisition process by manually inputting the data by the user is avoided, and the data acquisition efficiency and the data acquisition universality are improved.

And S240, generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task.

And obtaining page jump logic in the data acquisition process according to the dependent page, automatically generating a corresponding request message according to the page jump logic, and sending the request message to a target server to obtain message data to be acquired.

In some embodiments, the data collection task is generated according to a target data extraction step and a dependent page, and the target data extraction step is used for indicating how to extract the target collection data item from the message data to be collected after obtaining the message data to be collected fed back by the target server.

In some embodiments, generating a data collection task based on a dependency page includes: acquiring website information corresponding to the dependent page; acquiring data acquisition request information based on the website information corresponding to the dependent page; and generating a data acquisition task based on the data acquisition request information.

The data acquisition request information is used for acquiring corresponding page information from the target server, and it should be noted that one dependent page may correspond to a plurality of data acquisition request information, and parameters carried by the plurality of data acquisition request information may be different. For example, if the dependent page is a product list page, the data acquisition request information corresponding to the product list page may include data acquisition request information of a product a and data acquisition request information of a product B.

And further, generating a data acquisition task according to the data acquisition request information so as to acquire data according to the data acquisition task.

In some embodiments, the data collection task includes data collection request information and a target data extraction step; data acquisition is carried out according to a data acquisition task, and the data acquisition method comprises the following steps: acquiring a target page to be acquired based on the data acquisition request information; and acquiring target data in a target page to be acquired based on the target data extraction step, and storing the target data.

For example, the data acquisition request information is a request for a commodity detail page of the commodity a sent to the target server, the target server feeds back the commodity detail page of the commodity a to the data acquisition server according to the data acquisition request information, the data acquisition server acquires message data corresponding to the commodity detail page of the commodity a, then acquires target data in the message data according to the target data extraction step, and stores the target data.

The target data can be stored in the data acquisition server so as to be sent to the user terminal after a downloading request of the user terminal to the target data is received; the target data can also be directly stored in a storage position designated by a user in the user terminal; the target data can also be saved in a data cloud storage server. The storage mode and the storage position of the target data can be flexibly selected according to the actual situation, and the method is not limited by the application.

In some embodiments, data collection may be performed by the proxy server according to the data collection task by calling a preset proxy server, so as to replace an Internet Protocol (IP) address for accessing the target server by the proxy server.

In some embodiments, when data collection is performed according to a data collection task, there may be a case that authentication needs to be performed, and therefore, an operation record of a user when accessing a web page may be obtained, for example, information such as a password of the user is cached in a web browser, and an authentication operation is performed by obtaining the cached information such as the password of the user.

In some embodiments, multiple target acquisition data items may be acquired simultaneously to improve the efficiency of data acquisition.

In some embodiments, different presentation modes can be adopted for the acquired target data. The display mode includes but is not limited to the following one or more modes in combination: displaying the display target data in a list form; displaying the display target data in a histogram form; displaying the display target data in the form of a circular data analysis graph; displaying the display target data in the form of a data comparison analysis chart; and displaying the display target data in the form of a data trend analysis chart. For displaying the target data with the evolution process, the state of each time period can be shown in an animation mode, and the state can also be shown by adopting a function curve with time as a horizontal axis. Through view display, a user can intuitively acquire the data condition of the acquired target data according to the view so as to be convenient for analyzing the target data.

Referring to fig. 9, fig. 9 is a schematic flowchart of a data acquisition method according to an embodiment of the present disclosure. As shown in fig. 9, the method includes steps S910 to S950.

Step S910, the user terminal sends the target website input by the user to a data acquisition server;

step S920, the data acquisition server sends an acquisition request of a target page to the target server according to the target website;

step S930, the target server feeds back the target page to the data acquisition server according to the acquisition request of the target page;

step S940, the data acquisition server sends the target page to the user terminal for displaying;

step S950, the data acquisition server receives at least two selection operations executed by the user on the data which can be acquired by the target page in the user terminal to obtain a target acquisition data item;

step S960, acquiring a dependent page corresponding to the target acquisition data item in the target page based on the input parameters required by the target acquisition data item in the target page, and generating data acquisition request information according to the dependent page; and obtaining target data extraction step based on at least two selection operations; sending a data acquisition request to a target server according to the data acquisition request information;

step S970, the target server feeds back message data to the data acquisition server according to the data acquisition request;

and step S980, the data acquisition server extracts the message data according to the target data extraction step to obtain target data, and sends the target data to the user terminal.

The specific implementation process of step S910 to step S980 may refer to the technical solution of the foregoing embodiment, which is not described herein again.

The method comprises the steps of responding to a target website sent by a user, and displaying a target page corresponding to the target website; the target page comprises collectable data; receiving selection operation executed by a user on the data which can be acquired in the target page to obtain a target acquisition data item; acquiring a dependent page corresponding to a target acquisition data item in a target page based on input parameters required by the target acquisition data item in the target page; wherein, there are input parameters depending on the page; and generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task. The user can complete the automatic generation of the data acquisition task only by simple selection operation, so that the user can generate and define the data acquisition task without professional network data acquisition knowledge, and the data is acquired from the website through the data acquisition task, thereby improving the efficiency and the universality of data acquisition.

Referring to fig. 10, fig. 10 is a schematic block diagram of a data acquisition apparatus according to an embodiment of the present application, where the data acquisition apparatus may be configured in a server or a computer device for executing the foregoing data acquisition method.

As shown in fig. 10, the data acquisition apparatus includes: the system comprises a display module 1010, a target data acquisition module 1020, a dependent page acquisition module 1030 and a data acquisition module 1040.

The display module 1010 is configured to respond to a target website sent by a user and display a target page corresponding to the target website; the target page comprises collectable data;

the target data obtaining module 1020 is configured to receive a selection operation executed by a user on data collectable in a target page to obtain a target collection data item;

the dependent page acquiring module 1030 is configured to acquire a dependent page corresponding to a target acquisition data item in a target page based on an input parameter required by the target acquisition data item in the target page; wherein, there are input parameters depending on the page;

the data collection module 1040 is configured to generate a data collection task based on the dependency page, and collect data according to the data collection task.

In some embodiments, based on the foregoing scheme, receiving a selection operation performed by a user on collectable data in a target page to obtain a target collection data item, including: receiving at least two selection operations executed by a user on the data which can be collected in the target page to obtain first selection data and second selection data; respectively acquiring data item identifications of first selected data and second selected data; and if the data item identifications of the first selected data and the second selected data are the same, taking the data item identifications as target acquisition data items.

In some embodiments, based on the foregoing solution, the data collection task is generated based on a dependent page, including: if the data item identifications of the first selected data and the second selected data are the same, a first data extraction step of acquiring the first selected data and a second data extraction step of acquiring the second selected data; a target data extraction step of obtaining a target acquisition data item based on the first data extraction step and the second data extraction step; and generating a data acquisition task based on the target data extraction step and the dependent page.

In some embodiments, based on the foregoing scheme, the target data extraction step of obtaining the target acquisition data item based on the first data extraction step and the second data extraction step includes: and a target data extraction step of generating a target collected data item based on the same substeps included in the first data extraction step and the second data extraction step.

In some embodiments, based on the foregoing scheme, obtaining a dependent page corresponding to a target acquisition data item in a target page based on an input parameter required by the target acquisition data item in the target page includes: acquiring page path information associated with a target page based on input parameters required by a target acquisition data item in the target page, and taking all pages contained in the page path information as candidate pages; receiving parameter configuration operation executed by a user on a target acquisition data item in a target page to obtain a target data acquisition range; and screening the candidate pages based on the target data acquisition range to obtain the dependent pages corresponding to the target acquisition data items in the target pages.

In some embodiments, based on the foregoing scheme, generating the data collection task based on the dependent page includes: acquiring website information corresponding to the dependent page; acquiring data acquisition request information based on the website information corresponding to the dependent page; and generating a data acquisition task based on the data acquisition request information.

In some embodiments, based on the foregoing scheme, the data collection task includes data collection request information and a target data extraction step; data acquisition is carried out according to a data acquisition task, and the data acquisition method comprises the following steps: acquiring a target page to be acquired based on the data acquisition request information; and acquiring target data in a target page to be acquired based on the target data extraction step, and storing the target data.

It should be noted that, as will be clear to those skilled in the art, for convenience and brevity of description, the specific working processes of the apparatus, the modules and the units described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The methods, apparatus, and devices of the present application are operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

FIG. 11 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

It should be noted that the computer system 1100 of the electronic device shown in fig. 11 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 11, electronic device 1100 is embodied in the form of a general purpose computing device. The components of the electronic device 1100 may include, but are not limited to: the at least one processing unit 1110, the at least one memory unit 1120, a bus 1130 connecting different system components (including the memory unit 1120 and the processing unit 1110), and a display unit 1140.

Where the memory unit stores program code, the program code may be executed by the processing unit 1110 to cause the processing unit 1110 to perform the steps according to various exemplary embodiments of the present disclosure as described in the above-mentioned "exemplary methods" section of this specification.

The storage unit 1120 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)1121 and/or a cache memory unit 1122, and may further include a read-only memory unit (ROM) 1123.

The storage unit 1120 may also include a program/utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 1130 may be representative of one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 1100 may also communicate with one or more external devices 1170 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1100, and/or any devices (e.g., router, modem, etc.) that enable the electronic device 1100 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 1150. Also, the electronic device 1100 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 1160. As shown, the network adapter 1160 communicates with the other modules of the electronic device 1100 over the bus 1130. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. Which when executed by the processing unit 1110 performs various functions defined in the system of the present application.

It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

Another aspect of the present application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the foregoing data processing method based on the ESOP system. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being incorporated in the electronic device.

Another aspect of the application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method based on the ESOP system provided in the above embodiments.

The above description is only a preferred exemplary embodiment of the present application, and is not intended to limit the embodiments of the present application, and those skilled in the art can easily make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data acquisition, the method comprising:

responding to a target website sent by a user, and displaying a target page corresponding to the target website; wherein the target page comprises collectable data;

receiving selection operation executed by the user on the data which can be acquired in the target page to obtain a target acquisition data item;

acquiring a dependent page corresponding to a target acquisition data item in the target page based on input parameters required by the target acquisition data item in the target page; wherein the input parameters exist in the dependent page;

and generating a data acquisition task based on the dependence page, and acquiring data according to the data acquisition task.

2. The method according to claim 1, wherein the receiving a selection operation performed by the user on the collectible data in the target page to obtain a target collection data item comprises:

receiving at least two selection operations executed by the user on the data which can be collected in the target page to obtain first selection data and second selection data;

respectively acquiring data item identifiers of the first selected data and the second selected data;

and if the data item identifications of the first selected data and the second selected data are the same, taking the data item identifications as target acquisition data items.

3. The method of claim 2, wherein generating the data collection task based on the dependency page comprises:

if the data item identifications of the first selected data and the second selected data are the same, acquiring a first data extraction step of the first selected data and a second data extraction step of the second selected data;

a target data extraction step of obtaining the target acquisition data item based on the first data extraction step and the second data extraction step;

and generating the data acquisition task based on the target data extraction step and the dependent page.

4. The method of claim 3, wherein the step of extracting target data based on the first and second data extraction steps to obtain the target acquisition data item comprises:

a target data extraction step of generating the target collected data item based on the same sub-step included in the first data extraction step and the second data extraction step.

5. The method according to claim 1, wherein the obtaining a dependent page corresponding to a target acquisition data item in the target page based on an input parameter required by the target acquisition data item in the target page comprises:

acquiring page path information associated with the target page based on input parameters required by a target acquisition data item in the target page, and taking all pages contained in the page path information as candidate pages;

receiving parameter configuration operation executed by the user on a target acquisition data item in the target page to obtain a target data acquisition range;

and screening the candidate pages based on the target data acquisition range to obtain a dependent page corresponding to the target acquisition data item in the target page.

6. The method of claim 1, wherein generating a data collection task based on the dependency page comprises:

acquiring website information corresponding to the dependent page;

acquiring data acquisition request information based on the website information corresponding to the dependent page;

and generating a data acquisition task based on the data acquisition request information.

7. The method according to any one of claims 1 to 6, wherein the data collection task includes data collection request information and a target data extraction step; the data acquisition according to the data acquisition task comprises the following steps:

acquiring a target page to be acquired based on the data acquisition request information;

and acquiring target data in the target page to be acquired based on the target data extraction step, and storing the target data.

8. A data acquisition device, the device comprising:

the display module is configured to respond to a target website sent by a user and display a target page corresponding to the target website; wherein the target page comprises collectable data;

the target data acquisition module is configured to receive selection operation executed by the user on the data which can be acquired in the target page to obtain a target acquisition data item;

the dependent page acquisition module is configured to acquire a dependent page corresponding to a target acquisition data item in the target page based on input parameters required by the target acquisition data item in the target page; wherein the input parameters exist in the dependent page;

and the data acquisition module is configured to generate a data acquisition task based on the dependency page and acquire data according to the data acquisition task.

9. A computer device, wherein the computer device comprises a memory and a processor;

the memory for storing a computer program;

the processor for executing the computer program and implementing the data acquisition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to carry out a data acquisition method according to any one of claims 1 to 7.