CN107644021A - Information collecting method and information collecting device - Google Patents

Abstract

The present invention proposes an information collection method and an information collection device, wherein the information collection method includes: after logging into the web version of the application, obtaining the cookie of the application and the data published on the application using the target account list page link; obtain the list page according to the cookie and the list page link, and obtain at least one content page link in the list page; download the corresponding content page link of each content page link in the at least one content page link content page. Through the technical scheme of the present invention, it is possible to simulate human behavior to collect valuable data from massive data in applications, thereby improving the efficiency of information collection.

Description

Information collection method and information collection device

technical field

The present invention relates to the technical field of information processing, in particular to an information collection method and an information collection device.

Background technique

At present, WeChat has more than 10 million public accounts, with hundreds of millions of articles, and it is growing at a rate of millions every day, and the data value of articles published by public accounts is relatively high, so the collection of WeChat official account articles has become a An essential part of massive data collection.

Collecting articles from WeChat official accounts refers to obtaining articles posted by official accounts in real time. Compared with other collections, WeChat is associated with terminal devices such as mobile phones and tablets. Its collection method is unique, requiring simulated human behavior, and will be subject to strict bans.

Therefore, how to simulate human behavior to collect valuable data from the massive data of WeChat, so as to improve the efficiency of information collection has become an urgent problem to be solved.

Contents of the invention

Based on the above-mentioned problems, the present invention proposes a new technical solution, which can simulate human behavior to collect valuable data from massive data in applications, thereby improving the efficiency of information collection.

In view of this, the first aspect of the present invention proposes an information collection method, including: after logging into the web version of the application, obtaining the cookie of the application and the list page of the data published on the application using the target account linking; obtaining a list page according to the cookie and the list page link, and obtaining at least one content page link in the list page; downloading a content page corresponding to each content page link in the at least one content page link.

In this technical solution, after logging in to the web version of the application, for example, after using the selenium webdriver tool to log in to the web version of the application, by obtaining the cookie of the application and the list page link of the data published on the application using the target account, to Obtain the list page, then obtain at least one content page link in the list page, and finally obtain the data published by the target account on the application according to at least one content page link, thus realizing the simulation of human behavior in the massive data of the application Collect valuable data, thereby improving the efficiency of information collection.

For example, after using selenium webdriver to log in to the web version of WeChat, obtain the WeChat cookie and the list page link of the article published by the target public account for "Beijing", and obtain the list page according to the cookie and list page link, and there is an article title in the list page It is "Beijing 5-Day Travel Guide", "Top 10 Must-Visit Tourist Attractions in Beijing", and "Beijing Food Collection". Analyze the list page to obtain the content page links in the list page, that is, access to the content of articles such as "Beijing 5-day travel guide", "10 must-visit tourist attractions in Beijing", and "Beijing food collection" Finally, according to the links of the content of these articles, the content of these articles can be obtained.

In the above technical solution, preferably, the step of obtaining the list page according to the cookie and the link to the list page further includes: if the list page is not obtained according to the cookie and the link to the list page, then Obtaining other cookies of the application again, so as to obtain the list page according to the other cookies and the list page link.

In this technical solution, since the applied Cookie has a certain timeliness, if the list page cannot be obtained according to the Cookie and the link to the list page, it means that the Cookie is invalid, and other Cookies are obtained again, so as to obtain to the list page.

In any of the above technical solutions, preferably, further comprising: parsing the content page to obtain the content in the content page, and converting the content in the content page into data in a target format.

In this technical solution, by extracting each content in the content page, and converting each content in the content page into data in a unified target format for storage, for example, converting it into plain text data in TXT or WORD format, thereby It is convenient to carry out unified management on the content of the downloaded content page.

In any of the above technical solutions, preferably, further comprising: after logging into the web version of the application, periodically refreshing the web page of the application.

In this technical solution, because the application is not operated for a long time, the application is offline or logged out, the network of the application is refreshed periodically to ensure that the web version of the application is in an online state, and to avoid the application being disconnected Then log back into the app.

In any of the above technical solutions, preferably, the list page and the content page are data in JSON format.

In this technical solution, the obtained list page and content page are data in JSON format, so that at least one content page link in the list page and the content page link in the content page can be obtained by parsing the list page and content page according to the JSON library. content.

The second aspect of the present invention proposes an information collection device, including: a first acquisition unit, configured to acquire the cookie of the application and the data published on the application using the target account after logging in to the web version of the application the list page link; the second obtaining unit is used to obtain the list page according to the cookie and the list page link, and obtain at least one content page link in the list page; the download unit is used to download the at least one A content page corresponding to each content page link in the content page links.

In this technical solution, after logging in to the web version of the application, for example, after using the selenium webdriver tool to log in to the web version of the application, by obtaining the cookie of the application and the list page link of the data published on the application using the target account, to Obtain the list page, then obtain at least one content page link in the list page, and finally obtain the data published by the target account on the application according to at least one content page link, thus realizing the simulation of human behavior in the massive data of the application Collect valuable data, thereby improving the efficiency of information collection.

For example, after using selenium webdriver to log in to the web version of WeChat, obtain the WeChat cookie and the list page link of the article published by the target public account for "Beijing", and obtain the list page according to the cookie and list page link, and there is an article title in the list page It is "Beijing 5-Day Travel Guide", "Top 10 Must-Visit Tourist Attractions in Beijing", and "Beijing Food Collection". Analyze the list page to obtain the content page links in the list page, that is, access to the content of articles such as "Beijing 5-day travel guide", "10 must-visit tourist attractions in Beijing", and "Beijing food collection" Finally, according to the links of the content of these articles, the content of these articles can be obtained.

In the above technical solution, preferably, it also includes: a third obtaining unit, if the list page is not obtained according to the cookie and the link to the list page, re-acquire other cookies of the application, so as to obtain the other cookies of the application according to the Other cookies and said list page link fetch said list page.

In this technical solution, since the applied Cookie has a certain timeliness, if the list page cannot be obtained according to the Cookie and the link to the list page, it means that the Cookie is invalid, and other Cookies are obtained again, so as to obtain to the list page.

In any of the above technical solutions, preferably, further comprising: a conversion unit configured to parse the content page to obtain the content in the content page, and convert the content in the content page into a target format data.

In this technical solution, by extracting each content in the content page, and converting each content in the content page into data in a unified target format for storage, for example, converting it into plain text data in TXT or WORD format, thereby It is convenient to carry out unified management on the content of the downloaded content page.

In any of the above technical solutions, preferably, further comprising: a refreshing unit, configured to periodically refresh the web page of the application after logging in the web version of the application.

In this technical solution, because the application is not operated for a long time, the application is offline or logged out, the network of the application is refreshed periodically to ensure that the web version of the application is in an online state, and to avoid the application being disconnected Then log back into the app.

In any of the above technical solutions, preferably, the list page and the content page are data in JSON format.

In this technical solution, the obtained list page and content page are data in JSON format, so that at least one content page link in the list page and the content page link in the content page can be obtained by parsing the list page and content page according to the JSON library. content.

Through the technical solution of the present invention, it is possible to simulate human behavior to collect valuable data from massive data in applications, thereby improving the efficiency of information collection.

Description of drawings

FIG. 1 shows a schematic flow diagram of an information collection method according to an embodiment of the present invention;

FIG. 2 shows a schematic flow chart of an information collection method according to another embodiment of the present invention;

FIG. 3 shows a schematic structural diagram of an information collection device according to an embodiment of the present invention;

Fig. 4 shows a schematic structural diagram of an information collection device according to another embodiment of the present invention.

detailed description

In order to have a clearer understanding of the above objects, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described here. Therefore, the protection scope of the present invention is not limited by the specific details disclosed below. EXAMPLE LIMITATIONS.

Fig. 1 shows a schematic flowchart of an information collection method according to an embodiment of the present invention.

As shown in Figure 1, the information collection method according to one embodiment of the present invention includes:

Step 102, after logging into the web version of the application, obtain the cookie of the application and the list page link of the data published on the application using the target account.

Step 104, obtain a list page according to the cookie and the list page link, and obtain at least one content page link in the list page.

In step 104, since the applied Cookie has a certain timeliness, if the list page cannot be obtained according to the Cookie and the link to the list page, it means that the Cookie is invalid. If the list page is not acquired, other cookies of the application are acquired again, so as to acquire the list page according to the other cookies and the list page link.

Step 106, download the content page corresponding to each content page link in the at least one content page link.

After step 106, the content page is analyzed to obtain the content in the content page, and the content in the content page is converted into data in the target format, thereby converting each content in the content page into a unified format, for example, converted to plain text data in TXT or WORD format, so as to facilitate the unified management of the content of the downloaded content page.

In addition, the list page and the content page in steps 104 and 106 are data in JSON format, so that at least one content page link in the list page can be obtained by parsing the list page and the content page according to the JSON library. and the content in the content page.

In this technical solution, after logging in to the web version of the application (the application can be WeChat, Weibo, QQ), for example, after using the selenium webdriver tool to log in to the web version of the application, by obtaining the cookie of the application and using the target account in the The list page link of the data published on the application to obtain the list page, then obtain at least one content page link in the list page, and finally obtain the data published by the target account on the application according to at least one content page link, so as to realize In order to simulate human behavior and collect valuable data in the massive data of the application, the efficiency of information collection is improved.

For example, after using selenium webdriver to log in to the web version of WeChat, obtain the WeChat cookie and the list page link of the article published by the target public account for "Beijing", and obtain the list page according to the cookie and list page link, and there is an article title in the list page It is "Beijing 5-Day Travel Guide", "Top 10 Must-Visit Tourist Attractions in Beijing", and "Beijing Food Collection". Analyze the list page to obtain the content page links in the list page, that is, access to the content of articles such as "Beijing 5-day travel guide", "10 must-visit tourist attractions in Beijing", and "Beijing food collection" Finally, according to the links of the content of these articles, the content of these articles can be obtained.

In the above technical solution, preferably, further comprising: after logging into the web version of the application, periodically refreshing the web page of the application.

In this technical solution, because the application is not operated for a long time, the application is offline or logged out, the network of the application is refreshed periodically to ensure that the web version of the application is in an online state, and to avoid the application being disconnected Then log back into the app.

Fig. 2 shows a schematic flowchart of an information collection method according to another embodiment of the present invention.

As shown in Figure 2, the information collection method according to another embodiment of the present invention includes:

Step 202, Wechat_server sends Cookie request to Cookie_server.

In step 204, the Wechat_server obtains the WeChat cookie from the Cookie_server, and obtains a linked list page list of articles published by the target official account in WeChat.

Step 206, obtain the list page according to the cookie and the list page link list.

Step 208, judge whether the acquired Cookie is valid, if the judgment result is yes, go to step 210, otherwise, go to step 202 to re-acquire other cookies of WeChat. Wherein, if the list page is obtained according to the cookie and the list page link list, it is determined that the cookie is valid; if no list page or other content is obtained according to the cookie and the list page link list, then the cookie is determined to be invalid.

Step 210, parsing the list page to obtain multiple content page links in the list page.

Step 212, downloading content pages corresponding to multiple content page links.

Step 214, analyze the content page, and implement the information, specifically, analyze the content page to obtain the content in the content page, and convert the content in the content page into usable and formatted plain text data.

Fig. 3 shows a schematic structural diagram of an information collection device according to an embodiment of the present invention.

As shown in Figure 3, an information collection device 300 according to an embodiment of the present invention includes: a first acquisition unit 302, a second acquisition unit 304 and a download unit 306, wherein the first acquisition unit 302 is used to After the application of the application, obtain the cookie of the application and the list page link of the data published on the application using the target account; the second obtaining unit 304 is used to obtain the list page according to the cookie and the list page link, And acquire at least one content page link in the list page; the downloading unit 306 is configured to download a content page corresponding to each content page link in the at least one content page link.

In this technical solution, after logging in to the web version of the application, for example, after using the selenium webdriver tool to log in to the web version of the application, by obtaining the cookie of the application and the list page link of the data published on the application using the target account, to Obtain the list page, then obtain at least one content page link in the list page, and finally obtain the data published by the target account on the application according to at least one content page link, thus realizing the simulation of human behavior in the massive data of the application Collect valuable data, thereby improving the efficiency of information collection.

For example, after using selenium webdriver to log in to the web version of WeChat, obtain the WeChat cookie and the list page link of the article published by the target public account for "Beijing", and obtain the list page according to the cookie and list page link, and there is an article title in the list page It is "Beijing 5-Day Travel Guide", "Top 10 Must-Visit Tourist Attractions in Beijing", and "Beijing Food Collection". Analyze the list page to obtain the content page links in the list page, that is, access to the content of articles such as "Beijing 5-day travel guide", "10 must-visit tourist attractions in Beijing", and "Beijing food collection" Finally, according to the links of the content of these articles, the content of these articles can be obtained.

In the above technical solution, preferably, it further includes: a third obtaining unit 308, if the list page is not obtained according to the cookie and the link to the list page, re-acquire other cookies of the application to The other cookies mentioned above and the link to the list page are used to obtain the list page.

In this technical solution, since the applied Cookie has a certain timeliness, if the list page cannot be obtained according to the Cookie and the link to the list page, it means that the Cookie is invalid, and other Cookies are obtained again, so as to obtain to the list page.

In any of the above technical solutions, preferably, further comprising: a conversion unit 310, configured to parse the content page to obtain the content in the content page, and convert the content in the content page into a target format The data.

In this technical solution, by extracting each content in the content page, and converting each content in the content page into data in a unified target format for storage, for example, converting it into plain text data in TXT or WORD format, thereby It is convenient to carry out unified management on the content of the downloaded content page.

In any of the above technical solutions, preferably, further comprising: a refreshing unit 312, configured to periodically refresh the web page of the application after logging in the web version of the application.

In this technical solution, because the application is not operated for a long time, the application is offline or logged out, the network of the application is refreshed periodically to ensure that the web version of the application is in an online state, and to avoid the application being disconnected Then log back into the app.

In any of the above technical solutions, preferably, the list page and the content page are data in JSON format.

In this technical solution, the obtained list page and content page are data in JSON format, so that at least one content page link in the list page and the content page link in the content page can be obtained by parsing the list page and content page according to the JSON library. content.

Fig. 4 shows a schematic structural diagram of an information collection device according to another embodiment of the present invention.

As shown in FIG. 4 , an information collection device 400 according to another embodiment of the present invention includes: Cookie_server402 and Wechat_server404 . Cookie_server402 accesses the address https://wx.qq.com/ through the selenium webdriver tool, simulates the login of the web version of WeChat, obtains the Cookie of WeChat, and provides Cookie service for Wechat_server404. If Wechat_server404 requests Cookie from Cookie_server402, then Wechat_server404 can obtain the Cookie of WeChat from Cookie_server402. Then Wechat_server404 obtains the list page link of the article published by the target account on WeChat, obtains the list page according to the obtained Cookie and the list page link, parses the content page link in the list page, and then downloads the content page corresponding to the content page link, and converts the content page The content in is converted into usable, formatted plain text data.

The technical solution of the present invention has been described in detail above in conjunction with the accompanying drawings. Through the technical solution of the present invention, human behavior can be simulated to collect valuable data from massive data in applications, thereby improving the efficiency of information collection.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. An information collection method, characterized in that, comprising: After logging into the web version of the application, obtain the cookie of the application and the list page link of the data published on the application using the target account; Obtain a list page according to the Cookie and the list page link, and obtain at least one content page link in the list page; A content page corresponding to each content page link in the at least one content page link is downloaded. 2. The information collection method according to claim 1, wherein the step of obtaining the list page according to the cookie and the list page link further comprises: If the list page is not obtained according to the cookie and the link to the list page, reacquire other cookies of the application, so as to obtain the list page according to the other cookie and the link to the list page. 3. The information collection method according to claim 1, further comprising: The content page is parsed to obtain the content in the content page, and the content in the content page is converted into data in a target format. 4. The information collection method according to any one of claims 1 to 3, further comprising: After logging into the web version of the application, the web page of the application is periodically refreshed. 5. The information collection method according to any one of claims 1 to 3, wherein the list page and the content page are data in JSON format. 6. An information collection device, characterized in that it comprises: The first obtaining unit is used to obtain the cookie of the application and the list page link of the data published on the application using the target account after logging in the application of the web version; A second obtaining unit, configured to obtain a list page according to the cookie and the list page link, and obtain at least one content page link in the list page; A downloading unit, configured to download a content page corresponding to each content page link in the at least one content page link. 7. The information collection device according to claim 6, further comprising: The third obtaining unit, if the list page is not obtained according to the cookie and the link to the list page, re-acquire other cookies of the application, so as to obtain the list according to the other cookies and the link to the list page Page. 8. The information collection device according to claim 6, further comprising: The conversion unit is configured to parse the content page to obtain the content in the content page, and convert the content in the content page into data in a target format. 9. The information collection device according to any one of claims 6 to 8, further comprising: A refreshing unit, configured to periodically refresh the webpage of the application after logging into the webpage version of the application. 10. The information collection device according to any one of claims 6 to 8, wherein the list page and the content page are data in JSON format.