CN112732996A - Multi-platform distributed data crawling method based on asynchronous aiohttp - Google Patents
Multi-platform distributed data crawling method based on asynchronous aiohttp Download PDFInfo
- Publication number
- CN112732996A CN112732996A CN202110029155.8A CN202110029155A CN112732996A CN 112732996 A CN112732996 A CN 112732996A CN 202110029155 A CN202110029155 A CN 202110029155A CN 112732996 A CN112732996 A CN 112732996A
- Authority
- CN
- China
- Prior art keywords
- aiohttp
- asynchronous
- client
- platform
- data crawling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
Abstract
The invention relates to the technical field of data crawling, in particular to a multi-platform distributed data crawling method based on asynchronous aiohttp, which comprises the following steps: step one, a service center distributes url tasks to a client; secondly, the client reads configuration from the configuration center; step three, a plurality of clients download contents to a plurality of url tasks; step four, analyzing, cleaning and storing the downloaded content into a database; collecting logs by using a log center; and step six, opening the monitoring center to check the resource condition and the result. According to the invention, through a task multi-platform distribution and utilization kafka publish-subscribe message transfer mode, only a subscriber who subscribes to topic receives a message, so that the multi-platform can be distinguished, and the task deduplication principle utilizes the set characteristic of redis to carry out the deduplication function; according to the invention, the request is sent by utilizing async asynchronous characteristic, the network io cannot cause blockage, high concurrency and high availability are realized, and the data crawling speed can be effectively improved by the mode.
Description
Technical Field
The invention relates to the technical field of data crawling, in particular to a multi-platform distributed data crawling method based on asynchronous aiohttp.
Background
The industries such as finance, communication, catering, e-commerce, real estate and the like exist for a long time, data generated by the application of the industries is explosively increased, and people are more and more aware of the importance of the data to enterprises. If we have a plurality of platform data to wait to crawl, only use a thread and adopt serial mode to carry out, then can only wait to climb and just can continue next after one, efficiency can be very low, can not in time provide the data source, can produce speed problem, quality problems, if carry out many machines and crawl, maintenance difficulty, managerial inefficiency can appear in the back. The service requirements cannot be met. In view of this, we propose a multi-platform distributed data crawling method based on asynchronous aiohttp.
Disclosure of Invention
The invention aims to provide a multi-platform distributed data crawling method based on asynchronous aiohttp, so as to solve the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme:
a multi-platform distributed data crawling method based on asynchronous aiohttp comprises the following steps:
step one, a service center distributes url tasks to a client;
secondly, the client reads configuration from the configuration center;
step three, a plurality of clients download contents to a plurality of url tasks;
step four, analyzing, cleaning and storing the downloaded content into a database;
collecting logs by using a log center;
and step six, opening the monitoring center to check the resource condition and the result.
As a preferred technical scheme of the invention, the url task distribution in the step one is specifically operated as follows: creating a message theme, dividing a plurality of themes for a plurality of platforms, performing task multi-platform distribution, performing duplicate removal on a single platform url task, and sending url tasks on different themes.
As a preferred technical solution of the present invention, the read configuration function in the step two includes: client agent, failure retry, custom request information, synchronous, asynchronous operation mode selection, timeout control, request time, task white list, client middleware, database type selection, performance setting, request type.
As the preferred technical scheme of the invention, the specific operations of downloading the content in the step three are as follows: and sending a request to the url task by using the aiohttp library, and receiving returned data.
As a preferred technical scheme of the invention, the data cleaning in the step four comprises the following specific operations: analyzing the returned data, analyzing the contents of different types by using different libraries, analyzing the data in the json format by using the json library, analyzing the html format by using the xpath library, extracting other texts by using the re regular library, storing the pictures and video byte streams by using a binary system, and storing the cleaned data in the database.
As a preferred technical solution of the present invention, the log collection in the step five specifically operates as follows: logging modules are utilized to record logs in different modes for different log levels of the client, such as five levels of DEBUG, INFO, WARNING, ERROR and CRITICAL, log information is recorded by using files, HTTP GET/POST, SMTP and Socket modes are adopted for recording, and the file format is log files generally.
As a preferred technical solution of the present invention, the monitoring in the step six includes: the usability of the client resource can record the server problem and inform the server problem when the system is shut down, the server resource trend and the system activity are analyzed, the data amount condition of data crawling and warehousing is carried out, the log recording condition of the client is recorded, a WEB interface is provided for setting the client, and the result is checked.
As a preferred technical scheme of the invention, the task multi-platform distribution principle is that a kafka publishing and subscribing message delivery mode is utilized, a publisher sends a message to topic, and only a subscriber who subscribes to topic receives the message, so that multiple platforms can be distinguished, and the task deduplication principle utilizes the set characteristic of redis self-provided with the deduplication function.
As the preferred technical scheme of the invention, the aiohttp principle is to utilize async asynchronous characteristic to send the request, and the network io can not cause blockage, thereby realizing high concurrency and high availability.
As a preferred technical scheme of the invention, the principle of the configuration center is that one service is started as a service side, then each service needing to be configured is used as a client side to obtain configuration by the service side, the configuration of tens of thousands of client sides is unified, the platform is unified, the client sides are highly available, the maintenance cost is low, and the aiohttp, xpath, json, logging, kafka and redis belong to an open source library.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, through a task multi-platform distribution and subscription message issuing and subscribing mode by utilizing kafka, only a subscriber subscribing to topic receives a message, so that the multi-platform can be distinguished, and the task deduplication principle utilizes the set characteristic of redis self-provided with the deduplication function; the invention utilizes async asynchronous characteristic to send the request, the network io can not cause blockage, and high concurrency and high availability are realized; according to the method, one service is started to serve as a service side, and then each service needing to be configured is taken as a client side to obtain configuration by the service side, so that configuration of tens of thousands of client sides is unified, a platform is unified, the client sides are highly available, the maintenance cost is low, and the data crawling speed can be effectively increased through the method.
Drawings
FIG. 1 is a main flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
FIG. 2 is a url task distribution flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
FIG. 3 is a flow chart of a configuration center of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
FIG. 4 is a flow chart of url task downloading based on the asynchronous aiohttp multi-platform distributed data crawling method of the present invention;
FIG. 5 is a data cleaning flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
FIG. 6 is a flow chart of log collection in the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
FIG. 7 is a flowchart of a monitoring center of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;
fig. 8 is a data comparison diagram of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the following embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate orientations and positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the equipment or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be considered as limiting the present invention.
Examples
The technical scheme provided by the embodiment is as follows:
a multi-platform distributed data crawling method based on asynchronous aiohttp, as shown in fig. 1, includes the following steps:
step one, a service center distributes url tasks to a client;
secondly, the client reads configuration from the configuration center;
step three, a plurality of clients download contents to a plurality of url tasks;
step four, analyzing, cleaning and storing the downloaded content into a database;
collecting logs by using a log center;
and step six, opening the monitoring center to check the resource condition and the result.
As a preferred technical solution of this embodiment, as shown in fig. 2, url task distribution in step one is specifically performed: creating a message theme, dividing a plurality of themes for a plurality of platforms, performing task multi-platform distribution, performing duplicate removal on a single platform url task, and sending url tasks on different themes.
As a preferred technical solution of this embodiment, the read configuration function in step two includes: client agent, failure retry, custom request information, synchronous, asynchronous operation mode selection, timeout control, request time, task white list, client middleware, database type selection, performance setting, request type.
As a preferred technical solution of this embodiment, as shown in fig. 4, the specific operations of downloading content in step three are: and sending a request to the url task by using the aiohttp library, and receiving returned data.
As a preferred technical solution of this embodiment, as shown in fig. 5, the data cleansing in step four specifically operates: analyzing the returned data, analyzing the contents of different types by using different libraries, analyzing the data in the json format by using the json library, analyzing the html format by using the xpath library, extracting other texts by using the re regular library, storing the pictures and video byte streams by using a binary system, and storing the cleaned data in the database.
As a preferred technical solution of this embodiment, as shown in fig. 6, the log collection operation in step five is specifically: logging modules are utilized to record logs in different modes for different log levels of the client, such as five levels of DEBUG, INFO, WARNING, ERROR and CRITICAL, log information is recorded by using files, HTTP GET/POST, SMTP and Socket modes are adopted for recording, and the file format is log files generally.
As a preferred technical solution of this embodiment, as shown in fig. 7, the monitoring in step six includes: the usability of the client resource can record the server problem and inform the server problem when the system is shut down, the server resource trend and the system activity are analyzed, the data amount condition of data crawling and warehousing is carried out, the log recording condition of the client is recorded, a WEB interface is provided for setting the client, and the result is checked.
As a preferred technical solution of this embodiment, the task multi-platform distribution principle is that a kafka publish-subscribe message delivery mode is used, a publisher sends a message to topic, and only a subscriber who subscribes to topic receives the message, so that multiple platforms can be distinguished, and the task deduplication principle uses a set characteristic of redis to carry a deduplication function.
As a preferred technical solution of this embodiment, the aiohttp principle is to send a request by using async asynchronous characteristics, and the network io does not cause congestion, thereby achieving high concurrency and high availability.
As a preferred technical solution of this embodiment, as shown in fig. 3, a principle of the configuration center is to start a service as a service party, and then each service that needs to be configured is obtained and configured as a client by the service party, so that configuration of ten thousand clients is uniformly configured, a platform is uniform, the client is highly available, and maintenance cost is low.
As a preferred technical solution of this embodiment, aiohttp, xpath, json, logging, kafka, and redis all belong to an open source library.
The implementation environment of the asynchronous aiohttp-based multi-platform distributed data crawling method is as follows: the system comprises the following steps: windows10 system, CPU: i7-8700, memory: 24g, process: 1 process, 1 thread, 500 coroutines request at the same time.
Comparative example 1
A data crawling method based on single-platform synchronous operation comprises the following steps:
step one, a service center distributes url tasks to a client;
secondly, the client reads configuration from the configuration center;
step three, a single client downloads content to a single url task;
step four, analyzing, cleaning and storing the downloaded content into a database;
collecting logs by using a log center;
and step six, opening the monitoring center to check the resource condition and the result.
Comparative example 2
A data crawling method based on asynchronous aiohttp single platform operation comprises the following steps:
step one, a service center distributes url tasks to a client;
secondly, the client reads configuration from the configuration center;
step three, a single client downloads content to a single url task;
step four, analyzing, cleaning and storing the downloaded content into a database;
collecting logs by using a log center;
and step six, opening the monitoring center to check the resource condition and the result.
The examples and comparative example 1 differ in that: comparative example 1 does not employ the aiohttp principle, and the number of url tasks and the number of processing clients of comparative example 1 are different from those of the embodiment, and the rest are the same.
The difference between the example and the comparative example 2 is that: the number of url tasks and the number of processing clients of comparative example 2 are different from those of the example, and the rest are the same.
Experimental comparison is performed on the asynchronous aiohttp-based multi-platform distributed data crawling method provided by the invention, the traditional single-platform synchronous operation-based data crawling method and the traditional single-platform asynchronous operation-based data crawling method, and as shown in fig. 8, the following data are obtained:
according to the data of the table, compared with other two methods, the speed of the asynchronous aiohttp-based multi-platform distributed data crawling method provided by the invention is greatly improved.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and the preferred embodiments of the present invention are described in the above embodiments and the description, and are not intended to limit the present invention. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (10)
1. A multi-platform distributed data crawling method based on asynchronous aiohttp is characterized by comprising the following steps: the method comprises the following steps:
step one, a service center distributes url tasks to a client;
secondly, the client reads configuration from the configuration center;
step three, a plurality of clients download contents to a plurality of url tasks;
step four, analyzing, cleaning and storing the downloaded content into a database;
collecting logs by using a log center;
and step six, opening the monitoring center to check the resource condition and the result.
2. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the url task distribution in the step one comprises the following specific operations: creating a message theme, dividing a plurality of themes for a plurality of platforms, performing task multi-platform distribution, performing duplicate removal on a single platform url task, and sending url tasks on different themes.
3. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the reading configuration function in the second step comprises: client agent, failure retry, custom request information, synchronous, asynchronous operation mode selection, timeout control, request time, task white list, client middleware, database type selection, performance setting, request type.
4. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the specific operations of downloading the content in the third step are as follows: and sending a request to the url task by using the aiohttp library, and receiving returned data.
5. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: and the data cleaning in the fourth step comprises the following specific operations: analyzing the returned data, analyzing the contents of different types by using different libraries, analyzing the data in the json format by using the json library, analyzing the html format by using the xpath library, extracting other texts by using the re regular library, storing the pictures and video byte streams by using a binary system, and storing the cleaned data in the database.
6. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the log collection in the fifth step is specifically operated as follows: logging modules are utilized to record logs in different modes for different log levels of the client, such as five levels of DEBUG, INFO, WARNING, ERROR and CRITICAL, log information is recorded by using files, HTTP GET/POST, SMTP and Socket modes are adopted for recording, and the file format is log files generally.
7. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the monitoring in the sixth step comprises the following steps: the usability of the client resource can record the server problem and inform the server problem when the system is shut down, the server resource trend and the system activity are analyzed, the data amount condition of data crawling and warehousing is carried out, the log recording condition of the client is recorded, a WEB interface is provided for setting the client, and the result is checked.
8. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 2, wherein: the task multi-platform distribution principle is that a kafka publish-subscribe message transfer mode is utilized, a publisher sends a message to a topic, only a subscriber who subscribes to the topic receives the message, so that multiple platforms can be distinguished, and the task deduplication principle utilizes a set characteristic of redis to carry a deduplication function.
9. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the principle of the aiohttp is that a request is sent by utilizing async asynchronous characteristics, so that the network io cannot cause blockage, and high concurrency and high availability are realized.
10. The asynchronous aiohttp-based multi-platform distributed data crawling method described in claim 1, wherein: the principle of the configuration center is that one service is started as a service side, then each service needing to be configured is used as a client side to obtain configuration by the service side, so that the configuration of tens of thousands of client sides is unified, the platform is unified, the client sides are highly available, the maintenance cost is low, and the aiohttp, xpath, json, logging, kafka and redis belong to an open source library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110029155.8A CN112732996A (en) | 2021-01-11 | 2021-01-11 | Multi-platform distributed data crawling method based on asynchronous aiohttp |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110029155.8A CN112732996A (en) | 2021-01-11 | 2021-01-11 | Multi-platform distributed data crawling method based on asynchronous aiohttp |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112732996A true CN112732996A (en) | 2021-04-30 |
Family
ID=75590014
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110029155.8A Pending CN112732996A (en) | 2021-01-11 | 2021-01-11 | Multi-platform distributed data crawling method based on asynchronous aiohttp |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112732996A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100131554A1 (en) * | 2008-11-26 | 2010-05-27 | Yahoo! Inc. | System and method for publishing messages asynchronously in a distributed database |
| CN107451223A (en) * | 2017-07-17 | 2017-12-08 | 广州特道信息科技有限公司 | The big data acquisition system and method for a kind of high concurrent parallel computation |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN109284430A (en) * | 2018-09-07 | 2019-01-29 | 杭州艾塔科技有限公司 | Visualization subject web page content based on distributed structure/architecture crawls system and method |
| CN109657125A (en) * | 2018-12-14 | 2019-04-19 | 平安城市建设科技(深圳)有限公司 | Data processing method, device, equipment and storage medium based on web crawlers |
| CN109948026A (en) * | 2019-03-28 | 2019-06-28 | 深信服科技股份有限公司 | A kind of web data crawling method, device, equipment and medium |
| CN110209909A (en) * | 2019-04-19 | 2019-09-06 | 平安科技（深圳）有限公司 | Data crawling method, device, computer equipment and storage medium |
| CN111241373A (en) * | 2020-02-20 | 2020-06-05 | 山东爱城市网信息技术有限公司 | Webpage crawler system based on micro-service and implementation method |
| CN111814024A (en) * | 2020-08-14 | 2020-10-23 | 北京斗米优聘科技发展有限公司 | Distributed data acquisition method, system and storage medium |
-
2021
- 2021-01-11 CN CN202110029155.8A patent/CN112732996A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100131554A1 (en) * | 2008-11-26 | 2010-05-27 | Yahoo! Inc. | System and method for publishing messages asynchronously in a distributed database |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN107451223A (en) * | 2017-07-17 | 2017-12-08 | 广州特道信息科技有限公司 | The big data acquisition system and method for a kind of high concurrent parallel computation |
| CN109284430A (en) * | 2018-09-07 | 2019-01-29 | 杭州艾塔科技有限公司 | Visualization subject web page content based on distributed structure/architecture crawls system and method |
| CN109657125A (en) * | 2018-12-14 | 2019-04-19 | 平安城市建设科技(深圳)有限公司 | Data processing method, device, equipment and storage medium based on web crawlers |
| CN109948026A (en) * | 2019-03-28 | 2019-06-28 | 深信服科技股份有限公司 | A kind of web data crawling method, device, equipment and medium |
| CN110209909A (en) * | 2019-04-19 | 2019-09-06 | 平安科技（深圳）有限公司 | Data crawling method, device, computer equipment and storage medium |
| CN111241373A (en) * | 2020-02-20 | 2020-06-05 | 山东爱城市网信息技术有限公司 | Webpage crawler system based on micro-service and implementation method |
| CN111814024A (en) * | 2020-08-14 | 2020-10-23 | 北京斗米优聘科技发展有限公司 | Distributed data acquisition method, system and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| HUMANITY11: "kafka分布式爬虫系统-任务的发放", pages 1 - 4, Retrieved from the Internet <URL:https://blog.csdn.net/humanity11/article/details/86633721> * |
| SL_WORLD: "Python实战异步爬虫(协程)+分布式爬虫(多进程)", pages 1 - 9, Retrieved from the Internet <URL:https://blog.csdn.net/SL_World/article/details/86633611> * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107895009A (en) | One kind is based on distributed internet data acquisition method and system | |
| US20110087630A1 (en) | System analysis program, system analysis method, and system analysis apparatus | |
| CN105357273B (en) | SOCKET communications and management of process general-purpose platform and method under asynchronous communication model | |
| Martin-Flatin | Web-based management of IP networks and systems | |
| US9342396B2 (en) | Self-stabilizing network nodes in mobile discovery system | |
| CN103036961A (en) | Distributed collection and storage method of journal | |
| CN103533087A (en) | Cloud service platform middleware and cloud uploading method | |
| CN110737647B (en) | Internet big data cleaning method | |
| CA2397865A1 (en) | Improved method and system of state management for data communications | |
| CN110858192A (en) | Log query method and system, log checking system and query terminal | |
| CN1540945A (en) | Communication device and communication method | |
| CN100579126C (en) | A mail subscription method and server | |
| CN112732996A (en) | Multi-platform distributed data crawling method based on asynchronous aiohttp | |
| CN108512943A (en) | Embedded device data store system based on message queue and store method | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| Sah | A New Architecture for Managing Enterprise Log Data. | |
| CN112506886B (en) | Multi-source service operation log acquisition method and system | |
| CN115865680A (en) | Method, system and device for distributed equipment access, control and data transmission | |
| CN113194150B (en) | Long connection push service processing system | |
| CN103856359A (en) | Method and system for obtaining information | |
| CN116028247B (en) | File acquisition system based on message middleware | |
| KR100423840B1 (en) | method of data processing for dispersion database | |
| Nishida et al. | Congestion control in interconnected local area networks | |
| CN116962132A (en) | Data processing method, device, storage medium and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |