CN111224817A - Distributed micro-acquisition system and method thereof - Google Patents
Distributed micro-acquisition system and method thereof Download PDFInfo
- Publication number
- CN111224817A CN111224817A CN201911253785.2A CN201911253785A CN111224817A CN 111224817 A CN111224817 A CN 111224817A CN 201911253785 A CN201911253785 A CN 201911253785A CN 111224817 A CN111224817 A CN 111224817A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- data
- client
- acquired
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000007405 data analysis Methods 0.000 claims abstract description 4
- 238000013500 data storage Methods 0.000 claims abstract description 4
- 238000009826 distribution Methods 0.000 claims abstract description 4
- 238000004519 manufacturing process Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0213—Standardised network management protocols, e.g. simple network management protocol [SNMP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/10—Current supply arrangements
Abstract
The invention discloses a distributed micro-collection system and a method thereof, wherein the system comprises: the acquisition client acquires data through the data acquisition point and packages and sends the acquired data; the data processor is used for analyzing and processing the acquired data of the acquisition client and sending the result to a message queue; the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor; the registration center collects and stores the information of the collection client to form queue information for service; and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing. The system can acquire the performance of a large batch of POE equipment in a short time, has high data accuracy and low server cost, and can add an acquisition client according to the increase of the equipment, thereby reducing the operation cost.
Description
Technical Field
The invention relates to the technical field of distributed data acquisition, in particular to a distributed micro-acquisition system and a method thereof.
Background
With social progress and scientific development, people see more and more computer equipment used in life, most of the existing distributed micro-acquisition systems of POE equipment have single acquisition targets and are difficult to support POE equipment produced by multiple manufacturers, and if the distributed micro-acquisition systems acquire other equipment, program modification is needed. In addition, the currently adopted integrated acquisition technology is complex and does not realize the cluster of the acquisition server, when the equipment amount is large, the acquisition tasks are too many, the occupation of server resources is high, the acquisition period is very long, and if the acquisition server has problems, the service of the acquisition server cannot be used, so that the acquisition client is limited.
Disclosure of Invention
The invention aims to provide a distributed micro-acquisition system, which divides an acquisition function into independent acquisition programs, enables the acquisition programs to only perform acquisition-related work, reduces the complexity, adopts multi-thread execution, has high processing capacity, can be set as required by an acquisition client, and saves various costs.
To achieve the above object, the present invention provides a distributed micro-collection system, comprising:
the system comprises an acquisition client, a data acquisition server and a data transmission server, wherein the acquisition client comprises a plurality of data acquisition points and acquires data through the data acquisition points and packages and transmits the acquired data;
the data processor analyzes and processes the data acquired by the acquisition client and sends the result to a message queue, and simultaneously monitors the message queue and writes the result into a file;
the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor;
the registration center collects and stores the information of the collection client to form queue information for service;
and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing.
Preferably, in the above technical solution, the acquisition client receives the parameter data of the device to be acquired, which is sent by the acquisition server, and after the acquisition client obtains the parameter data of the device to be acquired in a multithreading manner, the acquisition client packages the parameter data into a character string in a specified format and sends the character string to a specified message queue in a queue manner;
wherein, the parameter data at least comprises an IP address, an MAC address and an acquisition serial number of the equipment.
Preferably, in the above technical solution, the data processor writes the acquisition result into the designated file in an NIO manner.
Preferably, in the above technical solution, the data processor is provided with a query database and a parameter setting database; the parameter setting database is a Resultprocess capable of providing a result processing parameter setting service, and the query database is a Dataprocess capable of providing an equipment production query service.
Preferably, in the above technical solution, the acquisition server acquires the data processing service of the registry through a Dubbo distributed framework, thereby acquiring a list of devices to be acquired, then, according to configured batch operands, batch-transfers acquisition tasks of the devices to be acquired to application processing in acquisition clients registered in the registry, and simultaneously invokes a parameter setting database service to transfer required parameters to a query database for application.
Another object of the present invention is to provide an acquisition method of a distributed micro-acquisition system, the method comprising the steps of:
step S001, starting a system;
step S002, after the system is started, the acquisition server inquires a list of the equipment to be acquired from the inquiry database according to the period of the configuration file, acquires the IP address, the MAC address and the on-line and off-line states of the equipment to be acquired, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server groups the devices to be acquired according to the number of the devices to be acquired online, calls acquisition client data by using a Dubbo frame, and the acquisition client executes an acquisition task by using a Collection-client service;
step S004, after the Collect-client service of the acquisition client is called, starting multithreading to process the acquisition work of the equipment to be acquired, acquiring the port information of the equipment to be acquired through an SNMP protocol, and then sending the acquisition result to a message queue;
step S005, the data processor monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
Preferably, in the above technical solution, in step S004, the acquisition server records the number of the devices to be acquired, and then sends the number to the data processor, the data processor subtracts 1 from the count after receiving a message, and when the count is 0, the data processor writes the acquisition result into the designated file.
Compared with the prior art, the invention has the following beneficial effects:
1. according to the system, the performance of large batches of POE equipment can be acquired in a short time, the accuracy of data is improved, the cost of the server is reduced, and the acquisition client can be added according to the increase of the equipment, so that the operation cost is reduced.
2. The system can adjust the number of the acquisition clients according to the number of the online devices, the acquisition server sends acquisition tasks to the acquisition clients for execution, and then the acquisition clients return results to the data processing module for data processing, so that the acquisition function originally integrated in the network management system is split into separate acquisition programs, the acquisition programs only perform acquisition-related work, and the functions are simplified.
3. The collection process of the system only needs collection related work and adopts multithread execution, so that the processing capacity is greatly improved, each collection client program can collect and process the performance of more than 5000 devices within 1 minute during application test, and when the device quantity is increased, only collection clients are properly added, so that the system is simple and convenient.
Drawings
FIG. 1 is a block diagram of a distributed micro-acquisition system according to the present invention;
FIG. 2 is a flow diagram of a distributed micro-acquisition system according to the present invention;
FIG. 3 is a block diagram of a registration of a distributed micro-acquisition system according to the present invention.
Description of the main reference numerals:
101-acquisition client, 102-message queue, 103-data processor, 1031-query database, 1032-parameter setting database, 104-registration center, 105-acquisition server.
Detailed Description
Detailed description of the preferred embodimentsthe following detailed description of the present invention will be made with reference to the accompanying drawings 1-3, although it should be understood that the scope of the present invention is not limited to the specific embodiments.
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1-3, a distributed micro-collection system of the present invention includes a collection client 101, a message queue 102, a data processor 103, a registry 104, and a collection server 105.
The acquisition server 105 is configured to acquire a data processing task of the registry 104, execute the processing task, and distribute the processing result to the acquisition client 101 for application processing. Specifically, a data processing service of the registry 104 is acquired through a distributed framework (preferably, a Dubbo framework is adopted to realize distributed calling), so as to acquire a list of devices to be acquired, then, according to configured batch operands, acquisition tasks of the devices to be acquired are given to a plurality of acquisition clients 101 registered in the registry 104 in batches for application processing, and meanwhile, a parameter setting database 1032 is called to serve to transmit required parameters to a query database 1031 for application;
and the data processor 103 is configured to analyze and process the acquired data of the acquisition client 101 and send the result to the message queue 102, and meanwhile, the data processor also monitors the message queue and writes the acquisition result into a file. Specifically, the acquired data of the acquisition client 101 is analyzed and processed, and the result is sent to the message queue 102 or stored in the database, and the message queue 102 is monitored, when a message of the acquisition result arrives in the message queue 102, the message is judged and processed, and when parameter data such as ip address, mac address, acquisition serial number and the like of the device transmitted by the acquisition server are met, the acquisition result is written into the designated file in an nio (nivanio) manner. Further, the data processor 103 of the present embodiment is provided with a query database 1031 and a parameter setting database 1032; among them, the parameter setting database 1032 is preferably such as Resultprocess that can provide a result processing parameter setting service, and the query database 1031 is preferably such as Dataprocess that can provide a device production query service.
The system comprises an acquisition Client 101 and a server, wherein the acquisition Client 101 comprises a plurality of data acquisition points (a plurality of Client machines), acquires data through the data acquisition points (the acquired data at least comprises equipment parameter data), and encapsulates and transmits the acquired data; specifically, data of a plurality of acquisition points (acquisition client 1 and acquisition client 2 … … acquire client n) is acquired, parameter data of the device to be acquired, which is sent by the acquisition server 105, is received, the acquisition client 101 accesses the device to be acquired in a multithreading manner, after the parameter data is obtained, the parameter data is packaged into a character string in a specified format, and the character string is sent to the specified message queue 102 in a queue manner.
And the message queue 102 is used for coordinating data analysis, storage and distribution work between the acquisition client 101 and the data processor 103.
And the registration center 104 is used for acquiring and storing the parameter data of the acquisition client to form queue information for service. Specifically, ZooKeeper is used as a registration center, parameter data such as IP addresses, MAC addresses, and collection serial numbers of all collection clients are collected and stored, and are provided to the collection server 105 to query a device list to be collected from a database according to a configuration file cycle, obtain IP addresses, MAC addresses, and an online and offline state of POE devices, and simultaneously respond to and maintain the data processor 103, and after receiving a request from the data processor, the data processor processes the request, and performs services according to different types of the request.
In addition, the system is also provided with a collection service interface (not shown in the figure) for providing an exposed interface of various services to connect all the components.
In specific application, the flow is shown in fig. 3, and includes the following steps:
step S001, starting a system, specifically starting an acquisition client 101, a message queue 102, a data processor 103, a registration center 104 and an acquisition server 105 at the same time;
step S002, after the acquisition server 105 is started, the acquisition server 105 queries the list of the devices to be acquired from the query database 1031 according to the configuration file period, obtains the IP address, the MAC address and the on-line and off-line states of the POE devices, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server divides the devices to be collected into a plurality of groups, for example, 5000 devices, according to the number of the devices to be collected online, calls the data of the collection client 101 by using a Dubbo frame, and the collection client 101 performs collection tasks by using a Collection-client service;
step S004, after the Collect-client service of the collection client 101 is called, starting multithreading to process the collection work of the equipment to be collected, simultaneously acquiring the port information of the equipment to be collected through an SNMP protocol (in particular, SNMP-hibernet), and then sending the collection result to the message queue 102;
in step S005, the data processor 103 monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
When the acquisition work starts, the acquisition server 105 records the number of the devices to be acquired, and then sends the number to the data processor 103, the data processor 103 subtracts 1 from the count after receiving a message, and when the count is 0, the data processor 103 generates an acquisition record and writes and saves the result in a designated file in an nio (java nio) manner. The acquisition server 105 is ensured to normally operate in the acquisition process, if the acquisition server 105 does not work, the whole system does not work, and the acquisition client side at least needs to ensure that one acquisition client side works normally.
In summary, the system of the present invention can adjust the number of the acquisition clients according to the number of the online devices, the acquisition server sends the acquisition tasks to the acquisition clients for execution, and then the acquisition clients return the results to the data processing module for data processing, thereby splitting the acquisition function originally integrated in the network management system into separate acquisition programs, allowing the acquisition programs to only perform acquisition-related work, and simplifying the functions. The collection process only needs collection related work and adopts multi-thread execution, so that the processing capacity is greatly improved, the performance of more than 5000 devices can be collected and processed within 1 minute by each collection client program during application test, and when the device quantity is increased, only collection clients are properly added.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.
Claims (7)
1. A distributed micro-acquisition system, comprising:
the system comprises an acquisition client, a data acquisition server and a data transmission server, wherein the acquisition client comprises a plurality of data acquisition points and acquires data through the data acquisition points and packages and transmits the acquired data;
the data processor analyzes and processes the data acquired by the acquisition client and sends the result to a message queue, and simultaneously monitors the message queue and writes the result into a file;
the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor;
the registration center collects and stores the information of the collection client to form queue information for service;
and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing.
2. The distributed micro-acquisition system of claim 1, wherein the acquisition client receives parameter data of the device to be acquired from the acquisition server, and the acquisition client packages the parameter data into a character string of a specified format after acquiring the parameter data of the device to be acquired in a multithreading manner, and sends the character string to a specified message queue in a queue manner;
wherein, the parameter data at least comprises an IP address, an MAC address and an acquisition serial number of the equipment.
3. The distributed micro-acquisition system of claim 1, wherein the data processor writes the acquisition results to a designated file by NIO.
4. The distributed micro-acquisition system of claim 3, wherein the data processor has disposed therein a query database and a parameter setting database; the parameter setting database is a Resultprocess capable of providing a result processing parameter setting service, and the query database is a Dataprocess capable of providing an equipment production query service.
5. The distributed micro-acquisition system of claim 1, wherein the acquisition server acquires the data processing services of the registry through a Dubbo distributed framework, thereby acquiring the list of the devices to be acquired, then batch-hands the acquisition tasks of the devices to be acquired to the application of the acquisition clients registered in the registry according to the configured batch operands, and simultaneously invokes the parameter setting database service to transfer the required parameters to the query database for application.
6. A collection method of a distributed micro-collection system, based on the distributed micro-collection system of any one of claims 1 to 5, the method comprising the steps of:
step S001, starting a system;
step S002, after the system is started, the acquisition server inquires a list of the equipment to be acquired from the inquiry database according to the period of the configuration file, acquires the IP address, the MAC address and the on-line and off-line states of the equipment to be acquired, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server groups the belt acquisition equipment according to the number of the online equipment to be acquired, calls acquisition client data by using a Dubbo frame, and the acquisition client executes an acquisition task by using a Collection-client service;
step S004, after the Collect-client service of the acquisition client is called, starting multithreading to process the acquisition work of the equipment to be acquired, acquiring the port information of the equipment to be acquired through an SNMP protocol, and then sending the acquisition result to a message queue;
step S005, the data processor monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
7. The acquisition method according to claim 6, wherein in step S004, the acquisition server records the number of the devices to be acquired and then sends the number to the data processor, the data processor receives a message and then decrements the count by 1, and when the count is 0, the data processor writes the result into a designated file.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911253785.2A CN111224817A (en) | 2020-04-21 | 2020-04-21 | Distributed micro-acquisition system and method thereof |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911253785.2A CN111224817A (en) | 2020-04-21 | 2020-04-21 | Distributed micro-acquisition system and method thereof |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111224817A true CN111224817A (en) | 2020-06-02 |
Family
ID=70827753
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911253785.2A Pending CN111224817A (en) | 2020-04-21 | 2020-04-21 | Distributed micro-acquisition system and method thereof |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111224817A (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113138900A (en) * | 2021-04-27 | 2021-07-20 | 上海淇玥信息技术有限公司 | Data acquisition processing method and device and electronic equipment |
CN114124695A (en) * | 2021-11-23 | 2022-03-01 | 上海帆一尚行科技有限公司 | Data acquisition method and device and electronic equipment |
CN114124643A (en) * | 2021-11-10 | 2022-03-01 | 中盈优创资讯科技有限公司 | PaaS-based network equipment flow acquisition method and device |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2007084436A2 (en) * | 2006-01-17 | 2007-07-26 | Reality Mobile Llc | System and method for remote data acquisition and distribution |
CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
CN106936618A (en) * | 2015-12-31 | 2017-07-07 | 北京神州泰岳软件股份有限公司 | A kind of collecting method and system |
CN108769115A (en) * | 2018-04-19 | 2018-11-06 | 中国科学院计算技术研究所 | Distributed RSS data acquisition method and system |
-
2020
- 2020-04-21 CN CN201911253785.2A patent/CN111224817A/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2007084436A2 (en) * | 2006-01-17 | 2007-07-26 | Reality Mobile Llc | System and method for remote data acquisition and distribution |
CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
CN106936618A (en) * | 2015-12-31 | 2017-07-07 | 北京神州泰岳软件股份有限公司 | A kind of collecting method and system |
CN108769115A (en) * | 2018-04-19 | 2018-11-06 | 中国科学院计算技术研究所 | Distributed RSS data acquisition method and system |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113138900A (en) * | 2021-04-27 | 2021-07-20 | 上海淇玥信息技术有限公司 | Data acquisition processing method and device and electronic equipment |
CN114124643A (en) * | 2021-11-10 | 2022-03-01 | 中盈优创资讯科技有限公司 | PaaS-based network equipment flow acquisition method and device |
CN114124643B (en) * | 2021-11-10 | 2023-08-01 | 中盈优创资讯科技有限公司 | PaaS-based network equipment flow collection method and device |
CN114124695A (en) * | 2021-11-23 | 2022-03-01 | 上海帆一尚行科技有限公司 | Data acquisition method and device and electronic equipment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111224817A (en) | Distributed micro-acquisition system and method thereof | |
CN111414416B (en) | Data processing method, device, equipment and storage medium | |
WO2022179140A1 (en) | Data processing method and system | |
CN101399715B (en) | Network test platform and test method for optical fiber channel | |
CN111580995A (en) | Synchronous communication method and system of distributed cloud platform and Internet of things intelligent terminal based on MQTT asynchronous communication scene | |
US10454795B1 (en) | Intermediate batch service for serverless computing environment metrics | |
US10853153B2 (en) | Message queue performance monitoring | |
CN112988485A (en) | Simulation test method and device for power Internet of things equipment | |
CN113110930A (en) | Cloud solution method, system, server and storage medium for decision problem | |
CN114745295A (en) | Data acquisition method, device, equipment and readable storage medium | |
CN112954006A (en) | Industrial Internet edge gateway design method supporting Web high-concurrency access | |
CN113609047B (en) | Data adapter and data adaptation method | |
CN108509256B (en) | Method and device for scheduling running device and running device | |
CN111984505A (en) | Operation and maintenance data acquisition engine and acquisition method | |
CN113342503B (en) | Real-time progress feedback method, device, equipment and storage medium | |
CN111756599B (en) | Service monitoring method and service monitoring system | |
CN111082964B (en) | Distribution method and device of configuration information | |
CN112380001A (en) | Log output method, load balancing device and computer readable storage medium | |
CN112463604A (en) | APP detection system based on electric power integration terminal | |
CN111930482A (en) | Task processing method, device and equipment based on node cluster and storage medium | |
CN111913784A (en) | Task scheduling method and device, network element and storage medium | |
CN115883639A (en) | Web real-time message pushing method and device, equipment and storage medium | |
CN113055493B (en) | Data packet processing method, device, system, scheduling device and storage medium | |
CN115080601A (en) | Domestic CPU and OS based heterogeneous terminal unified platform management method and system | |
CN116974655A (en) | Capability scheduling method and capability scheduling functional entity |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200602 |