CN111224817A - Distributed micro-acquisition system and method thereof - Google Patents

Distributed micro-acquisition system and method thereof Download PDF

Info

Publication number
CN111224817A
CN111224817A CN201911253785.2A CN201911253785A CN111224817A CN 111224817 A CN111224817 A CN 111224817A CN 201911253785 A CN201911253785 A CN 201911253785A CN 111224817 A CN111224817 A CN 111224817A
Authority
CN
China
Prior art keywords
acquisition
data
client
acquired
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911253785.2A
Other languages
Chinese (zh)
Inventor
陈跃飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Kunliang Technology Co ltd
Original Assignee
Chongqing Kunliang Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Kunliang Technology Co ltd filed Critical Chongqing Kunliang Technology Co ltd
Priority to CN201911253785.2A priority Critical patent/CN111224817A/en
Publication of CN111224817A publication Critical patent/CN111224817A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02Standardisation; Integration
    • H04L41/0213Standardised network management protocols, e.g. simple network management protocol [SNMP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00Data switching networks
    • H04L12/02Details
    • H04L12/10Current supply arrangements

Abstract

The invention discloses a distributed micro-collection system and a method thereof, wherein the system comprises: the acquisition client acquires data through the data acquisition point and packages and sends the acquired data; the data processor is used for analyzing and processing the acquired data of the acquisition client and sending the result to a message queue; the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor; the registration center collects and stores the information of the collection client to form queue information for service; and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing. The system can acquire the performance of a large batch of POE equipment in a short time, has high data accuracy and low server cost, and can add an acquisition client according to the increase of the equipment, thereby reducing the operation cost.

Description

Distributed micro-acquisition system and method thereof
Technical Field
The invention relates to the technical field of distributed data acquisition, in particular to a distributed micro-acquisition system and a method thereof.
Background
With social progress and scientific development, people see more and more computer equipment used in life, most of the existing distributed micro-acquisition systems of POE equipment have single acquisition targets and are difficult to support POE equipment produced by multiple manufacturers, and if the distributed micro-acquisition systems acquire other equipment, program modification is needed. In addition, the currently adopted integrated acquisition technology is complex and does not realize the cluster of the acquisition server, when the equipment amount is large, the acquisition tasks are too many, the occupation of server resources is high, the acquisition period is very long, and if the acquisition server has problems, the service of the acquisition server cannot be used, so that the acquisition client is limited.
Disclosure of Invention
The invention aims to provide a distributed micro-acquisition system, which divides an acquisition function into independent acquisition programs, enables the acquisition programs to only perform acquisition-related work, reduces the complexity, adopts multi-thread execution, has high processing capacity, can be set as required by an acquisition client, and saves various costs.
To achieve the above object, the present invention provides a distributed micro-collection system, comprising:
the system comprises an acquisition client, a data acquisition server and a data transmission server, wherein the acquisition client comprises a plurality of data acquisition points and acquires data through the data acquisition points and packages and transmits the acquired data;
the data processor analyzes and processes the data acquired by the acquisition client and sends the result to a message queue, and simultaneously monitors the message queue and writes the result into a file;
the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor;
the registration center collects and stores the information of the collection client to form queue information for service;
and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing.
Preferably, in the above technical solution, the acquisition client receives the parameter data of the device to be acquired, which is sent by the acquisition server, and after the acquisition client obtains the parameter data of the device to be acquired in a multithreading manner, the acquisition client packages the parameter data into a character string in a specified format and sends the character string to a specified message queue in a queue manner;
wherein, the parameter data at least comprises an IP address, an MAC address and an acquisition serial number of the equipment.
Preferably, in the above technical solution, the data processor writes the acquisition result into the designated file in an NIO manner.
Preferably, in the above technical solution, the data processor is provided with a query database and a parameter setting database; the parameter setting database is a Resultprocess capable of providing a result processing parameter setting service, and the query database is a Dataprocess capable of providing an equipment production query service.
Preferably, in the above technical solution, the acquisition server acquires the data processing service of the registry through a Dubbo distributed framework, thereby acquiring a list of devices to be acquired, then, according to configured batch operands, batch-transfers acquisition tasks of the devices to be acquired to application processing in acquisition clients registered in the registry, and simultaneously invokes a parameter setting database service to transfer required parameters to a query database for application.
Another object of the present invention is to provide an acquisition method of a distributed micro-acquisition system, the method comprising the steps of:
step S001, starting a system;
step S002, after the system is started, the acquisition server inquires a list of the equipment to be acquired from the inquiry database according to the period of the configuration file, acquires the IP address, the MAC address and the on-line and off-line states of the equipment to be acquired, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server groups the devices to be acquired according to the number of the devices to be acquired online, calls acquisition client data by using a Dubbo frame, and the acquisition client executes an acquisition task by using a Collection-client service;
step S004, after the Collect-client service of the acquisition client is called, starting multithreading to process the acquisition work of the equipment to be acquired, acquiring the port information of the equipment to be acquired through an SNMP protocol, and then sending the acquisition result to a message queue;
step S005, the data processor monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
Preferably, in the above technical solution, in step S004, the acquisition server records the number of the devices to be acquired, and then sends the number to the data processor, the data processor subtracts 1 from the count after receiving a message, and when the count is 0, the data processor writes the acquisition result into the designated file.
Compared with the prior art, the invention has the following beneficial effects:
1. according to the system, the performance of large batches of POE equipment can be acquired in a short time, the accuracy of data is improved, the cost of the server is reduced, and the acquisition client can be added according to the increase of the equipment, so that the operation cost is reduced.
2. The system can adjust the number of the acquisition clients according to the number of the online devices, the acquisition server sends acquisition tasks to the acquisition clients for execution, and then the acquisition clients return results to the data processing module for data processing, so that the acquisition function originally integrated in the network management system is split into separate acquisition programs, the acquisition programs only perform acquisition-related work, and the functions are simplified.
3. The collection process of the system only needs collection related work and adopts multithread execution, so that the processing capacity is greatly improved, each collection client program can collect and process the performance of more than 5000 devices within 1 minute during application test, and when the device quantity is increased, only collection clients are properly added, so that the system is simple and convenient.
Drawings
FIG. 1 is a block diagram of a distributed micro-acquisition system according to the present invention;
FIG. 2 is a flow diagram of a distributed micro-acquisition system according to the present invention;
FIG. 3 is a block diagram of a registration of a distributed micro-acquisition system according to the present invention.
Description of the main reference numerals:
101-acquisition client, 102-message queue, 103-data processor, 1031-query database, 1032-parameter setting database, 104-registration center, 105-acquisition server.
Detailed Description
Detailed description of the preferred embodimentsthe following detailed description of the present invention will be made with reference to the accompanying drawings 1-3, although it should be understood that the scope of the present invention is not limited to the specific embodiments.
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1-3, a distributed micro-collection system of the present invention includes a collection client 101, a message queue 102, a data processor 103, a registry 104, and a collection server 105.
The acquisition server 105 is configured to acquire a data processing task of the registry 104, execute the processing task, and distribute the processing result to the acquisition client 101 for application processing. Specifically, a data processing service of the registry 104 is acquired through a distributed framework (preferably, a Dubbo framework is adopted to realize distributed calling), so as to acquire a list of devices to be acquired, then, according to configured batch operands, acquisition tasks of the devices to be acquired are given to a plurality of acquisition clients 101 registered in the registry 104 in batches for application processing, and meanwhile, a parameter setting database 1032 is called to serve to transmit required parameters to a query database 1031 for application;
and the data processor 103 is configured to analyze and process the acquired data of the acquisition client 101 and send the result to the message queue 102, and meanwhile, the data processor also monitors the message queue and writes the acquisition result into a file. Specifically, the acquired data of the acquisition client 101 is analyzed and processed, and the result is sent to the message queue 102 or stored in the database, and the message queue 102 is monitored, when a message of the acquisition result arrives in the message queue 102, the message is judged and processed, and when parameter data such as ip address, mac address, acquisition serial number and the like of the device transmitted by the acquisition server are met, the acquisition result is written into the designated file in an nio (nivanio) manner. Further, the data processor 103 of the present embodiment is provided with a query database 1031 and a parameter setting database 1032; among them, the parameter setting database 1032 is preferably such as Resultprocess that can provide a result processing parameter setting service, and the query database 1031 is preferably such as Dataprocess that can provide a device production query service.
The system comprises an acquisition Client 101 and a server, wherein the acquisition Client 101 comprises a plurality of data acquisition points (a plurality of Client machines), acquires data through the data acquisition points (the acquired data at least comprises equipment parameter data), and encapsulates and transmits the acquired data; specifically, data of a plurality of acquisition points (acquisition client 1 and acquisition client 2 … … acquire client n) is acquired, parameter data of the device to be acquired, which is sent by the acquisition server 105, is received, the acquisition client 101 accesses the device to be acquired in a multithreading manner, after the parameter data is obtained, the parameter data is packaged into a character string in a specified format, and the character string is sent to the specified message queue 102 in a queue manner.
And the message queue 102 is used for coordinating data analysis, storage and distribution work between the acquisition client 101 and the data processor 103.
And the registration center 104 is used for acquiring and storing the parameter data of the acquisition client to form queue information for service. Specifically, ZooKeeper is used as a registration center, parameter data such as IP addresses, MAC addresses, and collection serial numbers of all collection clients are collected and stored, and are provided to the collection server 105 to query a device list to be collected from a database according to a configuration file cycle, obtain IP addresses, MAC addresses, and an online and offline state of POE devices, and simultaneously respond to and maintain the data processor 103, and after receiving a request from the data processor, the data processor processes the request, and performs services according to different types of the request.
In addition, the system is also provided with a collection service interface (not shown in the figure) for providing an exposed interface of various services to connect all the components.
In specific application, the flow is shown in fig. 3, and includes the following steps:
step S001, starting a system, specifically starting an acquisition client 101, a message queue 102, a data processor 103, a registration center 104 and an acquisition server 105 at the same time;
step S002, after the acquisition server 105 is started, the acquisition server 105 queries the list of the devices to be acquired from the query database 1031 according to the configuration file period, obtains the IP address, the MAC address and the on-line and off-line states of the POE devices, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server divides the devices to be collected into a plurality of groups, for example, 5000 devices, according to the number of the devices to be collected online, calls the data of the collection client 101 by using a Dubbo frame, and the collection client 101 performs collection tasks by using a Collection-client service;
step S004, after the Collect-client service of the collection client 101 is called, starting multithreading to process the collection work of the equipment to be collected, simultaneously acquiring the port information of the equipment to be collected through an SNMP protocol (in particular, SNMP-hibernet), and then sending the collection result to the message queue 102;
in step S005, the data processor 103 monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
When the acquisition work starts, the acquisition server 105 records the number of the devices to be acquired, and then sends the number to the data processor 103, the data processor 103 subtracts 1 from the count after receiving a message, and when the count is 0, the data processor 103 generates an acquisition record and writes and saves the result in a designated file in an nio (java nio) manner. The acquisition server 105 is ensured to normally operate in the acquisition process, if the acquisition server 105 does not work, the whole system does not work, and the acquisition client side at least needs to ensure that one acquisition client side works normally.
In summary, the system of the present invention can adjust the number of the acquisition clients according to the number of the online devices, the acquisition server sends the acquisition tasks to the acquisition clients for execution, and then the acquisition clients return the results to the data processing module for data processing, thereby splitting the acquisition function originally integrated in the network management system into separate acquisition programs, allowing the acquisition programs to only perform acquisition-related work, and simplifying the functions. The collection process only needs collection related work and adopts multi-thread execution, so that the processing capacity is greatly improved, the performance of more than 5000 devices can be collected and processed within 1 minute by each collection client program during application test, and when the device quantity is increased, only collection clients are properly added.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.

Claims (7)

1. A distributed micro-acquisition system, comprising:
the system comprises an acquisition client, a data acquisition server and a data transmission server, wherein the acquisition client comprises a plurality of data acquisition points and acquires data through the data acquisition points and packages and transmits the acquired data;
the data processor analyzes and processes the data acquired by the acquisition client and sends the result to a message queue, and simultaneously monitors the message queue and writes the result into a file;
the message queue is used for coordinating data analysis, storage and distribution work between the acquisition client and the data processor;
the registration center collects and stores the information of the collection client to form queue information for service;
and the acquisition server is used for acquiring the data processing tasks of the registration center, executing the processing tasks and distributing the processing results to the acquisition client for application processing.
2. The distributed micro-acquisition system of claim 1, wherein the acquisition client receives parameter data of the device to be acquired from the acquisition server, and the acquisition client packages the parameter data into a character string of a specified format after acquiring the parameter data of the device to be acquired in a multithreading manner, and sends the character string to a specified message queue in a queue manner;
wherein, the parameter data at least comprises an IP address, an MAC address and an acquisition serial number of the equipment.
3. The distributed micro-acquisition system of claim 1, wherein the data processor writes the acquisition results to a designated file by NIO.
4. The distributed micro-acquisition system of claim 3, wherein the data processor has disposed therein a query database and a parameter setting database; the parameter setting database is a Resultprocess capable of providing a result processing parameter setting service, and the query database is a Dataprocess capable of providing an equipment production query service.
5. The distributed micro-acquisition system of claim 1, wherein the acquisition server acquires the data processing services of the registry through a Dubbo distributed framework, thereby acquiring the list of the devices to be acquired, then batch-hands the acquisition tasks of the devices to be acquired to the application of the acquisition clients registered in the registry according to the configured batch operands, and simultaneously invokes the parameter setting database service to transfer the required parameters to the query database for application.
6. A collection method of a distributed micro-collection system, based on the distributed micro-collection system of any one of claims 1 to 5, the method comprising the steps of:
step S001, starting a system;
step S002, after the system is started, the acquisition server inquires a list of the equipment to be acquired from the inquiry database according to the period of the configuration file, acquires the IP address, the MAC address and the on-line and off-line states of the equipment to be acquired, and stores the IP address, the MAC address and the on-line and off-line states;
step S003, the data server groups the belt acquisition equipment according to the number of the online equipment to be acquired, calls acquisition client data by using a Dubbo frame, and the acquisition client executes an acquisition task by using a Collection-client service;
step S004, after the Collect-client service of the acquisition client is called, starting multithreading to process the acquisition work of the equipment to be acquired, acquiring the port information of the equipment to be acquired through an SNMP protocol, and then sending the acquisition result to a message queue;
step S005, the data processor monitors the message queue, and takes out the message for processing to obtain the target data once the message enters.
7. The acquisition method according to claim 6, wherein in step S004, the acquisition server records the number of the devices to be acquired and then sends the number to the data processor, the data processor receives a message and then decrements the count by 1, and when the count is 0, the data processor writes the result into a designated file.
CN201911253785.2A 2020-04-21 2020-04-21 Distributed micro-acquisition system and method thereof Pending CN111224817A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911253785.2A CN111224817A (en) 2020-04-21 2020-04-21 Distributed micro-acquisition system and method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911253785.2A CN111224817A (en) 2020-04-21 2020-04-21 Distributed micro-acquisition system and method thereof

Publications (1)

Publication Number Publication Date
CN111224817A true CN111224817A (en) 2020-06-02

Family

ID=70827753

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911253785.2A Pending CN111224817A (en) 2020-04-21 2020-04-21 Distributed micro-acquisition system and method thereof

Country Status (1)

Country Link
CN (1) CN111224817A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113138900A (en) * 2021-04-27 2021-07-20 上海淇玥信息技术有限公司 Data acquisition processing method and device and electronic equipment
CN114124695A (en) * 2021-11-23 2022-03-01 上海帆一尚行科技有限公司 Data acquisition method and device and electronic equipment
CN114124643A (en) * 2021-11-10 2022-03-01 中盈优创资讯科技有限公司 PaaS-based network equipment flow acquisition method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007084436A2 (en) * 2006-01-17 2007-07-26 Reality Mobile Llc System and method for remote data acquisition and distribution
CN103024014A (en) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 Method and system for distributing and processing mass data passing through message queues
CN106936618A (en) * 2015-12-31 2017-07-07 北京神州泰岳软件股份有限公司 A kind of collecting method and system
CN108769115A (en) * 2018-04-19 2018-11-06 中国科学院计算技术研究所 Distributed RSS data acquisition method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007084436A2 (en) * 2006-01-17 2007-07-26 Reality Mobile Llc System and method for remote data acquisition and distribution
CN103024014A (en) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 Method and system for distributing and processing mass data passing through message queues
CN106936618A (en) * 2015-12-31 2017-07-07 北京神州泰岳软件股份有限公司 A kind of collecting method and system
CN108769115A (en) * 2018-04-19 2018-11-06 中国科学院计算技术研究所 Distributed RSS data acquisition method and system

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113138900A (en) * 2021-04-27 2021-07-20 上海淇玥信息技术有限公司 Data acquisition processing method and device and electronic equipment
CN114124643A (en) * 2021-11-10 2022-03-01 中盈优创资讯科技有限公司 PaaS-based network equipment flow acquisition method and device
CN114124643B (en) * 2021-11-10 2023-08-01 中盈优创资讯科技有限公司 PaaS-based network equipment flow collection method and device
CN114124695A (en) * 2021-11-23 2022-03-01 上海帆一尚行科技有限公司 Data acquisition method and device and electronic equipment

Similar Documents

Publication Publication Date Title
CN111224817A (en) Distributed micro-acquisition system and method thereof
CN111414416B (en) Data processing method, device, equipment and storage medium
WO2022179140A1 (en) Data processing method and system
CN101399715B (en) Network test platform and test method for optical fiber channel
CN111580995A (en) Synchronous communication method and system of distributed cloud platform and Internet of things intelligent terminal based on MQTT asynchronous communication scene
US10454795B1 (en) Intermediate batch service for serverless computing environment metrics
US10853153B2 (en) Message queue performance monitoring
CN112988485A (en) Simulation test method and device for power Internet of things equipment
CN113110930A (en) Cloud solution method, system, server and storage medium for decision problem
CN114745295A (en) Data acquisition method, device, equipment and readable storage medium
CN112954006A (en) Industrial Internet edge gateway design method supporting Web high-concurrency access
CN113609047B (en) Data adapter and data adaptation method
CN108509256B (en) Method and device for scheduling running device and running device
CN111984505A (en) Operation and maintenance data acquisition engine and acquisition method
CN113342503B (en) Real-time progress feedback method, device, equipment and storage medium
CN111756599B (en) Service monitoring method and service monitoring system
CN111082964B (en) Distribution method and device of configuration information
CN112380001A (en) Log output method, load balancing device and computer readable storage medium
CN112463604A (en) APP detection system based on electric power integration terminal
CN111930482A (en) Task processing method, device and equipment based on node cluster and storage medium
CN111913784A (en) Task scheduling method and device, network element and storage medium
CN115883639A (en) Web real-time message pushing method and device, equipment and storage medium
CN113055493B (en) Data packet processing method, device, system, scheduling device and storage medium
CN115080601A (en) Domestic CPU and OS based heterogeneous terminal unified platform management method and system
CN116974655A (en) Capability scheduling method and capability scheduling functional entity

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200602