CN109586947B - Distributed equipment information acquisition system and method - Google Patents
Distributed equipment information acquisition system and method Download PDFInfo
- Publication number
- CN109586947B CN109586947B CN201811185588.7A CN201811185588A CN109586947B CN 109586947 B CN109586947 B CN 109586947B CN 201811185588 A CN201811185588 A CN 201811185588A CN 109586947 B CN109586947 B CN 109586947B
- Authority
- CN
- China
- Prior art keywords
- equipment information
- node
- task
- detection
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/04—Network management architectures or arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a system and a method for collecting distributed equipment information, which are characterized in that a collection task is obtained through user interaction, the collection task is divided to obtain a collection subtask, the collection subtask is distributed to a detection node, and an equipment information collection result uploaded by the detection node is received; and receiving the acquisition subtask, executing equipment information acquisition to obtain an equipment information acquisition result, and uploading the equipment information acquisition result to the management node. The detection node filters out an open port through port scanning, constructs a detection request, analyzes a returned response message, acquires equipment information, and returns the equipment information to the management node, wherein the acquired equipment information is uniformly stored and managed by the management node. Networking equipment in a large-scale network address range is subjected to rapid information acquisition, the situation that IP is forbidden due to abnormal flow is avoided, and efficient and accurate equipment information acquisition is realized; and distributed scheduling and cooperative working modes of the management nodes and the detection nodes are adopted, so that the scattered acquisition and the unified storage management of the equipment information are realized.
Description
Technical Field
The invention relates to the field of network security, in particular to a distributed equipment information acquisition system and a distributed equipment information acquisition method.
Background
The information acquisition of networking equipment is carried out on a large-scale network and even the whole Internet, the whole safety condition of the Internet space is favorably mastered, and information support is provided for protecting an important networking system, so that the equipment with problems is timely checked out before the network attack happens, or the influence range is determined and the response is quickly made when the network attack happens. At present, a distributed organization architecture is generally adopted for information acquisition of a large-scale network, so that the limitation of network bandwidth of single-point detection is avoided, and meanwhile, the information acquisition capability of the whole system can be conveniently expanded.
However, the existing information acquisition method has the problems that the IP address detected by each detection node basically belongs to the same subnet net segment or an adjacent subnet net segment, if rapid information acquisition is required, the behavior that a single detection node frequently sends network data packets to a corresponding network segment easily occurs, and the flows are possibly determined as abnormal flows by a target network segment, so that the IP address of the detection node is blocked by the target network segment, and further, the quantity of the acquired equipment information is far lower than the actual situation. How to achieve the purpose of rapidly acquiring information of networking equipment in a large-scale network address range and avoid detecting a blacklist of an access network segment in which a node IP address is brought into the target network segment in the information acquisition process is a problem to be solved urgently at present.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a distributed equipment information acquisition system and a distributed equipment information acquisition method.
The distributed equipment information acquisition system comprises a node management module and a node detection module, wherein the node management module is composed of management nodes, the node detection module is composed of detection nodes, and the management nodes and the detection nodes are communicated through a distributed task scheduling framework; a node management module: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node; a node detection module: and receiving the acquisition subtask, executing equipment information acquisition to obtain an equipment information acquisition result, and uploading the equipment information acquisition result to the management node.
Preferably, the management node comprises a user interaction module: acquiring an acquisition task and providing equipment information query by interacting with a user; a task segmentation module: dividing the collection task to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management module; a task management module: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling; the data receiving and managing module: and receiving the equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query.
Preferably, the probing node module includes a scheduling communication module: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node; a port scanning module: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port; the device information detection module: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
The invention provides a distributed equipment information acquisition method, which comprises the following steps: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node; a node detection step: and receiving the acquisition subtask, executing equipment information acquisition to obtain an equipment information acquisition result, and uploading the equipment information acquisition result to the management node.
Preferably, the node management step includes a user interaction step of: acquiring an acquisition task and providing equipment information query by interacting with a user; task segmentation step: dividing the collection task to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management step; and task management: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling; data receiving and managing: and receiving the equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query.
Preferably, the node probing step includes the scheduling communication step of: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node; port scanning step: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port; a device information detection step: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
Preferably, the collection task includes an IP address segment to be scanned and a port range to be scanned; the acquisition task segmentation adopts random rearrangement to obtain an acquisition subtask which is expressed in the following form:
{ip1,ip2,port1,port2,i*c,(i+1)*c,r}
wherein IP1 and IP2 represent IP address segments to be scanned;
port1, port2 indicates the range of ports to be scanned;
c represents the number of targets to be scanned of the acquisition subtask;
i represents the task number of the collection subtask, i is 0,1,2, … (ip2-ip1+1) (port2-port1+ 1)/c;
r represents the base value of the random rearrangement.
Preferably, the distributing and collecting subtasks manage and collect subtasks through two groups of task queues; the task queue mainly comprises a task queue which is being executed and a task queue waiting to be executed; the collection subtasks which are being executed are stored in the task queue which is being executed, and the collection subtasks which are waiting to be executed are stored in the task queue which is waiting to be executed.
Preferably, the information acquisition of the execution device adopts a multi-thread asynchronous mode; the multithreading asynchronous mode mainly comprises a sending thread and a receiving thread; the sending thread is used for calculating a detection target for equipment information acquisition, and constructing and sending a detection request; the receiving thread is used for monitoring a detection target and acquiring a result of equipment information.
Compared with the prior art, the invention has the following beneficial effects:
1. networking equipment in a large-scale network address range is subjected to rapid information acquisition, the situation that IP is forbidden due to abnormal flow judged by a target network segment is avoided, and efficient and accurate equipment information acquisition is realized;
2. and distributed scheduling and cooperative working modes of the management nodes and the detection nodes are adopted, so that the scattered acquisition and the unified storage management of the equipment information are realized.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a block diagram of a distributed device information acquisition system of the present invention;
fig. 2 is a flowchart of a distributed device information acquisition method according to the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
The invention provides a system and a method for collecting distributed equipment information, which can be used for collecting networking equipment information in a large-scale network address range. The management node acquires an information acquisition task and divides the task, so that a plurality of IP addresses of the same subnet net section are prevented from being concentrated in one subtask, and then different subtasks are handed over to different detection nodes to be completed, thereby ensuring the efficiency of information acquisition and reducing the possibility of being forbidden by a target network section. After receiving the subtasks issued by the management node, the detection node firstly filters out the open ports through port scanning, then judges the running application layer protocol of each open port, constructs a corresponding request according to the protocol and analyzes the returned response message, thereby acquiring the equipment information. And finally, the detection node returns the equipment information acquired by the subtasks to the management node, and the acquired equipment information is uniformly stored and managed by the management node.
The invention provides a distributed equipment information acquisition method, which comprises the following steps: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node; a node detection step: and receiving the acquisition subtask, executing equipment information acquisition to obtain an equipment information acquisition result, and uploading the equipment information acquisition result to the management node.
Specifically, the node management step includes a user interaction step: acquiring an acquisition task and providing equipment information query by interacting with a user; task segmentation step: dividing the collection task to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management step; and task management: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling; data receiving and managing: and receiving the equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query.
Specifically, the node probing step includes a scheduling communication step: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node; port scanning step: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port; a device information detection step: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
Specifically, the collection task includes an IP address segment to be scanned and a port range to be scanned; the acquisition task segmentation adopts random rearrangement to obtain an acquisition subtask which is expressed in the following form:
{ip1,ip2,port1,port2,i*c,(i+1)*c,r}
wherein IP1 and IP2 represent IP address segments to be scanned;
port1, port2 indicates the range of ports to be scanned;
c represents the number of targets to be scanned of the acquisition subtask;
i represents the task number of the collection subtask, i is 0,1,2, … (ip2-ip1+1) (port2-port1+ 1)/c;
r represents the base value of the random rearrangement.
Specifically, after acquiring the acquisition subtask, the probe node calculates the IP and port of actual scanning probe by using a random rearrangement algorithm. The range of the random rearrangement algorithm is (IP2-IP1+1) ((port 2-port1+1), x ∈ [ i × c, (i +1) × c), and the corresponding y ═ f (x) is calculated by the random rearrangement algorithm, so that the scan target IP is (IP1+ y/(port2-port1+1)) and the target port is (port1+ y% (port2-port1+ 1)).
Wherein, the random rearrangement algorithm should satisfy the following properties: let X ∈ X be an element in the sequentially arranged set X, Y ∈ Y be an element in the rearranged set Y, and X and Y are bijective relationships, and a value Y ═ f (X) corresponding to the rearranged position X can be obtained by a random rearrangement algorithm.
Specifically, the distribution and collection subtask manages and collects subtasks through two groups of task queues; the task queue mainly comprises a task queue which is being executed and a task queue waiting to be executed; the collection subtasks which are being executed are stored in the task queue which is being executed, and the collection subtasks which are waiting to be executed are stored in the task queue which is waiting to be executed.
Adding the divided acquisition subtasks into a task queue, wherein the initial state is 'waiting for issuing'; the management node issues the acquisition subtask in the 'waiting for issuing' state to the detection node, and the issued acquisition subtask is changed into an 'execution' state; and when the detection node finishes executing the issued acquisition subtask and returns the acquisition result to the management node, the management node deletes the acquisition subtask from the task queue.
Specifically, the information acquisition of the execution device adopts a multi-thread asynchronous mode; the multithreading asynchronous mode mainly comprises a sending thread and a receiving thread; the sending thread is used for calculating a detection target for equipment information acquisition, and constructing and sending a detection request; the receiving thread is used for monitoring a detection target and acquiring a result of equipment information. Preferably, the probe node collects the device information of the port in the open state, including that the probe node calls an application protocol probe to determine an application layer protocol operated by the open port; the detection node then constructs a corresponding detection request according to the determined application layer protocol format, sends the detection request to the open port and receives response data returned by the target port; and the detection node extracts the equipment information from the response data by adopting an analysis method corresponding to the determined application layer protocol. The detection node returns the device information acquisition result corresponding to the subtask to the management node, and the management node stores the acquisition result in a database in a proper way and provides a retrieval function of the stored device information for the user by the user interaction module.
The distributed equipment information acquisition system comprises a node management module and a node detection module, wherein the node management module is composed of management nodes, the node detection module is composed of detection nodes, and the management nodes and the detection nodes are communicated through a distributed task scheduling framework; a node management module: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node; a node detection module: and receiving the acquisition subtask, executing equipment information acquisition to obtain an equipment information acquisition result, and uploading the equipment information acquisition result to the management node.
Specifically, the management node includes a user interaction module: acquiring an acquisition task through interaction with a user, providing equipment information query, and providing a system management and equipment information query function for the user; a task segmentation module: dividing the collection task by adopting a segmentation algorithm to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management module; a task management module: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling; the data receiving and managing module: and receiving the equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query.
Specifically, the probing node module includes a scheduling communication module: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node; a port scanning module: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port; the device information detection module: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
Specifically, the data of the distributed task scheduling includes an information acquisition subtask and an equipment information acquisition result, wherein the information acquisition subtask is transmitted from the management node to the detection node, and the equipment information acquisition result is transmitted from the detection node to the management node.
The distributed equipment information acquisition system provided by the invention can be realized through the step flow of the distributed equipment information acquisition method. The distributed device information acquisition method can be understood as a preferred example of the distributed device information acquisition system by those skilled in the art.
Preferred embodiments of the present invention are further described below.
As shown in fig. 1, the distributed device information collection system is composed of a plurality of nodes, the nodes are divided into a management node and a detection node according to specific functions, and the nodes exchange data through message channels to cooperatively work. The management node acquires an equipment information acquisition task and transmits the divided subtasks to the detection node to complete the task; and the detection node collects the equipment information according to the issued subtasks and returns the result to the management node. Task scheduling and data communication between the management node and the detection node are realized by adopting a distributed task scheduling framework, and preferably, a Gearman distributed task scheduling framework is adopted.
The number of the management nodes and the number of the detection nodes in the distributed device information acquisition system are not limited, and preferably, one management node and a plurality of detection nodes are set. Meanwhile, each detection node in the equipment information acquisition system is generally positioned on different hosts, and the management node can independently operate on one host and can also share one host with a certain detection node.
As shown in fig. 2, the distributed device information acquisition method includes the following six steps, as shown in the flowchart:
step one, a management node acquires an IP address range and a port range of equipment information acquisition. Preferably, through a user interaction module of the management node, the user stores the device information acquisition task into the relational database through a WEB interface, the management node maintains the working state of each detection node, and when the detection node is in an idle state, the to-be-executed task stored in the relational database is taken out to perform the specific device information acquisition task.
And step two, the management node performs task segmentation on the equipment information acquisition task. The device information acquisition task to be executed generally relates to a larger target range, so that the target range of one task is divided into a plurality of subtasks with smaller target ranges, and the completion speed of the device information acquisition task is increased in a mode that a plurality of detection nodes work in parallel. Meanwhile, in order to avoid that the scanning speed of the system to a network segment is too fast to influence the normal operation of a target network or to cause the detection node to be blocked, a randomization strategy is introduced to the scanning target sequence in the task segmentation algorithm. Preferably, the black rock algorithm is adopted as a random rearrangement algorithm in task segmentation. This algorithm has the following properties: let X ∈ X be an element in the sequentially arranged set X, Y ∈ Y be an element in the rearranged set Y, and X and Y are bijective relationships, and a value Y ═ f (X) corresponding to the rearranged position X can be obtained by a random rearrangement algorithm. Assuming that the IP address segments to be scanned are IP 1-IP 2, the range of the scanned ports is port 1-port 2, the number of scanned targets of each sub-task after task division is c, and the seed value of the black rock algorithm is r, then one divided sub-task is expressed as: { ip1, ip2, port1, port2, i × c, (i +1) × c, r }, where i is the sequence number of the current subtask and ranges from 0,1,2 …, (ip2-ip1+1) ((port 2-port1+ 1)/c). After a certain detection node acquires a subtask, the IP and the port of actual scanning detection are calculated by using a random rearrangement algorithm. The range of the random rearrangement algorithm is (IP2-IP1+1) ((port 2-port1+1), x ∈ [ i × c, (i +1) × c), and the corresponding y ═ f (x) is calculated by the random rearrangement algorithm, so that the scan target IP is (IP1+ y/(port2-port1+1)) and the target port is (port1+ y% (port2-port1+ 1)).
And step three, the management node issues the divided subtasks to the detection node. And the management node manages the divided subtasks and the execution state thereof and distributes the unexecuted subtasks to the idle detection nodes for execution in a planned way. The management node manages the divided subtasks by adopting two task queues, namely a running task queue (running _ tasks) and a waiting task queue (waiting _ tasks), wherein the two task queues are empty in an initial system state, and the subtasks in the running _ tasks queue are deleted from the queues if the execution is finished. The work processing flow of the subtask queue is as follows: step 1, judging whether running _ tasks are not full, if so, entering step 2, otherwise, entering step 6; step 2, judging whether the waiting _ tasks queue is empty, if so, entering step 3, otherwise, entering step 5; step 3, judging whether an equipment information detection task to be executed issued by a user exists, if so, entering step 4, otherwise, entering step 6; step 4, dividing the task to be executed into a plurality of subtasks, and adding the divided subtasks into a waiting _ tasks queue; step 5, supplementing the subtasks in the waiting _ tasks queue to the running _ tasks queue, and sending the subtasks to the detection node for execution; and 6, repeating the step 1.
Preferably, the length of the queue of running tasks is the same as the number of probe nodes. And when the length of the rounding _ tasks queue is not full or a gap occurs due to the completion of task execution, supplementing the subtasks in the waiting _ tasks queue to the rounding _ tasks queue, and issuing the subtasks to the detection node for execution. And if the running _ tasks queue is empty and the waiting _ tasks queue has no subtasks to be executed, repeating the step of acquiring the to-be-executed equipment information detection task issued by the user in the step one.
And step four, the detection node receives the subtasks and carries out port scanning. When the port detection is carried out, two threads are operated simultaneously, one thread calculates a detection target from subtasks according to a random rearrangement algorithm when the tasks are divided, a detection SYN message is constructed and sent asynchronously, and the other thread is responsible for monitoring and identifying a legal SYN + ACK message and collecting an open port detection result. After the message sending thread finishes sending, the receiving thread delays for a period of time to wait for receiving the response of the detection message, and preferably, the delay time is set to be 8 seconds; and (3) identifying the SYN + ACK message of the port detection response: when constructing the detection message, data of a plurality of key fields, namely a source IP, a source port, a destination IP and a destination port, during port detection are spliced in sequence, and the spliced data is hashed by using a hash algorithm. Preferably, the Murmur3 hash algorithm is adopted to hash the spliced data, 32-bit output is obtained, and the output is filled into the TCP sequence number field; for the received SYN + ACK message, if the field values of a destination IP, a destination port, a source IP and a source port are sequentially spliced, and the result value obtained by the Murmur3 hash algorithm with the same seed is that the field value of the TCP acknowledgement number of the message is minus 1, the message is considered as a legal response message, and the message is recorded as an open port; and (3) carrying out duplicate removal check on port scanning results: the detection node stores the port opening result in a form of a single-direction linked list, before the port scanning receiving thread temporarily stores the result in the linked list in the memory, the existing result in the linked list is checked at first, and the current opening port is stored in the linked list only when the condition that the current opening port does not exist in the linked list is confirmed.
And step five, acquiring equipment information by the detection node. The detection node firstly calls an application protocol probe to determine an application layer protocol operated by the open port, then constructs a corresponding detection request according to the determined application layer protocol format, receives response data returned by the target port, and adopts an analysis method corresponding to the determined application layer protocol to extract equipment information from the response data. Preferably, the probing node maintains an application protocol probe library, establishes a mapping table between each application protocol probe and a port commonly used by the application protocol, and searches for a probe from the mapping table according to a port number for each port in an open state filtered by the step five-port scanning. If the corresponding probe is not found in the mapping table or the application protocol operated by the open port does not belong to the probe identification type, traversing each probe of the application protocol probe library, and if the protocol type can not be determined, skipping the processing of the port. In the embodiment, the application protocol probe library is expandable, and the application protocol identification capability of the probe node can be expanded by adding the probe. Optionally, because the protocol probe, the probe request constructing method and the response data parsing method are all related to a certain application protocol or a certain type of application protocol, the protocol probe, the probe request constructing method and the response data parsing method may be integrated into a probe plug-in, and the probe plug-in is responsible for invoking.
And step six, the detection node returns the equipment information acquisition result corresponding to the subtask to the management node. Preferably, the scheduling communication module of the probe node transmits the acquisition result in the JSON format, and the management node stores the acquisition result in the JSON format in the NoSQL-type database in a suitable manner. Through the user interaction module of the management node, the user can retrieve the stored equipment information through the WEB interface.
Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (6)
1. A distributed equipment information acquisition system is characterized by comprising a node management module and a node detection module, wherein the node management module is composed of management nodes, the node detection module is composed of detection nodes, and the management nodes and the detection nodes are communicated through a distributed task scheduling framework;
a node management module: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node;
a node detection module: receiving the collection subtask, executing equipment information collection to obtain an equipment information collection result, and uploading the equipment information collection result to the management node;
the management node includes:
a user interaction module: acquiring an acquisition task and providing equipment information query by interacting with a user;
a task segmentation module: dividing the collection task to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management module;
a task management module: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling;
the data receiving and managing module: receiving an equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query;
in order to avoid that the scanning speed of a network segment is too fast to influence the normal operation of a target network or to cause the detection node to be blocked, a randomization strategy is introduced to the sequence of a scanning target in a task segmentation algorithm, a black rock algorithm is adopted as a random rearrangement algorithm in the task segmentation, X belongs to X as an element in a set X which is sequentially arranged, Y belongs to Y as an element in a set Y which is rearranged, X and Y are bijective relations, and a value Y corresponding to the position X after rearrangement is obtained through the random rearrangement algorithm, wherein f (X); assuming that the IP address segments to be scanned are IP 1-IP 2, the range of the scanned ports is port 1-port 2, the number of scanned targets of each sub-task after task division is c, and the seed value of the black rock algorithm is r, then one divided sub-task is expressed as: { ip1, ip2, port1, port2, i × c, (i +1) × c, r }, where i is the sequence number of the current subtask and ranges from 0,1,2 …, (ip2-ip1+1) (port2-port1+ 1)/c; after a certain detection node obtains a subtask, calculating the IP and the port of actual scanning detection by using a random rearrangement algorithm; the range of the random rearrangement algorithm is (IP2-IP1+1) ((port 2-port1+1), x belongs to [ i × c, (i +1) × c), and the corresponding y is calculated by the random rearrangement algorithm to be f (x), so that the scanning target IP is (IP1+ y/(port2-port1+1)), and the target port is (port1+ y% (port2-port1+ 1));
networking equipment in a large-scale network address range is subjected to rapid information acquisition, the situation that IP is forbidden due to abnormal flow judged by a target network segment is avoided, and efficient and accurate equipment information acquisition is realized;
and distributed scheduling and cooperative working modes of the management nodes and the detection nodes are adopted, so that the scattered acquisition and the unified storage management of the equipment information are realized.
2. The distributed device information collection system of claim 1, wherein the node detection module comprises:
a scheduling communication module: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node;
a port scanning module: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port;
the device information detection module: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
3. A distributed equipment information acquisition method is characterized by comprising the following steps:
a node management step: acquiring an acquisition task through user interaction, dividing the acquisition task to obtain an acquisition subtask, distributing the acquisition subtask to a detection node, and receiving an equipment information acquisition result uploaded by the detection node;
a node detection step: receiving the collection subtask, executing equipment information collection to obtain an equipment information collection result, and uploading the equipment information collection result to the management node;
the node management step includes:
a user interaction step: acquiring an acquisition task and providing equipment information query by interacting with a user;
task segmentation step: dividing the collection task to obtain a plurality of collection subtasks, and sending the collection subtasks to a task management step;
and task management: monitoring node states of a plurality of detection nodes, monitoring task execution states of acquisition subtasks, and distributing the acquisition subtasks to be executed to the detection nodes by adopting distributed scheduling;
data receiving and managing: receiving an equipment information acquisition result uploaded by the detection node, storing the equipment information acquisition result to a database, and providing a data basis for equipment information query;
in order to avoid that the scanning speed of a network segment is too fast to influence the normal operation of a target network or to cause the detection node to be blocked, a randomization strategy is introduced to the sequence of a scanning target in a task segmentation algorithm, a black rock algorithm is adopted as a random rearrangement algorithm in the task segmentation, X belongs to X as an element in a set X which is sequentially arranged, Y belongs to Y as an element in a set Y which is rearranged, X and Y are bijective relations, and a value Y corresponding to the position X after rearrangement is obtained through the random rearrangement algorithm, wherein f (X); assuming that the IP address segments to be scanned are IP 1-IP 2, the range of the scanned ports is port 1-port 2, the number of scanned targets of each sub-task after task division is c, and the seed value of the black rock algorithm is r, then one divided sub-task is expressed as: { ip1, ip2, port1, port2, i × c, (i +1) × c, r }, where i is the sequence number of the current subtask and ranges from 0,1,2 …, (ip2-ip1+1) (port2-port1+ 1)/c; after a certain detection node obtains a subtask, calculating the IP and the port of actual scanning detection by using a random rearrangement algorithm; the range of the random rearrangement algorithm is (IP2-IP1+1) ((port 2-port1+1), x belongs to [ i × c, (i +1) × c), and the corresponding y is calculated by the random rearrangement algorithm to be f (x), so that the scanning target IP is (IP1+ y/(port2-port1+1)), and the target port is (port1+ y% (port2-port1+ 1));
networking equipment in a large-scale network address range is subjected to rapid information acquisition, the situation that IP is forbidden due to abnormal flow judged by a target network segment is avoided, and efficient and accurate equipment information acquisition is realized;
and distributed scheduling and cooperative working modes of the management nodes and the detection nodes are adopted, so that the scattered acquisition and the unified storage management of the equipment information are realized.
4. The distributed device information collection method according to claim 3, wherein the node detection step includes:
and scheduling communication: receiving an acquisition subtask issued by a management node, and uploading an equipment information acquisition result to the management node;
port scanning step: analyzing the collection subtask to obtain an IP address and port information, and scanning the IP address and the port information to obtain an open port;
a device information detection step: and sending the equipment detection request to the open port, and analyzing response data sent by the open port to obtain an equipment information acquisition result.
5. The distributed device information collection system according to claim 1 or the distributed device information collection method according to claim 3, wherein the distribution collection subtask manages a collection subtask through two sets of task queues;
the task queue mainly comprises a task queue which is being executed and a task queue waiting to be executed;
the collection subtasks which are being executed are stored in the task queue which is being executed, and the collection subtasks which are waiting to be executed are stored in the task queue which is waiting to be executed.
6. The distributed device information collection system according to claim 1 or the distributed device information collection method according to claim 3, wherein the execution device information collection is in a multi-threaded asynchronous manner;
the multithreading asynchronous mode mainly comprises a sending thread and a receiving thread;
the sending thread is used for calculating a detection target for equipment information acquisition, and constructing and sending a detection request; the receiving thread is used for monitoring a detection target and acquiring a result of equipment information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811185588.7A CN109586947B (en) | 2018-10-11 | 2018-10-11 | Distributed equipment information acquisition system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811185588.7A CN109586947B (en) | 2018-10-11 | 2018-10-11 | Distributed equipment information acquisition system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109586947A CN109586947A (en) | 2019-04-05 |
| CN109586947B true CN109586947B (en) | 2020-12-22 |
Family
ID=65920258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811185588.7A Active CN109586947B (en) | 2018-10-11 | 2018-10-11 | Distributed equipment information acquisition system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109586947B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112003758A (en) * | 2019-05-27 | 2020-11-27 | 北京白帽汇科技有限公司 | Method for identifying object characteristics in network space mapping process |
| CN111309830A (en) * | 2019-07-10 | 2020-06-19 | 浙江商安信息科技有限公司 | Method, system and storage medium for obtaining meal ordering shop information based on electronic map |
| CN110532060B (en) * | 2019-08-10 | 2022-05-03 | 佳都科技集团股份有限公司 | Hybrid network environment data acquisition method and system |
| CN110516738B (en) * | 2019-08-23 | 2022-09-16 | 佳都科技集团股份有限公司 | Distributed comparison clustering method and device, electronic equipment and storage medium |
| CN110955497B (en) * | 2019-11-04 | 2023-03-31 | 浙江工业大学 | Distributed graph computing system based on task segmentation |
| CN111369773A (en) * | 2020-02-14 | 2020-07-03 | 国网新疆电力有限公司电力科学研究院 | Electricity price parameter issuing method and system based on transparent task |
| CN111597053A (en) * | 2020-05-29 | 2020-08-28 | 广州万灵数据科技有限公司 | Cooperative operation and self-adaptive distributed computing engine |
| CN113852475A (en) * | 2020-06-28 | 2021-12-28 | 京东方科技集团股份有限公司 | Operation and maintenance method and system and master control node |
| CN111835596B (en) * | 2020-07-21 | 2022-03-18 | 北京长亭未来科技有限公司 | Target state detection method and device and electronic equipment |
| CN112436979A (en) * | 2020-11-04 | 2021-03-02 | 深圳供电局有限公司 | Cloud network traffic acquisition method and system |
| CN113938404B (en) * | 2021-10-12 | 2023-04-07 | 北京恒安嘉新安全技术有限公司 | Asset detection method, device, equipment, system and storage medium |
| CN115296891B (en) * | 2022-08-02 | 2023-12-22 | 中国电子科技集团公司信息科学研究院 | Data detection system and data detection method |
| CN115296888B (en) * | 2022-08-02 | 2023-11-17 | 中国电子科技集团公司信息科学研究院 | Data Radar Monitoring System |
| CN115296892B (en) * | 2022-08-02 | 2023-11-24 | 中国电子科技集团公司信息科学研究院 | Data information service system |
| CN115914328B (en) * | 2023-01-30 | 2023-06-23 | 天翼云科技有限公司 | Network health detection method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801778A (en) * | 2005-12-29 | 2006-07-12 | 中山大学 | Multidimensional queue dispatching and managing system for network data stream |
| CN102075365A (en) * | 2011-02-15 | 2011-05-25 | 中国工商银行股份有限公司 | Method and device for locating and protecting network attack source |
| CN106649732A (en) * | 2016-12-23 | 2017-05-10 | 金蝶软件（中国）有限公司 | Information pushing method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102375837B (en) * | 2010-08-19 | 2014-12-31 | 中国移动通信集团公司 | Data acquiring system and method |
| CN103384205B (en) * | 2012-05-02 | 2016-05-25 | 中国科学院计算机网络信息中心 | A kind of mass alarm data parallel acquisition system, device and method |
| CN105471950A (en) * | 2014-09-05 | 2016-04-06 | 鸿富锦精密工业(武汉)有限公司 | Distributed computing method and system |
| CN104735138B (en) * | 2015-03-09 | 2018-01-09 | 中国科学院计算技术研究所 | A kind of distributed acquisition method and system of user oriented generation content |
-
2018
- 2018-10-11 CN CN201811185588.7A patent/CN109586947B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801778A (en) * | 2005-12-29 | 2006-07-12 | 中山大学 | Multidimensional queue dispatching and managing system for network data stream |
| CN102075365A (en) * | 2011-02-15 | 2011-05-25 | 中国工商银行股份有限公司 | Method and device for locating and protecting network attack source |
| CN106649732A (en) * | 2016-12-23 | 2017-05-10 | 金蝶软件（中国）有限公司 | Information pushing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109586947A (en) | 2019-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109586947B (en) | Distributed equipment information acquisition system and method | |
| CN108667725B (en) | Industrial soft gateway based on multiple accesses and edge calculation and implementation method | |
| CN105631026B (en) | Safety data analysis system | |
| CN104794047B (en) | Method and system for correlation analysis of performance indicators | |
| US5819299A (en) | Process for distributed garbage collection | |
| CN107302450B (en) | Data acquisition task changing method and device | |
| DE112011101321B4 (en) | Querying performance data on a parallel computer system having compute nodes | |
| CN108737213B (en) | High-parallelism and high-throughput penetration test system and method based on FPGA | |
| US8797876B2 (en) | Identification of underutilized network devices | |
| US20050097300A1 (en) | Processing system and method including a dedicated collective offload engine providing collective processing in a distributed computing environment | |
| US10498817B1 (en) | Performance tuning in distributed computing systems | |
| US9733997B2 (en) | Event management method and distributed system | |
| CN110020046A (en) | A kind of data grab method and device | |
| US7899906B2 (en) | Multi-layered measurement model for data collection and method for data collection using same | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN109361576A (en) | A kind of PIM monitoring data processing method and system | |
| US10558391B2 (en) | Data processing system and data processing method | |
| CN109800081A (en) | A kind of management method and relevant device of big data task | |
| CN107436904A (en) | Data capture method and its equipment | |
| CN112559525A (en) | Data checking system, method, device and server | |
| US20220284043A1 (en) | Data analytical processing apparatus, data analytical processing method, and data analytical processing program | |
| CN115292176A (en) | Pressure testing method, device, equipment and storage medium | |
| CN109947798A (en) | A kind of processing method and processing device of stream event | |
| Alom et al. | Optimization of detected deadlock views of distributed database | |
| JP5530878B2 (en) | Data replication management method in distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |