CN106126346B - A kind of large-scale distributed data collection system and method - Google Patents
A kind of large-scale distributed data collection system and method Download PDFInfo
- Publication number
- CN106126346B CN106126346B CN201610522950.XA CN201610522950A CN106126346B CN 106126346 B CN106126346 B CN 106126346B CN 201610522950 A CN201610522950 A CN 201610522950A CN 106126346 B CN106126346 B CN 106126346B
- Authority
- CN
- China
- Prior art keywords
- node
- task
- acquisition
- online
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/501—Performance criteria
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention proposes a kind of large-scale distributed data collection system and method, belongs to data acquisition technology field, which includes task allocation node module, management node module, task warehouse module and acquisition node module；Management node receives and manages the registration information of acquisition node and task allocation node for providing distributed collaboration service, communicated according to certain frequency with acquisition node and task allocation node；Task allocation node is for setting acquisition tasks, being distributed and being monitored；Task warehouse is for receiving and storing the acquisition tasks that task allocation node is sent；Acquisition node is used to obtain acquisition tasks from task input warehouse, and completes the acquisition to corresponding data in field control station；The acquisition tasks that acquisition node executes can set dynamic change according to user, some or certain several failures in task allocation node and acquisition node will not influence current collection period collection result, have high flexibility, high fault tolerance and high reliability.
Description
Technical field
The invention belongs to data acquisition technology fields, and in particular to a kind of large-scale distributed data collection system and side
Method.
Background technique
With greatly increasing for the development of industrial enterprise, production scale expansion and data collection capacity, data are adopted by enterprise
The requirement for collecting reliability is higher and higher, guarantees that the difficulty of acquisition reliability is also increasing.
The patent in terms of large-scale distributed data acquisition is mainly that " (one kind is based on industry by CN105527948A at present
The large-scale distributed data collection system and method for process) ", which is multiple numbers using a data collection scheduling module
Acquisition tasks are distributed according to acquisition client, when acquiring client failure, next collection period utilizes collection scheduling mould
The mode that block carries out readjustment degree guarantees the reliability of next period and the acquisition of later time after failure.Although the patent is in certain journey
It can be improved the reliability of acquisition on degree, but still have certain limitation in collection process:
1. in the patent, collection scheduling module adopts acquisition client distribution according to the running state information of field control station
Set task, performed acquisition can only be carried out as unit of field control station, and can not be acquired as unit of data item,
And all acquisition tasks do not distinguish priority, acquisition flexibility is not high enough.For needing to acquire certain in all field control stations
Occasion or each acquisition tasks that one or a few item datas are analyzed are divided into the occasion of significance level height, this is special
Benefit is unable to satisfy requirement.
2. system described in the patent lacks independent coordinated management module to carry out accordingly node each in distributed system
Management and automatic load balancing, when needing multiple collection scheduling modules, system is difficult to realize dynamically extend, system spirit
Activity is not high enough.
3. the patent when acquiring client failure, only can guarantee next collection period normal acquisition, current for failure
Collection period does not provide good disaster tolerance treatment mechanism to guarantee to acquire reliability and fault-tolerance, and therefore, failure is once sent out
Raw, the data that the moment occurs for failure are easily lost, and the production occasion high for acquisition reliability requirement, this loss equally can be right
Enterprise causes certain economic loss.In addition, the patent distributes acquisition client only with a data collection scheduling module
Acquisition tasks, be not acquired scheduler module disaster tolerance processing, and the distribution of all acquisition tasks and to acquisition client
Management function be focused on a data collection scheduling module, function is more concentrated, once the collection scheduling module failure, whole
A system will be unable to work normally.
4. in the patent, collection scheduling module is directly interacted with all acquisition clients, the acquisition of distribution is appointed
Business is transmitted directly to all acquisition clients.When using multiple collection scheduling modules, collection scheduling module and acquisition client
Hold coupling it is larger, communicate therebetween it is lengthy and jumbled, when multiple collection scheduling modules to the same acquisition client transmission task when,
The time that the task of each collection scheduling module reaches acquisition client is different, and acquisition client waits the time of task distribution not
It determines, longer so as to cause the acquisition client waiting time, system effectiveness reduces.
In conclusion existing patent is in terms of data acquisition function, flexibility, high efficiency and guarantee data acquisition reliability
There are obvious deficiencies, it is difficult to meet currently manufactured enterprise into intelligent transformation process to real-time data acquisition the needs of.
Summary of the invention
In view of the deficiencies of the prior art, the present invention proposes a kind of large-scale distributed data collection system and method, to reach
The reliability of flexibility and scalability, raising management node and task warehouse to the system of raising improves task allocation node
Fault-tolerance to guarantee acquisition tasks distribution high reliability, improve acquisition node fault-tolerance to guarantee present sample week
The purpose for the long-term reliability that the fault-tolerance and high reliability and guarantee acquisition tasks of the data acquisition of phase execute；
A kind of large-scale distributed data collection system, the system include task allocation node module, management node module,
Task warehouse module and acquisition node module；
The task allocation node module includes multiple tasks distribution node；The management node module includes management
Node；The task warehouse module includes task warehouse；The acquisition node module includes multiple acquisition nodes；Wherein,
Task allocation node: the task for acquiring needed for setting obtains online acquisition node from management node in real time
List and other online task allocation node lists, to obtain the running state information of each online acquisition node and initial
The operating status threshold value that it is arranged follows acquisition tasks set by user one by one on each online task allocation node
Ring is divided equally, and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, successively will
Acquisition tasks are distributed to acquisition node online and in normal operating condition, and the acquisition tasks of target acquisition node will be distributed to
Lower two numbers online acquisition node adjacent thereto is given as backup cycle assignment, allocation result is sent to task warehouse
In；Distribution, execution to acquisition tasks on respective node in real time simultaneously and waiting situation are monitored；
Management node: for providing distributed collaboration service, receive and manage the note of acquisition node and task allocation node
Volume information, is numbered acquisition node according to registration order, is communicated according to certain frequency with acquisition node to obtain
Online acquisition node；Task allocation node is numbered according to registration order, according to certain frequency and task allocation node
It is communicated to obtain online task allocation node；
Task warehouse: for receiving the acquisition tasks of task allocation node transmission；
Acquisition node: it for obtaining other online acquisition node listings from management node, is obtained from task warehouse respective
The task of required acquisition, and be acquired according to data of the priority orders of acquisition tasks to field control station, acquisition is tied
Fruit is stored in database.
The task list of required acquisition is synchronized to other online task allocation nodes by the task allocation node
In.
The task allocation node includes task setting module, task allocating module and task monitoring module, wherein
Task setting module: for obtaining other online task allocation node lists in real time from management node, according to reality
Being acquired needed for demand setting for task, and the acquisition tasks in task list are recycled one by one and give all online task distribution
The task allocating module of node；
Task allocating module: for obtaining online acquisition node list and other online tasks in real time from management node
Distribution node list, thus the operating status threshold for obtaining the running state information of each online acquisition node and initially it being arranged
Value, according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, successively by acquisition tasks
It distributes to acquisition node online and in normal operating condition, and the acquisition tasks of target acquisition node will be distributed to as standby
Part cycle assignment gives lower two numbers online acquisition node adjacent thereto, sends task warehouse for allocation result；
Task monitoring module: for the distribution, execution in real time to acquisition tasks on respective node and situation is waited to supervise
Control.
The management node module, further includes management switch, and the management node uses redundant configuration mode, packet
Include main management node and standby management node；
Management switch: for capturing the fault message of main management node, realize main management node and standby management node it
Between switching and main management node and standby management node between data it is synchronous.
The task warehouse module, further includes task switcher, and the task warehouse uses redundant configuration mode, packet
Include main task warehouse and standby task warehouse；
Task switcher: for capturing the fault message in main task warehouse, realize main task warehouse and standby task warehouse it
Between switching and main task warehouse and standby task warehouse between data it is synchronous.
The collecting method carried out using large-scale distributed data collection system, comprising the following steps:
Step 1 carries out Initialize installation to system；
Step 2, the quantity and field device for configuring required field control station according to actual needs, and successively open and matched
The field device and field control station set；
Step 3 opens management node and task warehouse, and successively opens each acquisition node and task allocation node；
Registration information is sent from trend management node after step 4, the unlatching of each acquisition node, management node is suitable according to registering
Ordered pair acquisition node is numbered, and is communicated according to certain frequency with acquisition node to obtain online acquisition node；
Registration information is sent from trend management node after step 5, the unlatching of each task allocation node, management node is according to note
Task allocation node is numbered in volume sequence, is communicated according to certain frequency with task allocation node to obtain online appoint
Business distribution node；
Step 6: it is online that the task setting module of each online task allocation node obtains other in real time from management node respectively
Task allocation node list, task allocating module obtains online acquisition node list and other in real time from management node respectively
Online task allocation node list, each online acquisition node obtain other online acquisition nodes in real time from management node respectively
List；
Step 7, according to actual needs is set using the task setting module inside any one online task allocation node
The task of required acquisition, and the task that the acquisition tasks in task list are evenly distributed to each online task allocation node is divided
With module；
Task allocating module inside step 8, each online task allocation node obtains the operation shape of each online acquisition node
State information and the operating status threshold value that initially it is arranged, the priority of acquisition tasks according to acquired in it and each are adopted online
Collect the running state information of node, successively distributes acquisition tasks to acquisition node online and in normal operating condition, and
The acquisition tasks of target acquisition node will be distributed to as backup cycle assignment to lower two number online acquisitions adjacent thereto
Allocation result is sent in task warehouse by node；Task monitoring module simultaneously inside each online task allocation node is real-time
Distribution, execution and waiting situation to acquisition tasks on respective node are monitored；
The task that step 9, each online acquisition node acquire needed for obtaining from task warehouse respectively respectively；
Step 10, each online acquisition node adopt the data of field control station according to the priority orders of acquisition tasks
Collection, and collection result is stored in database.
Initialize installation is carried out to system described in step 1；Specifically: setting main management node, standby management node, each
Be engaged in distribution node, the IP address and port numbers in main task warehouse, standby task warehouse and each acquisition node, and be arranged clock it is synchronous and
The operating status threshold value of each acquisition node.
Described in step 7 according to actual needs, using the task setting mould inside any one online task allocation node
Being acquired needed for block setting for task, and the acquisition tasks in task list are averagely allocated to each online task allocation node
Task allocating module；Specific step is as follows:
Step 7-1, it according to actual needs, is set using the task setting module inside any one online task allocation node
Being acquired needed for fixed for task；
Step 7-2, the acquisition tasks in task list are recycled one by one and gives appointing for each online task allocation node
Business distribution module, until acquisition tasks all in task list are distributed on task allocating module.
Task allocating module inside each online task allocation node described in step 8 obtains the fortune of each online acquisition node
Row status information and the operating status threshold value that initially it is arranged, the priority of acquisition tasks according to acquired in it, and respectively exist
The running state information of line acquisition node successively distributes acquisition tasks to acquisition section online and in normal operating condition
Point, and the acquisition tasks for distributing to target acquisition node are online to lower two numbers adjacent thereto as backup cycle assignment
Allocation result is sent in task warehouse by acquisition node；Task monitoring module inside each online task allocation node simultaneously
Distribution, execution to acquisition tasks on respective node in real time and waiting situation are monitored；Specific step is as follows:
Step 8-1, the acquisition tasks that the task allocating module of each online task allocation node respectively gets it by
Be lined up according to priority, and according to online acquisition node listing, the running state information of each online acquisition node of poll and
The operating status threshold value that initially it is arranged；
Step 8-2, by comparing the running state information of each online acquisition node and the operation shape that initially it is arranged
State threshold value judges that each online acquisition node is up state or overlond running state, if at online acquisition node respectively
It is in normal operating condition, then online to this according to task priority sequence according to the running state information of the online acquisition node
Acquisition node distributes acquisition tasks；If online acquisition node is in overlond running state, which is not distributed
Acquisition tasks；
Step 8-3, the acquisition tasks of target acquisition node will be distributed to as backup cycle assignment under adjacent thereto
Two number online acquisition nodes；
Step 8-4, its allocation result is sent to task storehouse respectively by the task allocating module of each online task allocation node
In library；
Step 8-5, the task monitoring module of each online task allocation node is respectively to acquisition tasks on respective node
Distribution executes and situation is waited to be monitored；
Step 8-6, task allocation node breaks down if it exists, then above-mentioned task allocation node is off-line state, other
Online task allocation node obtains updated online task allocation node list, task setting module from management node in real time
Acquisition tasks in task list are recycled to the task allocating module for giving each online task allocation node, Zhi Daoren one by one
Until all acquisition tasks are distributed on task allocating module in business list, 8-1 is returned to step.
Each online acquisition node described in step 10 is according to the priority orders of acquisition tasks to the data of field control station
It is acquired, and collection result is stored in database；Specifically:
Step 10-1, each online acquisition node carries out the data of field control station according to the priority orders of acquisition tasks
Acquisition, under normal circumstances, the acquisition tasks that the lower two number online acquisition node adjacent with target acquisition node is backed up are only
Make local cache, is not stored in database；
Step 10-2, when there are acquisition node failure, which will be in off-line state, in current sample period
Interior, the collection result for backing up acquisition tasks is stored in database from local cache area by first adjacent number online acquisition node
In；If first adjacent at this time number online acquisition node also breaks down, second adjacent number online acquisition section
Point will back up the collection result of acquisition tasks from local cache area deposit database；
Step 10-3, when starting next sampling period, each online task allocation node is got by management node
Updated online acquisition node listing；Acquisition tasks are distributed into online acquisition node, phase according to priority orders again
The online acquisition node answered executes the acquisition tasks of its acquisition and implementing result is stored in database.
The invention has the advantages that
The present invention proposes that a kind of large-scale distributed data collection system and method, user can be created according to demand at any time
Acquisition tasks are built, modified and deleted, and different acquisition task can be arranged according to actual needs different priority and sampling
Period；Acquisition tasks are allocated and are dispatched by task allocation node, acquisition node data collected are no longer fixed, and have
There is certain flexibility；Management node is added as independent coordinated management module, so that task allocation node and acquisition node
It can dynamically increase and delete, substantially increase the flexibility and scalability of system；When some or certain several tasks
When distribution node failure, other online task allocation nodes can obtain in real time updated online task point from management node
With node listing, current online appoint is given so that the acquisition tasks in the task list of task setting module be recirculated
Business distribution node, improves the fault-tolerance of task allocation node, to ensure that the high reliability of acquisition tasks distribution.When a certain
When a or certain several acquisition node failure, since there are the backup of acquisition tasks, standby acquisition node is detected by management node
The collection result that main acquisition node corresponds to acquisition tasks is stored in database, ensure that present sample week by main acquisition node failure
The fault-tolerance and reliability of the data acquisition of phase；Since next collection period, task allocation node is obtained by management node
Updated online acquisition node listing is got, acquisition tasks are distributed into online acquisition section according to priority orders again
Point, and corresponding online acquisition node is called to execute acquisition tasks, to ensure that the long-term reliability that acquisition tasks execute；Pipe
It manages node and uses redundant configuration mode with task warehouse, improve the reliability of management node and task warehouse.
Detailed description of the invention
Fig. 1 is the large-scale distributed data collection system structural schematic diagram of an embodiment of the present invention；
Fig. 2 is the management node modular structure schematic diagram of an embodiment of the present invention；
Fig. 3 is the task warehouse modular structure schematic diagram of an embodiment of the present invention；
Fig. 4 is the task allocation node structural schematic diagram of an embodiment of the present invention；
Fig. 5 is the large-scale distributed collecting method flow chart of an embodiment of the present invention.
Specific embodiment
An embodiment of the present invention is described further with reference to the accompanying drawing.
As shown in Figure 1, large-scale distributed data collection system include: task allocation node module, management node module,
Task warehouse module and acquisition node module；In the embodiment of the present invention, the task allocation node module includes 3 tasks point
With node；The management node module includes management node；The task warehouse module includes task warehouse；Described adopts
Collecting node module includes 3 acquisition nodes；In addition, field device is 9 groups of wind-driven generators, for generating the dependency number of blower
According to；The title of 9 groups of wind-driven generators is respectively as follows: F15, F16, F17, F18, F20, F21, F22, F23, F24；Field control station is
The PLC of 9 Foxboro companies, for connecting 9 groups of blowers；
In the embodiment of the present invention, as shown in Fig. 2, the management node module further includes management switch, the pipe
It manages node and uses redundant configuration mode, including main management node and standby management node；As shown in figure 3, the task warehouse mould
Block further includes task switcher, and the task warehouse uses redundant configuration mode, including main task warehouse and standby task warehouse；
In the embodiment of the present invention, using tide sea of clouds server virtualization system V4.0 industry cloud platform, for disposing master
Management node, standby management node, 3 task allocation nodes, main task warehouse, standby task warehouse and 3 acquisition nodes, and to it
Operating condition is monitored, and checks the operation conditions of each node, modifies online, the off-line operation state of each node；The master
Management node, standby management node, 3 task allocation nodes, main task warehouse, standby task warehouse run on Windows764
Operating system and memory size are on the virtual machine or virtual machine of 4G；3 acquisition nodes are run on
Windows764 bit manipulation system and memory size are on the virtual machine or virtual machine of 8G；The management switch and
Task switcher is based on 2014 development platform of MyEclipse and realizes exploitation using java language；
In the embodiment of the present invention, 3 task allocation nodes: the task for being acquired needed for setting, it is real-time from management node
Online acquisition node list and other online task allocation node lists are obtained, to obtain each online acquisition node
Running state information and the operating status threshold value that initially it is arranged；By acquisition tasks list synchronization set by user to other
In line task allocation node, the acquisition tasks in the task list are recycled one by one on each online task allocation node
Respectively, it and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, will successively adopt
Set task is distributed to acquisition node online and in normal operating condition, and the acquisition tasks that will distribute to target acquisition node
Lower two numbers online acquisition node adjacent thereto is given as backup cycle assignment, allocation result is sent to task warehouse
In；Distribution, execution to acquisition tasks on respective node in real time simultaneously and waiting situation are monitored；
In the embodiment of the present invention, as shown in figure 4, being further described: the task allocation node includes that task is set
Cover half block, task allocating module and task monitoring module, specifically: task setting module: for being obtained in real time from management node
Other online task allocation node lists, acquiring needed for setting according to actual needs for task, user can create, modify, delete
Except acquisition tasks, the acquisition data item that each acquisition tasks include is set, and to each acquisition tasks or each acquisition data item
Priority and sampling period are set, and the acquisition tasks in task list are recycled one by one and give all online task distribution sections
The task allocating module of point；Task allocating module: for obtaining online acquisition node list and other in real time from management node
Online task allocation node list, to obtain the running state information of each online acquisition node and initially it is arranged
Operating status threshold value, according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, successively
Acquisition tasks are distributed to acquisition node online and in normal operating condition, and the acquisition that will distribute to target acquisition node
Task gives lower two numbers online acquisition node adjacent thereto as backup cycle assignment, sends task storehouse for allocation result
Library；Task monitoring module: for the distribution, execution in real time to acquisition tasks on respective node and situation is waited to be monitored.
In the embodiment of the present invention, management node: for providing distributed collaboration service, receiving and managing 3 acquisition nodes
With the registration information of 3 task allocation nodes, acquisition node is numbered according to registration order, primary frequency is communicated according to 5s
Rate is communicated with acquisition node to obtain online acquisition node；Task allocation node is numbered according to registration order, is pressed
Primary frequency is communicated according to 5s to be communicated with task allocation node to obtain online task allocation node；
In the embodiment of the present invention, switch is managed: for capturing the fault message of main management node, realizing main management node
The data between switching and main management node and standby management node between standby management node are synchronous；Under normal circumstances, it acquires
Node and task allocation node communicate primary frequency with 5s and are led to main management node to main management node registration information
Letter, task allocation node obtain online acquisition node list and other online task allocation nodes from main management node in real time
List, acquisition node obtain other online acquisition node lists from main management node in real time；Standby management node is in real time from supervisor
It manages node and obtains synchrodata.When management switch captures the fault message of main management node, standby management node is as new
Main management node continue to provide coordination service, acquisition node and task allocation node will to standby management node registration information, and
It communicates primary frequency with 5s to be communicated with standby management node, task allocation node obtains online in real time from standby management node
Acquisition node list and other online task allocation node lists, acquisition node obtain other online from standby management node in real time
Acquisition node list；
In the embodiment of the present invention, task warehouse: for receiving the acquisition tasks of task allocation node transmission；
In the embodiment of the present invention, task switcher: for capturing the fault message in main task warehouse, main task warehouse is realized
The data between switching and main task warehouse and standby task warehouse between standby task warehouse are synchronous.Under normal circumstances, system
It is worked using main task warehouse, i.e., acquisition tasks allocation result is sent master by the task allocating module of task allocation node
In task warehouse, the task that acquisition node acquires needed for obtaining from main task warehouse respectively, standby task warehouse is in real time from director
Business warehouse obtains synchrodata.When task switcher, which captures main task warehouse, to break down, standby task warehouse will start work
Make, the task allocating module of task allocation node sends acquisition tasks allocation result in standby task warehouse, acquisition node from
Being acquired needed for being obtained in standby task warehouse respectively for task.
In the embodiment of the present invention, 3 acquisition nodes: for obtaining other online acquisition node listings from management node, from
Being acquired needed for being obtained in task warehouse respectively for task, and according to the priority orders of acquisition tasks to the data of field control station
It is acquired, collection result is stored in database.
In the embodiment of the present invention, the collecting method carried out using large-scale distributed data collection system, method stream
Journey figure is as shown in Figure 5, comprising the following steps:
Step 1 carries out Initialize installation to system, i.e. setting main management node, standby management node, 3 task distribution sections
The IP address and port numbers of point, main task warehouse, standby task warehouse and 3 acquisition nodes, and it is synchronous that Internet clock is arranged
With the operating status threshold value of 3 acquisition nodes；
In the embodiment of the present invention, the IP address of main management node is respectively configured in the configuration file in management node are as follows:
192.168.3.44；The IP address of standby management node are as follows: 192.168.3.45；Configure the IP address point of 3 task allocation nodes
Not are as follows: 192.168.3.46,192.168.3.47 and 192.168.3.48；Configure the IP address in main task warehouse are as follows:
192.168.3.49；The IP address in standby task warehouse is 192.168.3.50；The IP address for configuring 3 acquisition nodes is respectively as follows:
192.168.3.51,192.168.3.52 and 192.168.3.53；Configure main management node, standby management node, 3 task distribution
Node, main task warehouse, standby task warehouse, 3 acquisition nodes port numbers be 5000；
In the embodiment of the present invention, the operating status threshold value that 3 acquisition nodes are arranged is default value, including CPU is normally transported
Row state threshold is the 3%~70% of CPU total amount, CPU overload state threshold is CPU total amount 75%~80%, memory is normal
Operating status threshold value is the 20%~70% of memory amount, memory overload threshold value is the 75%~80% of memory amount, firmly
Disk normal operating condition threshold value is the 21%~70% of hard disk total amount, hard disk overload threshold value is hard disk total amount 75%~
80%, network bandwidth normal operating condition threshold value is the 2%~70% of network bandwidth total amount, network bandwidth overload threshold value is
The 75%~80% of network bandwidth total amount.
Step 2 successively opens 9 field control stations and 9 groups of blowers；
Step 3 opens management node and task warehouse, and successively opens 3 acquisition nodes and 3 task allocation nodes；
Registration information is sent from trend management node after step 4,3 acquisition nodes unlatchings, management node is suitable according to registering
Ordered pair acquisition node is numbered, and it is 1,2,3 that acquisition node is numbered respectively according to registration order, and waits task to be allocated；Management
Node communicates primary frequency according to 5s and is communicated with acquisition node to obtain online acquisition node；Specifically:
Step 4-1, management switch obtains the running state information of main management node in real time；When main management node is in just
When normal operating status, from trend main management node registration information after 3 acquisition nodes unlatchings, and it is in and waits task status, 3
Acquisition node and main management node are primary every 5s communication；When main management node is in failure operation state, 3 acquisition nodes
To standby management node registration information, and in task status is waited, 3 acquisition nodes and standby management node are every 5s communication one
It is secondary；
Step 4-2, management node gets online acquisition node automatically；When 3 acquisition nodes all successfully open it
Afterwards, management node will get 3 online acquisition nodes automatically；After 3 acquisition node parts are successfully opened, management node
The online acquisition node less than 3 will be got automatically；
From trend management node registration information after step 5,3 task allocation nodes unlatchings, management node is suitable according to registering
Ordered pair task allocation node is numbered, and it is 1,2,3 that task allocation node is numbered respectively according to registration order, management node according to
5s communicates primary frequency and is communicated with task allocation node to obtain online task allocation node；Specifically:
Step 5-1, when main management node is in normal operating condition, from trend master after 3 task allocation nodes unlatchings
Management node registration information, 3 task allocation nodes and main management node are primary every 5s communication；When main management node is in event
When hindering operating status, 3 task allocation nodes are to standby management node registration information, 3 task allocation nodes and for management node
It is primary every 5s communication；
Step 5-2, management node gets online task allocation node automatically；When 3 task allocation nodes all at
After function is opened, management node will get 3 online task allocation nodes automatically.When 3 task allocation node part successes
After unlatching, management node will get online task allocation node less than 3 automatically.
It is online that step 6, the task setting module of 3 online task allocation nodes obtain other in real time from management node respectively
Task allocation node list, task allocating module obtains online acquisition node list and other in real time from management node respectively
Online task allocation node list, 3 online acquisition nodes obtain other online acquisition sections in real time from management node respectively
Point list；
In the embodiment of the present invention, when main management node is in normal operating condition, times of 3 online task allocation nodes
Business setting module obtains other online task allocation node lists, task allocating module difference in real time from main management node respectively
Obtain online acquisition node list and other online task allocation node lists in real time from main management node, 3 are adopted online
Collection node obtains other online acquisition node lists in real time from main management node respectively；When main management node is in failure operation
When state, the task setting module of 3 task allocation nodes obtains other online tasks point from standby management node respectively in real time
With node listing, task allocating module obtains online acquisition node list in real time from standby management node respectively and other are online
Task allocation node list, 3 online acquisition nodes obtain other online acquisition node column from standby management node respectively in real time
Table；
Step 7, according to actual needs is set using the task setting module inside any one online task allocation node
The task of required acquisition, and the task that the acquisition tasks in task list are averagely allocated to 3 online task allocation nodes is divided
With module；It is specific as follows:
Step 7-1, user sets acquisition tasks in the task setting module of No. 1 task allocation node, that is, creates
Acquisition tasks are built, modified or deleted, set the acquisition data item that each acquisition tasks include, and to each acquisition tasks or each
The priority and sampling period for acquiring data item are set；
In the embodiment of the present invention, successively created altogether in the task setting module of No. 1 task allocation node F15, F16,
This 9 acquisition tasks of F17, F18, F20, F21, F22, F23, F24, respectively correspond F15, F16, F17, F18, F20, F21, F22,
The acquisition of data, the acquisition data item of each acquisition tasks are 404 on this 9 Fans of F23, F24, wherein Single
Float type data 31, Double Float type data 52, Boolean type data 319, Unsigned Integer type
Data 2；2 are set by the priority of F15, F16, F17 this 3 tasks, by the priority of this 3 tasks of F18, F20, F21
It is set as 1, indicates highest priority task, set 3 for the priority of F22, F23, F24 this 3 tasks, indicate low priority
Task.The sampling period of all tasks is disposed as 1000ms.
Step 7-2, the acquisition tasks in task list are recycled to the giving 3 online task allocation nodes of the task one by one
Distribution module, until acquisition tasks all in task list are distributed on task allocating module；
In the embodiment of the present invention, the acquisition tasks in task list are recycled one by one and give 3 online task distribution sections
The task allocating module of point, until acquisition tasks all in task list are distributed on task allocating module, Mei Geren
The task allocating module for distribution node of being engaged in only has a distribution permission to the acquisition tasks acquired in it, in present embodiment, acquisition
Task F15, F18, F22 are allocated by the task allocating module of No. 1 task allocation node, and acquisition tasks F16, F20, F23 are by 2
The task allocating module of number task allocation node is allocated, and acquisition tasks F17, F21, F24 are appointed by No. 3 task allocation nodes
Business distribution module is allocated.
Task allocating module inside step 8, each online task allocation node obtains the operation of 3 online acquisition nodes
Status information and the operating status threshold value that initially it is arranged, the priority of acquisition tasks according to acquired in it and 3 are online
The running state information of acquisition node successively distributes acquisition tasks to acquisition node online and in normal operating condition,
And it is adopted online using the acquisition tasks for distributing to target acquisition node as backup cycle assignment to lower two numbers adjacent thereto
Collect node, allocation result is sent in task warehouse；Task monitoring module inside online task allocation node each simultaneously
Distribution, execution to acquisition tasks on respective node in real time and waiting situation are monitored；Specifically:
Step 8-1, the acquisition tasks that the task allocating module of each online task allocation node respectively gets it by
Be lined up according to priority, and according to online acquisition node listing, the running state information of each online acquisition node of poll and
The operating status threshold value that initially it is arranged；
In the embodiment of the present invention, running state information, that is, each online acquisition node CPU of each online acquisition node,
Memory, hard disk, the total amount of network broadband, utilization rate and surplus, operating status threshold value, that is, each acquisition section of each acquisition node
CPU, memory, hard disk, network broadband normal operating condition threshold value and the overlond running state threshold of point；
Step 8-2, by comparing the running state information of each online acquisition node and the operation shape that initially it is arranged
State threshold value judges that 3 online acquisition nodes are up state or overlond running state, if at online acquisition node respectively
It is in normal operating condition, then online to this according to task priority sequence according to the running state information of the online acquisition node
Acquisition node distributes acquisition tasks；If online acquisition node is in overlond running state, which is not distributed
Acquisition tasks；
In the embodiment of the present invention, by comparing the use of the CPU, memory, hard disk, network bandwidth of each online acquisition node
Rate and the CPU that initially it is arranged, memory, hard disk, network bandwidth normal operating condition threshold value and overlond running state threshold, point
Do not judge that 3 online acquisition nodes are up state or overlond running state, if the CPU of online acquisition node, memory,
Hard disk, network bandwidth utilization rate be in the normal operating condition threshold range that initially it is arranged, then judge that this is online
Acquisition node is normal operating condition, and the task allocating module of 3 task allocation nodes is all in accordance with the online acquisition node at this time
Running state information, distribute it acquisition tasks, CPU, memory, hard disk, network bandwidth according to acquisition tasks priority orders
The acquisition tasks that the bigger online acquisition node of surplus obtains are more；If the CPU of online acquisition node, memory, hard disk, Netowrk tape
Any one of wide utilization rate is within the scope of the overlond running state threshold that initially it is arranged, then judges the online acquisition
Node is overlond running state, and the task allocating module of 3 task allocation nodes does not distribute acquisition tasks to it at this time；
Step 8-3, the acquisition tasks of target acquisition node will be distributed to as backup cycle assignment under adjacent thereto
Two number online acquisition nodes；
In the embodiment of the present invention, 2, No. 3 acquisition sections are distributed to using the acquisition tasks for distributing to No. 1 acquisition node as backup
The acquisition tasks for distributing to No. 2 acquisition nodes are distributed to 3, No. 1 acquisition nodes by point, will distribute to No. 3 acquisition sections
The acquisition tasks of point distribute to 1, No. 2 acquisition node as backup.Main acquisition node is known as master node, the acquisition section of backup
Point is referred to as slave1 node, slave2 node；
Step 8-4, its allocation result is sent to task respectively by the task allocating module of each online task allocation node
In warehouse；
In the embodiment of the present invention, the operating condition in task switcher poll main task warehouse, when main task warehouse is in just
When normal operating status, the task allocating module in each online task allocation node sends main task for its allocation result respectively
Warehouse；Task allocating module difference when main task warehouse is in failure operation state, in each online task allocation node
Standby task warehouse is sent by its allocation result；
Step 8-5, the task monitoring module of each online task allocation node is respectively to acquisition tasks on respective node
Distribution executes and situation is waited to be monitored；
Step 8-6, task allocation node breaks down if it exists, then above-mentioned task allocation node is off-line state, other
Online task allocation node obtains updated online task allocation node list, task setting module from management node in real time
Acquisition tasks in task list are recycled to the task allocating module for giving each online task allocation node, Zhi Daoren one by one
Until all acquisition tasks are distributed on task allocating module in business list, 8-1 is returned to step.
The task that step 9, each online acquisition node acquire needed for obtaining from task warehouse respectively respectively；
In the embodiment of the present invention, when main task warehouse is in normal operating condition, each online acquisition node respectively from
The acquisition tasks that main task warehouse acquires needed for obtaining respectively；It is each online when main task warehouse is in failure operation state
The acquisition tasks that acquisition node acquires needed for obtaining respectively from standby task warehouse respectively；
Step 10, each online acquisition node carry out the data of field control station according to the priority orders of acquisition tasks
Acquisition, and collection result is stored in database；Specifically:
Step 10-1, each online acquisition node according to acquisition tasks priority orders to the data of field control station into
Row acquisition, under normal circumstances, the acquisition tasks that the lower two number online acquisition node adjacent with target acquisition node is backed up
Only make local cache, is not stored in database；
In the embodiment of the present invention, under normal circumstances, after the completion of the acquisition of backup acquisition tasks on slave1, slave2 only into
Row local cache, is not stored in database.
Step 10-2, when there are acquisition node failure, which will be in off-line state, in current sample period
Interior, the collection result for backing up acquisition tasks is stored in database from local cache area by first adjacent number online acquisition node
In；If first adjacent at this time number online acquisition node also breaks down, second adjacent number online acquisition section
Point will back up the collection result of acquisition tasks from local cache area deposit database；
In the embodiment of the present invention, when acquisition node failure acquires when the data of master are stored in failed database currently
Period, slave1 detect the fault message of master by management node, will back up the collection results of acquisition tasks from local
Buffer area reads in database, if the acquisition node of slave1 backup at this time also breaks down, slave2 is detected by management node
To the fault message of master and slave1, the collection result for backing up acquisition tasks is read in into database from local cache area.Often
A collection period, buffer area content refresh are primary.
Step 10-3, when starting next sampling period, each online task allocation node is obtained by management node
Updated online acquisition node listing is got, acquisition tasks are distributed into online acquisition section according to priority orders again
Point, corresponding online acquisition node execute the acquisition tasks of its acquisition and implementing result are stored in database.
Claims (10)
1. a kind of large-scale distributed data collection system, which is characterized in that the system includes task allocation node module, management
Node module, task warehouse module and acquisition node module；
The task allocation node module includes multiple tasks distribution node；The management node module includes management section
Point；The task warehouse module includes task warehouse；The acquisition node module includes multiple acquisition nodes；Wherein,
Task allocation node: the task for acquiring needed for setting obtains online acquisition node list from management node in real time
The task allocation node list online with other, to obtain the running state information of each online acquisition node and initially to it
The operating status threshold value of setting carries out acquisition tasks set by user to recycle on each online task allocation node one by one
Point, and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, it successively will acquisition
Task is distributed to online and be in the acquisition node of normal operating condition, will distribute to the acquisition tasks of target acquisition node as
Backup cycle assignment gives lower two numbers online acquisition node adjacent thereto, and allocation result is sent in task warehouse；Together
When the distribution, execution to acquisition tasks on respective node in real time and wait situation be monitored；
Management node: for providing distributed collaboration service, receive and manage the registration letter of acquisition node and task allocation node
Breath, is numbered acquisition node according to the registration order of acquisition node, is communicated according to certain frequency with acquisition node
To obtain online acquisition node；Task allocation node is numbered according to the registration order of task allocation node, according to certain
Frequency communicated with task allocation node to obtain online task allocation node；
Task warehouse: for receiving the acquisition tasks of task allocation node transmission；
Acquisition node: for obtaining other online acquisition node listings from management node, needed for being obtained respectively from task warehouse
The task of acquisition, and be acquired according to data of the priority orders of acquisition tasks to field control station, collection result is deposited
Enter in database.
2. large-scale distributed data collection system according to claim 1, which is characterized in that task allocation node,
The task list of required acquisition is synchronized in other online task allocation nodes.
3. large-scale distributed data collection system according to claim 1, which is characterized in that task allocation node includes
Task setting module, task allocating module and task monitoring module, wherein
Task setting module: for obtaining other online task allocation node lists in real time from management node, according to actual needs
Being acquired needed for setting for task, and the acquisition tasks in task list are recycled one by one and give all online task allocation nodes
Task allocating module；
Task allocating module: for obtaining online acquisition node list and other online task distribution in real time from management node
Node listing, thus the operating status threshold value for obtaining the running state information of each online acquisition node and initially it being arranged,
According to the priority of acquired acquisition tasks and the running state information of each online acquisition node, successively acquisition tasks are distributed
It is followed to online and be in the acquisition node of normal operating condition, and using the acquisition tasks for distributing to target acquisition node as backing up
Ring distributes to lower two numbers online acquisition node adjacent thereto, sends task warehouse for allocation result；
Task monitoring module: for the distribution, execution in real time to acquisition tasks on respective node and situation is waited to be monitored.
4. large-scale distributed data collection system according to claim 1, which is characterized in that the management node mould
Block, further includes management switch, and the management node uses redundant configuration mode, including main management node and standby management section
Point；
Management switch: it for capturing the fault message of main management node, realizes between main management node and standby management node
Data between switching and main management node and standby management node are synchronous.
5. large-scale distributed data collection system according to claim 1, which is characterized in that the task warehouse mould
Block further includes task switcher, and the task warehouse uses redundant configuration mode, including main task warehouse and standby task storehouse
Library；
Task switcher: it for capturing the fault message in main task warehouse, realizes between main task warehouse and standby task warehouse
Data between switching and main task warehouse and standby task warehouse are synchronous.
6. the collecting method carried out using large-scale distributed data collection system described in claim 1, feature are existed
In, comprising the following steps:
Step 1 carries out Initialize installation to system；
Step 2, the quantity and field device for configuring required field control station according to actual needs, and successively open and configured
Field device and field control station；
Step 3 opens management node and task warehouse, and successively opens each acquisition node and task allocation node；
Registration information is sent from trend management node after step 4, the unlatching of each acquisition node, management node is according to registration order pair
Acquisition node is numbered, and is communicated according to certain frequency with acquisition node to obtain online acquisition node；
Registration information is sent from trend management node after step 5, the unlatching of each task allocation node, management node is suitable according to registering
Ordered pair task allocation node is numbered, and is communicated according to certain frequency with task allocation node to obtain online task point
With node；
Step 6, the task setting module of each online task allocation node obtain other online timess in real time from management node respectively
Business distribution node list, it is online with other that task allocating module obtains online acquisition node list in real time from management node respectively
Task allocation node list, each online acquisition node obtains other online acquisition nodes column in real time from management node respectively
Table；
Step 7, according to actual needs, using needed for the task setting module setting inside any one online task allocation node
The task of acquisition, and distribute the task that the acquisition tasks in task list are evenly distributed to each online task allocation node to mould
Block；
Task allocating module inside step 8, each online task allocation node obtains the operating status letter of each online acquisition node
Breath and the operating status threshold value that initially it is arranged, the priority of acquisition tasks according to acquired in it and each online acquisition section
Acquisition tasks are successively distributed to acquisition node online and in normal operating condition, and will divided by the running state information of point
The acquisition tasks of dispensing target acquisition node give lower two numbers online acquisition node adjacent thereto as backup cycle assignment,
Allocation result is sent in task warehouse；The task monitoring module inside each online task allocation node is in real time to respective simultaneously
The distribution, execution of acquisition tasks and waiting situation are monitored on node；
The task that step 9, each online acquisition node acquire needed for obtaining from task warehouse respectively respectively；
Step 10, each online acquisition node are acquired the data of field control station according to the priority orders of acquisition tasks,
And collection result is stored in database.
7. collecting method according to claim 6, which is characterized in that initialized described in step 1 to system
Setting；Specifically: setting main management node, standby management node, each task allocation node, main task warehouse, standby task warehouse and
The IP address and port numbers of each acquisition node, and the synchronous operating status threshold value with each acquisition node of clock is set.
8. collecting method according to claim 6, which is characterized in that described in step 7 according to actual needs, use
Being acquired needed for the task setting module setting inside any one online task allocation node for task, and will be in task list
Acquisition tasks are averagely allocated to the task allocating module of each online task allocation node；Specific step is as follows:
Step 7-1, institute according to actual needs, is set using the task setting module inside any one online task allocation node
Need to acquiring for task；
Step 7-2, the acquisition tasks in task list are recycled to the task point for giving each online task allocation node one by one
With module, until acquisition tasks all in task list are distributed on task allocating module.
9. collecting method according to claim 6, which is characterized in that each online task distribution section described in step 8
The operating status that task allocating module inside point obtains the running state information of each online acquisition node and initially it is arranged
Threshold value, the running state information of the priority of acquisition tasks according to acquired in it and each online acquisition node successively will acquisition
Task is distributed to acquisition node online and in normal operating condition, and the acquisition tasks that will distribute to target acquisition node are made
Lower two numbers online acquisition node adjacent thereto is given for backup cycle assignment, allocation result is sent in task warehouse；
Simultaneously the task monitoring module inside each online task allocation node in real time to the distribution of acquisition tasks on respective node, execute and
Situation is waited to be monitored；Specific step is as follows:
Step 8-1, the acquisition tasks that the task allocating module of each online task allocation node respectively gets it are according to excellent
First grade is lined up, and according to online acquisition node listing, the running state information of each online acquisition node of poll and initial
The operating status threshold value that it is arranged；
Step 8-2, by comparing the running state information of each online acquisition node and the operating status threshold that initially it is arranged
Value, judges that each online acquisition node is up state or overlond running state respectively, if online acquisition node is in just
Normal operating status, then according to the running state information of the online acquisition node, according to task priority sequence to the online acquisition
Node distributes acquisition tasks；If online acquisition node is in overlond running state, which is not distributed and acquired
Task；
Step 8-3, the acquisition tasks of target acquisition node will be distributed to as backup cycle assignment to adjacent thereto lower two
Number online acquisition node；
Step 8-4, its allocation result is sent in task warehouse by the task allocating module of each online task allocation node respectively；
Step 8-5, the task monitoring module of each online task allocation node respectively to the distribution of acquisition tasks on respective node,
It executes and situation is waited to be monitored；
Step 8-6, task allocation node breaks down if it exists, then above-mentioned task allocation node is off-line state, other are online
Task allocation node obtain updated online task allocation node list in real time from management node, task setting module will appoint
Acquisition tasks in business list recycle the task allocating module for giving each online task allocation node one by one, until task arranges
Until all acquisition tasks are distributed on task allocating module in table, 8-1 is returned to step.
10. collecting method according to claim 6, which is characterized in that each online acquisition node described in step 10
The data of field control station are acquired according to the priority orders of acquisition tasks, and collection result is stored in database；
Specifically:
Step 10-1, each online acquisition node adopts the data of field control station according to the priority orders of acquisition tasks
Collection, under normal circumstances, the acquisition tasks that the lower two number online acquisition node adjacent with target acquisition node is backed up only are made
Local cache is not stored in database；
Step 10-2, when there are acquisition node failure, which will be in off-line state, in current sample period,
First adjacent number online acquisition node will back up the collection result of acquisition tasks from local cache area deposit database；
If first adjacent at this time number online acquisition node also breaks down, second adjacent number online acquisition node will
The collection result of backup acquisition tasks is from local cache area deposit database；
Step 10-3, when starting next sampling period, each online task allocation node gets update by management node
Online acquisition node listing afterwards；Acquisition tasks are distributed into online acquisition node according to priority orders again, accordingly
Online acquisition node executes the acquisition tasks of its acquisition and implementing result is stored in database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610522950.XA CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610522950.XA CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106126346A CN106126346A (en) | 2016-11-16 |
| CN106126346B true CN106126346B (en) | 2019-02-26 |
Family
ID=57468566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610522950.XA Active CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106126346B (en) |
Families Citing this family (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241528B (en) * | 2017-01-19 | 2020-10-09 | 上海直真君智科技有限公司 | Dynamic acquisition method for mass network security data customized by user |
| CN108810042B (en) * | 2017-04-28 | 2021-06-29 | 华为技术有限公司 | Task processing method, related equipment and system |
| CN107154973A (en) * | 2017-05-08 | 2017-09-12 | 成都汇智远景科技有限公司 | Distributed disease detection method |
| CN107704320B (en) * | 2017-05-12 | 2018-08-17 | 贵州白山云科技有限公司 | A kind of method for allocating tasks and system of distributed system |
| CN107147525A (en) * | 2017-05-12 | 2017-09-08 | 山东浪潮商用系统有限公司 | Based on the method for realizing business datum recovery across station switching collection |
| CN107291544B (en) * | 2017-08-03 | 2020-07-28 | 浪潮云信息技术股份公司 | Task scheduling method and device and distributed task execution system |
| CN107464056A (en) * | 2017-08-03 | 2017-12-12 | 武汉远众科技有限公司 | A kind of offline real-time task type collecting method based on mobile phone |
| CN109542595B (en) * | 2017-09-21 | 2023-02-24 | 阿里巴巴集团控股有限公司 | Data acquisition method, device and system |
| CN107886692B (en) * | 2017-10-30 | 2020-08-25 | 皖西学院 | Sensor loRa wireless network communication system for bioengineering |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| CN108769115A (en) * | 2018-04-19 | 2018-11-06 | 中国科学院计算技术研究所 | Distributed RSS data acquisition method and system |
| CN110389856B (en) * | 2018-04-20 | 2023-07-11 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer readable medium for migrating data |
| CN108536539B (en) * | 2018-04-26 | 2021-12-17 | 东北大学 | Task scheduling method in industrial distributed data acquisition system |
| CN108540353B (en) * | 2018-06-14 | 2021-11-05 | 平安科技（深圳）有限公司 | Method and system for monitoring network node |
| CN111092921B (en) * | 2018-10-24 | 2022-05-10 | 北大方正集团有限公司 | Data acquisition method, device and storage medium |
| CN109495365B (en) * | 2018-11-20 | 2022-02-18 | 新智能源系统控制有限责任公司 | Single-ring network type SCADA system |
| CN109412862A (en) * | 2018-11-20 | 2019-03-01 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of looped network type SCADA system |
| CN109560962A (en) * | 2018-11-20 | 2019-04-02 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of monocycle net type SCADA system |
| CN109542061A (en) * | 2018-11-20 | 2019-03-29 | 新智能源系统控制有限责任公司 | A kind of looped network type SCADA system |
| CN109711779A (en) * | 2018-12-26 | 2019-05-03 | 广东嘉腾机器人自动化有限公司 | A kind of order allocation method and storage device and order distribution system |
| CN109656707B (en) * | 2018-12-26 | 2020-03-31 | 南方电网数字电网研究院有限公司 | Metering data acquisition method and device, computer equipment and storage medium |
| CN111381931A (en) * | 2018-12-29 | 2020-07-07 | 中兴通讯股份有限公司 | Disaster recovery method, device and system |
| CN111581033B (en) * | 2019-02-19 | 2023-10-27 | 青岛海信网络科技股份有限公司 | Load balancing method, system and device |
| CN110401698A (en) * | 2019-06-27 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of monitoring data collection methods, devices and systems of task based access control asynchronous distribution |
| CN111614716B (en) * | 2020-04-10 | 2022-12-16 | 厦门网宿有限公司 | Method and system for collecting remote equipment data |
| CN112714464A (en) * | 2020-12-25 | 2021-04-27 | 中国工程物理研究院电子工程研究所 | Large-scale distributed monitoring method based on position information matching |
| CN112637368B (en) * | 2021-03-10 | 2021-05-14 | 江苏金恒信息科技股份有限公司 | Distributed industrial data acquisition system and method |
| CN113419451B (en) * | 2021-06-29 | 2022-08-16 | 浙江邮电职业技术学院 | Distributed data acquisition method and system for communication network |
| CN113515082A (en) * | 2021-06-30 | 2021-10-19 | 青岛海尔空调电子有限公司 | Polling communication method and device |
| CN113626161B (en) * | 2021-07-09 | 2023-12-22 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN114124958A (en) * | 2021-11-26 | 2022-03-01 | 智昌科技集团股份有限公司 | Operation method of distributed data acquisition system |
| CN116720645A (en) * | 2022-10-21 | 2023-09-08 | 荣耀终端有限公司 | Task distribution method, distribution equipment and task distribution system |
| CN117640667A (en) * | 2023-12-20 | 2024-03-01 | 河北网新科技集团股份有限公司 | Transaction information access method and device, terminal equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622062A (en) * | 2003-11-26 | 2005-06-01 | 中国科学院沈阳自动化研究所 | Distributed underwater robot control system |
| CN101873005A (en) * | 2010-06-17 | 2010-10-27 | 深圳市科陆电子科技股份有限公司 | Method for realizing balanced acquisition of electric energy |
| CN102045402A (en) * | 2010-12-28 | 2011-05-04 | 云海创想信息技术（北京）有限公司 | Data acquisition method and data acquisition system of distributed file system |
| CN103369054A (en) * | 2013-07-30 | 2013-10-23 | 北京搜狐新媒体信息技术有限公司 | Acquisition task management method and system |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN105430057A (en) * | 2015-11-03 | 2016-03-23 | 上海新储集成电路有限公司 | Internet of Things (IoT) structure and data transmission method thereof |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN105527948A (en) * | 2015-12-11 | 2016-04-27 | 东北大学 | Large scale distributed data acquisition system and method based on industrial process |
| CN105549556A (en) * | 2015-12-17 | 2016-05-04 | 东北大学 | Data collection transmission system and method for real-time monitoring of industrial process |
| CN105553699A (en) * | 2015-12-09 | 2016-05-04 | 东北大学 | Wind turbine generator set state monitoring private cloud system and method built from point of data processing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11546230B2 (en) * | 2014-09-19 | 2023-01-03 | Impetus Technologies, Inc. | Real time streaming analytics platform |
-
2016
- 2016-07-05 CN CN201610522950.XA patent/CN106126346B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622062A (en) * | 2003-11-26 | 2005-06-01 | 中国科学院沈阳自动化研究所 | Distributed underwater robot control system |
| CN101873005A (en) * | 2010-06-17 | 2010-10-27 | 深圳市科陆电子科技股份有限公司 | Method for realizing balanced acquisition of electric energy |
| CN102045402A (en) * | 2010-12-28 | 2011-05-04 | 云海创想信息技术（北京）有限公司 | Data acquisition method and data acquisition system of distributed file system |
| CN103369054A (en) * | 2013-07-30 | 2013-10-23 | 北京搜狐新媒体信息技术有限公司 | Acquisition task management method and system |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN105430057A (en) * | 2015-11-03 | 2016-03-23 | 上海新储集成电路有限公司 | Internet of Things (IoT) structure and data transmission method thereof |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN105553699A (en) * | 2015-12-09 | 2016-05-04 | 东北大学 | Wind turbine generator set state monitoring private cloud system and method built from point of data processing |
| CN105527948A (en) * | 2015-12-11 | 2016-04-27 | 东北大学 | Large scale distributed data acquisition system and method based on industrial process |
| CN105549556A (en) * | 2015-12-17 | 2016-05-04 | 东北大学 | Data collection transmission system and method for real-time monitoring of industrial process |
Non-Patent Citations (1)
| Title |
|---|
| 分布式数据采集系统的设计和实现;韩国政，邱洪泽;《华中电力》;20060913;第2006卷(第4期);第33-36页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106126346A (en) | 2016-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106126346B (en) | A kind of large-scale distributed data collection system and method | |
| CN108921551B (en) | Alliance block chain system based on Kubernetes platform | |
| CN107066319B (en) | Multi-dimensional scheduling system for heterogeneous resources | |
| CN109379420B (en) | Comprehensive energy service platform system based on distributed architecture | |
| CN103685486B (en) | Distributed system monitoring method stepping over data center clusters and system | |
| KR102508817B1 (en) | High availability distribution intelligence system using message transmission bus | |
| CN105703940A (en) | Multistage dispatching distributed parallel computing-oriented monitoring system and monitoring method | |
| CN106790436B (en) | Traffic system monitoring method based on cloud architecture and control center cloud server | |
| CN107341205A (en) | A kind of intelligent distribution system based on big data platform | |
| CN105071994B (en) | A kind of mass data monitoring system | |
| CN108848132B (en) | Power distribution scheduling main station system based on cloud | |
| CN105592122A (en) | Cloud platform monitoring method and cloud platform monitoring system | |
| CN112966906A (en) | Centralized monitoring platform of spacecraft ground measurement and control station resource pool architecture | |
| CN107682411A (en) | A kind of extensive SDN controllers cluster and network system | |
| CN102902669A (en) | Distribution information capturing method based on internet system | |
| CN111064672A (en) | Cloud platform communication system, election method and resource scheduling management method | |
| CN109845192A (en) | Computer system and method for the network that dynamically internet adapter software defines | |
| CN108073696A (en) | GIS application processes based on distributed memory database | |
| CN103546572A (en) | Cloud storage device and multi-cloud storage networking system and method | |
| CN111541599B (en) | Cluster software system and method based on data bus | |
| CN112437129A (en) | Cluster management method and cluster management device | |
| CN110704465A (en) | Method, device and storage medium for processing service work order table | |
| CN116489214A (en) | Unified service scheduling method, device, medium and equipment based on micro service gateway | |
| CN104484228B (en) | Distributed parallel task processing system based on Intelli DSC | |
| CN110011984A (en) | A kind of distributed cluster system and method based on REST and RPC |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20161116 Assignee: Maogong Branch of Fushun Hanwang Aoniu Mining Co., Ltd. Assignor: Northeast University Contract record no.: X2019210000006 Denomination of invention: Large-scale distributed data collection system and method Granted publication date: 20190226 License type: Common License Record date: 20190916 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20161116 Assignee: LIAONING ATS INTELLIGENT TRANSPORTATION TECHNOLOGY Co.,Ltd. Assignor: Northeastern University Contract record no.: X2020210000028 Denomination of invention: A large scale distributed data acquisition system and method Granted publication date: 20190226 License type: Common License Record date: 20201127 |