CN106126346A - A kind of large-scale distributed data collecting system and method - Google Patents
A kind of large-scale distributed data collecting system and method Download PDFInfo
- Publication number
- CN106126346A CN106126346A CN201610522950.XA CN201610522950A CN106126346A CN 106126346 A CN106126346 A CN 106126346A CN 201610522950 A CN201610522950 A CN 201610522950A CN 106126346 A CN106126346 A CN 106126346A
- Authority
- CN
- China
- Prior art keywords
- node
- task
- acquisition
- online
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/501—Performance criteria
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
Abstract
The present invention proposes a kind of large-scale distributed data collecting system and method, belongs to data acquisition technology field, and this system includes task allocation node module, management node module, task warehouse module and acquisition node module；Management node is used for providing distributed collaboration service, receives and manages the log-on message of acquisition node and task allocation node, communicate with acquisition node and task allocation node according to certain frequency；Task allocation node is for being set acquisition tasks, distribute and monitoring；Task warehouse sends, for receiving and deposit task allocation node, the acquisition tasks of coming；Acquisition node is for obtaining acquisition tasks from task input warehouse, and completes the collection of corresponding data in field control station；The acquisition tasks that acquisition node performs can set dynamically change according to user, and certain or certain the several faults in task allocation node and acquisition node do not interfere with current collection period collection result, have high flexibility, high fault tolerance and high reliability.
Description
Technical field
The invention belongs to data acquisition technology field, be specifically related to a kind of large-scale distributed data collecting system and side
Method.
Background technology
Along with the development of industrial undertaking, production scale expand and being greatly increased of data acquisition amount, enterprise is to data acquisition
The requirement of collection reliability is more and more higher, it is ensured that the difficulty gathering reliability is the most increasing.
Mainly " CN105527948A is (a kind of based on industry for patent in terms of large-scale distributed data acquisition at present
The large-scale distributed data collecting system of process and method) ", one data collection scheduling module of this patent utilization is many numbers
According to gathering client distribution acquisition tasks, when gathering client and breaking down, next collection period, utilize collection scheduling mould
Block carries out the mode of readjustment degree and ensures next cycle and the reliability of time collection later after fault.Although this patent is in certain journey
The reliability of collection can be improved on degree, but in gatherer process, yet suffer from certain limitation:
1., in this patent, collection scheduling module is adopted gathering client distribution according to the running state information of field control station
Set task, performed collection can only be to carry out in units of field control station, and cannot be acquired in units of data item,
And all acquisition tasks does not differentiates between priority, gather motility not high enough.For needing to gather certain in all field control stations
The occasion that one or a few item data are analyzed, or each acquisition tasks have significance level height point occasion, this is special
Profit cannot meet requirement.
2. described in this patent, node each in distributed system is carried out accordingly by the independent coordinated management module of system shortage
Management and automatic load balancing, when needs multiple collection scheduling module, system is difficult to extend dynamically, system spirit
Activity is not high enough.
3. this patent is when gathering client failure, only can guarantee that next collection period normal acquisition, current for fault
Collection period, does not provide good disaster tolerance treatment mechanism to ensure to gather reliability and fault-tolerance, and therefore, fault is once sent out
Raw, fault occurs the data in moment to be easily lost, and the production occasion high for gathering reliability requirement, this loss equally can be right
Enterprise causes certain economic loss.Additionally, this patent is distributed gathering client only with a data collection scheduling module
Acquisition tasks, be not acquired scheduler module disaster tolerance process, and the distribution of all acquisition tasks and to gather client
Management function be focused on a data collection scheduling module, function is more concentrated, and once this collection scheduling module failure, whole
Individual system will be unable to normally work.
4., in this patent, the direct and all of collection client of collection scheduling module interacts, and the collection of distribution is appointed
Business is transmitted directly to all of collection client.When using multiple collection scheduling module, collection scheduling module and collection client
End coupling is relatively big, communicates therebetween lengthy and jumbled, when multiple collection scheduling modules send task to same collection client,
The task of each collection scheduling module arrives the time difference gathering client, gathers client and waits the time of task distribution not
Determining, thus it is longer to cause gathering the client waiting time, system effectiveness reduces.
In sum, existing patent is in terms of data acquisition function, motility, high efficiency and guarantee data acquisition reliability
Exist the most not enough, it is difficult to meet the currently manufactured enterprise demand to real-time data acquisition in intelligent transformation process.
Summary of the invention
For the deficiencies in the prior art, the present invention proposes a kind of large-scale distributed data collecting system and method, to reach
Motility and extensibility, raising management node and the reliability in task warehouse, raising task allocation node to raising system
Fault-tolerance thus ensure high reliability that acquisition tasks distributes, improve the fault-tolerance of acquisition node thus ensure present sample week
The purpose of the long-term reliability that the fault-tolerance of the data acquisition of phase and high reliability and the acquisition tasks that ensures perform；
A kind of large-scale distributed data collecting system, this system include task allocation node module, management node module,
Task warehouse module and acquisition node module；
Described task allocation node module includes multiple task allocation node；Described management node module includes management
Node；Described task warehouse module includes task warehouse；Described acquisition node module includes multiple acquisition node；Wherein,
Task allocation node: for setting the task of required collection, obtain online acquisition node in real time from management node
List and other online task allocation node lists, thus obtain the running state information of each online acquisition node and initial
The running status threshold value arranging it, follows on each online task allocation node one by one by the acquisition tasks that user sets
Ring is divided equally, and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, successively will
Acquisition tasks distribution, to online and be in the acquisition node of normal operating condition, will distribute to the acquisition tasks of target acquisition node
Give, as backup cycle assignment, lower two the numbering online acquisition nodes being adjacent, allocation result is sent to task warehouse
In；Simultaneously in real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored；
Management node: be used for providing distributed collaboration service, receive and manage the note of acquisition node and task allocation node
Volume information, is numbered acquisition node according to registration order, communicates to obtain according to certain frequency and acquisition node
Online acquisition node；According to registration order, task allocation node is numbered, according to certain frequency and task allocation node
Communicate to obtain online task allocation node；
Task warehouse: for receiving the acquisition tasks that task allocation node sends；
Acquisition node: for obtaining other online acquisition node listings from management node, obtain each from task warehouse
Required gathering of task, and according to the priority orders of acquisition tasks, the data of field control station are acquired, collection is tied
Fruit is stored in data base.
Described task allocation node, the task list of required collection is synchronized to other online task allocation nodes by it
In.
Described task allocation node includes task setting module, task allocating module and Mission Monitor module, wherein,
Task setting module: for obtaining other online task allocation node lists in real time, according to reality from management node
Required gathering of the task of requirements set, and the acquisition tasks in task list is circulated one by one all give the distribution of all online tasks
The task allocating module of node；
Task allocating module: for obtaining online acquisition node list and other online tasks in real time from management node
Distribution node list, thus obtain the running state information of each online acquisition node and the initial running status threshold that it is arranged
Value, according to priority and the running state information of each online acquisition node of acquired acquisition tasks, successively by acquisition tasks
Distribute to online and be in the acquisition node of normal operating condition, and using distribute to target acquisition node acquisition tasks as standby
Part cycle assignment gives lower two the numbering online acquisition nodes being adjacent, and allocation result is sent to task warehouse；
Mission Monitor module: in real time to the distribution of acquisition tasks on respective node, perform and wait situation is supervised
Control.
Described management node module, also includes managing switch, and described management node uses redundant configuration mode, bag
Include main management node and standby management node；
Management switch: for catching the fault message of main management node, it is achieved main management node and standby management node it
Between switching and main management node and standby management node between data syn-chronization.
Described task warehouse module, also includes that task switcher, described task warehouse use redundant configuration mode, bag
Include main task warehouse and standby task warehouse；
Task switcher: for catching the fault message in main task warehouse, it is achieved main task warehouse and standby task warehouse it
Between switching and main task warehouse and standby task warehouse between data syn-chronization.
The collecting method using large-scale distributed data collecting system to carry out, comprises the following steps:
Step 1, system is carried out Initialize installation；
Step 2, the quantity configuring required field control station according to actual needs and field apparatus, and unlatching is joined successively
The field apparatus put and field control station；
Step 3, unlatching management node and task warehouse, and open each acquisition node and task allocation node successively；
Step 4, each acquisition node send log-on message from trend management node after opening, and management node is suitable according to registration
Ordered pair acquisition node is numbered, and communicates to obtain online acquisition node according to certain frequency and acquisition node；
Step 5, each task allocation node send log-on message from trend management node after opening, and management node is according to note
Task allocation node is numbered by volume order, communicates to obtain online appointing according to certain frequency and task allocation node
Business distribution node；
Step 6: it is online that the task setting module of each online task allocation node obtains other in real time from management node respectively
Task allocation node list, task allocating module respectively from management node obtain in real time online acquisition node list and other
Online task allocation node list, each online acquisition node obtains other online acquisition nodes in real time from management node respectively
List；
Step 7, according to the actual requirements, uses any one task setting module settings within online task allocation node
Required gathering of task, and the task that the acquisition tasks in task list is evenly distributed to each online task allocation node divides
Join module；
Task allocating module within step 8, each online task allocation node obtains the operation shape of each online acquisition node
State information and the running status threshold value initially arranged it, according to the priority of acquisition tasks acquired in it, and each adopt online
The running state information of collection node, successively by acquisition tasks distribution to online and be in the acquisition node of normal operating condition, and
Give, as backup cycle assignment, lower two the numbering online acquisition being adjacent using the acquisition tasks distributing to target acquisition node
Node, sends allocation result to task warehouse；Mission Monitor module within the most each online task allocation node is real-time
To the distribution of acquisition tasks on respective node, perform and wait situation is monitored；
Step 9, each online acquisition node obtain the most required gathering of task respectively from task warehouse；
The data of field control station are adopted by step 10, each online acquisition node according to the priority orders of acquisition tasks
Collection, and collection result is stored in data base.
Described in step 1, system is carried out Initialize installation；Particularly as follows: arrange main management node, standby management node, each
Business distribution node, main task warehouse, standby task warehouse and the IP address of each acquisition node and port numbers, and arrange clock synchronize and
The running status threshold value of each acquisition node.
Described in step 7 according to the actual requirements, any one task setting mould within online task allocation node is used
Block sets required gathering of task, and the acquisition tasks in task list is averagely allocated to each online task allocation node
Task allocating module；Specifically comprise the following steps that
Step 7-1, according to the actual requirements, uses any one task setting module within online task allocation node to set
Fixed required gathering of task；
Step 7-2, the acquisition tasks in task list is circulated one by one all give each online task allocation node appoint
Business distribution module, until all of acquisition tasks is distributed on task allocating module in task list.
The task allocating module within each online task allocation node described in step 8 obtains the fortune of each online acquisition node
Row status information and the running status threshold value initially arranged it, according to the priority of acquisition tasks acquired in it, and respectively exist
The running state information of line acquisition node, successively by acquisition tasks distribution to online and be in the collection joint of normal operating condition
Point, and the acquisition tasks distributing to target acquisition node is given as backup cycle assignment lower two numberings being adjacent online
Acquisition node, sends allocation result to task warehouse；Mission Monitor module within the most each online task allocation node
In real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored；Specifically comprise the following steps that
Step 8-1, each online task allocation node task allocating module respectively to its acquisition tasks got by
Rank according to priority, and according to online acquisition node listing, the running state information of poll each online acquisition node and
The initial running status threshold value that it is arranged；
Step 8-2, by the running state information of relatively each online acquisition node and the operation shape that initially it arranged
State threshold value, judges that each online acquisition node is up state or overlond running state, respectively if at online acquisition node
In normal operating condition, then according to the running state information of this online acquisition node, online to this according to task priority order
Acquisition node distribution acquisition tasks；If online acquisition node is in overlond running state, this online acquisition node is not distributed
Acquisition tasks；
Step 8-3, using the acquisition tasks distributing to target acquisition node as backup cycle assignment give be adjacent down
Two numbering online acquisition nodes；
Its allocation result is sent to task storehouse by step 8-4, the task allocating module of each online task allocation node respectively
In storehouse；
Step 8-5, each online task allocation node Mission Monitor module respectively to acquisition tasks on respective node
Distribution, perform and wait situation is monitored；
Breaking down if step 8-6 exists task allocation node, the most above-mentioned task allocation node is off-line state, other
The online task allocation node list after management node obtains renewal in real time of the online task allocation node, task setting module
Acquisition tasks in task list is circulated one by one the task allocating module all giving each online task allocation node, until appointing
Till all of acquisition tasks is distributed on task allocating module in business list, returns and perform step 8-1.
Each online acquisition node described in step 10 is according to data to field control station of the priority orders of acquisition tasks
It is acquired, and collection result is stored in data base；Particularly as follows:
The data of field control station are carried out by step 10-1, each online acquisition node according to the priority orders of acquisition tasks
Gathering, under normal circumstances, the acquisition tasks that lower two the numbering online acquisition nodes adjacent with target acquisition node are backed up is only
Make local cache, be not stored in data base；
Step 10-2, when there is acquisition node fault, this acquisition node will be in off-line state, at current sample period
In, the collection result of backup acquisition tasks is stored in data base from local cache district by first adjacent numbering online acquisition node
In；If the most adjacent first numbering online acquisition node also breaks down, second the most adjacent numbering online acquisition joint
The collection result of backup acquisition tasks is stored in data base by point from local cache district；
Step 10-3, when starting the next sampling period, each online task allocation node is got by management node
Online acquisition node listing after renewal；Acquisition tasks is distributed to online acquisition node, phase according to priority orders again
The online acquisition node answered performs its acquisition tasks obtained and execution result is stored in data base.
The invention have the advantages that
The present invention proposes a kind of large-scale distributed data collecting system and method, and user can create the most according to demand
Build, revise and delete acquisition tasks, and according to the actual requirements different acquisition task can be arranged different priority and sampling
Cycle；Being allocated acquisition tasks by task allocation node and dispatch, the data that acquisition node is gathered are the most fixing, and have
There is certain motility；Add management node as independent coordinated management module so that task allocation node and acquisition node
All can dynamically increase and delete, substantially increase motility and the extensibility of system；When some or certain several tasks
During distribution node fault, other online task allocation nodes can divide from the online task after management node obtains renewal in real time
Join node listing, thus the acquisition tasks in the task list of task setting module is recirculated and all gives the most online appointing
Business distribution node, improves the fault-tolerance of task allocation node, thus ensure that the high reliability that acquisition tasks is distributed.When a certain
During individual or certain several acquisition node fault, owing to there is the backup of acquisition tasks, standby acquisition node is arrived by management nodal test
Main acquisition node fault, is stored in data base by the collection result of main acquisition node correspondence acquisition tasks, it is ensured that present sample week
The fault-tolerance of the data acquisition of phase and reliability；From the beginning of next collection period, task allocation node is obtained by management node
Get the online acquisition node listing after renewal, acquisition tasks is distributed to online collection joint according to priority orders again
Point, and call corresponding online acquisition node execution acquisition tasks, thus ensure that the long-term reliability that acquisition tasks performs；Pipe
Reason node and and task warehouse employing redundant configuration mode, raising management node and the reliability in task warehouse.
Accompanying drawing explanation
Fig. 1 is the large-scale distributed data collecting system structural representation of an embodiment of the present invention；
Fig. 2 is the management node module structural representation of an embodiment of the present invention；
Fig. 3 is the task warehouse modular structure schematic diagram of an embodiment of the present invention；
Fig. 4 is the task allocation node structural representation of an embodiment of the present invention；
Fig. 5 is the large-scale distributed collecting method flow chart of an embodiment of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawings an embodiment of the present invention is described further.
As it is shown in figure 1, large-scale distributed data collecting system includes: task allocation node module, management node module,
Task warehouse module and acquisition node module；In the embodiment of the present invention, described task allocation node module includes that 3 tasks are divided
Join node；Described management node module includes managing node；Described task warehouse module includes task warehouse；Described adopts
Collection node module includes 3 acquisition nodes；Additionally, field apparatus is 9 groups of wind-driven generators, for producing the dependency number of blower fan
According to；The title of 9 groups of wind-driven generators is respectively as follows: F15, F16, F17, F18, F20, F21, F22, F23, F24；Field control station is
The PLC of 9 Foxboro companies, for connecting 9 groups of blower fans；
In the embodiment of the present invention, as in figure 2 it is shown, described management node module also includes managing switch, described pipe
Reason node uses redundant configuration mode, including main management node and standby management node；As it is shown on figure 3, described task warehouse mould
Block also includes that task switcher, described task warehouse use redundant configuration mode, including main task warehouse and standby task warehouse；
In the embodiment of the present invention, use tide sea of clouds server virtualization system V4.0 industry cloud platform, be used for disposing master
Management node, standby management node, 3 task allocation nodes, main task warehouse, standby task warehouse and 3 acquisition nodes, and to it
Ruuning situation is monitored, and checks the operation conditions of each node, revises online, the off-line operation state of each node；Described master
Management node, standby management node, 3 task allocation nodes, main task warehouse, standby task warehouses all run on Windows764 position
On operating system and virtual machine that memory size is 4G or virtual machine；3 described acquisition nodes run on
On Windows764 bit manipulation system and virtual machine that memory size is 8G or virtual machine；Described management switch and
Task switcher uses java language to realize exploitation based on MyEclipse 2014 development platform；
In the embodiment of the present invention, 3 task allocation nodes: for setting the task of required collection, real-time from management node
Obtain online acquisition node list and other online task allocation node lists, thus obtain each online acquisition node
Running state information and the running status threshold value initially it arranged；By user set acquisition tasks list synchronization to other
In line task allocation node, the acquisition tasks in this task list is circulated on each online task allocation node one by one
Divide equally, and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, will adopt successively
Set task distribution is to online and be in the acquisition node of normal operating condition, and will distribute to the acquisition tasks of target acquisition node
Give, as backup cycle assignment, lower two the numbering online acquisition nodes being adjacent, allocation result is sent to task warehouse
In；Simultaneously in real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored；
In the embodiment of the present invention, as shown in Figure 4, it is further described: described task allocation node includes that task sets
Cover half block, task allocating module and Mission Monitor module, particularly as follows: task setting module: for obtaining in real time from management node
Other online task allocation node lists, set required gathering of task according to the actual requirements, and user can create, revises, delete
Except acquisition tasks, set the collection data item that each acquisition tasks comprises, and to each acquisition tasks or each collection data item
Set priority and sampling period, and the acquisition tasks in task list is circulated one by one all give all online tasks distribution joint
The task allocating module of point；Task allocating module: for from management node obtain in real time online acquisition node list and other
Online task allocation node list, thus obtain the running state information of each online acquisition node and initially it arranged
Running status threshold value, according to priority and the running state information of each online acquisition node of acquired acquisition tasks, successively
By acquisition tasks distribution to online and be in the acquisition node of normal operating condition, and the collection of target acquisition node will be distributed to
Task gives, as backup cycle assignment, lower two the numbering online acquisition nodes being adjacent, and allocation result is sent to task storehouse
Storehouse；Mission Monitor module: in real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored.
In the embodiment of the present invention, manage node: be used for providing distributed collaboration service, receive and manage 3 acquisition nodes
With the log-on message of 3 task allocation nodes, according to registration order, acquisition node is numbered, according to 5s communication frequency once
Rate and acquisition node communicate to obtain online acquisition node；According to registration order, task allocation node is numbered, presses
Communicate to obtain online task allocation node with task allocation node according to 5s communication frequency once；
In the embodiment of the present invention, manage switch: for catching the fault message of main management node, it is achieved main management node
And the data syn-chronization between switching and main management node and standby management node between standby management node；Under normal circumstances, gather
Node and task allocation node are to main management node log-on message, and lead to main management node with 5s communication frequency once
Letter, task allocation node obtains online acquisition node list and other online task allocation nodes in real time from main management node
List, acquisition node obtains the acquisition node list that other are online in real time from main management node；Standby management node is in real time from supervisor
Reason node obtains synchrodata.When managing the fault message that switch captures main management node, standby management node is as newly
Main management node continue to provide coordination service, acquisition node and task allocation node will to standby management Node registry information, and
Communicating with standby management node with 5s communication frequency once, task allocation node obtains online from standby management node in real time
Acquisition node list and other online task allocation node lists, it is online that acquisition node obtains other in real time from standby management node
Acquisition node list；
In the embodiment of the present invention, task warehouse: for receiving the acquisition tasks that task allocation node sends；
In the embodiment of the present invention, task switcher: for catching the fault message in main task warehouse, it is achieved main task warehouse
And the data syn-chronization between switching and main task warehouse and standby task warehouse between standby task warehouse.Under normal circumstances, system
Use main task warehouse to be operated, i.e. acquisition tasks allocation result is sent to main by the task allocating module of task allocation node
In task warehouse, acquisition node obtains the most required gathering of task from main task warehouse, and standby task warehouse is in real time from director
Business warehouse obtains synchrodata.When task switcher capture main task warehouse break down time, standby task warehouse will start work
Making, acquisition tasks allocation result is sent in standby task warehouse by the task allocating module of task allocation node, acquisition node from
Standby task warehouse obtains the most required gathering of task.
In the embodiment of the present invention, 3 acquisition nodes: for obtaining other online acquisition node listings from management node, from
Task warehouse obtains each required gathering of task, and according to the priority orders of the acquisition tasks data to field control station
It is acquired, collection result is stored in data base.
In the embodiment of the present invention, use the collecting method that large-scale distributed data collecting system is carried out, method stream
Journey figure is as it is shown in figure 5, comprise the following steps:
Step 1, system is carried out Initialize installation, main management node, standby management node, 3 tasks distribution joints are i.e. set
Point, main task warehouse, standby task warehouse and the IP address of 3 acquisition nodes and port numbers, and the synchronization of Internet clock is set
Running status threshold value with 3 acquisition nodes；
In the embodiment of the present invention, the IP address being respectively configured main management node in the configuration file in management node is:
192.168.3.44；The IP address of standby management node is: 192.168.3.45；The IP address of 3 task allocation nodes of configuration is divided
It is not: 192.168.3.46,192.168.3.47 and 192.168.3.48；The IP address in configuration main task warehouse is:
192.168.3.49；The IP address in standby task warehouse is 192.168.3.50；The IP address of 3 acquisition nodes of configuration is respectively as follows:
192.168.3.51,192.168.3.52 and 192.168.3.53；Configuration main management node, standby management node, 3 task distribution
Node, main task warehouse, standby task warehouse, the port numbers of 3 acquisition nodes are 5000；
In the embodiment of the present invention, the running status threshold value arranging 3 acquisition nodes is default value, normally transports including CPU
Row state threshold is 3%～the 70% of CPU total amount, CPU overload state threshold is the 75%～80% of CPU total amount, and internal memory is normal
Running status threshold value is 20%～the 70% of memory amount, internal memory overload threshold value is the 75%～80% of memory amount, firmly
Dish normal operating condition threshold value is 21%～the 70% of hard disk total amount, hard disk overload threshold value be hard disk total amount 75%～
80%, network bandwidth normal operating condition threshold value is 2%～the 70% of network bandwidth total amount, network bandwidth overload threshold value is
The 75%～80% of network bandwidth total amount.
Step 2, successively 9 field control stations of unlatching and 9 groups of blower fans；
Step 3, unlatching management node and task warehouse, and open 3 acquisition nodes and 3 task allocation nodes successively；
Step 4,3 acquisition nodes send log-on message from trend management node after opening, and management node is suitable according to registration
Ordered pair acquisition node is numbered, acquisition node according to registration order respectively numbered 1,2,3, and etc. task to be allocated；Management
The frequency that node communicates once according to 5s and acquisition node communicate to obtain online acquisition node；Particularly as follows:
Step 4-1, management switch obtain the running state information of main management node in real time；When main management node is just in
Often during running status, from trend main management node log-on message after 3 acquisition nodes unlatchings, and it is in wait task status, 3
Acquisition node communicates once every 5s with main management node；When main management node is in failure operation state, 3 acquisition nodes
To standby management Node registry information, and being in wait task status, 3 acquisition nodes and standby management node communicate one every 5s
Secondary；
Step 4-2, management node get online acquisition node automatically；When 3 acquisition nodes the most successfully open it
After, management node will get 3 online acquisition nodes automatically；After 3 acquisition node parts are successfully opened, manage node
To automatically get the online acquisition node less than 3；
Step 5,3 task allocation nodes manage Node registry information from trend after opening, and management node is suitable according to registration
Ordered pair task allocation node is numbered, task allocation node according to registration order respectively numbered 1,2,3, management node according to
5s communication frequency once communicates to obtain online task allocation node with task allocation node；Particularly as follows:
Step 5-1, when main management node is in normal operating condition, 3 task allocation nodes open after from trend master
Management Node registry information, 3 task allocation nodes communicate once every 5s with main management node；When main management node is in event
During barrier running status, 3 task allocation nodes are to standby management Node registry information, 3 task allocation nodes and standby management node
Communicate once every 5s；
Step 5-2, management node get online task allocation node automatically；When 3 task allocation nodes all become
After merit is opened, management node will get 3 online task allocation nodes automatically.When 3 task allocation node part successes
After unlatching, management node will get the online task allocation node less than 3 automatically.
Step 6, from management node, to obtain other in real time online respectively for the task setting module of 3 online task allocation nodes
Task allocation node list, task allocating module respectively from management node obtain in real time online acquisition node list and other
Online task allocation node list, 3 online acquisition nodes obtain other online collection joints in real time from management node respectively
Point list；
In the embodiment of the present invention, when main management node is in normal operating condition, appointing of 3 online task allocation nodes
Business setting module obtains other online task allocation node lists respectively in real time from main management node, and task allocating module is respectively
Obtain online acquisition node list and other online task allocation node lists from main management node in real time, adopt online for 3
Collection node obtains other online acquisition node lists respectively in real time from main management node；When main management node is in failure operation
During state, the task setting module of 3 task allocation nodes obtains other online tasks in real time from standby management node respectively and divides
Joining node listing, task allocating module obtains online acquisition node list in real time from standby management node respectively and other are online
Task allocation node list, 3 online acquisition nodes obtain other online acquisition node row in real time from standby management node respectively
Table；
Step 7, according to the actual requirements, uses any one task setting module settings within online task allocation node
Required gathering of task, and the task that the acquisition tasks in task list is averagely allocated to 3 online task allocation nodes divides
Join module；Specific as follows:
Acquisition tasks is set in the task setting module of No. 1 task allocation node by step 7-1, user, i.e. creates
Build, revise or delete acquisition tasks, set the collection data item that each acquisition tasks comprises, and to each acquisition tasks or each
The priority and the sampling period that gather data item are set；
In the embodiment of the present invention, create the most altogether in the task setting module of No. 1 task allocation node F15, F16,
These 9 acquisition tasks of F17, F18, F20, F21, F22, F23, F24, respectively corresponding F15, F16, F17, F18, F20, F21, F22,
The collection of data on this 9 Fans of F23, F24, the collection data item of each acquisition tasks is 404, wherein Single
Float type data 31, Double Float type data 52, Boolean type data 319, Unsigned Integer type
Data 2；The priority of these 3 tasks of F15, F16, F17 is set to 2, by the priority of these 3 tasks of F18, F20, F21
It is set to 1, represents limit priority task, the priority of these 3 tasks of F22, F23, F24 is set to 3, represent low priority
Task.The sampling period of all tasks is disposed as 1000ms.
Step 7-2, the acquisition tasks in task list is circulated one by one the task of all giving 3 online task allocation nodes
Distribution module, until all of acquisition tasks is distributed on task allocating module in task list；
In the embodiment of the present invention, the acquisition tasks in task list is circulated one by one and all gives 3 online task distribution joints
The task allocating module of point, until all of acquisition tasks is distributed on task allocating module in task list, Mei Geren
The task allocating module of business distribution node only has distribution authority to the acquisition tasks that it is acquired, in present embodiment, gathers
Task F15, F18, F22 are allocated by the task allocating module of No. 1 task allocation node, and acquisition tasks F16, F20, F23 are by 2
The task allocating module of number task allocation node is allocated, acquisition tasks F17, F21, F24 appointing by No. 3 task allocation nodes
Business distribution module is allocated.
Task allocating module within step 8, each online task allocation node obtains the operation of 3 online acquisition nodes
Status information and the running status threshold value initially arranged it, according to the priority of acquisition tasks acquired in it, and 3 online
The running state information of acquisition node, successively by acquisition tasks distribution to online and be in the acquisition node of normal operating condition,
And adopt to lower two numberings being adjacent online as backup cycle assignment the acquisition tasks distributing to target acquisition node
Collection node, sends allocation result to task warehouse；Mission Monitor module within the most each online task allocation node
In real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored；Particularly as follows:
Step 8-1, each online task allocation node task allocating module respectively to its acquisition tasks got by
Rank according to priority, and according to online acquisition node listing, the running state information of poll each online acquisition node and
The initial running status threshold value that it is arranged；
In the embodiment of the present invention, the CPU of running state information the most each online acquisition node of each online acquisition node,
Internal memory, hard disk, the total amount of network broadband, utilization rate and surplus, the running status threshold value of each acquisition node is the most each gathers joint
CPU, internal memory, hard disk, network broadband normal operating condition threshold value and the overlond running state threshold of point；
Step 8-2, by the running state information of relatively each online acquisition node and the operation shape that initially it arranged
State threshold value, judges that 3 online acquisition nodes are up state or overlond running state, respectively if at online acquisition node
In normal operating condition, then according to the running state information of this online acquisition node, online to this according to task priority order
Acquisition node distribution acquisition tasks；If online acquisition node is in overlond running state, this online acquisition node is not distributed
Acquisition tasks；
In the embodiment of the present invention, by the relatively CPU of each online acquisition node, internal memory, hard disk, the use of the network bandwidth
Rate and initial to its CPU arranged, internal memory, hard disk, network bandwidth normal operating condition threshold value and overlond running state threshold, point
Do not judge that 3 online acquisition nodes are up state or overlond running state, if the CPU of online acquisition node, internal memory,
Hard disk, the utilization rate of the network bandwidth are in initial in its normal operating condition threshold range arranged, then judge that this is online
Acquisition node is normal operating condition, and now the task allocating module of 3 task allocation nodes is all in accordance with this online acquisition node
Running state information, distribute acquisition tasks, CPU, internal memory, hard disk, the network bandwidth according to acquisition tasks priority orders to it
The acquisition tasks that the online acquisition node that surplus is the biggest obtains is the most；If the CPU of online acquisition node, internal memory, hard disk, Netowrk tape
In the range of any one is in initially to its overlond running state threshold arranged in wide utilization rate, then judge this online acquisition
Node is overlond running state, and now it is not distributed acquisition tasks by the task allocating module of 3 task allocation nodes；
Step 8-3, using the acquisition tasks distributing to target acquisition node as backup cycle assignment give be adjacent down
Two numbering online acquisition nodes；
In the embodiment of the present invention, the acquisition tasks distributing to No. 1 acquisition node is distributed to 2, No. 3 as backup and gathers joint
Point, distributes to 3, No. 1 acquisition node using the acquisition tasks distributing to No. 2 acquisition nodes as backup, will distribute to No. 3 and gather joint
The acquisition tasks of point distributes to 1, No. 2 acquisition nodes as backup.Main acquisition node is referred to as master node, the collection joint of backup
Point is called slave1 node, slave2 node；
Its allocation result is sent to task by step 8-4, the task allocating module of each online task allocation node respectively
In warehouse；
In the embodiment of the present invention, the ruuning situation in task switcher poll main task warehouse, when main task warehouse is just in
Often during running status, its allocation result is sent to main task by task allocating module in each online task allocation node respectively
Warehouse；When main task warehouse is in failure operation state, the task allocating module in each online task allocation node is respectively
Its allocation result is sent to standby task warehouse；
Step 8-5, each online task allocation node Mission Monitor module respectively to acquisition tasks on respective node
Distribution, perform and wait situation is monitored；
Breaking down if step 8-6 exists task allocation node, the most above-mentioned task allocation node is off-line state, other
The online task allocation node list after management node obtains renewal in real time of the online task allocation node, task setting module
Acquisition tasks in task list is circulated one by one the task allocating module all giving each online task allocation node, until appointing
Till all of acquisition tasks is distributed on task allocating module in business list, returns and perform step 8-1.
Step 9, each online acquisition node obtain the most required gathering of task respectively from task warehouse；
In the embodiment of the present invention, when main task warehouse is in normal operating condition, each online acquisition node respectively from
Main task warehouse obtains the most required acquisition tasks gathered；When main task warehouse is in failure operation state, each online
Acquisition node obtains the most required acquisition tasks gathered from standby task warehouse respectively；
The data of field control station are carried out by step 10, each online acquisition node according to the priority orders of acquisition tasks
Gather, and collection result is stored in data base；Particularly as follows:
The data of field control station are entered by step 10-1, each online acquisition node according to the priority orders of acquisition tasks
Row gathers, under normal circumstances, and the acquisition tasks that lower two the numbering online acquisition node adjacent with target acquisition node is backed up
Only make local cache, be not stored in data base；
In the embodiment of the present invention, under normal circumstances, the backup acquisition tasks collection on slave1, slave2 is only entered after completing
Row local cache, is not stored in data base.
Step 10-2, when there is acquisition node fault, this acquisition node will be in off-line state, at current sample period
In, the collection result of backup acquisition tasks is stored in data base from local cache district by first adjacent numbering online acquisition node
In；If the most adjacent first numbering online acquisition node also breaks down, second the most adjacent numbering online acquisition joint
The collection result of backup acquisition tasks is stored in data base by point from local cache district；
In the embodiment of the present invention, when acquisition node fault, when the data of master are stored in failed database, gather current
In the cycle, slave1 is by the fault message of management nodal test to master, by the collection result of backup acquisition tasks from this locality
Buffer area reads in data base, if now slave1 backup acquisition node also breaks down, slave2 is by management nodal test
To the fault message of master and slave1, the collection result of backup acquisition tasks is read in from local cache district data base.Often
Individual collection period, buffer area content refresh is once.
Step 10-3, when starting the next sampling period, each online task allocation node is all obtained by management node
Get the online acquisition node listing after renewal, acquisition tasks is distributed to online collection joint according to priority orders again
Point, corresponding online acquisition node performs its acquisition tasks obtained and execution result is stored in data base.
Claims (10)
1. a large-scale distributed data collecting system, it is characterised in that this system includes task allocation node module, management
Node module, task warehouse module and acquisition node module；
Described task allocation node module includes multiple task allocation node；Described management node module includes management joint
Point；Described task warehouse module includes task warehouse；Described acquisition node module includes multiple acquisition node；Wherein,
Task allocation node: for setting the task of required collection, obtain online acquisition node list in real time from management node
The task allocation node list online with other, thus obtain the running state information of each online acquisition node and initially to it
The running status threshold value arranged, is circulated the acquisition tasks that user sets all on each online task allocation node one by one
Point, and according to the priority of acquired acquisition tasks and the running state information of each online acquisition node, will gather successively
Task distribution to online and be in the acquisition node of normal operating condition, using the acquisition tasks of distributing to target acquisition node as
Backup cycle assignment gives lower two the numbering online acquisition nodes being adjacent, and allocation result is sent to task warehouse；With
Time in real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored；
Management node: be used for providing distributed collaboration service, receives and manages the registration letter of acquisition node and task allocation node
Breath, is numbered acquisition node according to registration order, communicates to obtain online according to certain frequency and acquisition node
Acquisition node；According to registration order, task allocation node is numbered, carries out with task allocation node according to certain frequency
Communication is to obtain online task allocation node；
Task warehouse: for receiving the acquisition tasks that task allocation node sends；
Acquisition node: for obtaining other online acquisition node listings from management node, obtain the most required from task warehouse
Gathering of task, and according to the priority orders of acquisition tasks, the data of field control station are acquired, collection result is deposited
Enter in data base.
Large-scale distributed data collecting system the most according to claim 1, it is characterised in that described task distribution joint
Point, the task list of required collection is synchronized in other online task allocation nodes by it.
Large-scale distributed data collecting system the most according to claim 1, it is characterised in that described task distribution joint
Point includes task setting module, task allocating module and Mission Monitor module, wherein,
Task setting module: for obtaining other online task allocation node lists in real time, according to the actual requirements from management node
Set required gathering of task, and the acquisition tasks in task list is circulated one by one all give all online task allocation nodes
Task allocating module；
Task allocating module: for obtaining online acquisition node list and other online task distribution in real time from management node
Node listing, thus obtain the running state information of each online acquisition node and the initial running status threshold value that it is arranged,
Priority according to acquired acquisition tasks and the running state information of each online acquisition node, distribute acquisition tasks successively
To online and be in the acquisition node of normal operating condition, and the acquisition tasks distributing to target acquisition node is followed as backup
Ring distributes to lower two the numbering online acquisition nodes being adjacent, and allocation result is sent to task warehouse；
Mission Monitor module: in real time to the distribution of acquisition tasks on respective node, perform and wait situation is monitored.
Large-scale distributed data collecting system the most according to claim 1, it is characterised in that described management node mould
Block, also includes managing switch, and described management node uses redundant configuration mode, including main management node and standby management joint
Point；
Management switch: for catching the fault message of main management node, it is achieved between main management node and standby management node
Data syn-chronization between switching and main management node and standby management node.
Large-scale distributed data collecting system the most according to claim 1, it is characterised in that described task warehouse mould
Block, also includes that task switcher, described task warehouse use redundant configuration mode, including main task warehouse and standby task storehouse
Storehouse；
Task switcher: for catching the fault message in main task warehouse, it is achieved between main task warehouse and standby task warehouse
Data syn-chronization between switching and main task warehouse and standby task warehouse.
6. using the collecting method that the large-scale distributed data collecting system described in claim 1 is carried out, its feature exists
In,
Comprise the following steps:
Step 1, system is carried out Initialize installation；
Step 2, the quantity configuring required field control station according to actual needs and field apparatus, and open successively and configured
Field apparatus and field control station；
Step 3, unlatching management node and task warehouse, and open each acquisition node and task allocation node successively；
Step 4, each acquisition node send log-on message from trend management node after opening, and management node is according to registration order pair
Acquisition node is numbered, and communicates to obtain online acquisition node according to certain frequency and acquisition node；
Step 5, each task allocation node send log-on message from trend management node after opening, and management node is suitable according to registration
Ordered pair task allocation node is numbered, and communicates to obtain online task according to certain frequency and task allocation node and divides
Join node；
Step 6, the task setting module of each online task allocation node obtain other online appointing in real time from management node respectively
Business distribution node list, task allocating module obtains online acquisition node list in real time from management node respectively and other are online
Task allocation node list, each online acquisition node obtains other online acquisition nodes row in real time from management node respectively
Table；
Step 7, according to the actual requirements, needed for using any one task setting module settings within online task allocation node
Gathering of task, and the acquisition tasks in task list is evenly distributed to the task distribution mould of each online task allocation node
Block；
Task allocating module within step 8, each online task allocation node obtains the running status letter of each online acquisition node
Breath and the running status threshold value initially it arranged, according to the priority of acquisition tasks acquired in it, and each online acquisition joint
The running state information of point, successively by acquisition tasks distribution to online and be in the acquisition node of normal operating condition, and will point
The acquisition tasks of dispensing target acquisition node gives, as backup cycle assignment, lower two the numbering online acquisition nodes being adjacent,
Allocation result is sent to task warehouse；Mission Monitor module within the most each online task allocation node is in real time to each
The distribution of acquisition tasks on node, perform and wait situation is monitored；
Step 9, each online acquisition node obtain the most required gathering of task respectively from task warehouse；
The data of field control station are acquired by step 10, each online acquisition node according to the priority orders of acquisition tasks,
And collection result is stored in data base.
Collecting method the most according to claim 6, it is characterised in that system is initialized described in step 1
Arrange；Particularly as follows: arrange main management node, standby management node, each task allocation node, main task warehouse, standby task warehouse and
The IP address of each acquisition node and port numbers, and clock synchronization and the running status threshold value of each acquisition node are set.
Collecting method the most according to claim 6, it is characterised in that described in step 7 according to the actual requirements, uses
Required gathering of the task of any one task setting module settings within online task allocation node, and by task list
Acquisition tasks is averagely allocated to the task allocating module of each online task allocation node；Specifically comprise the following steps that
Step 7-1, according to the actual requirements, uses any one task setting module settings institute within online task allocation node
Need to gathering of task；
Step 7-2, the acquisition tasks in task list is circulated one by one and all gives the task of each online task allocation node and divide
Join module, until all of acquisition tasks is distributed on task allocating module in task list.
Collecting method the most according to claim 6, it is characterised in that each online task distribution joint described in step 8
Task allocating module within Dian obtains the running state information of each online acquisition node and the initial running status arranging it
Threshold value, according to the priority of acquisition tasks acquired in it, and the running state information of each online acquisition node, will gather successively
Task distribution is to online and be in the acquisition node of normal operating condition, and the acquisition tasks that will distribute to target acquisition node is made
Give, for backup cycle assignment, lower two the numbering online acquisition nodes being adjacent, allocation result is sent to task warehouse；
Mission Monitor module within the most each online task allocation node in real time to the distribution of acquisition tasks on respective node, execution and
Wait situation is monitored；Specifically comprise the following steps that
Step 8-1, each online task allocation node task allocating module respectively to its acquisition tasks got according to excellent
First level is ranked, and according to online acquisition node listing, the running state information of poll each online acquisition node and initial
The running status threshold value that it is arranged；
Step 8-2, by the running state information of relatively each online acquisition node and the running status threshold that initially it arranged
Value, judges that each online acquisition node is up state or overlond running state, respectively if online acquisition node is just in
Often running status, then according to the running state information of this online acquisition node, according to task priority order to this online acquisition
Node distribution acquisition tasks；If online acquisition node is in overlond running state, the distribution of this online acquisition node is not gathered
Task；
Step 8-3, give be adjacent lower two using the acquisition tasks distributing to target acquisition node as backup cycle assignment
Numbering online acquisition node；
Its allocation result is sent to task warehouse by step 8-4, the task allocating module of each online task allocation node respectively；
Step 8-5, each online task allocation node Mission Monitor module respectively to the distribution of acquisition tasks on respective node,
Perform and wait situation is monitored；
Breaking down if step 8-6 exists task allocation node, the most above-mentioned task allocation node is off-line state, and other are online
Task allocation node obtain the online task allocation node list after renewal in real time from management node, task setting module will be appointed
Acquisition tasks in business list circulates the task allocating module all giving each online task allocation node one by one, until task arranges
Till all of acquisition tasks is distributed on task allocating module in table, returns and perform step 8-1.
Collecting method the most according to claim 6, it is characterised in that each online acquisition node described in step 10
According to the priority orders of acquisition tasks, the data of field control station are acquired, and collection result is stored in data base；
Particularly as follows:
The data of field control station are adopted by step 10-1, each online acquisition node according to the priority orders of acquisition tasks
Collection, under normal circumstances, the acquisition tasks that lower two the numbering online acquisition nodes adjacent with target acquisition node are backed up only is made
Local cache, is not stored in data base；
Step 10-2, when there is acquisition node fault, this acquisition node will be in off-line state, in current sample period,
The collection result of backup acquisition tasks is stored in data base by first adjacent numbering online acquisition node from local cache district；
If the most adjacent first numbering online acquisition node also breaks down, second the most adjacent numbering online acquisition node will
The collection result of backup acquisition tasks is stored in data base from local cache district；
Step 10-3, when starting the next sampling period, each online task allocation node gets renewal by management node
After online acquisition node listing；Acquisition tasks is distributed to online acquisition node according to priority orders again, accordingly
Online acquisition node performs its acquisition tasks obtained and execution result is stored in data base.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610522950.XA CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610522950.XA CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106126346A true CN106126346A (en) | 2016-11-16 |
| CN106126346B CN106126346B (en) | 2019-02-26 |
Family
ID=57468566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610522950.XA Active CN106126346B (en) | 2016-07-05 | 2016-07-05 | A kind of large-scale distributed data collection system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106126346B (en) |
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107147525A (en) * | 2017-05-12 | 2017-09-08 | 山东浪潮商用系统有限公司 | Based on the method for realizing business datum recovery across station switching collection |
| CN107154973A (en) * | 2017-05-08 | 2017-09-12 | 成都汇智远景科技有限公司 | Distributed disease detection method |
| CN107291544A (en) * | 2017-08-03 | 2017-10-24 | 山东浪潮云服务信息科技有限公司 | Method and device, the distributed task scheduling execution system of task scheduling |
| CN107464056A (en) * | 2017-08-03 | 2017-12-12 | 武汉远众科技有限公司 | A kind of offline real-time task type collecting method based on mobile phone |
| CN107704320A (en) * | 2017-05-12 | 2018-02-16 | 贵州白山云科技有限公司 | The method for allocating tasks and system of a kind of distributed system |
| CN107886692A (en) * | 2017-10-30 | 2018-04-06 | 皖西学院 | A kind of bioengineering sensor LoRa wireless network communication systems |
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| CN108536539A (en) * | 2018-04-26 | 2018-09-14 | 东北大学 | A kind of method for scheduling task in industrial allocation formula data collecting system |
| CN108769115A (en) * | 2018-04-19 | 2018-11-06 | 中国科学院计算技术研究所 | Distributed RSS data acquisition method and system |
| CN108810042A (en) * | 2017-04-28 | 2018-11-13 | 华为技术有限公司 | A kind of task processing method, relevant device and system |
| CN109412862A (en) * | 2018-11-20 | 2019-03-01 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of looped network type SCADA system |
| CN109495365A (en) * | 2018-11-20 | 2019-03-19 | 新智能源系统控制有限责任公司 | A kind of monocycle net type SCADA system |
| CN109542061A (en) * | 2018-11-20 | 2019-03-29 | 新智能源系统控制有限责任公司 | A kind of looped network type SCADA system |
| CN109542595A (en) * | 2017-09-21 | 2019-03-29 | 阿里巴巴集团控股有限公司 | A kind of collecting method, device and system |
| CN109560962A (en) * | 2018-11-20 | 2019-04-02 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of monocycle net type SCADA system |
| CN109656707A (en) * | 2018-12-26 | 2019-04-19 | 鼎信信息科技有限责任公司 | Measurement data acquisition method, apparatus, computer equipment and storage medium |
| CN109711779A (en) * | 2018-12-26 | 2019-05-03 | 广东嘉腾机器人自动化有限公司 | A kind of order allocation method and storage device and order distribution system |
| CN110389856A (en) * | 2018-04-20 | 2019-10-29 | 伊姆西Ip控股有限责任公司 | Method, equipment and computer program product for migrating data |
| CN110401698A (en) * | 2019-06-27 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of monitoring data collection methods, devices and systems of task based access control asynchronous distribution |
| WO2019237531A1 (en) * | 2018-06-14 | 2019-12-19 | 平安科技（深圳）有限公司 | Network node monitoring method and system |
| CN111092921A (en) * | 2018-10-24 | 2020-05-01 | 北大方正集团有限公司 | Data acquisition method, device and storage medium |
| CN111581033A (en) * | 2019-02-19 | 2020-08-25 | 青岛海信网络科技股份有限公司 | Load balancing method, system and device |
| CN111614716A (en) * | 2020-04-10 | 2020-09-01 | 厦门网宿有限公司 | Method and system for collecting remote equipment data |
| CN112637368A (en) * | 2021-03-10 | 2021-04-09 | 江苏金恒信息科技股份有限公司 | Distributed industrial data acquisition system and method |
| CN112714464A (en) * | 2020-12-25 | 2021-04-27 | 中国工程物理研究院电子工程研究所 | Large-scale distributed monitoring method based on position information matching |
| CN113419451A (en) * | 2021-06-29 | 2021-09-21 | 浙江邮电职业技术学院 | Distributed data acquisition method and system for communication network |
| CN113626161A (en) * | 2021-07-09 | 2021-11-09 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN114374695A (en) * | 2021-11-26 | 2022-04-19 | 智昌科技集团股份有限公司 | Operation method and system of distributed data acquisition system |
| WO2023273528A1 (en) * | 2021-06-30 | 2023-01-05 | 青岛海尔空调电子有限公司 | Polling communication method and apparatus |
| CN116720645A (en) * | 2022-10-21 | 2023-09-08 | 荣耀终端有限公司 | Task distribution method, distribution equipment and task distribution system |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622062A (en) * | 2003-11-26 | 2005-06-01 | 中国科学院沈阳自动化研究所 | Distributed underwater robot control system |
| CN101873005A (en) * | 2010-06-17 | 2010-10-27 | 深圳市科陆电子科技股份有限公司 | Method for realizing balanced acquisition of electric energy |
| CN102045402A (en) * | 2010-12-28 | 2011-05-04 | 云海创想信息技术（北京）有限公司 | Data acquisition method and data acquisition system of distributed file system |
| CN103369054A (en) * | 2013-07-30 | 2013-10-23 | 北京搜狐新媒体信息技术有限公司 | Acquisition task management method and system |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN105430057A (en) * | 2015-11-03 | 2016-03-23 | 上海新储集成电路有限公司 | Internet of Things (IoT) structure and data transmission method thereof |
| US20160085399A1 (en) * | 2014-09-19 | 2016-03-24 | Impetus Technologies, Inc. | Real Time Streaming Analytics Platform |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN105527948A (en) * | 2015-12-11 | 2016-04-27 | 东北大学 | Large scale distributed data acquisition system and method based on industrial process |
| CN105553699A (en) * | 2015-12-09 | 2016-05-04 | 东北大学 | Wind turbine generator set state monitoring private cloud system and method built from point of data processing |
| CN105549556A (en) * | 2015-12-17 | 2016-05-04 | 东北大学 | Data collection transmission system and method for real-time monitoring of industrial process |
-
2016
- 2016-07-05 CN CN201610522950.XA patent/CN106126346B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622062A (en) * | 2003-11-26 | 2005-06-01 | 中国科学院沈阳自动化研究所 | Distributed underwater robot control system |
| CN101873005A (en) * | 2010-06-17 | 2010-10-27 | 深圳市科陆电子科技股份有限公司 | Method for realizing balanced acquisition of electric energy |
| CN102045402A (en) * | 2010-12-28 | 2011-05-04 | 云海创想信息技术（北京）有限公司 | Data acquisition method and data acquisition system of distributed file system |
| CN103369054A (en) * | 2013-07-30 | 2013-10-23 | 北京搜狐新媒体信息技术有限公司 | Acquisition task management method and system |
| US20160085399A1 (en) * | 2014-09-19 | 2016-03-24 | Impetus Technologies, Inc. | Real Time Streaming Analytics Platform |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN105430057A (en) * | 2015-11-03 | 2016-03-23 | 上海新储集成电路有限公司 | Internet of Things (IoT) structure and data transmission method thereof |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN105553699A (en) * | 2015-12-09 | 2016-05-04 | 东北大学 | Wind turbine generator set state monitoring private cloud system and method built from point of data processing |
| CN105527948A (en) * | 2015-12-11 | 2016-04-27 | 东北大学 | Large scale distributed data acquisition system and method based on industrial process |
| CN105549556A (en) * | 2015-12-17 | 2016-05-04 | 东北大学 | Data collection transmission system and method for real-time monitoring of industrial process |
Non-Patent Citations (1)
| Title |
|---|
| 韩国政，邱洪泽: "分布式数据采集系统的设计和实现", 《华中电力》 * |
Cited By (43)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
| CN108241528B (en) * | 2017-01-19 | 2020-10-09 | 上海直真君智科技有限公司 | Dynamic acquisition method for mass network security data customized by user |
| CN108810042A (en) * | 2017-04-28 | 2018-11-13 | 华为技术有限公司 | A kind of task processing method, relevant device and system |
| CN107154973A (en) * | 2017-05-08 | 2017-09-12 | 成都汇智远景科技有限公司 | Distributed disease detection method |
| CN107704320A (en) * | 2017-05-12 | 2018-02-16 | 贵州白山云科技有限公司 | The method for allocating tasks and system of a kind of distributed system |
| CN107147525A (en) * | 2017-05-12 | 2017-09-08 | 山东浪潮商用系统有限公司 | Based on the method for realizing business datum recovery across station switching collection |
| WO2018205890A1 (en) * | 2017-05-12 | 2018-11-15 | 贵州白山云科技有限公司 | Task assignment method and system of distributed system, computer readable storage medium and computer device therefor |
| CN107291544A (en) * | 2017-08-03 | 2017-10-24 | 山东浪潮云服务信息科技有限公司 | Method and device, the distributed task scheduling execution system of task scheduling |
| CN107464056A (en) * | 2017-08-03 | 2017-12-12 | 武汉远众科技有限公司 | A kind of offline real-time task type collecting method based on mobile phone |
| CN107291544B (en) * | 2017-08-03 | 2020-07-28 | 浪潮云信息技术股份公司 | Task scheduling method and device and distributed task execution system |
| CN109542595B (en) * | 2017-09-21 | 2023-02-24 | 阿里巴巴集团控股有限公司 | Data acquisition method, device and system |
| CN109542595A (en) * | 2017-09-21 | 2019-03-29 | 阿里巴巴集团控股有限公司 | A kind of collecting method, device and system |
| CN107886692A (en) * | 2017-10-30 | 2018-04-06 | 皖西学院 | A kind of bioengineering sensor LoRa wireless network communication systems |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| CN108769115A (en) * | 2018-04-19 | 2018-11-06 | 中国科学院计算技术研究所 | Distributed RSS data acquisition method and system |
| CN110389856A (en) * | 2018-04-20 | 2019-10-29 | 伊姆西Ip控股有限责任公司 | Method, equipment and computer program product for migrating data |
| CN108536539A (en) * | 2018-04-26 | 2018-09-14 | 东北大学 | A kind of method for scheduling task in industrial allocation formula data collecting system |
| CN108536539B (en) * | 2018-04-26 | 2021-12-17 | 东北大学 | Task scheduling method in industrial distributed data acquisition system |
| WO2019237531A1 (en) * | 2018-06-14 | 2019-12-19 | 平安科技（深圳）有限公司 | Network node monitoring method and system |
| CN111092921A (en) * | 2018-10-24 | 2020-05-01 | 北大方正集团有限公司 | Data acquisition method, device and storage medium |
| CN109560962A (en) * | 2018-11-20 | 2019-04-02 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of monocycle net type SCADA system |
| CN109542061A (en) * | 2018-11-20 | 2019-03-29 | 新智能源系统控制有限责任公司 | A kind of looped network type SCADA system |
| CN109495365A (en) * | 2018-11-20 | 2019-03-19 | 新智能源系统控制有限责任公司 | A kind of monocycle net type SCADA system |
| CN109412862A (en) * | 2018-11-20 | 2019-03-01 | 新智能源系统控制有限责任公司 | A kind of multinode collecting method of looped network type SCADA system |
| CN109495365B (en) * | 2018-11-20 | 2022-02-18 | 新智能源系统控制有限责任公司 | Single-ring network type SCADA system |
| CN109711779A (en) * | 2018-12-26 | 2019-05-03 | 广东嘉腾机器人自动化有限公司 | A kind of order allocation method and storage device and order distribution system |
| CN109656707A (en) * | 2018-12-26 | 2019-04-19 | 鼎信信息科技有限责任公司 | Measurement data acquisition method, apparatus, computer equipment and storage medium |
| CN109656707B (en) * | 2018-12-26 | 2020-03-31 | 南方电网数字电网研究院有限公司 | Metering data acquisition method and device, computer equipment and storage medium |
| CN111581033A (en) * | 2019-02-19 | 2020-08-25 | 青岛海信网络科技股份有限公司 | Load balancing method, system and device |
| CN111581033B (en) * | 2019-02-19 | 2023-10-27 | 青岛海信网络科技股份有限公司 | Load balancing method, system and device |
| CN110401698A (en) * | 2019-06-27 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of monitoring data collection methods, devices and systems of task based access control asynchronous distribution |
| CN111614716B (en) * | 2020-04-10 | 2022-12-16 | 厦门网宿有限公司 | Method and system for collecting remote equipment data |
| CN111614716A (en) * | 2020-04-10 | 2020-09-01 | 厦门网宿有限公司 | Method and system for collecting remote equipment data |
| CN112714464A (en) * | 2020-12-25 | 2021-04-27 | 中国工程物理研究院电子工程研究所 | Large-scale distributed monitoring method based on position information matching |
| CN112637368B (en) * | 2021-03-10 | 2021-05-14 | 江苏金恒信息科技股份有限公司 | Distributed industrial data acquisition system and method |
| CN112637368A (en) * | 2021-03-10 | 2021-04-09 | 江苏金恒信息科技股份有限公司 | Distributed industrial data acquisition system and method |
| CN113419451A (en) * | 2021-06-29 | 2021-09-21 | 浙江邮电职业技术学院 | Distributed data acquisition method and system for communication network |
| WO2023273528A1 (en) * | 2021-06-30 | 2023-01-05 | 青岛海尔空调电子有限公司 | Polling communication method and apparatus |
| CN113626161A (en) * | 2021-07-09 | 2021-11-09 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN113626161B (en) * | 2021-07-09 | 2023-12-22 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN114374695A (en) * | 2021-11-26 | 2022-04-19 | 智昌科技集团股份有限公司 | Operation method and system of distributed data acquisition system |
| CN114374695B (en) * | 2021-11-26 | 2022-06-28 | 智昌科技集团股份有限公司 | Operation method and system of distributed data acquisition system |
| CN116720645A (en) * | 2022-10-21 | 2023-09-08 | 荣耀终端有限公司 | Task distribution method, distribution equipment and task distribution system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106126346B (en) | 2019-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106126346A (en) | A kind of large-scale distributed data collecting system and method | |
| CN103236949B (en) | Monitoring method, device and the system of a kind of server cluster | |
| CN102739775B (en) | The monitoring of internet of things data acquisition server cluster and management method | |
| CN103685486B (en) | Distributed system monitoring method stepping over data center clusters and system | |
| CN109451065A (en) | A kind of soft load balancing shunts automated system and its operation method | |
| CN106357426A (en) | Large-scale distribution intelligent data collection system and method based on industrial cloud | |
| CN101571813B (en) | Master/slave scheduling method in multimachine assembly | |
| CN103343681B (en) | The integrated double loop network coupling control system of rig based on IPC and PLC | |
| CN102497279B (en) | Component management method applicable to tract traffic comprehensive monitoring system | |
| CN106790436B (en) | Traffic system monitoring method based on cloud architecture and control center cloud server | |
| CN107682411A (en) | A kind of extensive SDN controllers cluster and network system | |
| CN113055232B (en) | Network configuration deployment method, device and equipment | |
| CN106375480A (en) | Electric energy data real-time acquisition system and method based on distributed system | |
| CN112134293A (en) | Station terminal public unit and wide area autonomous method based on edge calculation | |
| CN109845192A (en) | Computer system and method for the network that dynamically internet adapter software defines | |
| CN112423365A (en) | MESH networking communication method for LoRa communication of power distribution station equipment | |
| CN105892364B (en) | Water utilities Centralized Monitoring management system | |
| CN111262723B (en) | Edge intelligent computing platform based on modularized hardware and software definition | |
| CN101247206B (en) | Layered transmission method and control system of peer-to-peer network | |
| CN112788798A (en) | Master-slave negotiation multi-gateway intelligent Zigbee network system | |
| CN107918273A (en) | A kind of application system of mobile technology of Internet of things on halved belt sorter | |
| CN105872055A (en) | Online monitoring method and system for computer systems in network distributed deployment mode | |
| CN101587351B (en) | Reliable communication method of intelligent electronic device | |
| CN114338670B (en) | Edge cloud platform and network-connected traffic three-level cloud control platform with same | |
| CN105450739A (en) | Monitoring and managing method for storage resource off-site and synchronous sharing in storage dual-active environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20161116 Assignee: Maogong Branch of Fushun Hanwang Aoniu Mining Co., Ltd. Assignor: Northeast University Contract record no.: X2019210000006 Denomination of invention: Large-scale distributed data collection system and method Granted publication date: 20190226 License type: Common License Record date: 20190916 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20161116 Assignee: LIAONING ATS INTELLIGENT TRANSPORTATION TECHNOLOGY Co.,Ltd. Assignor: Northeastern University Contract record no.: X2020210000028 Denomination of invention: A large scale distributed data acquisition system and method Granted publication date: 20190226 License type: Common License Record date: 20201127 |
|
| EE01 | Entry into force of recordation of patent licensing contract |