CN109981318A - The method and apparatus of data acquisition and procession - Google Patents
The method and apparatus of data acquisition and procession Download PDFInfo
- Publication number
- CN109981318A CN109981318A CN201711452146.XA CN201711452146A CN109981318A CN 109981318 A CN109981318 A CN 109981318A CN 201711452146 A CN201711452146 A CN 201711452146A CN 109981318 A CN109981318 A CN 109981318A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- module
- information
- time data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to network data management fields, and in particular to a kind of method and apparatus of data acquisition and procession.For the bad problem of the overall stability of the disposed of in its entirety speed and system of existing network information collection and processing, the device of network information gathering provided by the invention and processing, it is made of configuration management module, real-time data acquisition module, real time data processing module, data buffering module, data memory module, with loose couplings, scalability is strong, easy to maintain.The method of network information gathering provided by the invention and processing, real-time data acquisition module uses distributed type assemblies acquisition scheme, acquisition tasks N (N > 1) a program is distributed to by configuration strategy to go to execute, real time data processing module uses distributive type Computational frame, the calculating and processing to a large amount of acquisition data are completed by N (N > 1) a program, it can be ensured that data processing speed is faster.
Description
Technical field
The present invention relates to network data management fields, and in particular to a kind of method and apparatus of data acquisition and procession.
Background technique
With being constantly increasing for server in current IT environment, number of applications and type, each laboratory IT needs to pass through
Scientific and efficient means as far as possible in detail, in real time and accurately obtain in entire framework specific to each server, each system
The system even details of each application work, and accessed initial data can be analyzed, drawn and be counted, it is
Reference frame is established in subsequent Performance tuning, construction adjustment and all types of misarrangements, and therefore, data source is very more, type
Many samples, storage and data processing are in great demand, and the requirement to data exhibiting is also very high, focuses on the height of data processing
Effect property and availability are the characteristic features of current IT monitoring field.
Traditional data acquisition source is single, and storage, management and analysis data volume are also relatively small, mostly use relationship type greatly
Database and parallel data warehouse processing.Traditional data acquisition and processing (DAP) mainly include configuration management module, data acquisition with
Processing module and data memory module, referring to Fig. 1, configuration management module is mainly responsible for acquisition item management and Portable Batch System,
By visualized management interface, acquisition item and acquisition tasks are configured and dispatched, and dispatch command is sent to data and is adopted
In collection and processing module；Data acquisition and procession module, which is mainly responsible for, is managed Collection agent, according to acquisition tasks to adopting
Collection agency's distribution acquisition index, while the data of acquisition being calculated and being handled；Data memory module is mainly responsible for acquisition
Item, task schedule information, collecting data information carry out persistent storage.Data processing speed side is promoted to by parallel computation
For face, traditional parallel database technology pursues high consistency and fault-tolerance, it is difficult to guarantee its availability and scalability.
Traditional big data processing method needs to take data-centered mould centered on processor, and under big data environment
Formula reduces the mobile bring expense of data.Therefore, traditional data processing method has not adapted to the demand of big data.
Summary of the invention
In view of the deficiencies of the prior art, the present invention provides a kind of method and apparatus of data acquisition and procession.
The device of network information gathering provided by the invention and processing, by configuration management module, real time data buffer module,
Real-time data acquisition module, real time data processing module and memory module composition.Configuration management module mainly to experimental duties,
The essential information of equipment, the configuration for acquiring item and the configuration of scene discovery strategy are managed；Real time data buffer module is mainly right
The communication message of each module is stored and is distributed, and message communication and number between each module are handled using message-oriented middleware
According to caching；Real-time data acquisition module be mainly responsible for according to frequency acquisition under cyberspace environment host/server,
The facility information of router, interchanger and firewall is acquired in real time, while being carried out frequency conversion automatically according to alarm status and being adopted
Collection, the module complete the parallel acquisition of real time data based on distributed service framework；Real time data processing module is mainly responsible for pair
The data collected are calculated and are handled, and the information that acquisition is distinguished by the threshold values or amplitude of setting is in just
Often, alarm or alert status, real time data processing module is based on distributive type computing technique, to real-time data acquisition mould
The information of block acquisition carries out parallel processing；Memory module includes database service, mainly carries out persistent storage to data.
Preferably, real-time data acquisition module monitors submodule, configuration and distribution submodule, acquisition submodule by acquisition tasks
Block composition.(1) acquisition tasks submodule is mainly responsible for obtains task status information in real time, and is scheduled to task；(2) match
It sets to be mainly responsible for distribution submodule and the initialization registration that acquisition tasks monitor submodule and acquire submodule is configured, and according to
Acquisition index is distributed in each acquisition submodule by scheduler task；(3) acquisition submodule is made of N (N > 1) a capture program,
It is mainly responsible for and is communicated with the network equipment, acquisition tasks are completed according to acquisition index, the collecting data information that will acquire carries out
Storage.
The method of network information gathering provided by the invention and processing, comprising the following steps:
(1) configuration management module to memory module send management request, management request include information management, network device management,
Acquire item configuration management, discovery strategy configuration management etc.；Configuration management module is established database with memory module and is connect, and executes
Data manipulation, data manipulation include data increase, data deletion, data modification and data query, and configuration management module is by depositing
Store up the cache interface access cache data of module；The database server or caching of configuration management module reception memory module
The data information or status information that server returns, while storage optimization is passed through to the data information of acquisition and record information
Device interface is visualized, and management operation is completed.Configuration management module is communicated based on the API of real time data buffer module
Interface sends task schedule request to real time data buffer module, and task schedule request includes the stopping of task, suspends or open
It is dynamic；
(2) real-time data acquisition module receives task schedule request based on the API communication interface of real time data buffer module, and
It is operated according to task schedule request, if it is pause or is stopped, real-time data acquisition module will suspend or stop acquisition and appoint
Business；If it is starting, real-time data acquisition module starts acquisition tasks, is distributed, is passed through by item is acquired according to frequency acquisition
Distributed type assemblies acquisition technique synchronizes acquisition by N (N > 1) a program；Real-time data acquisition module is slow by real time data
The API communication interface of die block, the data of acquisition are cached；
(3) real time data processing module is obtained from message queue in real time based on the API communication interface of real time data buffer module
The network equipment acquires information, by distributive type Computational frame, acquisition item is calculated and is located using N (N > 1) a program
Reason, is verified and is recorded according to the information that configuration strategy acquires the network equipment, and pass through preset threshold values or vibration
Width come verify the network equipment acquisition information；It verifies to obtain network equipment warning message by threshold values, verifies to obtain by amplitude
Network equipment warning information；Real time data processing module is arrived record storage by real time data buffer module API communication interface
In message queue；Real-time data acquisition module is based on real time data buffer module API communication interface and obtains record, and according to note
Record state, the automatic frequency acquisition that changes are acquired the acquisition item that scene occurs；Real-time data acquisition module is by the number of acquisition
According to by real time data buffer module API communication interface by record storage into message queue；Real time data processing module is based on
The API communication interface of real time data buffer module obtains acquisition information in real time from message queue；It is calculated by distributive type
Frame is calculated and is handled to acquisition item using N (N > 1) a program；The information that the network equipment is acquired according to configuration strategy
It is verified and is recorded, and verify acquisition network device state by the threshold values of setting and amplitude: a) not reaching threshold values
Or information belongs to normal information in amplitude range, and in order to facilitate historical information inquiry, normal information is saved in database,
B) it verifies to obtain network equipment warning message by threshold values, on the one hand warning message can provide prompting message in visualization interface,
On the one hand the acquisition of automatic frequency-conversion rate, c are carried out according to alarm condition) it verifies to obtain network equipment warning information by amplitude range；
When based on storage optimization device interface, warning information is shown.
Network equipment state information record is saved in database by database interface；
(4) configuration management module sends data inquiry request to data memory module, and inquiry request includes that the network equipment is believed substantially
Breath, record information, recovery information etc.；Data memory module completes data query operation returned data information according to request；And base
It in storage optimization device interface, is visualized with list, patterned form, completes data manipulation.
Preferably, step (2) is specific as follows:
(i) acquisition tasks monitor submodule initialization block, and monitoring end registration information is submitted in configuration and distribution submodule and is carried out
Service registration；
(ii) submodule initialization block is acquired, collection terminal registration information is submitted in configuration and distribution submodule and carries out service note
Volume；
(iii) acquisition tasks monitor submodule and monitor internet message in real time, obtain task status information in real time, and according to task
It is scheduled；
(iv) on the one hand configuration is managed registration information with distribution submodule, is on the one hand advised by load and cluster configuration
Then, task acquisition index is distributed to N (N > 1) a capture program；
(v) acquisition index issued is carried out data acquisition according to snmp protocol by acquisition submodule, and is believed the data of acquisition
Breath carries out persistent storage.
The device of network information gathering provided by the invention and processing is by configuration management module, real-time data acquisition mould
Block, real time data processing module, data buffering module, data memory module composition, have loose couplings, scalability is strong, easily
Maintenance.The method of network information gathering provided by the invention and processing, based on the design of big data frame, real-time data acquisition
Module uses distributed type assemblies acquisition scheme, and acquisition tasks are distributed to N (N > 1) a program by configuration strategy and go to execute, can
Do not repeated with ensuring to acquire content, acquire data complete, acquisition performance reaches the second grade.Real time data processing module is using distribution
Formula streaming computing frame completes calculating and processing to a large amount of acquisition data by N (N > 1) a program, it can be ensured that data
Processing speed is faster.
Detailed description of the invention
Fig. 1 is traditional network information collection and process flow diagram.
Fig. 2 is the schematic device of the network information gathering that the embodiment of the present invention 1 provides and processing.
Fig. 3 is the schematic diagram for the real-time data acquisition module that the embodiment of the present invention 2 and embodiment 4 provide.
Fig. 4 is the data acquisition and procession flow chart for the normal frequency that the embodiment of the present invention 3 and embodiment 4 provide.
Fig. 5 is the data acquisition and procession flow chart for the automatic frequency-conversion that the embodiment of the present invention 3 and embodiment 4 provide.
Specific embodiment
To facilitate the understanding of the present invention, below in conjunction with Figure of description, the invention will be further described.
The device of embodiment 1 a kind of network information gathering and processing, referring to fig. 2, network information gathering provided by the invention
With the device of processing, by configuration management module, real time data buffer module, real-time data acquisition module, real time data processing mould
Block and memory module composition.Configuration management module is mainly to experimental duties, the essential information of equipment, the configuration for acquiring item and field
The configuration of scape discovery strategy is managed；Real time data buffer module is mainly stored and is distributed to the communication message of each module,
The caching of the message communication and data between each module is handled using message-oriented middleware；Real-time data acquisition module is mainly born
Duty according to frequency acquisition under cyberspace environment host/server, router, interchanger and firewall facility information
It is acquired in real time, while automatic frequency-conversion acquisition is carried out according to alarm status, real-time data acquisition module is based on Distributed Services
The parallel acquisition of frame completion real time data；Real time data processing module is mainly responsible for the calculating and processing of acquisition data, passes through
The threshold values or amplitude of setting are in normal, alarm, alert status, real time data processing module come the information for distinguishing acquisition
Based on distributive type computing technique, parallel processing is carried out to the information of real-time data acquisition module acquisition；Memory module includes
Database service mainly carries out persistent storage to data.
The device of the present embodiment is the explanation to the data acquisition and procession under normal mode, passes through big data frame skill
Art will optimize the mode of data acquisition and procession, make network information gathering and the performance of processing faster, network information gathering with
The data information of processing is more, and system can also carry out automatic frequency-conversion operation according to warning message under abnormal patterns.
The device of a kind of network information automatic frequency-conversion of embodiment 2 acquisition and processing, referring to figs. 2 and 3, by configuration management mould
Block, real time data buffer module, real-time data acquisition module, real time data processing module and memory module composition.Configuration management
Module is mainly managed experimental duties, the essential information of equipment, the configuration for acquiring item and the configuration of scene discovery strategy；It is real
When data buffering module mainly the communication message of each module is stored and is distributed, each mould is handled using message-oriented middleware
The caching of message communication and data between block；Real-time data acquisition module is mainly responsible for according to frequency acquisition to cyberspace ring
The facility information of host/server, router, interchanger and firewall under border is acquired in real time, while according to alarm shape
State carries out automatic frequency-conversion acquisition；Real-time data acquisition module is monitored submodule, configuration by acquisition tasks and is distributed submodule, adopts
Collect submodule composition.(1) task status information is obtained when acquisition tasks submodule mainly bears reality, and task is scheduled；
(2) configuration is mainly responsible for acquisition tasks monitoring submodule with distribution submodule and acquisition submodule initialization block registration configures, and root
Acquisition index is distributed in each acquisition submodule according to scheduler task；(3) acquisition submodule is made of N (N > 1) a capture program,
It is mainly responsible for and is communicated with the network equipment, acquisition tasks are completed according to acquisition index, the collecting data information that will acquire carries out
Storage can realize that acquisition tasks are synchronous, acquire data by this real-time data acquisition module and not repeat；Real time data processing mould
Block is mainly responsible for the calculating and processing of acquisition data, and the information that acquisition is distinguished by the threshold values or amplitude of setting is in just
Often, alarm, alert status, real time data processing module are based on distributive type computing technique, adopt to real-time data acquisition module
The information of collection carries out parallel processing；Memory module includes database service, mainly carries out persistent storage to data.
The device of the present embodiment is mainly for the explanation of real-time data acquisition module, and real-time data acquisition module is based on SNMP
Agreement (i.e. Simple Network Management Protocol) accessing network equipment obtains network state information, passes through Distributed Services framework, it is ensured that
The acquisition data of real time data are complete, do not repeat.
A kind of method of embodiment 3 network information gathering and processing, referring to fig. 4 and Fig. 5, configuration management module is to storage mould
Block sends management request, and management request includes information management, network device management, acquisition item configuration management, discovery strategy configuration
Management etc.；Configuration management module is established database with memory module and is connect, and data manipulation is executed, and data manipulation includes that data increase
Add, data are deleted, data modification and data query, configuration management module pass through the cache interface access cache number of memory module
According to；The data information or state letter that configuration management module receives the database server of memory module or cache server returns
Breath, while the data information of acquisition and record information are visualized by storage optimization device interface, complete management
Operation.API communication interface of the configuration management module based on real time data buffer module sends task to real time data buffer module
Scheduling request, task schedule request include stopping, pause or the starting of task；
Real-time data acquisition module receives task schedule based on the API communication interface of real time data buffer module and requests, and according to
Task schedule request is operated, and if it is pause or is stopped, and acquisition tasks line will be suspended or be stopped to real-time data acquisition module
Journey；If it is starting, real-time data acquisition module starts acquisition tasks, is distributed, is passed through by item is acquired according to frequency acquisition
Distributed type assemblies acquisition technique synchronizes acquisition by N (N > 1) a program；Real-time data acquisition module is slow by real time data
The API communication interface of die block, the data of acquisition are cached.
Real time data processing module is obtained from message queue in real time based on the API communication interface of real time data buffer module
The network equipment acquires information, by distributive type Computational frame, acquisition item is calculated and is located using N (N > 1) a program
Reason, is verified and is recorded according to the information that configuration strategy acquires the network equipment, and pass through preset threshold values or vibration
Width come verify the network equipment acquisition information；It verifies to obtain network equipment warning message by threshold values, verifies to obtain by amplitude
Network equipment warning information；Real time data processing module is arrived record storage by real time data buffer module API communication interface
In message queue；Real-time data acquisition module is based on real time data buffer module API communication interface and obtains record, and according to note
Record state, the automatic frequency acquisition that changes are acquired the acquisition item that scene occurs；Real-time data acquisition module is by the number of acquisition
According to by real time data buffer module API communication interface by record storage into message queue；Real time data processing module is based on
The API communication interface of real time data buffer module obtains acquisition information in real time from message queue；It is calculated by distributive type
Frame is calculated and is handled to acquisition item using N (N > 1) a program；The information that the network equipment is acquired according to configuration strategy
It is verified and is recorded, and verify acquisition network device state by the threshold values of setting and amplitude:
(1) belong to normal information not reaching information in threshold values or amplitude range, it, will in order to facilitate historical information inquiry
Normal information is saved in database；
(2) it verifies to obtain network equipment warning message by threshold values, on the one hand warning message can provide message in visualization interface
It reminds, the acquisition of automatic frequency-conversion rate is on the one hand carried out according to alarm condition；
(3) it verifies to obtain network equipment warning information by amplitude range；Based on storage optimization device interface by warning information into
Row is shown；
Acquisition information record is saved in database by database interface.
Configuration management module sends data inquiry request to data memory module, and inquiry request includes that the network equipment is believed substantially
Breath, record information, recovery information etc.；Data memory module completes data query operation, and returned data information according to request；
Configuration management module receives data information, completes data processing or configuration management module receives data information, and based on visual
Change browser interface, visualized with list, patterned form, completes data processing.
The present embodiment is mainly the method explanation to normal acquisition and automatic frequency-conversion acquisition aspect.
A kind of method of embodiment 4 network information gathering and processing, referring to Fig. 3-Fig. 5, configuration management module is to storage mould
Block sends management request, and management request includes information management, network device management, acquisition item configuration management, discovery strategy configuration
Management etc.；Configuration management module is established database with memory module and is connect, and data manipulation is executed, and data manipulation includes that data increase
Add, data are deleted, data modification and data query, configuration management module pass through the cache interface access cache number of memory module
According to；The data information or state letter that configuration management module receives the database server of memory module or cache server returns
Breath, while the data information of acquisition and record information are visualized by storage optimization device interface, complete management
Operation.API communication interface of the configuration management module based on real time data buffer module sends task to real time data buffer module
Scheduling request, task schedule request include stopping, pause, the starting of task；
Real-time data acquisition module receives task schedule request based on the API communication interface of real time data buffer module and completes in real time
Data collection task.Be embodied as follows: (i) acquisition tasks monitor submodule initialization block, will monitor end registration information and are submitted to
Service registration is carried out in configuration and distribution submodule；(ii) submodule initialization block is acquired, collection terminal registration information is submitted to and is matched
It sets and carries out service registration with distribution submodule；(iii) acquisition tasks monitor submodule and monitor scheduler task message in real time, and
Task status information is obtained in real time.Task schedule state includes the starting, stopping, pause of task, and state is to stop or suspend
When, acquisition tasks monitor submodule mission thread is stopped in time, pausing operation, state be starting when, acquisition tasks prison
It listens submodule to start acquisition tasks thread, sends data to configuration and distribution submodule；(iv) configuration and distribution one side of submodule
It is managed in face of registration information, on the one hand by load and cluster configuration rule, task acquisition index is adopted to N (N > 1) is a
Collection program is distributed；(v) acquisition submodule is according to snmp protocol, and the acquisition index issued is carried out data acquisition, and to adopting
The data information of collection carries out persistent storage.
Real time data processing module is obtained from message queue in real time based on the API communication interface of real time data buffer module
The network equipment acquires information, by distributive type Computational frame, acquisition item is calculated and is located using N (N > 1) a program
Reason, is verified and is recorded according to the information that configuration strategy acquires the network equipment, and pass through preset threshold values or vibration
Width come verify the network equipment acquisition information；It verifies to obtain network equipment warning message by threshold values, verifies to obtain by amplitude
Network equipment warning information；Real time data processing module is arrived record storage by real time data buffer module API communication interface
In message queue；Real-time data acquisition module is based on real time data buffer module API communication interface and obtains record, and according to note
Record state, the automatic frequency acquisition that changes are acquired the acquisition item that scene occurs；Real-time data acquisition module is by the number of acquisition
According to by real time data buffer module API communication interface by record storage into message queue；Real time data processing module is based on
The API communication interface of real time data buffer module obtains acquisition information in real time from message queue；It is calculated by distributive type
Frame is calculated and is handled to acquisition item using N (N > 1) a program；The information that the network equipment is acquired according to configuration strategy
It is verified and is recorded, and verify acquisition network device state by the threshold values of setting and amplitude:
(1) belong to normal information not reaching information in threshold values or amplitude range, it, will just in order to facilitate historical information inquiry
Normal information preservation is to database；
(2) it verifies to obtain network equipment warning message by threshold values, on the one hand warning message can provide message in visualization interface
It reminds, the acquisition of automatic frequency-conversion rate is on the one hand carried out according to alarm condition；
(3) it verifies to obtain network equipment warning information by amplitude range；And storage optimization device interface is based on by warning information
It is shown；
Information record is saved in database by database interface.
Configuration management module sends data inquiry request to data memory module, and inquiry request includes that the network equipment is believed substantially
Breath, record information, recovery information etc.；Data memory module completes data query operation, and returned data information according to request；
Configuration management module receives data information, completes data processing or configuration management module receives data information, and based on visual
Change browser interface, visualized with list, patterned form, completes data processing.
The present embodiment is the optimization to embodiment 3, is explained the submodule working principle of real-time data acquisition module
It states.
Claims (3)
1. the device of a kind of network information gathering and processing, by configuration management module, real time data buffer module, real time data
Acquisition module, real time data processing module and memory module composition: configuration management module mainly to experimental duties, equipment it is basic
Information, the configuration for acquiring item and the configuration of scene discovery strategy are managed；Real time data buffer module mainly leads to each module
News message is stored and is distributed, and the caching of the message communication and data between each module is handled using message-oriented middleware；It is real
When data acquisition module be mainly responsible for according to frequency acquisition to the host/server under cyberspace environment, router, exchange
The facility information of machine and firewall is acquired in real time, while carrying out automatic frequency-conversion acquisition according to alarm status, and real time data is adopted
Collect the parallel acquisition that module completes real time data based on distributed service framework；Real time data processing module is mainly responsible for acquisition
The calculating and processing of data, the information that acquisition is distinguished by the threshold values or amplitude of setting is in normal, alarm, early warning
State, real time data processing module are based on distributive type computing technique, carry out to the information of real-time data acquisition module acquisition
Parallel processing；Memory module includes database service, mainly carries out persistent storage to data.
2. the device of network information gathering according to claim 1 and processing, characterized in that the real-time data acquisition
Module is monitored submodule, configuration and distribution submodule, acquisition submodule by acquisition tasks and is formed；Acquisition tasks submodule is mainly born
Task status information is obtained when real, and task is scheduled, and is configured and is mainly responsible for acquisition tasks monitoring with distribution submodule
Submodule and acquisition submodule initialization block registration configuration, and acquisition index is distributed to by each acquisition submodule according to scheduler task
In, acquisition submodule is made of N (N > 1) a capture program, is mainly responsible for and is communicated with the network equipment, according to acquisition index
Acquisition tasks are completed, the collecting data information that will acquire is stored.
3. a kind of method of network information gathering and processing, comprising the following steps:
(1) configuration management module sends management request to memory module, and management request includes experiment scene information management, network
Equipment management, acquisition item configuration management, experiment scene discovery strategy configuration management etc.；Configuration management module and memory module are established
Database connection executes data manipulation, and data manipulation includes data increase, data deletion, data modification and data query, matches
Set the cache interface access cache data that management module passes through memory module；The database of configuration management module reception memory module
The data information or status information that server or cache server return, while to the data information and experiment scene of acquisition
Record information is visualized by storage optimization device interface, completes management operation；Configuration management module is based on real-time
The API communication interface of data buffering module sends task schedule request, task schedule request packet to real time data buffer module
Include the stopping, pause or starting of task；
(2) real-time data acquisition module receives task schedule request based on the API communication interface of real time data buffer module, and
It is operated according to task schedule request, if it is pause or is stopped, real-time data acquisition module will suspend or stop acquisition and appoint
Business；If it is starting, real-time data acquisition module starts acquisition tasks, is distributed, is passed through by item is acquired according to frequency acquisition
Distributed type assemblies acquisition technique synchronizes acquisition by N (N > 1) a program；Real-time data acquisition module is slow by real time data
The API communication interface of die block, the data of acquisition are cached；
(3) real time data processing module is obtained from message queue in real time based on the API communication interface of real time data buffer module
The network equipment acquires information, by distributive type Computational frame, acquisition item is calculated and is located using N (N > 1) a program
Reason, is verified and is recorded according to the information that experiment scene configuration strategy acquires the network equipment, and passes through preset valve
Value or amplitude verify the information of network equipment acquisition；It verifies to obtain experiment scene network equipment warning message by threshold values,
It verifies to obtain experiment scene network equipment warning information by amplitude；Real time data processing module buffers mould by real time data
Block API communication interface is by experiment scene record storage into message queue；Real-time data acquisition module is buffered based on real time data
Modules A PI communication interface obtains experiment scene record, and according to experiment scene recording status, the automatic frequency acquisition that changes is to field
The acquisition item that scape occurs is acquired；Real-time data acquisition module leads to the data of acquisition by real time data buffer module API
Communication interface is by experiment scene record storage into message queue；Real time data processing module is based on real time data buffer module
API communication interface obtains experiment scene acquisition information in real time from message queue；By distributive type Computational frame, using N
(N > 1) a program is calculated and is handled to acquisition item；The information that the network equipment is acquired according to experiment scene configuration strategy into
Row, which is verified, simultaneously to be recorded, and acquisition network device state is verified by the threshold values of setting and amplitude: a) do not reach threshold values or
Information belongs to normal information in person's amplitude range, and in order to facilitate historical information inquiry, normal information is saved in database, b)
It verifies to obtain experiment scene network equipment warning message by threshold values, which can provide message in visualization interface and mention
Wake up, and the acquisition of automatic frequency-conversion rate, c carried out according to the warning message) it verifies to obtain the experiment scene network equipment by amplitude range
Warning information, warning information only provide prompting message on visualization interface；Experiment scene information record passes through database interface
It is saved in database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711452146.XA CN109981318A (en) | 2017-12-28 | 2017-12-28 | The method and apparatus of data acquisition and procession |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711452146.XA CN109981318A (en) | 2017-12-28 | 2017-12-28 | The method and apparatus of data acquisition and procession |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109981318A true CN109981318A (en) | 2019-07-05 |
Family
ID=67072027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711452146.XA Pending CN109981318A (en) | 2017-12-28 | 2017-12-28 | The method and apparatus of data acquisition and procession |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109981318A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581196A (en) * | 2020-04-29 | 2020-08-25 | 深圳市双合电气股份有限公司 | Supply and distribution power grid intelligent data acquisition and arrangement system based on intelligent factory framework |
| CN112041867A (en) * | 2020-06-22 | 2020-12-04 | 西安市双合软件技术有限公司 | Intelligent factory data acquisition platform and implementation method thereof |
| CN112884942A (en) * | 2021-01-29 | 2021-06-01 | 中汽创智科技有限公司 | Data recording and playback system and playback method thereof |
| CN113114508A (en) * | 2021-04-15 | 2021-07-13 | 上海理想信息产业（集团）有限公司 | Multistage variable-frequency network monitoring data acquisition method and device |
-
2017
- 2017-12-28 CN CN201711452146.XA patent/CN109981318A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581196A (en) * | 2020-04-29 | 2020-08-25 | 深圳市双合电气股份有限公司 | Supply and distribution power grid intelligent data acquisition and arrangement system based on intelligent factory framework |
| CN112041867A (en) * | 2020-06-22 | 2020-12-04 | 西安市双合软件技术有限公司 | Intelligent factory data acquisition platform and implementation method thereof |
| CN112884942A (en) * | 2021-01-29 | 2021-06-01 | 中汽创智科技有限公司 | Data recording and playback system and playback method thereof |
| CN113114508A (en) * | 2021-04-15 | 2021-07-13 | 上海理想信息产业（集团）有限公司 | Multistage variable-frequency network monitoring data acquisition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104506373A (en) | Device and method for collecting and processing network information | |
| CN109981318A (en) | The method and apparatus of data acquisition and procession | |
| CN105224445B (en) | Distributed tracking system | |
| CN202058147U (en) | Distribution type real-time database management system | |
| CN106709003A (en) | Hadoop-based mass log data processing method | |
| CN105376110B (en) | The analysis method and system of network packet are realized with big data streaming technology | |
| CN111258979B (en) | Cloud protection log system and working method thereof | |
| CN109271412A (en) | The real-time streaming data processing method and system of smart city | |
| CN107390650A (en) | A kind of data collecting system based on Internet of Things and the data compression method based on the system | |
| CN102064975B (en) | Network equipment supervision method and system | |
| CN111464336A (en) | High-concurrency data processing method and system based on electric power communication machine room | |
| CN110995783B (en) | Remote monitoring system | |
| CN108804686A (en) | A kind of whole solution of internet of things data acquisition processing and analysis application | |
| CN112865311B (en) | Method and device for monitoring message bus of power system | |
| CN110597057A (en) | Data processing system in industrial application scene | |
| CN110347663A (en) | Dynamic environment monitoring method and system based on Complex event processing engine | |
| CN102142971B (en) | Realize supervisory control system and the method for supervising of subscription-collection mechanism | |
| CN106789270A (en) | Method and system for realizing centralized operation and maintenance management of information system | |
| CN111080500A (en) | Spark streaming based real-time rule deployment and control early warning method and device | |
| CN109347703A (en) | A kind of CPS node failure detection device and method | |
| CN108182263A (en) | A kind of date storage method of data center's total management system | |
| CN105260479A (en) | observer pattern-based real-time database real time data publishing method | |
| CN112422663B (en) | Data centralization system of industrial Internet of things and data aggregation, processing and storage method | |
| CN209400916U (en) | A kind of job Shop Management System based on Ethernet | |
| TWI448975B (en) | Dispersing-type algorithm system applicable to image monitoring platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190705 |