CN110188017A - Network computer room server and network equipment big data acquisition device and method - Google Patents
Network computer room server and network equipment big data acquisition device and method Download PDFInfo
- Publication number
- CN110188017A CN110188017A CN201910450454.1A CN201910450454A CN110188017A CN 110188017 A CN110188017 A CN 110188017A CN 201910450454 A CN201910450454 A CN 201910450454A CN 110188017 A CN110188017 A CN 110188017A
- Authority
- CN
- China
- Prior art keywords
- data
- network
- server
- acquisition
- network equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012423 maintenance Methods 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 4
- 239000013307 optical fiber Substances 0.000 claims description 3
- 230000000306 recurrent effect Effects 0.000 claims description 3
- 239000007858 starting material Substances 0.000 claims description 3
- 230000007704 transition Effects 0.000 abstract description 5
- 238000010276 construction Methods 0.000 abstract description 2
- 230000010354 integration Effects 0.000 abstract 1
- 238000012544 monitoring process Methods 0.000 description 7
- 230000036461 convulsion Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000006317 isomerization reaction Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of network computer room server and network equipment big data acquisition device and method, device includes: power module, CPU processor, RAM memory, ROM memory, flash storage, Network Interface Module.Acquisition method are as follows: check device state；Acquire journal file；Data uninterrupted is judged, the pass hull closure network interface if flow is excessive, if flow normally if carry out Data Integration；The data integrated and daily record data are stored in Hadoop big data platform.The present invention has the advantages that managing transition from labor management to Artificial intelligent acquisition, dependence of the computer room data collector to ability and experience is reduced, keeps an eye on the acquisition of network computer room big data to unattended transition from someone.Reduce network computer room server and the network equipment bring mistake and risk in data acquisition.Promote smart city informatization.It improves smart city service level, accelerate smart city and digital Construction with positive meaning.
Description
Technical field
The present invention relates to network big data technical field, in particular to a kind of network computer room server and the big number of the network equipment
According to acquisition device and method.
Background technique
With the continuous development of IT application in enterprises, the information system of operation is continuously increased, and some large enterprises and organ are single
The various software and hardware resources of position On-premise are continuously increased.During business system maintenance, the pressure of data processing can be got over
Come bigger, the workload and difficulty of system maintenance are continuously increased.Router, interchanger, PC server, various large-scale application systems
System, database, the various hardware numbers that the computer network facilities such as center monitoring equipment (including: various environment monitoring equipments) generate
According to, software process data, power consumption data, network log, system log, various application logs are continuously increased.
The various data that server in machine room and the network equipment generate (include: hardware data, software process data, various days
Will file), as to system application, the network equipment, the equipments of recording of user's operation are guaranteeing system security, removal system
Failure, many aspects such as monitoring system and network operation situation all play very important effect.
The various data that server in machine room and the network equipment generate (include: hardware data, software process data, various days
Will file) it analyzes to summarize and checks out system failure source, user's operation behavior, many information such as equipment running status.Such as
By checking that user's registration login record, customer incident record can find whether user is normal condition.To network configuration, if
Standby parameter, the record analysis of the behaviors such as interface variation, can obtain the information of network operation state.If can be in real time generating
The various data (including: hardware data, software process data, various journal files) that server in machine room and the network equipment generate are done
To real-time full dose acquire, so that it may carry out data analyze in real time or big data store.Network is formed by the various data of acquisition
Computer room big data training neural network diagnoses the various servers of network computer room and network equipment failure from Artificial Diagnosis to calculating
Machine artificial intelligence diagnosis' transition.The various big datas that computer room generates are converted to a kind of number that enterprise easily receives through the invention
According to form is showed, the value of data more can be clearly embodied.
With increasing rapidly for network demand, network application scale is increasingly huge, the high complexity and isomerization of network
Degree also increasingly improves, and the otherness of different vendor equipments and equipment management means makes network management more complicated, right
Effective maintenance of network, guarantees the running quality of network, has become a huge challenge.How data are more effectively carried out
Acquisition, and effectively support network management, the service quality of network is improved, is put in face of network management researcher
Property problem.
Effective monitoring to the state of network and its various pieces is the important means for guaranteeing network and working normally.Have
Effect monitoring network, and all data of network is accurately acquired, handled and analyzed also just be particularly important.Existing number
Bigger challenge is faced with according to acquisition system:
On the one hand, due to the surge of Internet resources quantity, the information content of data acquisition also increases with it, thus needs to go
Effective data collection strategy and method, in time by collected information formation data be submitted to backstage store, to guarantee
The high efficiency and reliability of acquisition.On the other hand, the network equipment is increasingly diversified, causes network type, manufacturer, device model
Etc. factors it is different, message form is ever-changing, how in the case where new device type emerges one after another, by effective information
Acquisition comes out, and improves the versatility of data acquisition, is one of main problem of data collecting field.
The shortcomings that prior art
1. the data that any server failure or the network equipment will likely influence acquisition upload, belongs to system bottleneck and ask
Topic.
2. also problems faced in technical staff's maintenance process is that information system management requirement basic data is more accurate steady
It is fixed, and existing acquisition equipment is also increasing with the fault point that acquisition data point increases appearance step by step；
3. different manufacturers all develops the Network Management System for being suitable for oneself equipment, acquires and communicate without uniform data
Interface lacks the corresponding coordination system.
4. various device data flows dynamically can not be monitored and be controlled in collection process.
5. generated due to server in machine room and the network equipment various data (include: hardware data, software process data,
Various journal files) total class is many and diverse does not carry out uniformly classification to data.
6. acquisition data procedures in the prior art can not monitoring room server and network equipment situation, acquire data mistake
A large amount of server in machine room and network device hardware resource and Internet resources and easy delay machine are occupied in journey.
Summary of the invention
The present invention in view of the drawbacks of the prior art, provides a kind of network computer room server and network equipment big data acquires
Device and method can effectively solve the above-mentioned problems of the prior art.
In order to realize the above goal of the invention, the technical solution adopted by the present invention is as follows:
A kind of network computer room server and network equipment big data acquisition device, including shell；
The shell leading portion installs six network interfaces 1 and an equipment debugging interface 2；
Shell back segment installs two network interfaces, 1, power interface 3 and a ups power interface 4；
So-called interior of shell is equipped with power module, CPU processor, RAM memory, ROM memory, flash storage, net
Network interface module, operating system, device management software, data acquisition software, data traffic control software；
Power module: the management for power supply and ups power；
CPU processor: central processing unit is for managing acquisition device；
RAM memory: for managing the calculator memory of acquisition device；
ROM memory: for the starting and maintenance of system, it is equivalent to computer BIOS；
Flash storage: some temporary files are stored and are equivalent to hard disc of computer；
Network Interface Module: 10,000,000,000 network interface of gigabit either optical fiber interface is provided；
Operating system: the hardware of management equipment；
Device management software: initializing the present apparatus and managed, and is formatted to flash storage；
Data acquisition software: using Apache Kafka, is responsible for the collecting work of data；
Data traffic controls software: being responsible for flow when control data acquisition.
The invention also discloses a kind of network computer room servers and network equipment big data acquisition method
Include the following steps:
S1: starter initialization；
S2: check device state, device jump to S1 extremely, and device normally enters S3；
S3: respectively from the journal file of server and network equipment acquisition data and server network device；
Log collection: server and network equipment log are acquired using Apache Flume；
Data format is as follows:
{ cpu frequency, memory usage, disk occupancy, network traffic data, process are living for collection of server data format
Jerk }；
The network equipment acquires data format { cpu frequency, memory usage, network traffic data }；
Power supply acquires devices collect data format { server power consumption, network equipment power consumption }；
Data acquisition finishes S4；
S4: judging data uninterrupted by logical recurrent neural network, if flow is excessive to arrive S5, if stream
Amount normally arrives S6；
S5: flow is reduced by flow equalization pass hull closure network interface, is accomplished to S3；
S6: collected data are integrated；
Integral data format:
{ cpu frequency, memory usage, disk occupancy, network traffic data, process are living for collection of server data format
Jerk, server power consumption }
The network equipment acquires data format { cpu frequency, memory usage, network traffic data, network equipment power consumption }
It is accomplished to S7；
S7: the data integrated and daily record data are stored in Hadoop big data platform.After return to S3.
Compared with prior art the present invention has the advantages that the various servers of network computer room and the network equipment is made to acquire data
Process manages transition, real-time monitoring server in machine room and network in collection process from labor management to Artificial intelligent acquisition
Device hardware resource and network resource status reduce computer room data collector to ability by Neural Network Optimization frequency acquisition
With the dependence of experience, keep an eye on the acquisition of network computer room big data to unattended transition from someone.Reduce network computer room server
And the network equipment bring mistake and risk in data acquisition.Promote smart city informatization.Improve wisdom city
City's service level accelerates smart city and digital Construction with positive meaning.
Detailed description of the invention
The main view of Fig. 1 acquisition device of the embodiment of the present invention；
The rearview of Fig. 2 acquisition device of the embodiment of the present invention；
Fig. 3 acquisition device of embodiment of the present invention internal hardware and module diagram；
Fig. 4 acquisition device of the embodiment of the present invention connects building-block of logic；
The acquisition method flow chart of Fig. 5 acquisition device of the embodiment of the present invention.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention more comprehensible, by the following examples, to the present invention do into
One step is described in detail.
As shown in Figure 1, 2, a kind of network computer room server and network equipment big data acquisition device, including shell；
The shell leading portion installs six network interfaces 1 and an equipment debugging interface 2；
Shell back segment installs two network interfaces, 1, power interface 3 and a ups power interface 4；
As shown in figure 3, so-called interior of shell be equipped with power module, CPU processor, RAM memory, ROM memory,
Flash storage, Network Interface Module, operating system, device management software, data acquisition software, data traffic control software；
Power module: the management for power supply and ups power；
CPU processor: central processing unit；
RAM memory: mainly running gear is interim, is equivalent to calculator memory；
ROM memory: for the starting and maintenance of system, it is equivalent to computer BIOS；
Flash storage: some temporary files are stored and are equivalent to hard disc of computer；
Network Interface Module: 10,000,000,000 network interface of gigabit (RJ45) either optical fiber interface is provided；
Operating system: the hardware of management equipment；
Device management software: initializing the present apparatus and managed, and is formatted to flash storage；
Data acquisition software: using Apache Kafka, is responsible for the collecting work of data；
Data traffic controls software: being responsible for flow when control data acquisition.
As shown in figure 4, the described acquisition device is for connecting each server of network computer room and the network equipment, big data array
Also it is connected with the device.
Big data array: it is responsible for storage network computer room server and network device hardware operation information.Such as: CPU usage
And information, memory service condition, hard disk movable progress information, network activity progress information, various application program service programs into
Journey information.
As shown in figure 5, the invention also discloses a kind of network computer room server and network equipment big data acquisition method, packet
Include following steps:
S1: starter initialization；
S2: check device state, device jump to S1 extremely, and device normally enters S3；
S3: respectively from the day of server, the network equipment, electric quantity of power supply acquisition devices collect data and server network device
Will file；
Log collection: server and network equipment log are acquired using Apache Flume；
Data format is as follows:
{ cpu frequency, memory usage, disk occupancy, network traffic data, process are living for collection of server data format
Jerk }
The network equipment acquires data format { cpu frequency, memory usage, network traffic data }
Power supply acquires devices collect data format { server power consumption, network equipment power consumption }
Data acquisition finishes S4
S4: judging data uninterrupted by logical recurrent neural network, if flow is excessive to arrive S5, if stream
Amount normally arrives S6；
S5: flow is reduced by flow equalization pass hull closure network interface, is accomplished to S3；
S6: collected data are integrated；
Integral data format:
{ cpu frequency, memory usage, disk occupancy, network traffic data, process are living for collection of server data format
Jerk, server power consumption }
The network equipment acquires data format { cpu frequency, memory usage, network traffic data, network equipment power consumption }
It is accomplished to S7
S7: the data integrated and daily record data are stored in Hadoop big data platform.After return to S3.
Those of ordinary skill in the art will understand that the embodiments described herein, which is to help reader, understands this hair
Bright implementation method, it should be understood that protection scope of the present invention is not limited to such specific embodiments and embodiments.Ability
The those of ordinary skill in domain disclosed the technical disclosures can make its various for not departing from essence of the invention according to the present invention
Its various specific variations and combinations, these variations and combinations are still within the scope of the present invention.
Claims (2)
1. a kind of network computer room server and network equipment big data acquisition device, including shell；
It is characterized by: the shell leading portion installs six network interfaces (1) and an equipment debugging interface (2)；
Shell back segment installs two network interfaces (1), a power interface (3) and a ups power interface (4)；
So-called interior of shell is equipped with power module, CPU processor, RAM memory, ROM memory, flash storage, network and connects
Mouth mold block, operating system, device management software, data acquisition software, data traffic control software；
Power module: the management for power supply and ups power；
CPU processor: central processing unit is for managing acquisition device；
RAM memory: for managing the calculator memory of acquisition device；
ROM memory: for the starting and maintenance of system, it is equivalent to computer BIOS；
Flash storage: some temporary files are stored and are equivalent to hard disc of computer；
Network Interface Module: 10,000,000,000 network interface of gigabit either optical fiber interface is provided；
Operating system: the hardware of management equipment；
Device management software: initializing the present apparatus and managed, and is formatted to flash storage；
Data acquisition software: using Apache Kafka, is responsible for the collecting work of data；
Data traffic controls software: being responsible for flow when control data acquisition.
2. the big data of a kind of network computer room server according to claim 1 and network equipment big data acquisition device is adopted
Set method, which comprises the steps of:
S1: starter initialization；
S2: check device state, device jump to S1 extremely, and device normally enters S3；
S3: respectively from the journal file of server and network equipment acquisition data and server network device；
Log collection: server and network equipment log are acquired using Apache Flume；
Data format is as follows:
{ cpu frequency, memory usage, disk occupancy, network traffic data, process are active for collection of server data format
Degree }；
The network equipment acquires data format { cpu frequency, memory usage, network traffic data }；
Power supply acquires devices collect data format { server power consumption, network equipment power consumption }；
Data acquisition finishes S4；
S4: judging data uninterrupted by logical recurrent neural network, if flow is excessive to arrive S5, if flow is just
Often arrive S6；
S5: flow is reduced by flow equalization pass hull closure network interface, is accomplished to S3；
S6: collected data are integrated；
Integral data format:
Collection of server data format cpu frequency, memory usage, disk occupancy, network traffic data, process liveness,
Server power consumption }；
The network equipment acquires data format { cpu frequency, memory usage, network traffic data, network equipment power consumption }；
It is accomplished to S7；
S7: the data integrated and daily record data are stored in Hadoop big data platform；After return to S3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910450454.1A CN110188017A (en) | 2019-05-28 | 2019-05-28 | Network computer room server and network equipment big data acquisition device and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910450454.1A CN110188017A (en) | 2019-05-28 | 2019-05-28 | Network computer room server and network equipment big data acquisition device and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110188017A true CN110188017A (en) | 2019-08-30 |
Family
ID=67718189
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910450454.1A Pending CN110188017A (en) | 2019-05-28 | 2019-05-28 | Network computer room server and network equipment big data acquisition device and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110188017A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933169A (en) * | 2016-07-04 | 2016-09-07 | 江苏飞搏软件股份有限公司 | Efficient, robust and safe large data polymerization system and method |
| CN106152394A (en) * | 2015-05-11 | 2016-11-23 | 李彬 | Data center's entirety energy consuming solution and system |
| CN108199922A (en) * | 2018-01-11 | 2018-06-22 | 承德石油高等专科学校 | A kind of system and method for diagnosing and repairing for the network equipment and server failure |
| CN108718306A (en) * | 2018-05-10 | 2018-10-30 | 北京邮电大学 | A kind of abnormal flow behavior method of discrimination and device |
| CN108880901A (en) * | 2018-06-29 | 2018-11-23 | 合肥微商圈信息科技有限公司 | System and method for diagnosing and repairing network equipment and server fault |
| CN108989129A (en) * | 2018-08-23 | 2018-12-11 | 承德石油高等专科学校 | A kind of device and method based on the storage of network big data, acquisition and analysis |
-
2019
- 2019-05-28 CN CN201910450454.1A patent/CN110188017A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106152394A (en) * | 2015-05-11 | 2016-11-23 | 李彬 | Data center's entirety energy consuming solution and system |
| CN105933169A (en) * | 2016-07-04 | 2016-09-07 | 江苏飞搏软件股份有限公司 | Efficient, robust and safe large data polymerization system and method |
| CN108199922A (en) * | 2018-01-11 | 2018-06-22 | 承德石油高等专科学校 | A kind of system and method for diagnosing and repairing for the network equipment and server failure |
| CN108718306A (en) * | 2018-05-10 | 2018-10-30 | 北京邮电大学 | A kind of abnormal flow behavior method of discrimination and device |
| CN108880901A (en) * | 2018-06-29 | 2018-11-23 | 合肥微商圈信息科技有限公司 | System and method for diagnosing and repairing network equipment and server fault |
| CN108989129A (en) * | 2018-08-23 | 2018-12-11 | 承德石油高等专科学校 | A kind of device and method based on the storage of network big data, acquisition and analysis |
Non-Patent Citations (1)
| Title |
|---|
| 柯艳霞等: "基于Internet的远程水环境监测系统研究与开发", 《承德石油高等专科学校学报》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107508722B (en) | Service monitoring method and device | |
| CN104022904B (en) | Distributed computer room information technoloy equipment management platform | |
| CN107330056B (en) | Wind power plant SCADA system based on big data cloud computing platform and operation method thereof | |
| CN108092813A (en) | Data center's total management system server hardware Governance framework and implementation method | |
| CN109034580B (en) | Information system overall health degree evaluation method based on big data analysis | |
| CN107343041B (en) | Accurate poverty alleviation management system and method based on cloud computing | |
| CN106375328A (en) | Adaptive optimization operation method of large-scale data distribution system | |
| Liu et al. | Availability prediction and modeling of high mobility oscar cluster | |
| CN113282635A (en) | Micro-service system fault root cause positioning method and device | |
| CN112579288A (en) | Cloud computing-based intelligent security data management system | |
| CN111124830B (en) | Micro-service monitoring method and device | |
| CN110827164A (en) | Intelligent aquaculture management system and method based on edge cloud | |
| Demirbaga et al. | Autodiagn: An automated real-time diagnosis framework for big data systems | |
| CN111858251A (en) | Big data computing technology-based data security audit method and system | |
| CN114675956B (en) | Method for configuration and scheduling of Pod between clusters based on Kubernetes | |
| CN109215158A (en) | A kind of PON equipment routing inspection method, apparatus and storage medium | |
| CN115619162A (en) | Power supply service command system based on cloud platform and micro-service architecture | |
| CN116340110A (en) | Automatic inspection method and system for cloud platform cluster monitoring index data | |
| Popović et al. | A novel cloud-based advanced distribution management system solution | |
| CN110188017A (en) | Network computer room server and network equipment big data acquisition device and method | |
| CN110415136B (en) | Service capability evaluation system and method for power dispatching automation system | |
| EP4242848B1 (en) | Method and computer system for capture and analysis of repetitive actions generated by the employee-computer interaction | |
| CN106649034A (en) | Visual intelligent operation and maintenance method and platform | |
| CN114090382B (en) | Health inspection method and device for super-converged cluster | |
| CN115495231A (en) | Dynamic resource scheduling method and system under complex scene of high concurrent tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190830 |