CN108173679A - A kind of storage system for IT system O＆M monitoring data - Google Patents
A kind of storage system for IT system O＆M monitoring data Download PDFInfo
- Publication number
- CN108173679A CN108173679A CN201711398481.6A CN201711398481A CN108173679A CN 108173679 A CN108173679 A CN 108173679A CN 201711398481 A CN201711398481 A CN 201711398481A CN 108173679 A CN108173679 A CN 108173679A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- polymerization
- initial
- aggregated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
- H04L43/067—Generation of reports using time frame reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of storage system for IT system O＆M monitoring data, including initial data receiving module, initial data memory module, initial data polymerization memory module, oligomerization level data polymerization memory module.The initial data that initial data receiving module passes through interface O＆M monitoring system, initial data is stored in raw data table by initial data memory module, initial data is polymerize and is stored to first order aggregated data table by initial data polymerization memory module, oligomerization level data polymerization memory module repeatedly polymerize the data in first order aggregated data table again, generates and store the aggregated data of higher level.
Description
Technical field
The present invention relates to the technologies such as network monitoring, data storage technology more particularly to a kind of IT system O＆M that is used for monitor
The storage system of data.
Background technology
With the continuous expansion of enterprises and institutions' scale, enterprises and institutions' portfolio increases, and the scale of information system constantly expands, and protects
Efficient, the safe and stable operation of barrier information system then becomes the most important thing of Enterprise Informatization system O＆M monitoring work.Work
It is intended to be apt to its thing, must first sharpen his tools, a set of effective O＆M monitoring system can effectively mitigate the burden of manual control flow, simultaneously
Also it plays a multiplier effect to improving business service quality.O＆M monitoring system common at present generally can be complete well
Into data collection task, but in terms of data storage mostly by the way of directly initial data is stored, this mode logarithm
Huge challenge is proposed according to storage service.In the case of 100 monitored objects, when monitored index is more improved its
The data storage capacity of every month is up to GB up to a hundred.The reading and writing data of O＆M monitoring system is different from general information system simultaneously,
The characteristics of but very high frequency small there are a secondary amounts is written in monitoring data, and monitoring data read-write is low there are frequency but inquiry data volume is big
And the characteristics of substantially along with a large amount of data mathematical operation, the prior art can not support this offer well.Therefore, it studies
It is a kind of can efficient storage and the storage system of O＆M monitoring data that efficiently reads be a preferable selection.
Invention content
In order to solve above-mentioned the deficiencies in the prior art, the present invention proposes a kind of depositing for IT system O＆M monitoring data
Storage system.For improving the reading efficiency of the memory capacity of data and data.
The present invention includes at least a database server, using relational data library storage data.Data storage uses
Original monitoring data short time storage, and data aggregate program often is performed by specific time, to initial data using repeatedly poly-
It closes, to compress the memory capacity of its occupancy, and maximum, minimum and average value prior to calculating data before data query.Pass through
To initial data, polymerization generates the aggregated data of minimum extent of polymerization for the first time, and polymerize generation again to oligomerization level data
The data of high extent of polymerization.According to different demands, multi-step polymerization can be divided, the difference of different degrees of aggregated data table exists
In the time span of each data representative and the holding time of data.When the data of different extent of polymerizations store different
Between, the higher time data memory of extent of polymerization is longer, and the lower time data memory of extent of polymerization is shorter.
Common monitoring data is divided into two classes according to its feature, in polymerization using different polymerization methods.The first kind, number
Value type data, such monitor control index represents the height of some performance of equipment, size degree, such as resource usage amount 50%；Second
Class, state type data, such monitor control index represents some state of equipment, as network is not connected with.During polymerization, with time span
It for foundation, sorts to all data according to acquisition time, all initial data in a time span or low degree is polymerize
For data aggregate into a data, primary sources calculate maximum value, minimum value, average value, and secondary sources calculate each number
The occurrence number of value.
System stores data by two categories below table：
Raw data table, for preserving the original monitoring data of O＆M monitoring system acquisition.There are multiple raw data tables, lead to
Spending the date distinguishes, and every table preserves the initial data of one day, and the initial data of same date is not stored in different original numbers
According in table.The field of preservation includes device id, monitor control index ID, monitoring numerical value, acquisition time.After system receives monitoring data
Data are directly preserved to the table.
Aggregated data table, for preserving monitoring data after polymerization.The aggregated data that first kind monitoring data preserves includes setting
Standby ID, monitor control index ID, average value, maximum value, minimum value, time, the aggregated data that the second class monitoring data preserves include setting
Standby ID, monitor control index ID, monitoring numerical value, occurrence number, number, time are always monitored.System is inserted by polymerizeing task to the table
Data.
The data source of raw data table in O＆M monitoring system, the data source of aggregated data table in initial data or
The aggregated data of oligomerization degree.
Specific embodiment
Below in conjunction with the drawings and specific embodiments, the present invention will be described in detail, but not as a limitation of the invention.
A kind of storage system for IT system O＆M monitoring data provided by the invention, Fig. 1 are specific for the method for the present invention
Implement schematic diagram, as shown in Figure 1, the system includes：
Initial data receiving module S1：Storage system passes through the initial data with interface O＆M monitoring system；
Initial data memory module S2：Initial data is stored in raw data table, there are multiple raw data tables, passes through
Date distinguishes, and every table preserves the initial data of one day, and the initial data of same date is not stored in different initial data
In table.The field of preservation includes device id, monitor control index ID, monitoring numerical value, acquisition time.System receives straight after monitoring data
It connects and preserves data to the table, data at most preserve 7 days.
Initial data polymerization memory module S3：An initial data polymerization is performed within every 1 hour, by the original number of first 1 hour
According to being polymerize.The time span of polymerization is 1 minute, is stored after polymerization to first order aggregated data table, it is small then to delete 12
When pre-polymerization first order aggregated data.
Oligomerization level data polymerization memory module S4：Data in first order aggregated data table are repeatedly gathered again
It closes, generates and store the aggregated data of higher level.
Wherein, the detailed process schematic diagram of Fig. 2 the method for the present invention specific implementation module S4, such as Fig. 2, oligomerization level data
Polymerization memory module S4 is specifically included：
First order aggregated data polymerization memory module S41：A first order aggregated data polymerization is performed within every 6 hours, by preceding 6
The initial data of hour is polymerize.The time span of polymerization is 5 minutes, is stored after polymerization to second level aggregated data table, so
The second level aggregated data of 2 days pre-polymerizations is deleted afterwards.
Second level aggregated data polymerization memory module S42：A second level aggregated data polymerization is performed within every 1 day, by first 1 day
Initial data polymerize.The time span of polymerization is 30 minutes, is stored after polymerization to third level aggregated data table, then deleted
Except the third level aggregated data of 7 days pre-polymerizations.
Third level aggregated data polymerization memory module S43：A third level aggregated data polymerization is performed within every 3 days, by first 3 days
Initial data polymerize.The time span of polymerization is 3 hours, is stored after polymerization to fourth stage aggregated data table, then deleted
Except the fourth stage aggregated data of 3 months pre-polymerizations.
Fourth stage aggregated data polymerization memory module S44：A fourth stage aggregated data polymerization is performed within every 1 month, by preceding 1
The initial data of a month is polymerize.The time span of polymerization is 1 day, is stored after polymerization to level V aggregated data table, then
Delete the level V aggregated data of five years ago polymerization.
Claims (3)
1. a kind of storage system for IT system O＆M monitoring data, which is characterized in that including：
Initial data receiving module：The initial data that storage system passes through interface O＆M monitoring system；
Initial data memory module：Initial data is stored in raw data table, there is multiple raw data tables, by the date into
Row is distinguished, and every table preserves the initial data of one day, and the initial data of same date is not stored in different raw data tables.It protects
The field deposited includes device id, monitor control index ID, monitoring numerical value, acquisition time.System directly preserves after receiving monitoring data
Data to the table, data only preserve certain time.
Initial data polymerization memory module S3：An initial data polymerization is performed per a period of time, by the period memory storage
Initial data is polymerize.It is stored after polymerization to first order aggregated data table, then deletes unwanted premature polymerization
First order aggregated data.
Oligomerization level data polymerization memory module S4：Data in first order aggregated data table are repeatedly polymerize again, are produced
Give birth to and store the aggregated data of higher level.
2. according to claim 1 in the storage system of IT system O＆M monitoring data, it is characterised in that by common monitoring number
It is divided into two classes according to according to its feature, in polymerization using different polymerization methods.The first kind, numeric type data, such monitoring refer to
Mark represents the height of some performance of equipment, size degree, such as resource usage amount 50%；Second class, state type data, such prison
Some state of index expression equipment is controlled, as network is not connected with.During polymerization, using time span as foundation, all data are pressed
It sorts according to acquisition time, all initial data in a time span or low degree aggregated data is aggregated into a data,
Primary sources calculate maximum value, minimum value, average value, and secondary sources calculate the occurrence number of each numerical value.
3. the storage system according to claim 1 for IT system O＆M monitoring data, which is characterized in that the system
Data are stored by two categories below table：
Raw data table, for preserving the original monitoring data of O＆M monitoring system acquisition.There are multiple raw data tables, pass through day
Phase distinguishes, and every table preserves the initial data of one day, and the initial data of same date is not stored in different raw data tables
In.The field of preservation includes device id, monitor control index ID, monitoring numerical value, acquisition time.System is received after monitoring data directly
Data are preserved to the table.
Aggregated data table, for preserving monitoring data after polymerization.First kind monitoring data preserve aggregated data include device id,
Monitor control index ID, average value, maximum value, minimum value, time, the second class monitoring data preserve aggregated data include device id,
Monitor control index ID, monitoring numerical value, occurrence number, number, time are always monitored.System is inserted into data by polymerizeing task to the table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711398481.6A CN108173679A (en) | 2017-12-21 | 2017-12-21 | A kind of storage system for IT system O＆M monitoring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711398481.6A CN108173679A (en) | 2017-12-21 | 2017-12-21 | A kind of storage system for IT system O＆M monitoring data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108173679A true CN108173679A (en) | 2018-06-15 |
Family
ID=62523479
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711398481.6A Pending CN108173679A (en) | 2017-12-21 | 2017-12-21 | A kind of storage system for IT system O＆M monitoring data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108173679A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955654A (en) * | 2018-09-26 | 2020-04-03 | 北京国双科技有限公司 | Multi-dimensional index calculation method and device |
-
2017
- 2017-12-21 CN CN201711398481.6A patent/CN108173679A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955654A (en) * | 2018-09-26 | 2020-04-03 | 北京国双科技有限公司 | Multi-dimensional index calculation method and device |
| CN110955654B (en) * | 2018-09-26 | 2023-10-31 | 北京国双科技有限公司 | Multi-dimensional index calculation method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103827826B (en) | Adaptively determining response time distribution of transactional workloads | |
| US9372889B1 (en) | Incremental statistics update | |
| US9665632B2 (en) | Managing activities over time in an activity graph | |
| US20130006996A1 (en) | Clustering E-Mails Using Collaborative Information | |
| CN104504116B (en) | A kind of storage method of real-time data base | |
| Agrawal et al. | Low-latency analytics on colossal data streams with summarystore | |
| CN101963982A (en) | Method for managing metadata of redundancy deletion and storage system based on location sensitive Hash | |
| US20180102952A1 (en) | Systems and methods for analyzing report access data | |
| US20140071135A1 (en) | Managing activities over time in an activity graph | |
| CN105446990B (en) | A kind of business data processing method and equipment | |
| CN106611064A (en) | Data processing method and device for distributed relational database | |
| CN108009290A (en) | A kind of data modeling and storage method of track traffic command centre gauze big data | |
| CN105677549A (en) | Software testing management method and system | |
| CN108415990B (en) | Data quality monitoring method and device, computer equipment and storage medium | |
| CN112241420A (en) | Government affair service item recommendation method based on association rule algorithm | |
| CN106844320A (en) | A kind of financial statement integration method and equipment | |
| CN104731716B (en) | A kind of date storage method | |
| CN108173679A (en) | A kind of storage system for IT system O＆M monitoring data | |
| US20170097787A1 (en) | Dynamic two-tier data storage utilization | |
| CN110019195A (en) | A kind of storage method and device of data | |
| US20140052591A1 (en) | Joining Of Logistical And Financial Data Related to Procurement Spending | |
| Lv et al. | Classification of data granularity in data warehouse | |
| CN113610629A (en) | Method and device for screening client data features from large-scale feature set | |
| CN112927071A (en) | Post-loan behavior feature processing method and device | |
| KR101221992B1 (en) | Apparatus for logging data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180615 |