CN107247804A - O＆M big data analysis method, apparatus and system - Google Patents
O＆M big data analysis method, apparatus and system Download PDFInfo
- Publication number
- CN107247804A CN107247804A CN201710525504.9A CN201710525504A CN107247804A CN 107247804 A CN107247804 A CN 107247804A CN 201710525504 A CN201710525504 A CN 201710525504A CN 107247804 A CN107247804 A CN 107247804A
- Authority
- CN
- China
- Prior art keywords
- data
- required operation
- maintenance
- key
- maintenance data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides a kind of O＆M big data analysis method, apparatus and system, wherein, the O＆M big data analysis method comprises the following steps：Required operation/maintenance data is obtained from client；The required operation/maintenance data key value is changed to form, the required operation/maintenance data of key-value pair form is obtained；Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, the required operation/maintenance data after being encapsulated；Data mining is carried out to the required operation/maintenance data after the encapsulation, data mining results are obtained；The data mining results are shown.The program, which instead of, traditional artificial checks daily record, the mode of chart, the main points that automatization judgement data need to be paid close attention to.Greatly save manpower than manual type, while avoid produced in possible omission and the work of attendant's long term repeatability slack, make the work such as performance evaluation, health assessment conscientiously generate benefit.
Description
Technical field
The present invention relates to big data digging technology field, more particularly to a kind of O＆M big data analysis method, device and it is
System.
Background technology
At present, China's level of informatization is improved constantly, and Various types of data center is emerged in large numbers like the mushrooms after rain.Accordingly, IT is transported
The data volume of dimension also rises into geometry speed.On the one hand, the growth of operation maintenance personnel does not catch up with server, the increasing of data volume much
It is long；On the other hand, company reduces the demand of cost but more and more higher for improving IT system stability.This is required constantly
Improve the automatization level of O＆M.
Currently on the market and industry, monitoring, the automated production of program deployment relatively enrich, but operation/maintenance data are utilized
It is not enough.Because developer lacks O＆M experience, the algorithm of operation/maintenance data excavation is not known about；And operation maintenance personnel is due to time and skill
The limitation of art level, it is impossible to be confined to manually check daily record, property by thinking commercialization, therefore the existing O＆M to big data more
Energy icon, this manual method efficiency is low and can not cover Servers-all.
In addition, big data is also further extensive in the application of all trades and professions, number has been createed with hundred billion market.How to improve
The utilization ratio of O＆M big data, therefrom excavates and arrives gold, the problem of finding in existing IT system, reduction equipment and personnel into
This, and future is made prediction, a major issue as current IT maintenance departments.
The content of the invention
The embodiments of the invention provide a kind of O＆M big data analysis method, manpower is greatly saved than manual type,
The utilization ratio of O＆M big data is improved, is produced while avoiding in possible omission and the work of attendant's long term repeatability
It is raw slack, make the work such as performance evaluation, health assessment conscientiously generate benefit.The O＆M big data analysis method includes：
Required operation/maintenance data is obtained from client；
The required operation/maintenance data key value is changed to form, the required operation/maintenance data of key-value pair form is obtained；
Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, the institute after being encapsulated
Need operation/maintenance data；
Data mining is carried out to the required operation/maintenance data after the encapsulation, data mining results are obtained；
The data mining results are shown.
In one embodiment, the required operation/maintenance data includes operating system aspect data, log recording data and answered
Use interface data；
Required operation/maintenance data is obtained from client, including：
Listening port is set, the operating system in the required operation/maintenance data sent in client active is obtained by listening port
Plane data and log recording data；
Pass through the application interface data in application programming interfaces operation/maintenance data needed for client is obtained.
In one embodiment, the operating system aspect packet includes CPU usage, I/O data reading rate, I/O data
Writing rate, serve port set up connection number, network traffics；
The log recording data include trading volume, transaction the response time, JVM utilization rates, single garbage reclamation duration and
With the time interval duration of last garbage reclamation；
The application interface data include current data source connection pool size, current Web ccontainer thread pool sizes,
Datasource connection pool maximum, Webccontainer thread pool maximums, data source are averaged and Webccontainer thread pools
Active line number of passes.
In one embodiment, place is being packaged to the required operation/maintenance data of the key-value pair form according to preset format
Before reason, in addition to：
The required operation/maintenance data of the key-value pair form is screened, log buffer data and database caches number is obtained
According to；
Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, including：
The log buffer data are packaged according to form needed for log buffer, the database caches data are pressed
It is packaged according to form needed for database caches.
In one embodiment, it is packaged by the log buffer data according to form needed for log buffer, by institute
After database caches data are stated according to form is packaged needed for database caches, in addition to：
Log buffer data after encapsulation recorded in the journal file specified, by the database caches data after encapsulation
It is stored in corresponding database table.
In one embodiment, place is packaged to the required operation/maintenance data of the key-value pair form according to preset format
Reason, including：
The required operation/maintenance data of the key-value pair form is resolved to the required operation/maintenance data of MAP array formats；
Processing is packaged to the required operation/maintenance data of MAP array formats according to preset format.
In one embodiment, the data mining results are shown, including：
The data mining results are shown in the form of mail or EXCEL tables.
The embodiments of the invention provide a kind of O＆M big data analytical equipment, manpower is greatly saved than manual type,
The utilization ratio of O＆M big data is improved, is produced while avoiding in possible omission and the work of attendant's long term repeatability
It is raw slack, make the work such as performance evaluation, health assessment conscientiously generate benefit.The O＆M big data analytical equipment includes：
Data acquisition module, operation/maintenance data needed for for being obtained from client；
Format converting module, for the required operation/maintenance data key value to be changed to form, obtains key assignments plaid matching
The required operation/maintenance data of formula；
Data package module, for being packaged place to the required operation/maintenance data of the key-value pair form according to preset format
Reason, the required operation/maintenance data after being encapsulated；
Data-mining module, for carrying out data mining to the required operation/maintenance data after the encapsulation, obtains data mining
As a result；
Display module, for the data mining results to be shown.
In one embodiment, the required operation/maintenance data includes operating system aspect data, log recording data and answered
Use interface data；
The data acquisition module specifically for：
As follows required operation/maintenance data is obtained from client：
Listening port is set, the operating system in the required operation/maintenance data sent in client active is obtained by listening port
Plane data and log recording data；
Pass through the application interface data in application programming interfaces operation/maintenance data needed for client is obtained.
In one embodiment, the operating system aspect packet includes CPU usage, I/O data reading rate, I/O data
Writing rate, serve port set up connection number, network traffics；
The log recording data include trading volume, transaction the response time, JVM utilization rates, single garbage reclamation duration, with
The time interval duration of last garbage reclamation；
The application interface data include current data source connection pool size, current Web ccontainer thread pool sizes,
Datasource connection pool maximum, Webccontainer thread pool maximums, data source are averaged and Webccontainer thread pools
Active line number of passes.
In one embodiment, in addition to：Screening module, for the required operation/maintenance data of the key-value pair form to be carried out
Screening, obtains log buffer data and database caches data；
The data package module specifically for：The log buffer data are sealed according to form needed for log buffer
Dress, the database caches data are packaged according to form needed for database caches.
In one embodiment, in addition to：Record storage module, for the log buffer data after encapsulation to recorded into finger
In fixed journal file, the database caches data after encapsulation are stored in corresponding database table.
In one embodiment, the data package module specifically for：
The required operation/maintenance data of institute's key-value pair form is resolved to the required operation/maintenance data of MAP array formats；
Processing is packaged to the required operation/maintenance data of MAP array formats according to preset format.
In one embodiment, the display module specifically for：
The data mining results are shown in the form of mail or EXCEL tables.
The embodiment of the present invention additionally provides a kind of O＆M big data analysis system, including：Client and fortune as described above
Tie up big data analytical equipment.
In embodiments of the present invention, first, required operation/maintenance data is obtained from client, then will the required operation/maintenance data
Key value is changed to form, obtains the required operation/maintenance data of key-value pair form；According to preset format to the key assignments plaid matching
The required operation/maintenance data of formula is packaged processing, the required operation/maintenance data after being encapsulated, then, needed for after the encapsulation
Operation/maintenance data carries out data mining, obtains data mining results, finally, the data mining results are shown.The program
It instead of and traditional artificial check daily record, the mode of chart, the main points that automatization judgement data need to be paid close attention to.Come compared to manual type
Say, the inventive method greatly saves manpower, improve the utilization ratio of O＆M big data, while avoiding possible omission
And produced in the work of attendant's long term repeatability it is slack, make the work such as performance evaluation, health assessment generating conscientiously
Benefit.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is a kind of O＆M big data analysis method flow chart provided in an embodiment of the present invention；
Fig. 2 is a kind of O＆M big data analytical equipment structural representation provided in an embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.Based on this
Embodiment in invention, the every other reality that those of ordinary skill in the art are obtained under the premise of creative work is not made
Example is applied, the scope of protection of the invention is belonged to.
In the prior art, because of server system, and its operation/maintenance data produced is increasingly various, using manually checking day
Will, the method for chart and existing instrument can not meet the automation of production O＆M, the management requirement of precision well, especially
In terms of problem finds with cost reduction.
The present invention is refined most representative and perspective data target, artificial data is dug based on O＆M experience for many years
The thinking of pick is converted to algorithm, and O＆M historical data is counted and excavated, and combines historical failure information and performance bottleneck
Confirmed, propose and realize a kind of new more automation, intelligent, expansible O＆M big data method for digging, dress
Put and system.The application system can effectively anticipation fault in production and performance bottleneck, find the wasting of resources, improve operation maintenance personnel
Operating efficiency, for banking and other possess a large amount of operation/maintenance datas, pay attention to IT system quality, while want to reduce the enterprise of cost again
Industry, has important practical significance.
In embodiments of the present invention there is provided a kind of O＆M big data analysis method, as shown in figure 1, this method includes：
Step 101：Required operation/maintenance data is obtained from client；
Step 102：The required operation/maintenance data key value is changed to form, the required fortune of key-value pair form is obtained
Dimension data；
Step 103：Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, sealed
Required operation/maintenance data after dress；
Step 104：Data mining is carried out to the required operation/maintenance data after the encapsulation, data mining results are obtained；
Step 105：The data mining results are shown.
When it is implemented, the data obtained from client mainly include three kinds：Operating system aspect data, log recording number
According to application interface data.These three data are obtained in the following way：Listening port is set, is obtained by listening port
Operating system aspect data and log recording data in the required operation/maintenance data sent in client active；Connect by application program
Application interface data in mouth operation/maintenance data needed for client is obtained.Then operation/maintenance data key value is to form needed for will be described
Changed, obtain the required operation/maintenance data of key-value pair form.
Specifically, (1) from client by system command to CPU, connection number, IO, the operating system aspect such as network traffics
Data acquisition；It is that (key-value pair is exactly that can obtain corresponding one according to a key assignments to key-value pair form by data preliminary treatment
Value.For example, being exactly with a series of often using Form.Enable=True or False in general object-oriented editor
Encapsulation of the api function to a value.), and actively report and submit service end.Wherein, table 1 is the pass of the operating system aspect data obtained
Key field list：
Table 1
(2) when client extracts trading volume, transaction response time, JVM utilization rates, gc by text-processing from daily record
Between, the data such as gc intervals.JVM is Java Virtual Machine (Java Virtual Machine) abbreviation, and JVM is a kind of for calculating
The specification of equipment, it is one and fabricates computer out, is by the various computers of analogue simulation on actual computer
Function is realized.Key-value pair form is processed data into, and actively reports and submits service end.Wherein, table 2 is the log recording obtained
The critical field list of data：
Table 2
(3) actively initiated by service end, by application programming interfaces, obtain client data (application interface data), visitor
Family end not actively on send data.Wherein, table 3 is the critical field list of the application interface data obtained：
Table 3
When it is implemented, needs read the configuration file of the present invention in the application program launching that this method is related to, obtain
Parameters configuration of the present invention；After more new configuration file, configuration file content is loaded onto internal memory by hand.Table 4 is each for configuration
The critical field list of item parameter：
Table 4
When it is implemented, being packaged processing to the required operation/maintenance data of the key-value pair form according to preset format
Before (step 103), in addition to：The required operation/maintenance data of the key-value pair form is screened, log buffer data are obtained
With database caches data.
That step 103 is exactly specifically to be packaged the log buffer data according to form needed for log buffer, will
The database caches data are packaged according to form needed for database caches.By the log buffer data according to daily record
Caching needed for form be packaged, by the database caches data according to form is packaged needed for database caches after,
Also need to recorded the log buffer data after encapsulation in the journal file specified, the database caches data after encapsulation are deposited
Enter in corresponding database table, as shown in table 5., can be with when log buffer data recorded in the journal file specified
By the indicative information in operating process, system output record write-in Disk Logs file.
Table 5
In the log buffer data after by encapsulation recorded the journal file specified, by the database caches number after encapsulation
During according to being stored in corresponding database table, it is related to database manipulation and file read-write operations.Database manipulation includes：Open number
According to Kuku, self-built datasource connection pool realizes that cleaning etc. is closed in the establishment of data source connection；File read-write operations include：File
Read-write, has rewritten files classes and has realized the demand of itself；Journal format is adjusted, and it is self-defined interior that rewriting daily record class adds timestamp etc.
Hold.Journal file can be handled by external programs such as tivoli.
When it is implemented, also needing to resolve to the required operation/maintenance data of the key-value pair form to complete step 103
The required operation/maintenance data of MAP array formats, and carry out capital and small letter, length transition；Then according still further to preset format to MAP array cells
The required operation/maintenance data of formula is packaged processing.
When it is implemented, after above-mentioned data encapsulation is completed, it is necessary to carry out data digging to the data after encapsulation
Dig (step 104).Specifically, data mining can be completed for different parameters and index respectively according to following several ways：
(1) some data have fixed upper limit.Such as CPU upper limits are 100, and the data source upper limit is data source maximum.For
These data can set a stationary window (that is to say threshold value), according to different application, take each application main business period to put down
Average, such as, when the main business period mainly includes morning 9-11, during 14-17 in afternoon, the certain applications server batch period
Deng.Threshold value is set according to application actual conditions.
(2) some data are not fixed without fixed upper limit, i.e. threshold value.A sliding window can be set for these data
Mouthful, it is adaptable to need the situation with the contrast of a certain period before.The present invention was mainly contrasted with the last week same period, during part
Section is contrasted with the last month same period.Such as, require too wide for the general no maximum requirement such as all kinds of connection numbers or the upper limit
The index of pine, using sliding window, finds the change of performance indications suddenly.
(3) some data are directed to, mainly find that its average value is relatively low, but there is the situation of bottleneck in peak period performance.Such as,
A) find occur the secondary situations high more than threshold value (different according to application) wink of N (N is different according to application) in a period of time；B) find
Peak period transaction has substantially slow situation, and baseline threshold is 5 seconds, the investigation being accustomed to according to user, the request response more than 5 seconds
Time is enough to make user leave the page.
The thinking of people can also be simulated, by setting complex parameters, based on the data mining algorithm (initial algorithm) on basis,
The data mining of complex scene is realized, wherein, initial algorithm and threshold value are mainly rule of thumb set.Mainly include：
A) contrasted by visit capacity highest ip and other ip in a period of time, and the ip history accesses situation and access
Total amount, note abnormalities access ip.Such as abnormal access request time is obviously prolonged or major part is failed transactions, then by problem liter
Level.
B) contrasted by each average values of jvm after garbage reclamation, and with reference to jvm maximums, find RAM leakage.
C) by the data mining results will method progress data mining acquisition planted using (1), (2), (3), and outside
The connection number alarm of portion's application, is contrasted with the trading situation of corresponding time, assesses the reasonability that threshold value is set.
After data mining results are obtained, it is found that present analysis result (data mining results) does not have and accurately reflect
IT system running situation, can combine the unreasonable place that Consumer's Experience constantly finds alarm, add the parameter of personal customization, with reference to
The algorithm generation personal customization algorithm on basis.Then the good personal customization algorithm of proving effect is preserved, and next time can be direct
Use.The parameter of personal customization can include：The A optional excavation periods；B can be by main frame；The optional application types of C；D may specify basis
Algorithm；E adjustable thresholds.
The leading indicator of data mining is as shown in table 6：
Table 6
When it is implemented, it is necessary to by result by various forms, intuitively show after data mining results are obtained
In front of the user.Such as, after data mining results (being a result set) can be encapsulated, result is sent into mail.This knot
It can be that timing is performed or checked at any time that fruit, which sends mail,.It can also be shown in the form of EXCEL tables
Mail format is similar as follows：Hereinafter the server uprushed before number is contrasted one week is connected for WEB server：
| Apply Names | Host name | IP | Web connection numbers (same day) | Web connections number (before one week) |
Based on same inventive concept, a kind of O＆M big data analytical equipment is additionally provided in the embodiment of the present invention, it is such as following
Embodiment described in.Because the principle that O＆M big data analytical equipment solves problem is similar to O＆M big data analysis method, because
The implementation of this O＆M big data excavating gear may refer to the implementation of O＆M big data analysis method, repeats part and repeats no more.
Used below, term " unit " or " module " can realize the combination of the software and/or hardware of predetermined function.Although with
Device described by lower embodiment is preferably realized with software, but hardware, or the combination of software and hardware realization
May and it be contemplated.
Fig. 2 is a kind of structured flowchart of the O＆M big data analytical equipment of the embodiment of the present invention, as shown in Fig. 2 changing O＆M
Big data analysis includes：
Data acquisition module 201, operation/maintenance data needed for for being obtained from client；
Format converting module 202, for the required operation/maintenance data key value to be changed to form, obtains key-value pair
The required operation/maintenance data of form；
Data package module 203, for being sealed according to preset format to the required operation/maintenance data of the key-value pair form
Dress processing, the required operation/maintenance data after being encapsulated；
Data-mining module 204, for carrying out data mining to the required operation/maintenance data after the encapsulation, obtains data and digs
Dig result；
Display module 205, for the data mining results to be shown.
The structure is illustrated below.
Connect when it is implemented, operation/maintenance data needed for described includes operating system aspect data, log recording data and application
Mouth data；
The data acquisition module 201 specifically for：
As follows required operation/maintenance data is obtained from client：
Listening port is set, the operating system in the required operation/maintenance data sent in client active is obtained by listening port
Plane data and log recording data；
Pass through the application interface data in application programming interfaces operation/maintenance data needed for client is obtained.
When it is implemented, the operating system aspect packet, which includes CPU usage, I/O data reading rate, I/O data, writes speed
Degree, serve port set up connection number, network traffics；
The log recording data include trading volume, transaction the response time, JVM utilization rates, single garbage reclamation duration, with
The time interval duration of last garbage reclamation；
The application interface data include current data source connection pool size, current Web ccontainer thread pool sizes,
Datasource connection pool maximum, Webccontainer thread pool maximums, data source are averaged and Webccontainer thread pools
Active line number of passes.
When it is implemented, also including：Screening module, for the required operation/maintenance data of the key-value pair form to be sieved
Choosing, obtains log buffer data and database caches data；
The data package module 203 specifically for：The log buffer data are entered according to form needed for log buffer
Row encapsulation, the database caches data are packaged according to form needed for database caches.
When it is implemented, also including：Record storage module, for the log buffer data after encapsulation to recorded into what is specified
In journal file, the database caches data after encapsulation are stored in corresponding database table.
When it is implemented, the data package module 203 specifically for：
The required operation/maintenance data of institute's key-value pair form is resolved to the required operation/maintenance data of MAP array formats；
Processing is packaged to the required operation/maintenance data of MAP array formats according to preset format.
When it is implemented, the display module 205 specifically for：
The data mining results are shown in the form of mail or EXCEL tables.
When it is implemented, the present invention also proposes a kind of O＆M big data analysis system, including：Client and above-described
O＆M big data analytical equipment.
In summary, automation proposed by the present invention, expansible O＆M big data analysis method, apparatus and system, take
Manually daily record, the mode of chart, the main points that automatization judgement data need to be paid close attention to are checked for traditional.Than manual type greatly
Manpower is saved, while avoiding produced in possible omission and the work of attendant's long term repeatability slack, makes performance
The work such as analysis, health assessment conscientiously generate benefit.
O＆M experience for many years is additionally based on, using original creation, the data mining algorithm of standing practice test, simulation is artificial
Thinking during judgement, realizes that intelligent big data is excavated.The present invention is an open system, realizes data mining, data
Show, interpretation of result, algorithm generation, then to the closed loop of data mining, allow the algorithm of data mining in practice constantly from
I optimizes extension, reaches more perfect effect.The present invention also supports the extension of external program, and such as tivoli obtains integrated, can
Alarm content is showed by tivoli.By the constantly improve of the present invention, the utilization of resource can be more accurately carried out, it is to avoid
Possible failure risk, effectively reduces the resource use cost and disorderly closedown cost of enterprise, to enterprise IT O＆M levels
Raising has important realistic meaning.
It should be understood by those skilled in the art that, embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the present invention can be using the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the present invention can be used in one or more computers for wherein including computer usable program code
The computer program production that usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The present invention is the flow with reference to method according to embodiments of the present invention, equipment (system) and computer program product
Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention, for the skill of this area
For art personnel, the embodiment of the present invention can have various modifications and variations.Within the spirit and principles of the invention, made
Any modification, equivalent substitution and improvements etc., should be included in the scope of the protection.
Claims (15)
1. a kind of O＆M big data analysis method, it is characterised in that including：
Required operation/maintenance data is obtained from client；
The required operation/maintenance data key value is changed to form, the required operation/maintenance data of key-value pair form is obtained；
Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, the required fortune after being encapsulated
Dimension data；
Data mining is carried out to the required operation/maintenance data after the encapsulation, data mining results are obtained；
The data mining results are shown.
2. O＆M big data analysis method as claimed in claim 1, it is characterised in that operation/maintenance data includes operation needed for described
System level data, log recording data and application interface data；
Required operation/maintenance data is obtained from client, including：
Listening port is set, the operating system aspect in the required operation/maintenance data sent in client active is obtained by listening port
Data and log recording data；
Pass through the application interface data in application programming interfaces operation/maintenance data needed for client is obtained.
3. O＆M big data analysis method as claimed in claim 2, it is characterised in that the operating system aspect packet is included
CPU usage, I/O data reading rate, I/O data writing rate, serve port set up connection number, network traffics；
The log recording data include trading volume, transaction the response time, JVM utilization rates, single garbage reclamation duration and with it is upper
The time interval duration of garbage reclamation；
The application interface data include current data source connection pool size, current Web ccontainer thread pool sizes, data
Source connection pool maximum, Webccontainer thread pool maximums, data source are average and Webccontainer thread pools are movable
Thread Count.
4. O＆M big data analysis method as claimed in claim 1, it is characterised in that according to preset format to the key assignments
Required operation/maintenance data to form is packaged before processing, in addition to：
The required operation/maintenance data of the key-value pair form is screened, log buffer data and database caches data are obtained；
Processing is packaged to the required operation/maintenance data of the key-value pair form according to preset format, including：
The log buffer data are packaged according to form needed for log buffer, by the database caches data according to number
It is packaged according to form needed for banked cache.
5. O＆M big data analysis method as claimed in claim 4, it is characterised in that by the log buffer data according to
Form is packaged needed for log buffer, and the database caches data are packaged into it according to form needed for database caches
Afterwards, in addition to：
Log buffer data after encapsulation recorded in the journal file specified, the database caches data after encapsulation are stored in
In corresponding database table.
6. O＆M big data analysis method as claimed in claim 1, it is characterised in that according to preset format to the key-value pair
The required operation/maintenance data of form is packaged processing, including：
The required operation/maintenance data of the key-value pair form is resolved to the required operation/maintenance data of MAP array formats；
Processing is packaged to the required operation/maintenance data of MAP array formats according to preset format.
7. O＆M big data analysis method as claimed in claim 1, it is characterised in that the data mining results are subjected to exhibition
Show, including：
The data mining results are shown in the form of mail or EXCEL tables.
8. a kind of O＆M big data analytical equipment, it is characterised in that including：
Data acquisition module, operation/maintenance data needed for for being obtained from client；
Format converting module, for the required operation/maintenance data key value to be changed to form, obtains key-value pair form
Required operation/maintenance data；
Data package module, for being packaged processing to the required operation/maintenance data of the key-value pair form according to preset format,
Required operation/maintenance data after being encapsulated；
Data-mining module, for carrying out data mining to the required operation/maintenance data after the encapsulation, obtains data mining results；
Display module, for the data mining results to be shown.
9. O＆M big data analytical equipment as claimed in claim 8, it is characterised in that operation/maintenance data includes operation needed for described
System level data, log recording data and application interface data；
The data acquisition module specifically for：
As follows required operation/maintenance data is obtained from client：
Listening port is set, the operating system aspect in the required operation/maintenance data sent in client active is obtained by listening port
Data and log recording data；
Pass through the application interface data in application programming interfaces operation/maintenance data needed for client is obtained.
10. O＆M big data analytical equipment as claimed in claim 9, it is characterised in that the operating system aspect packet
Include CPU usage, I/O data reading rate, I/O data writing rate, serve port and set up connection number, network traffics；
The log recording data include trading volume, transaction response time, JVM utilization rates, single garbage reclamation duration and upper one
The time interval duration of secondary garbage reclamation；
The application interface data include current data source connection pool size, current Web ccontainer thread pool sizes, data
Source connection pool maximum, Webccontainer thread pool maximums, data source are average and Webccontainer thread pools are movable
Thread Count.
11. O＆M big data analytical equipment as claimed in claim 8, it is characterised in that also include：Screening module, for inciting somebody to action
The required operation/maintenance data of the key-value pair form is screened, and obtains log buffer data and database caches data；
The data package module specifically for：The log buffer data are packaged according to form needed for log buffer,
The database caches data are packaged according to form needed for database caches.
12. O＆M big data analytical equipment as claimed in claim 11, it is characterised in that also include：Record storage module, is used
It is recorded in the log buffer data after by encapsulation in the journal file specified, the database caches data after encapsulation be stored in phase
In the database table answered.
13. O＆M big data analytical equipment as claimed in claim 8, it is characterised in that the data package module is specifically used
In：
The required operation/maintenance data of the key-value pair form is resolved to the required operation/maintenance data of MAP array formats；
Processing is packaged to the required operation/maintenance data of MAP array formats according to preset format.
14. O＆M big data analytical equipment as claimed in claim 8, it is characterised in that the display module specifically for：
The data mining results are shown in the form of mail or EXCEL tables.
15. a kind of O＆M big data analysis system, it is characterised in that including：O＆M described in client and claim 8 to 14
Big data analytical equipment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710525504.9A CN107247804B (en) | 2017-06-30 | 2017-06-30 | Operation and maintenance big data analysis method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710525504.9A CN107247804B (en) | 2017-06-30 | 2017-06-30 | Operation and maintenance big data analysis method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107247804A true CN107247804A (en) | 2017-10-13 |
| CN107247804B CN107247804B (en) | 2020-09-15 |
Family
ID=60013739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710525504.9A Active CN107247804B (en) | 2017-06-30 | 2017-06-30 | Operation and maintenance big data analysis method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107247804B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109522193A (en) * | 2018-10-22 | 2019-03-26 | 网宿科技股份有限公司 | A kind of processing method of operation/maintenance data, system and device |
| CN111984505A (en) * | 2020-08-21 | 2020-11-24 | 豪越科技有限公司 | Operation and maintenance data acquisition engine and acquisition method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104038957A (en) * | 2014-06-13 | 2014-09-10 | 杭州大光明通信系统集成有限公司 | 4G base station operation maintenance information analysis process method based on integration structure |
| CN106777021A (en) * | 2016-12-08 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of data analysing method and device based on automation operation platform |
-
2017
- 2017-06-30 CN CN201710525504.9A patent/CN107247804B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104038957A (en) * | 2014-06-13 | 2014-09-10 | 杭州大光明通信系统集成有限公司 | 4G base station operation maintenance information analysis process method based on integration structure |
| CN106777021A (en) * | 2016-12-08 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of data analysing method and device based on automation operation platform |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109522193A (en) * | 2018-10-22 | 2019-03-26 | 网宿科技股份有限公司 | A kind of processing method of operation/maintenance data, system and device |
| CN111984505A (en) * | 2020-08-21 | 2020-11-24 | 豪越科技有限公司 | Operation and maintenance data acquisition engine and acquisition method |
| CN111984505B (en) * | 2020-08-21 | 2023-06-16 | 豪越科技有限公司 | Fortune dimension data acquisition device and acquisition method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107247804B (en) | 2020-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6393805B2 (en) | Efficient query processing using histograms in the columnar database | |
| US10838982B2 (en) | System and method for aggregating values through risk dimension hierarchies in a multidimensional database environment | |
| US10116534B2 (en) | Systems and methods for WebSphere MQ performance metrics analysis | |
| US10558515B2 (en) | Policy based dynamic data collection for problem analysis | |
| CN103827826B (en) | Adaptively determining response time distribution of transactional workloads | |
| CN107315776A (en) | A kind of data management system based on cloud computing | |
| CN106294521A (en) | Date storage method and data warehouse | |
| CN113538154B (en) | Risk object identification method and device, storage medium and electronic equipment | |
| CN109657914A (en) | Information-pushing method, device, computer equipment and storage medium | |
| Agosto et al. | COVID-19 contagion and digital finance | |
| US20120303325A1 (en) | Inferring effects of configuration on performance | |
| JP2006048702A (en) | Automatic configuration of transaction-based performance model | |
| US20230351401A1 (en) | Automated Rules Execution Testing And Release System | |
| US20180096301A1 (en) | Systems and methods for generating customized reports based on operational stage rules | |
| CN111800292A (en) | Early warning method and device based on historical flow, computer equipment and storage medium | |
| CN111210332A (en) | Method and device for generating post-loan management strategy and electronic equipment | |
| Gupta et al. | Long range dependence in cloud servers: a statistical analysis based on google workload trace | |
| CN107247804A (en) | O＆M big data analysis method, apparatus and system | |
| CN103678322B (en) | The integration system of a kind of sample data and method | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| CN112907362A (en) | Loan transaction processing method and device, electronic equipment and storage medium | |
| US11954106B2 (en) | Dynamic data aggregation | |
| CN106408407A (en) | Comprehensive accounting matters managing system | |
| CN110532153A (en) | A kind of business level user's operation experience visualization system | |
| WO2018188352A1 (en) | Method, device, and apparatus for determining resource balance, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20201231 Address after: 100140, 55, Fuxing Avenue, Xicheng District, Beijing Patentee after: INDUSTRIAL AND COMMERCIAL BANK OF CHINA Patentee after: ICBC Technology Co.,Ltd. Address before: 100140, 55, Fuxing Avenue, Xicheng District, Beijing Patentee before: INDUSTRIAL AND COMMERCIAL BANK OF CHINA |