CN111400335A - Analysis method and system for cloud environment operation data - Google Patents
Analysis method and system for cloud environment operation data Download PDFInfo
- Publication number
- CN111400335A CN111400335A CN201811618288.3A CN201811618288A CN111400335A CN 111400335 A CN111400335 A CN 111400335A CN 201811618288 A CN201811618288 A CN 201811618288A CN 111400335 A CN111400335 A CN 111400335A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud environment
- analysis
- analyzing
- machine learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses an analysis method for cloud environment operation data, which comprises the following steps: collecting cloud environment operation data and cloud environment log data; storing and processing the collected data; analyzing the data through machine learning; displaying the result; the cloud environment operation real situation is clearly known by the aid of business analysis of the cloud environment operation data, and the cloud environment operation real situation is not set and configured simply according to a unified and non-personalized threshold mode. Through machine learning, the real situation of the system can be accurately grasped, and the setting and configuration of a specific working performance curve range and the frequency of events are facilitated.
Description
Technical Field
The invention relates to the technical field of cloud data, in particular to an analysis method and system for cloud environment operation data.
Background
At present, operation analysis of cloud environments is basically divided into two types, firstly, each cloud manufacturer is limited by the design of manufacturer products based on the usage amount of resource layer data needed to be used by the product operation and maintenance management of the cloud manufacturer, and only the operation data of the platform of the cloud manufacturer is concerned, so that the operation analysis of the whole environment cannot be matched with the service scene needs of users using the products. Secondly, whether the operation is good or not is judged based on the simplest threshold value judgment of basic indexes such as the usage amount, and the like, namely that the system with less usage amount is good in operation, the system with more usage amount is poor in operation, the flexibility for coping with various scenes is lacked, and the user can not really find out the root cause of the service problem except seeing some numerical values and colors.
And the second method is to use the traditional network management protocol for monitoring, and to acquire and display data in a conventional IT system management mode. The traditional IT management mode is not compatible with the cloud environment, and is limited by the difficulty of supporting the operation analysis point concerned by the management mode for the cloud environment. In addition, the traditional mode is difficult to adapt under the development trend of the current mixed cloud. The main points are as follows: 1. and the monitoring breadth is not enough, so that the public and private layers of the hybrid cloud and the non-operation data of each resource under the cloud cannot be covered. 2. The monitoring depth is not enough, the data acquisition only stops at the acquisition, the data is simply spread and presented, secondary processing and delivery are not available, association with specific services is not available, the data significance is not large, and no help is provided for solving the specific service problem of a user.
Meanwhile, both the two modes only stop at the step of taking the file for the data at the log level, and further regulation and analysis for the log file are not performed, not to mention processing for unstructured data in the log. All data are isolated, so that a uniform relation cannot be formed, and higher-level associated service data analysis cannot be performed.
Disclosure of Invention
In view of the existing defects, the invention provides an analysis method for cloud environment operation data, which performs uniform structural processing on system operation data, log data and the like, and further uses machine learning to analyze the data, so that the data related to the cloud environment can be linked, and a user is helped to know the real operation situation of the cloud environment very clearly through business analysis.
In order to achieve the above purpose, the embodiment of the invention adopts the following technical scheme:
a method of analyzing operational data for a cloud environment, the method comprising the steps of:
collecting cloud environment operation data and cloud environment log data;
storing and processing the collected data;
analyzing the data through machine learning;
and displaying the result.
According to one aspect of the invention, the storing and processing the collected data comprises: storing the raw operational data and unstructured data.
According to one aspect of the invention, the method comprises the steps of: and transmitting the collected data through an http mode interface.
According to one aspect of the invention, the method comprises the steps of:
collecting log files;
performing preliminary analysis on the log file;
judging whether the specific content of the file supports direct formatting or not;
if yes, directly classifying parameters;
if not, firstly carrying out semantic recognition, carrying out semantic word segmentation according to the recognition, and then carrying out parameter classification;
and carrying out data structural transformation and storing the result in a warehouse.
According to one aspect of the invention, the method comprises the steps of:
exploring and preprocessing the stored data;
performing data cleaning, data conversion and standard data supplement;
generating training data and test data;
generating a training model and a testing model and optimizing to form a determined service model;
the deployment model is further subjected to data analysis.
According to one aspect of the invention, the method comprises the steps of: the machine learning method is used for judging the system running state by applying a practice rule and finally giving a machine learning result.
An analysis system for cloud environment operational data, the system comprising:
the collection engine is used for collecting cloud environment operation data and cloud environment log data;
the data layer module is used for storing and processing the collected data;
the analysis and display platform is used for analyzing the data through machine learning;
and the display module is used for displaying the result.
According to one aspect of the invention, the collection engine is deployed and configured in a development and deployment manner of the microservice, and interfaces with the analysis presentation platform through the API interface.
In accordance with one aspect of the invention, the analytical presentation platform includes a machine learning module.
According to one aspect of the invention, the presentation module includes a UI interface.
The implementation of the invention has the advantages that: the invention relates to an analysis method for cloud environment operation data, which comprises the following steps: collecting cloud environment operation data and cloud environment log data; storing and processing the collected data; analyzing the data through machine learning; displaying the result; the cloud environment operation real situation is clearly known by the aid of business analysis of the cloud environment operation data, and the cloud environment operation real situation is not set and configured simply according to a unified and non-personalized threshold mode. Through machine learning, the real situation of the system can be accurately grasped, and the setting and configuration of a specific working performance curve range and the frequency of events are facilitated. And further finding out the root of the problems of system short boards, faults and the like, and recommending a relevant reference KB for the set knowledge base system, so that troubleshooting and problem solving are facilitated. And finally, opening an imagination space for business layer data analysis, and analyzing the operation future trend of the cloud environment.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of a complete architecture of an analysis system according to the present invention;
FIG. 2 is a flowchart illustrating the processing of a log file according to the present invention;
fig. 3 is a flowchart illustrating a process of machine learning according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, 2 and 3, a method for analyzing cloud environment operation data, the method comprising the steps of:
collecting cloud environment operation data and cloud environment log data;
storing and processing the collected data;
analyzing the data through machine learning;
and displaying the result.
As shown in fig. 1, a complete architecture of an analysis system for cloud environment operation data is built, specifically as follows:
and 101, building a central analysis display platform and configuring the analysis display platform.
And 102, deploying a collection engine, performing initial configuration, and butting an analysis display platform.
And 103, transmitting the data to the analysis display platform by the collection engine through an http interface.
And 104, storing the original operation data and the unstructured data by the analysis and display platform.
And 105, processing the data by the analysis and display platform.
At step 106, the machine learning performs further analysis of the data.
Step 107, the user displays and views the business analysis result through the UI.
And a development and deployment mode of micro-services is adopted, so that the expansion is flexible.
Collecting operation data of the cloud environment, finishing by butting an API (application programming interface) interface by a collection engine, and sending the operation data to an analysis display platform at regular time;
and collecting the cloud environment logs, finishing the collection of the cloud environment logs through a log collection engine, and sending the collected cloud environment logs to an analysis display platform at regular time.
The data processing module of the analysis display platform formats the data;
a machine learning module of the analysis display platform learns the performance data, the state data and the log content, and learns the performance probability, the reasonable operation space and the future development trend of the object. At the same time, by learning, it can also be known that: associations and possible inferences when a problem occurs. By the machine learning method, the operation state of the system can be judged by applying the practice rules, and the machine learning structure is finally given.
And a display module of the analysis display platform displays the resource statistics condition, the running state and the machine learning result of the cloud environment.
As shown in fig. 2, a flowchart of the log file processing is depicted. The method comprises the following specific steps:
step 201, the collection engine collects log files.
Step 202, performing a preliminary analysis on the log file.
Step 203, determine whether the specific content of the file supports direct formatting.
And 204, if so, directly classifying the parameters, if not, firstly performing semantic recognition, performing semantic word segmentation according to the recognition, and then classifying the parameters.
Step 205, then data structure transformation is performed, and the result is stored in a warehouse.
As shown in fig. 3, a process flow diagram for machine learning is described. The method comprises the following specific steps:
step 301, exploration and preprocessing of stored data are performed.
Step 302, further performing data cleaning, data conversion and standard data supplement.
The main function here is to perform a refined analysis conversion on the data so that the data can be applied to the training model and the testing model according to a specified format.
Step 303, generating training data and test data.
Step 304, a training model and a testing model are generated, which is a process of continuously perfecting and optimizing the training model. The training model needs to be adjusted optimally according to the results of the test model. Finally, a determined business model is formed.
Step 305, the final model is deployed, which can be used for further data analysis.
1. Unstructured data structuring method.
Collecting log data, formatting dates, objects and operation contents in the logs, and storing the dates, the objects and the operation contents into a basic database; after the log database is preliminarily formatted, the log database is classified and stored similar to performance and state data. Queries, presentations and secondary analysis may be provided.
2. A machine learning model of cloud environment data.
The detailed analysis is carried out on the log content, firstly, the semantic recognition is carried out on the characters of the content, and the semantic word segmentation which can be recognized by a machine is formed through a machine learning mode and artificial learning guidance. And further processing is entered.
And (3) carrying out second-layer classification on different semantic participles and parameters contained in the related log contents, namely: and the specific parameters of the specific object are automatically classified, so that the specific parameters can be put into a warehouse together with the performance data and the state data. In this case, machine learning is performed on the performance data, the status data, and the log content, and the purpose of learning is to: and (4) knowing the performance probability, the reasonable operation space and the future development trend of the object. At the same time, by learning, it can also be known that: associations and possible inferences when a problem occurs. By the machine learning method, the operation state of the system can be judged by applying the practice rules, and the machine learning structure is finally given.
3. A machine learning model of cloud environment data.
The data of different entries are collected in a business mode, machine learning can be conducted on each business system under the operation of the cloud environment according to needs through business dimension, architecture dimension and management responsibility dimension, therefore, root causes of problems of system short boards, faults and the like can be found, relevant reference KB is recommended to the set knowledge base system, and troubleshooting and problem solving are facilitated. Meanwhile, the feedback of the fault after the problem and the fault are solved each time needs specialized evaluation and suggestion improvement on the machine learning data and judgment basis, so that the improvement is more obvious.
The implementation of the invention has the advantages that: the invention relates to an analysis method for cloud environment operation data, which comprises the following steps: collecting cloud environment operation data and cloud environment log data; storing and processing the collected data; analyzing the data through machine learning; displaying the result; the cloud environment operation real situation is clearly known by the aid of business analysis of the cloud environment operation data, and the cloud environment operation real situation is not set and configured simply according to a unified and non-personalized threshold mode. Through machine learning, the real situation of the system can be accurately grasped, and the setting and configuration of a specific working performance curve range and the frequency of events are facilitated. And further finding out the root of the problems of system short boards, faults and the like, and recommending a relevant reference KB for the set knowledge base system, so that troubleshooting and problem solving are facilitated. And finally, opening an imagination space for business layer data analysis, and analyzing the operation future trend of the cloud environment.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention disclosed herein are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A method for analyzing cloud environment operational data, the method comprising the steps of:
collecting cloud environment operation data and cloud environment log data;
storing and processing the collected data;
analyzing the data through machine learning;
and displaying the result.
2. The method for analyzing cloud environment operating data according to claim 1, wherein the storing and processing the collected data comprises: storing the raw operational data and unstructured data.
3. The method for analyzing cloud environment operation data according to claim 1, wherein the method comprises the following steps: and transmitting the collected data through an http mode interface.
4. The method for analyzing cloud environment operation data according to one of claims 1 to 3, wherein the method comprises the following steps:
collecting log files;
performing preliminary analysis on the log file;
judging whether the specific content of the file supports direct formatting or not;
if yes, directly classifying parameters;
if not, firstly carrying out semantic recognition, carrying out semantic word segmentation according to the recognition, and then carrying out parameter classification;
and carrying out data structural transformation and storing the result in a warehouse.
5. The method for analyzing cloud environment operation data according to claim 4, wherein the method comprises the following steps:
exploring and preprocessing the stored data;
performing data cleaning, data conversion and standard data supplement;
generating training data and test data;
generating a training model and a testing model and optimizing to form a determined service model;
the deployment model is further subjected to data analysis.
6. The method for analyzing cloud environment operation data according to claim 5, wherein the method comprises the following steps: the machine learning method is used for judging the system running state by applying a practice rule and finally giving a machine learning result.
7. An analysis system for cloud environment operational data, the system comprising:
the collection engine is used for collecting cloud environment operation data and cloud environment log data;
the data layer module is used for storing and processing the collected data;
the analysis and display platform is used for analyzing the data through machine learning;
and the display module is used for displaying the result.
8. The system for analyzing cloud environment operating data according to claim 7, wherein the collection engine is deployed and configured in a development and deployment manner of micro-services, and interfaces the analysis presentation platform through an API interface.
9. The analysis system for cloud environment operational data according to claim 7, wherein the analysis presentation platform comprises a machine learning module.
10. The analysis system for cloud environment operational data according to claim 7, wherein the presentation module comprises a UI interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811618288.3A CN111400335B (en) | 2018-12-28 | Analysis method and system for cloud environment operation data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811618288.3A CN111400335B (en) | 2018-12-28 | Analysis method and system for cloud environment operation data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400335A true CN111400335A (en) | 2020-07-10 |
| CN111400335B CN111400335B (en) | 2023-09-19 |
Family
ID=
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109754804A (en) | Voice control method and device, storage medium and intelligent home system | |
| CN104185840B (en) | It is used for being prioritized the mthods, systems and devices of multiple tests in lasting deployment streamline | |
| CN105975392A (en) | Duplicated code detection method and device based on abstract syntax tree | |
| CN112579789A (en) | Equipment fault diagnosis method and device and equipment | |
| WO2022048668A1 (en) | Knowledge graph construction method and apparatus, check method and storage medium | |
| CN114791846B (en) | Method for realizing observability aiming at cloud-originated chaos engineering experiment | |
| CN110648172B (en) | Identity recognition method and system integrating multiple mobile devices | |
| CN103440243A (en) | Teaching resource recommendation method and device thereof | |
| US20220247638A1 (en) | Automated generation of standard network device configurations | |
| CN103049365A (en) | Monitoring and evaluating method for information and application resource operating states | |
| CN109145301A (en) | Information classification approach and device, computer readable storage medium | |
| CN101589387B (en) | Information handling | |
| CN114066418B (en) | Fire control data processing system based on data center | |
| CN118132750A (en) | Processing method and device for customer service data in power industry | |
| CN106815219A (en) | The edit methods and device of database engine | |
| CN116911386B (en) | Knowledge graph construction method of hydroelectric equipment based on knowledge context service-oriented scene | |
| CN106447207A (en) | Green building pre-evaluation system based on large data analysis | |
| CN114117134A (en) | Abnormal feature detection method, device, equipment and computer readable medium | |
| CN106055641A (en) | Human-computer interaction method and device oriented to intelligent robot | |
| CN111400335A (en) | Analysis method and system for cloud environment operation data | |
| CN111400335B (en) | Analysis method and system for cloud environment operation data | |
| CN115562191B (en) | Industrial digital twin-based intelligent presumption analysis method for productivity center | |
| CN111784192A (en) | Industrial park emergency plan executable system based on dynamic evolution | |
| CN116126852A (en) | BIM-based intelligent management data storage method for fabricated building | |
| CN115508089A (en) | Method and device for identifying working condition of main bearing and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |