CN108197251A - A kind of big data operation and maintenance analysis method, device and server - Google Patents
A kind of big data operation and maintenance analysis method, device and server Download PDFInfo
- Publication number
- CN108197251A CN108197251A CN201711484351.4A CN201711484351A CN108197251A CN 108197251 A CN108197251 A CN 108197251A CN 201711484351 A CN201711484351 A CN 201711484351A CN 108197251 A CN108197251 A CN 108197251A
- Authority
- CN
- China
- Prior art keywords
- services
- information
- data
- node
- maintenance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention proposes a kind of big data operation and maintenance analysis method, device and server, the method includes：The information on services of each node in collecting computer cluster；The information on services of each node of acquisition is subjected to pretreatment operation；Using machine learning algorithm construction strategy model；According to the Policy model, according to pretreated information on services, generating run maintenance strategy；Each node in the operation and maintenance policy distribution to cluster is performed.Above-mentioned technical proposal can be realized：Using machine learning algorithm construction strategy model, so that the Policy model is according to pretreated information on services, generating run maintenance strategy, and each node in the operation and maintenance policy distribution to cluster is performed, since the strategy issued is obtained according to the information on services of node each in cluster, therefore the resource between major data service can be made to accomplish the flexible and shared of automatic elastic, improve the overall utilization rate of cluster resource.
Description
Technical field
The present invention relates to big data operation and maintenance technical field, specially a kind of big data operation and maintenance analysis method, dress
It puts and server.
Background technology
With the raising of China's level of informatization, Various types of data center continues to bring out, correspondingly, the data of IT operation and maintenance
Amount also rises into geometry speed.In IT operation and maintenance monitoring fields, an effective operation maintenance system can help operation and maintenance
Personnel have found the hidden danger of the system failure in time.Operation maintenance system needs to include the acquisition of achievement data, displaying, then divide to data
Analysis etc., the design of links will ensure efficient stable, this is particularly important in big data operation maintenance system.
Traditional operation and maintenance analysis system is managed only for single data statistics item, according to single data statistics
Item is for statistical analysis.For example, only for single data statistics item given threshold, whether threshold value is reached according to the data statistics item
To issue operation maintenance management strategy；For another example issue operation and maintenance strategy according only to the record of network log.This mode
The incidence relation between data is had ignored, the overall utilization rate for leading to cluster resource is relatively low.
Invention content
The embodiment of the present invention provides a kind of big data operation and maintenance analysis method, device and server, existing at least to solve
There is the above technical problem in technology.
In a first aspect, an embodiment of the present invention provides a kind of big data operation and maintenance analysis method, including：
The information on services of each node in collecting computer cluster；
The information on services of each node of acquisition is subjected to pretreatment operation；
Using machine learning algorithm construction strategy model；
According to the Policy model, according to pretreated information on services, generating run maintenance strategy；
Each node in the operation and maintenance policy distribution to cluster is performed.
It is with reference to first aspect, of the invention in the first embodiment of first aspect,
The information on services of each node of acquisition is subjected to pretreatment operation, including：
Data cleansing is carried out to the information on services, to filter out undesirable data；
Data pick-up is carried out to the information on services after cleaning, to obtain the useful properties of the information on services；
The useful feature obtained after data pick-up is subjected to data conversion, is suitable for the Policy model training to obtain
Data type.
The first embodiment with reference to first aspect, using machine learning algorithm construction strategy model, including：
Each node is to the monitoring data of the information on services before pretreatment in collecting computer cluster, by the monitoring number
According to as data sample；
According to the monitoring data, the Policy model is built using machine learning algorithm；
Obtain manual feedback result；
According to the manual feedback as a result, being optimized to the Policy model.
With reference to first aspect, the present invention is in the second embodiment of first aspect, each node in collecting computer cluster
Information on services after, the method further includes：
The information on services of each node in storage cluster；
Inquiry service is provided for the information on services, wherein the inquiry service includes batch query service and real-time query
Service.
Second aspect, an embodiment of the present invention provides a kind of big data operation and maintenance analytical equipment, including：
Acquisition module is configured to the information on services of each node in collecting computer cluster；
Preprocessing module is configured to the information on services of each node of acquisition carrying out pretreatment operation；
Modeling module is configured to using machine learning algorithm construction strategy model；
Generation strategy module is configured to according to the Policy model, according to pretreated information on services, generating run dimension
Shield strategy；
Distributing policy module is configured to perform each node in the operation and maintenance policy distribution to cluster.
With reference to second aspect, in the first embodiment of second aspect, the preprocessing module includes the present invention：
Data cleansing submodule is configured to carry out data cleansing to the information on services, undesirable to filter out
Data；
Data pick-up submodule is configured to carry out data pick-up to the information on services after cleaning, is believed with obtaining the service
The useful properties of breath；
Data conversion submodule, data conversion is carried out by the useful feature obtained after data pick-up, is suitable for institute to obtain
State the data type of Policy model training.
With reference to the first embodiment of second aspect,
The generation strategy module includes：
Submodule is acquired, is configured in collecting computer cluster each node to the monitoring number of the information on services before pretreatment
According to using the monitoring data as data sample；
Submodule is built, is configured to according to the monitoring data, the Policy model is built using machine learning algorithm；
Acquisition submodule is configured to obtain manual feedback result；
Optimize submodule, be configured to according to the manual feedback as a result, being optimized to the Policy model.
With reference to second aspect, in the second embodiment of second aspect, the preprocessing module further includes the present invention：
Sub-module stored is configured to the information on services of each node in storage computer cluster；
Submodule is inquired, is configured to provide inquiry service for the information on services, wherein the inquiry service includes batch
Inquiry service and real-time query service.
The third aspect, the embodiment of the present invention provide a kind of server, and the server includes：
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are performed by one or more of processors so that one or more of places
It manages device and realizes above-mentioned big data operation and maintenance analysis method.
Fourth aspect, an embodiment of the present invention provides a kind of computer readable storage medium, for storing big data operation
Computer software instructions used in maintenance analysis device, including dividing for performing big data operation and maintenance in above-mentioned first aspect
Program involved by analysis method.
A technical solution in above-mentioned technical proposal has the following advantages that or advantageous effect：Using machine learning algorithm structure
Build Policy model so that the Policy model is according to pretreated information on services, generating run maintenance strategy, and will described in
Each node in operation and maintenance policy distribution to cluster is performed, since the strategy issued is according to node each in cluster
What information on services obtained, therefore the resource between major data service can be made to accomplish the flexible and shared of automatic elastic, it improves
The overall utilization rate of cluster resource.
Above-mentioned general introduction is merely to illustrate that the purpose of book, it is not intended to be limited in any way.Except foregoing description
Schematical aspect, except embodiment and feature, by reference to attached drawing and the following detailed description, the present invention is further
Aspect, embodiment and feature will be what is be readily apparent that.
Description of the drawings
In the accompanying drawings, unless specified otherwise herein, otherwise represent the same or similar through the identical reference numeral of multiple attached drawings
Component or element.What these attached drawings were not necessarily to scale.It should be understood that these attached drawings are depicted only according to the present invention
Some disclosed embodiments, and should not serve to limit the scope of the present invention.
Fig. 1 is the flow chart of the big data operation and maintenance analysis method of the embodiment of the present invention one；
Fig. 2 is the flow chart of the big data operation and maintenance analysis method of the embodiment of the present invention two；
Fig. 3 is the structure chart of the big data operation and maintenance analysis and management system of the embodiment of the present invention two；
Fig. 4 is the schematic diagram of the big data operation and maintenance analytical equipment of the embodiment of the present invention three；
Fig. 5 is the schematic diagram of the big data operation and maintenance analytical equipment of the embodiment of the present invention four；
Fig. 6 is the schematic diagram of the server of the embodiment of the present invention five.
Specific embodiment
Hereinafter, certain exemplary embodiments are simply just described.As one skilled in the art will recognize that
Like that, without departing from the spirit or scope of the present invention, described embodiment can be changed by various different modes.
Therefore, attached drawing and description are considered essentially illustrative rather than restrictive.
Embodiment one
The embodiment of the present invention provides a kind of big data operation and maintenance analysis method.As shown in Figure 1, for the embodiment of the present invention
The flow chart of big data operation and maintenance analysis method.The big data operation and maintenance analysis method of the embodiment of the present invention includes following step
Suddenly：
S101, the information on services of each node in collecting computer cluster.
Cluster refers to a parallel or distributed system being made of the computer that some are mutually connected to each other.These meters
Calculation machine works and runs a series of common application programs together, meanwhile, single system is provided and is reflected for user and application program
It penetrates.Externally, they are only a system, externally provide unified service.Computer in cluster physically through
Cable connection is then connected by clustered software in program.These connections allow computer to meet an urgent need and load balance work(using failure
Can, and it is impossible that failure, which is met an urgent need with load balance function on unit,.Wherein, each computer in cluster is known as
One node.
Wherein, the computer that the present invention implements to be related to refers to the equipment with operational capability, that is to say, that the equipment need not
So include the external equipments such as mouse, sound equipment, as long as with operational capability.
The information on services of each node operation in acquisition cluster of the embodiment of the present invention, including the application disposed on each node
The data such as operation information, daily record and the operating index of program.
The information on services of each node of acquisition is carried out pretreatment operation by S102.
Wherein, pretreatment operation is carried out to information on services and mainly includes the behaviour such as data cleansing, data pick-up and data conversion
Make.
S103, using machine learning algorithm construction strategy model.
The embodiment of the present invention is substantially exactly the association for extracting multi-dimensional data using machine learning algorithm Training strategy model
Model, therefore the result of model output is also that the result of analysis acquisition is carried out according to associated multi-dimensional data.
Index (Metrics) represents the performance parameter of computer application level.Below to acquire the index on each node
Data instance introduces construction strategy model and the process using Policy model.For example, first, acquire the index of each node；So
Afterwards, carry out pretreatment operation to the index that is acquired, specifically for the index that is acquired using three rank exponential smoothing scheduling algorithms into
Row time series analysis so that more recent data influence the value that model exports bigger；Subsequently, under being surveyed according to Policy model
The index at a moment, according to model to predict node lower a moment index when, if the index at lower a moment of actual acquisition and prediction
Lower a moment index difference it is larger, illustrate the Indexes Abnormality at practical lower a moment；Alarm is finally generated according to abnormal index
Wait strategies.
Be above to construction strategy model from it is longitudinal for example, below construction strategy model is lifted from transverse direction again
Example explanation.First, acquisition and the relevant index of service operation, for example acquire processor (CPU), memory and the network of a node and refer to
The index generated when mark and service self-operating；The pretreatment operations such as then cleaned, extracted to the index；Finally,
Machine learning algorithm, such as SVM algorithm are used according to data above, Training strategy model is enabled to according to the Policy model
Whether generation needs to restart the strategy of computer.This Policy model training when be according to multiple associated data into
Row training, therefore the relevance between data is considered, it can also be according to newly-increased data point reuse optimisation strategy mould in the later stage
Type.S104, according to the Policy model, according to pretreated information on services, generating run maintenance strategy.
In the introduction of above step, strategy of the strategies such as alarm ultimately generated and restarting computer etc. is
For the operation and maintenance strategy generated according to Policy model.
S105 performs each node in the operation and maintenance policy distribution to computer cluster.
Machine learning algorithm construction strategy model may be used in the embodiment of the present invention, each into cluster by Policy model
Node issues operation and maintenance strategy.It can be by acquiring the operation and maintenance data of history, as sample, construction strategy model, so
Pass through Policy model distributing policy afterwards.
A technical solution in above-mentioned technical proposal has the following advantages that or advantageous effect：Using machine learning algorithm structure
Build Policy model so that the Policy model is according to pretreated information on services, generating run maintenance strategy, and will described in
Each node in operation and maintenance policy distribution to cluster is performed, since the strategy issued is according to node each in cluster
What information on services obtained, therefore the resource between major data service can be made to accomplish the flexible and shared of automatic elastic, it improves
The overall utilization rate of cluster resource.
Embodiment two
On the basis of embodiment one, the embodiment of the present invention provides a kind of big data operation and maintenance analysis method.Such as Fig. 2 institutes
Show, the flow chart of the big data operation and maintenance analysis method of the embodiment of the present invention.The big data operation and maintenance of the embodiment of the present invention
Analysis method includes the following steps：
S201, the information on services of each node in collecting computer cluster.
The embodiment of the present invention is also provided with storing and inquires service, after step S201, further includes：A, storage data turn
Information on services after changing；B provides inquiry service for the information on services after the data conversion, wherein the inquiry service includes
Batch query service and real-time query service.
The embodiment of the present invention can store transformed information on services, for inquiry.Wherein, refer to can for batch query service
Multiple information on services are inquired, while export query result；Real-time query service refers to that can inquire information on services runs shape in real time
Condition.
S202 carries out data cleansing, to filter out undesirable data to the information on services.
Wherein, data cleansing (Data cleaning) refers to the process of carry out data to examine and verify again, and purpose exists
In mistake existing for deletion duplicate message, correction, and provide data consistency.In the embodiment of the present invention, due to collecting computer
In cluster during the information on services of each node, for example, certain node log of acquisition is imperfect, which can be deleted
It removes.
S203 carries out data pick-up, to obtain the useful properties of the information on services to the information on services after cleaning.
Since the characteristic of the different required information on services of Training scene is different, according to practical Training scene need
The useful properties of information on services are extracted, for example, only counting the number of starts of certain program on certain node, then in data pick-up
When, only extract the number of starts of corresponding information on services.
S204, data conversion is carried out by the useful feature obtained after data pick-up, is suitable for the Policy model to obtain
Trained data type.
It is to be suitble to the data class of model training that data conversion, which is primarily referred to as the data type conversion of unsuitable model training,
Type.
S205, in collecting computer cluster each node to the monitoring data of the information on services before pretreatment, will described in
Monitoring data is as data sample.
The embodiment of the present invention can build engineering using the monitoring data of multiple dimensions of node each in cluster as sample
Practise model.And associate monitoring data with current operating situation, to generate next step operation and maintenance strategy.
According to the monitoring data, the Policy model is built using machine learning algorithm by S206.
Specifically, an embodiment of the present invention provides the engineerings classified including logistic regression, SVM, Bayes etc. and clustered
Practise algorithms library.It is exactly the correlation model for extracting multi-dimensional data using machine learning algorithm Training strategy model, in combination with going through
History Data Data index is not completely cured according to the update of latest data dynamic and correction model, therefore the result of model output is also root
The result of analysis acquisition is carried out according to associated multi-dimensional data.Specifically refer to the explanation of embodiment a pair of step S103.
S207 obtains manual feedback result.
The embodiment of the present invention is to improve the accuracy of Policy model, can add in some human interventions.When Policy model is defeated
When the result gone out has deviation, manual feedback can be inputted to Policy model as a result, with adjustable strategies model.For example, when tactful mould
Include the information on services of 4 nodes in the result of type output, and 5 nodes of physical presence, it at this moment can manually export 5 sections
The information on services of point, the result of Policy model output are consistent with reality.
S208, according to the manual feedback as a result, being optimized to the Policy model.
Refer to the explanation to step S207.
S209, according to the Policy model, according to pretreated information on services, generating run maintenance strategy.
Refer to the explanation of embodiment a pair of step S103.
S210 performs each node in the operation and maintenance policy distribution to cluster.
With reference to Fig. 3, first in data acquisition phase, the data of each node in cluster are acquired, including daily record, service letter
Breath and index etc., the host node being then sent to by proxy server in cluster carry out data preprocessing operation；Data are located in advance
The reason stage carries out data conversion to information on services, to be converted into being suitble to the data type of model training, then stores transformed
Information on services, while inquiry service can also be provided for information on services；On the one hand pretreated information on services can be passed through
Interface is forwarded to network, on the other hand may be used as the data sample of model training, and pretreated information on services is dug
Pick forms strategy, while also receives artificial feedback result, and model is adjusted.Block arrow represents policy flow, thin arrow in Fig. 3
Represent data flow.
A technical solution in above-mentioned technical proposal has the following advantages that or advantageous effect：Using machine learning algorithm structure
Build Policy model so that the Policy model is according to pretreated information on services, generating run maintenance strategy, and will described in
Each node in operation and maintenance policy distribution to cluster is performed, since the strategy issued is according to node each in cluster
What information on services obtained, therefore the resource between major data service can be made to accomplish the flexible and shared of automatic elastic, it improves
The overall utilization rate of cluster resource.
Embodiment three
The embodiment of the present invention provides a kind of big data operation and maintenance analytical equipment.As shown in figure 4, the embodiment of the present invention is big
The schematic diagram of data run maintenance analysis device.The big data operation and maintenance analytical equipment of the embodiment of the present invention includes：
Acquisition module 41 is configured to the information on services of each node in collecting computer cluster；
Preprocessing module 42 is configured to the information on services of each node of acquisition carrying out pretreatment operation；
Modeling module 43 is configured to using machine learning algorithm construction strategy model；
Generation strategy module 44 is configured to according to the Policy model, according to pretreated information on services, generating run
Maintenance strategy；
Distributing policy module 45 is configured to perform each node in the operation and maintenance policy distribution to cluster.
The embodiment of the present invention, which can be realized, makes the resource between major data service accomplish the flexible and shared of automatic elastic,
The advantageous effect of the overall utilization rate of cluster resource is improved, identical with embodiment one, details are not described herein.
Example IV
On the basis of embodiment three, the embodiment of the present invention provides a kind of big data operation and maintenance analytical equipment.Such as Fig. 5 institutes
Show, the schematic diagram of the big data operation and maintenance analytical equipment of the embodiment of the present invention.The big data operation and maintenance of the embodiment of the present invention
Analytical equipment includes：
The preprocessing module 42 includes：
Data cleansing submodule 421 is configured to carry out data cleansing to the information on services, undesirable to filter out
Data；
Data pick-up submodule 422 is configured to carry out data pick-up to the information on services after cleaning, to obtain the service
The useful properties of information；
The useful feature obtained after data pick-up is carried out data conversion, to be suitable for by data conversion submodule 423
The data type of the Policy model training.
Further, the generation strategy module 44 includes：
Submodule 441 is acquired, is configured in collecting computer cluster each node to the prison of the information on services before pretreatment
Data are controlled, using the monitoring data as data sample；
Submodule 442 is built, is configured to according to the monitoring data, using the machine learning algorithm structure tactful mould
Type；
Acquisition submodule 443 is configured to obtain manual feedback result；
Optimize submodule 444, be configured to according to the manual feedback as a result, being optimized to the Policy model.
Further, described device further includes：
Memory module 46 is configured to the information on services of each node in storage computer cluster；
Enquiry module 47 is configured to provide inquiry service for the information on services, wherein the inquiry service includes batch
Inquiry service and real-time query service.
The embodiment of the present invention, which can be realized, makes the resource between major data service accomplish the flexible and shared of automatic elastic,
The advantageous effect of the overall utilization rate of cluster resource is improved, identical with embodiment two, details are not described herein.
Embodiment five
The embodiment of the present invention provides a kind of server, as shown in fig. 6, the server includes：Memory 61 and processor
62,61 memory of memory contains the computer program that can be run on the processor 62.Processor 62 performs the computer program
Information classification approach in Shi Shixian above-described embodiments.The quantity of memory 61 and processor 62 can be one or more.
The equipment further includes：
Communication interface 63, for the communication between memory 61 and processor 62 and external equipment.
Memory 61 may include high-speed RAM memory, it is also possible to further include nonvolatile memory (non-volatile
Memory), a for example, at least magnetic disk storage.
If memory 61, processor 62 and the independent realization of communication interface 63, memory 61, processor 62 and communication connect
Mouth 63 can be connected with each other by bus and complete mutual communication.The bus can be industry standard architecture
(ISA, Industry Standard Architecture) bus, external equipment interconnection (PCI, Peripheral
Component) bus or extended industry-standard architecture (EISA, Extended Industry Standard
Component) bus etc..The bus can be divided into address bus, data/address bus, controlling bus etc..For ease of representing, Fig. 6
In only represented with a thick line, it is not intended that an only bus or a type of bus.
Optionally, in specific implementation, if memory 61, processor 62 and communication interface 63 are integrated in chip piece
On, then memory 61, processor 62 and communication interface 63 can complete mutual communication by internal interface.
Embodiment six
The embodiment of the present invention provides a kind of computer readable storage medium, is stored with computer program, which is characterized in that
The method as described in any embodiment in Fig. 1-3 is realized when the program is executed by processor.In the description of this specification, reference
The description of term " one embodiment ", " some embodiments ", " example ", " specific example " or " some examples " etc. means to combine
The embodiment or example particular features, structures, materials, or characteristics described be contained at least one embodiment of the present invention or
In example.Moreover, particular features, structures, materials, or characteristics described can be in any one or more of the embodiments or examples
It combines in an appropriate manner.In addition, without conflicting with each other, those skilled in the art can will retouch in this specification
The different embodiments or examples and the feature of different embodiments or examples stated are combined.
In addition, term " first ", " second " are only used for description purpose, and it is not intended that instruction or hint relative importance
Or the implicit quantity for indicating indicated technical characteristic." first " is defined as a result, the feature of " second " can be expressed or hidden
Include at least one this feature containing ground.In the description of the present invention, " multiple " are meant that two or more, unless otherwise
It is clearly specific to limit.
Any process described otherwise above or method description are construed as in flow chart or herein, represent to include
Module, segment or the portion of the code of the executable instruction of one or more the step of being used to implement specific logical function or process
Point, and the range of the preferred embodiment of the present invention includes other realization, wherein can not press shown or discuss suitable
Sequence, including according to involved function by it is basic simultaneously in the way of or in the opposite order, to perform function, this should be of the invention
Embodiment person of ordinary skill in the field understood.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use
In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, for
Instruction execution system, device or equipment (such as computer based system, including the system of processor or other can be held from instruction
The system of row system, device or equipment instruction fetch and execute instruction) it uses or combines these instruction execution systems, device or set
It is standby and use.For the purpose of this specification, " computer-readable medium " can any can be included, store, communicate, propagate or pass
Defeated program is for instruction execution system, device or equipment or the dress used with reference to these instruction execution systems, device or equipment
It puts.
Computer-readable medium described in the embodiment of the present invention can be that computer-readable signal media or computer can
Storage medium either the two is read arbitrarily to combine.The more specific example of computer readable storage medium is at least (non-poor
Property list to the greatest extent) including following：Electrical connection section (electronic device) with one or more wiring, portable computer diskette box (magnetic
Device), random access memory (RAM), read-only memory (ROM), erasable edit read-only storage (EPROM or flash
Memory), fiber device and portable read-only memory (CDROM).In addition, computer readable storage medium even can be with
It is the paper or other suitable media that can print described program on it, because can be for example by being carried out to paper or other media
Optical scanner is then handled described electronically to obtain into edlin, interpretation or when necessary with other suitable methods
Program is then stored in computer storage.
In embodiments of the present invention, computer-readable signal media can be included in a base band or as a carrier wave part
The data-signal of propagation, wherein carrying computer-readable program code.The data-signal of this propagation may be used a variety of
Form, including but not limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media is also
Can be any computer-readable medium other than computer readable storage medium, which can send, pass
Either transmission is broadcast for instruction execution system, input method or device use or program in connection.Computer can
Reading the program code included on medium can be transmitted with any appropriate medium, including but not limited to：Wirelessly, electric wire, optical cable, penetrate
Frequently (Radio Frequency, RF) etc. or above-mentioned any appropriate combination.
It should be appreciated that each section of the present invention can be realized with hardware, software, firmware or combination thereof.Above-mentioned
In embodiment, software that multiple steps or method can in memory and by suitable instruction execution system be performed with storage
Or firmware is realized.If for example, with hardware come realize in another embodiment, can be under well known in the art
Any one of row technology or their combination are realized：With for the logic gates to data-signal realization logic function
Discrete logic, have suitable combinational logic gate circuit application-specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
Those skilled in the art are appreciated that realize all or part of step that above-described embodiment method carries
Suddenly it is that relevant hardware can be instructed to complete by program, the program can be stored in a kind of computer-readable storage medium
In matter, the program when being executed, one or a combination set of the step of including embodiment of the method.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, it can also
That each unit is individually physically present, can also two or more units be integrated in a module.Above-mentioned integrated mould
The form that hardware had both may be used in block is realized, can also be realized in the form of software function module.The integrated module is such as
Fruit is realized in the form of software function module and is independent product sale or in use, can also be stored in a computer
In readable storage medium storing program for executing.The storage medium can be read-only memory, disk or CD etc..
The above description is merely a specific embodiment, but protection scope of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can readily occur in its various change or replacement,
These should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with the guarantor of the claim
It protects subject to range.
Claims (10)
1. a kind of big data operation and maintenance analysis method, which is characterized in that
The information on services of each node in collecting computer cluster；
The information on services of each node of acquisition is subjected to pretreatment operation；
Using machine learning algorithm construction strategy model；
According to the Policy model, according to pretreated information on services, generating run maintenance strategy；
Each node in the operation and maintenance policy distribution to cluster is performed.
2. according to the method described in claim 1, it is characterized in that, the information on services of each node of acquisition is located in advance
Reason operation, including：
Data cleansing is carried out to the information on services, to filter out undesirable data；
Data pick-up is carried out to the information on services after cleaning, to obtain the useful properties of the information on services；
The useful feature obtained after data pick-up is subjected to data conversion, to obtain the data for being suitable for the Policy model training
Type.
3. according to the method described in claim 1, it is characterized in that, using machine learning algorithm construction strategy model, including：
The monitoring data is made the monitoring data of the information on services before pretreatment by each node in collecting computer cluster
For data sample；
According to the monitoring data, the Policy model is built using machine learning algorithm；
Obtain manual feedback result；
According to the manual feedback as a result, being optimized to the Policy model.
4. according to the method described in claim 2, it is characterized in that, in collecting computer cluster each node information on services it
Afterwards, the method further includes：
The information on services of each node in storage cluster；
Inquiry service is provided for the information on services, wherein the inquiry service includes batch query service and real-time query clothes
Business.
5. a kind of big data operation and maintenance analytical equipment, which is characterized in that including：
Acquisition module is configured to the information on services of each node in collecting computer cluster；
Preprocessing module is configured to the information on services of each node of acquisition carrying out pretreatment operation；
Modeling module is configured to using machine learning algorithm construction strategy model；
Generation strategy module is configured to according to the Policy model, and according to pretreated information on services, plan is safeguarded in generating run
Slightly；
Distributing policy module is configured to perform each node in the operation and maintenance policy distribution to cluster.
6. device according to claim 5, which is characterized in that the preprocessing module includes：
Data cleansing submodule is configured to carry out data cleansing to the information on services, to filter out undesirable data；
Data pick-up submodule is configured to carry out data pick-up to the information on services after cleaning, to obtain the information on services
Useful properties；
Data conversion submodule, data conversion is carried out by the useful feature obtained after data pick-up, is suitable for the plan to obtain
The slightly data type of model training.
7. device according to claim 5, which is characterized in that the generation strategy module includes：
Submodule is acquired, is configured to the monitoring data of each node in collecting computer cluster to the information on services before pretreatment,
Using by the monitoring data as data sample；
Submodule is built, is configured to according to the monitoring data, the Policy model is built using machine learning algorithm；
Acquisition submodule is configured to obtain manual feedback result；
Optimize submodule, be configured to according to the manual feedback as a result, being optimized to the Policy model.
8. device according to claim 6, which is characterized in that described device further includes：
Memory module is configured to the information on services of each node in storage computer cluster；
Enquiry module is configured to provide inquiry service for the information on services, wherein the inquiry service includes batch query clothes
Business and real-time query service.
9. a kind of server, which is characterized in that the server includes：
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are performed by one or more of processors so that one or more of processors
Realize the method as described in any in claim 1-4.
10. a kind of computer readable storage medium, is stored with computer program, which is characterized in that the program is held by processor
The method as described in any in claim 1-4 is realized during row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711484351.4A CN108197251A (en) | 2017-12-29 | 2017-12-29 | A kind of big data operation and maintenance analysis method, device and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711484351.4A CN108197251A (en) | 2017-12-29 | 2017-12-29 | A kind of big data operation and maintenance analysis method, device and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108197251A true CN108197251A (en) | 2018-06-22 |
Family
ID=62586717
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711484351.4A Pending CN108197251A (en) | 2017-12-29 | 2017-12-29 | A kind of big data operation and maintenance analysis method, device and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108197251A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067619A (en) * | 2018-09-25 | 2018-12-21 | 海南民航凯亚有限公司 | A kind of elasticity volume dispatching method and processing terminal that micro services are administered |
| CN109614978A (en) * | 2018-09-29 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN109684455A (en) * | 2018-12-27 | 2019-04-26 | 深圳前海微众银行股份有限公司 | The implementation method and device of intelligent customer service system, equipment, readable storage medium storing program for executing |
| CN110032592A (en) * | 2019-03-06 | 2019-07-19 | 国网辽宁省电力有限公司信息通信分公司 | A kind of energy big data integration system and method |
| CN111581001A (en) * | 2020-04-27 | 2020-08-25 | 佛山科学技术学院 | Operation maintenance method and device for big data cluster |
| CN111985755A (en) * | 2019-05-21 | 2020-11-24 | 国际商业机器公司 | Minimizing risk using machine learning techniques |
| CN112101562A (en) * | 2019-06-18 | 2020-12-18 | 第四范式（北京）技术有限公司 | Method and system for realizing machine learning modeling process |
| CN112346860A (en) * | 2020-10-27 | 2021-02-09 | 四川长虹电器股份有限公司 | Method and system for elastically deploying service based on machine learning |
| CN113296840A (en) * | 2020-02-20 | 2021-08-24 | 银联数据服务有限公司 | Cluster operation and maintenance method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140129000A1 (en) * | 2012-11-06 | 2014-05-08 | General Electric Company | Systems and Methods For Dynamic Risk Derivation |
| CN104038957A (en) * | 2014-06-13 | 2014-09-10 | 杭州大光明通信系统集成有限公司 | 4G base station operation maintenance information analysis process method based on integration structure |
| CN105630648A (en) * | 2015-12-02 | 2016-06-01 | 武汉物联远科技有限公司 | Multidimensional data deep learning-based data center intelligent control method and system |
| CN106371975A (en) * | 2016-08-31 | 2017-02-01 | 国信优易数据有限公司 | Automatic operation and maintenance early-warning method and system |
| CN106790565A (en) * | 2016-12-27 | 2017-05-31 | 中国电子科技集团公司第五十二研究所 | A kind of network attached storage group system |
| CN107239835A (en) * | 2017-07-12 | 2017-10-10 | 华北石油通信公司 | Build the method and system that oil gas ground produces different kinds of process flow differentiation grade O＆M |
-
2017
- 2017-12-29 CN CN201711484351.4A patent/CN108197251A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140129000A1 (en) * | 2012-11-06 | 2014-05-08 | General Electric Company | Systems and Methods For Dynamic Risk Derivation |
| CN104038957A (en) * | 2014-06-13 | 2014-09-10 | 杭州大光明通信系统集成有限公司 | 4G base station operation maintenance information analysis process method based on integration structure |
| CN105630648A (en) * | 2015-12-02 | 2016-06-01 | 武汉物联远科技有限公司 | Multidimensional data deep learning-based data center intelligent control method and system |
| CN106371975A (en) * | 2016-08-31 | 2017-02-01 | 国信优易数据有限公司 | Automatic operation and maintenance early-warning method and system |
| CN106790565A (en) * | 2016-12-27 | 2017-05-31 | 中国电子科技集团公司第五十二研究所 | A kind of network attached storage group system |
| CN107239835A (en) * | 2017-07-12 | 2017-10-10 | 华北石油通信公司 | Build the method and system that oil gas ground produces different kinds of process flow differentiation grade O＆M |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067619A (en) * | 2018-09-25 | 2018-12-21 | 海南民航凯亚有限公司 | A kind of elasticity volume dispatching method and processing terminal that micro services are administered |
| CN109614978A (en) * | 2018-09-29 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN109684455A (en) * | 2018-12-27 | 2019-04-26 | 深圳前海微众银行股份有限公司 | The implementation method and device of intelligent customer service system, equipment, readable storage medium storing program for executing |
| CN110032592B (en) * | 2019-03-06 | 2023-07-28 | 国网辽宁省电力有限公司信息通信分公司 | Energy big data integration system and method |
| CN110032592A (en) * | 2019-03-06 | 2019-07-19 | 国网辽宁省电力有限公司信息通信分公司 | A kind of energy big data integration system and method |
| CN111985755B (en) * | 2019-05-21 | 2024-06-07 | 国际商业机器公司 | Method and system for minimizing risk using machine learning techniques |
| CN111985755A (en) * | 2019-05-21 | 2020-11-24 | 国际商业机器公司 | Minimizing risk using machine learning techniques |
| CN112101562A (en) * | 2019-06-18 | 2020-12-18 | 第四范式（北京）技术有限公司 | Method and system for realizing machine learning modeling process |
| CN112101562B (en) * | 2019-06-18 | 2024-01-30 | 第四范式（北京）技术有限公司 | Implementation method and system of machine learning modeling process |
| CN113296840A (en) * | 2020-02-20 | 2021-08-24 | 银联数据服务有限公司 | Cluster operation and maintenance method and device |
| CN111581001A (en) * | 2020-04-27 | 2020-08-25 | 佛山科学技术学院 | Operation maintenance method and device for big data cluster |
| CN112346860B (en) * | 2020-10-27 | 2022-02-08 | 四川长虹电器股份有限公司 | Method and system for elastically deploying service based on machine learning |
| CN112346860A (en) * | 2020-10-27 | 2021-02-09 | 四川长虹电器股份有限公司 | Method and system for elastically deploying service based on machine learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197251A (en) | A kind of big data operation and maintenance analysis method, device and server | |
| WO2023093177A1 (en) | Device fault diagnosis method and apparatus, and electronic device and storage medium | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN107688591B (en) | A kind of actuarial treating method and apparatus | |
| CN109491857B (en) | Data monitoring method and system based on rule engine and terminal equipment | |
| CN109508355A (en) | A kind of data pick-up method, system and terminal device | |
| CN107463661A (en) | The introduction method and device of data | |
| JP2016100005A (en) | Reconcile method, processor and storage medium | |
| CN109389518A (en) | Association analysis method and device | |
| CN111159897A (en) | Target optimization method and device based on system modeling application | |
| CN111783893A (en) | Method and system for generating combined features of machine learning samples | |
| CN110969172A (en) | Text classification method and related equipment | |
| CN104657437A (en) | Monitoring method and monitoring device for promotion status data | |
| CN108696559B (en) | Stream processing method and device | |
| CN115482419B (en) | Data acquisition and analysis method and system for marine fishery products | |
| CN111475375A (en) | Data state updating method and equipment | |
| CN108089871A (en) | Automatic updating method of software, device, equipment and storage medium | |
| CN117290744A (en) | Electrical node aggregation method and device based on graph theory | |
| CN111695077A (en) | Asset information pushing method, terminal equipment and readable storage medium | |
| WO2024065776A1 (en) | Method for data processing, apparatus for data processing, electronic device, and storage medium | |
| US11782923B2 (en) | Optimizing breakeven points for enhancing system performance | |
| CN118057327A (en) | Information Technology (IT) system alarm data processing method and device based on knowledge graph | |
| CN114969187A (en) | Data analysis system and method | |
| CN111130921B (en) | Method and device for processing performance index of core network element | |
| CN113987186A (en) | Method and device for generating marketing scheme based on knowledge graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180622 |
|
| RJ01 | Rejection of invention patent application after publication |