CN112346860B - Method and system for elastically deploying service based on machine learning - Google Patents
Method and system for elastically deploying service based on machine learning Download PDFInfo
- Publication number
- CN112346860B CN112346860B CN202011163987.0A CN202011163987A CN112346860B CN 112346860 B CN112346860 B CN 112346860B CN 202011163987 A CN202011163987 A CN 202011163987A CN 112346860 B CN112346860 B CN 112346860B
- Authority
- CN
- China
- Prior art keywords
- service
- server
- closing
- model
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Abstract
The invention relates to the technical field of computers, aims to solve the problems of untimely service deployment and waste of manpower and resources, and provides a method and a system for flexibly deploying services based on machine learning, wherein the scheme is summarized as follows: acquiring log information called by a server interface and use information of hardware resources of a server when a user issues a command for starting and closing the service; taking log information corresponding to the instructions for opening and closing the service and the use information of the hardware resource as samples of machine learning, and respectively training to obtain a service opening model and a service closing model; and acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result. The invention can save service resources and human resources.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a system for flexibly deploying services based on machine learning.
Background
At present, various network parameters and hardware use conditions of a server are generally monitored by using a zabbix monitoring system, the zabbix can only be used for providing a monitoring alarm function of the server, the problems of response timeout of a service interface and the like when the user quantity of the service is suddenly increased cannot be solved, when the load condition of the service is too high, the number of service clusters needs to be manually increased if the stability of the service needs to be ensured, when the load of the service is lower, some services need to be manually closed to reduce the waste of resources, and if the server is added or subtracted based on manual judgment, untimely service deployment and too much labor can be caused.
Disclosure of Invention
The invention aims to solve the problems of untimely service deployment and waste of manpower and resources in the prior art, and provides a method and a system for flexibly deploying services based on machine learning.
The technical scheme adopted by the invention for solving the technical problems is as follows: the method for elastically deploying the service based on the machine learning comprises the following steps:
step 1, acquiring log information called by a server interface and use information of hardware resources of a server when a user issues a command for starting and closing the service;
step 2, taking log information corresponding to the instructions for opening and closing the service and the use information of the hardware resource as machine learning samples, and respectively training to obtain a service opening model and a service closing model;
and 3, acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
Further, the log information called by the server interface at least includes: the time of the current interface call, the call volume of the interface and the response time of the interface.
Further, the usage information of the hardware resource at least includes: the server has used the memory size, the CPU load of the server, and the network card traffic of the server.
Further, the service opening model and the service closing model are obtained based on logistic regression algorithm training.
Further, step 3 is preceded by: and calculating the accuracy of the service model opening and closing based on a recall rate algorithm, entering the step 3 when the accuracy reaches a preset value, and otherwise, entering the step 1.
Further, the log information called by the server interface is acquired by using a logstack component.
Further, the use information of the hardware resource of the server is acquired by using a zabbix component.
The invention also provides a system for elastically deploying services based on machine learning, which comprises:
the acquisition unit is used for acquiring log information called by the server interface and the use information of the hardware resources of the server in real time, and acquiring the log information called by the server interface and the use information of the hardware resources of the server when a user issues an instruction for starting and closing the service;
the training unit is used for respectively training log information corresponding to the service opening and service closing instructions and the use information of the hardware resources as machine learning samples to obtain a service opening model and a service closing model;
and the service deployment unit is used for judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
Further, the log information called by the server interface at least includes: the time of the current interface call, the call volume of the interface and the response time of the interface.
Further, the usage information of the hardware resource at least includes: the server has used the memory size, the CPU load of the server, and the network card traffic of the server.
The invention has the beneficial effects that: according to the method and the system for elastically deploying the service based on the machine learning, disclosed by the invention, when the service is increased or decreased, the resource use condition of the server is calculated through the machine learning method, so that a more and more accurate model is trained, the service resource can be more accurately and reasonably deployed, and the service resource and the human resource are saved.
Drawings
Fig. 1 is a schematic flowchart of a method for elastically deploying a service based on machine learning according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a platform of a method for implementing a machine learning-based flexible deployment service according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a system for elastically deploying a service based on machine learning according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
The invention discloses a method for flexibly deploying services based on machine learning, which comprises the following steps: step 1, acquiring log information called by a server interface and use information of hardware resources of a server when a user issues a command for starting and closing the service; step 2, taking log information corresponding to the instructions for opening and closing the service and the use information of the hardware resource as machine learning samples, and respectively training to obtain a service opening model and a service closing model; and 3, acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
In particular, the user can analyze the stability and reliability of the service through the corresponding event of the call of the interface log, when the response time of the service reaches a certain threshold value, the user can manually issue corresponding instructions to start or close some services, when the user manually issues the instructions to start and close the services, collecting the log information called by the corresponding server interface and the use information of the hardware resource of the server, storing the log information and the use information in a database as a machine learning sample, then, the log information called by the server interface corresponding to the instruction for opening the service is manually sent by the user and the use information of the hardware resource of the server are used for training to obtain a service opening model, the method comprises the steps that a service closing model is obtained through training by log information called by a server interface corresponding to a service closing instruction manually issued by a user and using information of hardware resources of a server; and when subsequent services are deployed, acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, respectively inputting the log information and the use information into the service opening model and the service closing model to obtain a result of whether the services are opened or closed, and finally automatically opening or closing the services according to the output result.
Example 1
The method for elastically deploying services based on machine learning is implemented based on the following platform: the system comprises a server resource platform, a log acquisition platform, a search engine platform, a server monitoring platform and an elastic deployment service platform.
A server resource platform refers to a group of servers that can be used to deploy a service. The log collection platform collects log information called by a service interface by using a logstack component. The search engine platform refers to using an elastic search component to search for needed log information. The server monitoring platform is used for monitoring the use information of the hardware resource of the server by using a zabbix component. The flexible deployment service platform refers to a platform capable of issuing an instruction to start or close a service, and configuring information of server resources and a configuration issuing instruction and the like.
As shown in fig. 2, the log collection platform can analyze the size of the service interface call amount by collecting the call log of the service interface on the server resource platform, and determine whether the service is stably providing the interface, and the server monitoring platform is used for monitoring the service condition of the server resource. The search engine platform mainly acquires data of the log acquisition platform and provides the data to the elastic deployment service platform, the elastic deployment service platform analyzes log information and monitoring information, hardware resources of the server are reasonably distributed according to the running condition of the service, and the intelligent opening and closing of the service can be achieved through a long-term training model by adopting a machine learning algorithm.
The method for elastically deploying services based on machine learning, as shown in fig. 1, includes the following steps:
step S1, acquiring the log information called by the server interface and the use information of the hardware resource of the server when the user issues the instruction of opening and closing the service;
wherein, the log information called by the server interface at least comprises: the calling time of the current interface, the calling amount of the interface and the response time of the interface; the usage information of the hardware resources at least comprises: the server has used the memory size, the CPU load of the server, and the network card traffic of the server.
The realization process is as follows: firstly, a web configuration platform needs to be developed to configure hardware information of a management server resource platform, such as: host name, ip, group name, service name deployed on the server, port corresponding to the service, path deployed by the service, and the like, and information that the zabbix monitoring platform needs to monitor, such as information of memory of the server, flow of the network card, load of the cpu, and the like.
And the log acquisition platform acquires the calling information of the service interface and transmits the calling information to the search engine for storage.
The flexible deployment service platform acquires the interface calling condition at a certain time in real time through a search engine, analyzes the stability, reliability and the like of the service through the calling response time of an interface log, and when the response time of the service exceeds a certain threshold, a user can manually issue an instruction to start or close the service, and some information when the service is currently started or closed is as follows: the time of the current interface call, the call volume of the interface, the response time of the interface, and the state of whether to start or close the service are saved in a database.
Step S2, taking log information corresponding to the instructions for opening and closing the service and the use information of the hardware resource as machine learning samples, and respectively training to obtain a service opening model and a service closing model;
specifically, the flexible deployment service platform obtains samples of machine learning, such as: and training a service opening model and a service closing model based on the characteristics of time points of opening or closing the service, the interface calling quantity of the service, the response time of the interface, the memory of the server, the cpu of the server, the network card flow of the server and the like by using a logistic regression algorithm.
In order to ensure the accuracy of the model, the method further comprises the following steps: and calculating the accuracy of the service model opening and closing based on a recall rate algorithm, and entering step S3 when the accuracy reaches a preset value, or entering step S1 otherwise.
Step S3, collecting the log information called by the server interface and the use information of the hardware resource of the server in real time, judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
Example 2
The following example mainly describes how to dynamically add a service.
(1) Firstly, a service is deployed in a group of server resources, wherein the service is named as spring-boot-fm, and the service is used for simulating that the deployment service is dynamically increased when the interface access amount is increased, and the service is dynamically closed when the interface access amount is low.
When a user accesses the spring-boot-fm service, log information of the interface service is generated, and part of the information is as follows:
(2) the log collection platform is built by using a logstack component and is used for collecting log information called by the spring-boot-fm interface, the search engine platform is built by using an elastic search component and is used for storing data collected by the log collection platform and retrieving and counting the number of times of calling the spring-boot-fm interface per second through the elastic search on the elastic deployment service platform, and the number of times of counting the response time of the interface is more than 800 ms.
(3) The server monitoring platform is built by using the zabbix assembly and is used for monitoring the memory utilization rate of the server, the cpu load and the network card flow information, and the server hardware information monitored by the zabbix is as follows:
(4) the specification is defined on the flexible deployment service platform, when the interface call response time exceeds 15 seconds continuously (exceeds 1 second), the memory utilization rate exceeds 80%, the CPU load exceeds 80%, the network card inflow flow exceeds 200Mbps, and the network card outflow flow exceeds 200Mbps, services are dynamically deployed on an idle server, and the response time of the service interface, the memory utilization rate, the CPU load, the network card flow and the like are reduced through load balancing.
(5) And the flexible deployment service platform judges that the calling amount of the service interface with the server ip of 192.168.0.101 is large according to the collected log information and the hardware use information of the server, the use of the hardware resource of the server exceeds a specified threshold value, and detects that the server with the server ip of 192.168.0.102 is idle, so that the service can be manually deployed on the idle server, and the stability and the reliability of the service are ensured through load balancing.
(6) And when the deployment service is started, storing the hardware resource information and the log information called by the interface into a database, wherein the recorded information characteristics comprise the name of the service, the interface response time information of the service, the memory utilization rate of the hardware resource, the load of the CPU, the flow information of the network card and the state of starting the service.
(7) The method includes training a model to start deploying the service using machine learning, training using a logistic regression algorithm that uses a sigmod function to calculate a probability that the service starts deploying the service.
(9) And after a large amount of data samples are trained, a model with better precision is obtained, when the response time of a transmission parameter interface is 1200ms, a command for starting a new service is obtained when the utilization rate of the CPU of the server is 82% in the model, and further, the automatic deployment of the service is realized.
Based on the above technical solution, the present invention further provides a system for elastically deploying services based on machine learning, as shown in fig. 3, including:
the acquisition unit is used for acquiring log information called by the server interface and the use information of the hardware resources of the server in real time, and acquiring the log information called by the server interface and the use information of the hardware resources of the server when a user issues an instruction for starting and closing the service;
the training unit is used for respectively training log information corresponding to the service opening and service closing instructions and the use information of the hardware resources as machine learning samples to obtain a service opening model and a service closing model;
and the service deployment unit is used for judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
It can be understood that, since the machine learning based flexible deployment service system according to the embodiment of the present invention is a system for implementing the machine learning based flexible deployment service method according to the embodiment, for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is simpler, and for relevant points, reference may be made to the partial description of the method.
Claims (5)
1. The method for flexibly deploying the service based on the machine learning is characterized by comprising the following steps:
step 1, acquiring log information called by a server interface and use information of hardware resources of a server when a user issues an instruction for starting and closing the service, wherein the log information called by the server interface at least comprises the following steps: the time of the current interface call, the call volume of the interface and the response time of the interface, and the use information of the hardware resources at least comprises: the size of the used memory of the server, the CPU load of the server and the network card flow of the server;
step 2, using log information corresponding to the instructions for opening and closing the service and using information of hardware resources as machine learning samples, respectively training to obtain a service opening model and a service closing model, wherein the service opening model and the service closing model are obtained by training based on a logistic regression algorithm;
and 3, acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
2. The method for machine learning-based resilient deployment of services as claimed in claim 1, wherein said step 3 is preceded by further comprising: and calculating the accuracy of the service model opening and closing based on a recall rate algorithm, entering the step 3 when the accuracy reaches a preset value, and otherwise, entering the step 1.
3. The method for machine learning-based resilient deployment of services according to claim 1, wherein log information of the server interface calls is collected using a logstack component.
4. The method of machine learning-based resilient deployment service of claim 1, wherein usage information of hardware resources of the server is acquired using a zabbix component.
5. A system for machine learning based resilient deployment of services, comprising:
the acquisition unit is used for acquiring log information called by a server interface and the use information of the hardware resources of the server in real time, and acquiring the log information called by the server interface and the use information of the hardware resources of the server when a user issues an instruction for starting and closing the service, wherein the log information called by the server interface at least comprises: the time of the current interface call, the call volume of the interface and the response time of the interface, and the use information of the hardware resources at least comprises: the size of the used memory of the server, the CPU load of the server and the network card flow of the server;
the training unit is used for respectively training log information corresponding to the instructions for opening and closing the service and using information of hardware resources as machine learning samples to obtain an opening service model and a closing service model, wherein the opening service model and the closing service model are obtained by training based on a logistic regression algorithm;
and the service deployment unit is used for judging whether to start or close the service based on the service starting model and the service closing model, and starting or closing the service according to the judgment result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011163987.0A CN112346860B (en) | 2020-10-27 | 2020-10-27 | Method and system for elastically deploying service based on machine learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011163987.0A CN112346860B (en) | 2020-10-27 | 2020-10-27 | Method and system for elastically deploying service based on machine learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112346860A CN112346860A (en) | 2021-02-09 |
| CN112346860B true CN112346860B (en) | 2022-02-08 |
Family
ID=74360284
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011163987.0A Active CN112346860B (en) | 2020-10-27 | 2020-10-27 | Method and system for elastically deploying service based on machine learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112346860B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197251A (en) * | 2017-12-29 | 2018-06-22 | 百度在线网络技术（北京）有限公司 | A kind of big data operation and maintenance analysis method, device and server |
| CN108595301A (en) * | 2018-03-26 | 2018-09-28 | 中国科学院计算技术研究所 | A kind of server energy consumption prediction technique and system based on machine learning |
| CN110209467A (en) * | 2019-05-23 | 2019-09-06 | 华中科技大学 | A kind of flexible resource extended method and system based on machine learning |
| CN110990138A (en) * | 2019-12-04 | 2020-04-10 | 北京三快在线科技有限公司 | Resource scheduling method, device, server and storage medium |
| CN111309539A (en) * | 2020-03-26 | 2020-06-19 | 北京奇艺世纪科技有限公司 | Abnormity monitoring method and device and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8560465B2 (en) * | 2009-07-02 | 2013-10-15 | Samsung Electronics Co., Ltd | Execution allocation cost assessment for computing systems and environments including elastic computing systems and environments |
-
2020
- 2020-10-27 CN CN202011163987.0A patent/CN112346860B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197251A (en) * | 2017-12-29 | 2018-06-22 | 百度在线网络技术（北京）有限公司 | A kind of big data operation and maintenance analysis method, device and server |
| CN108595301A (en) * | 2018-03-26 | 2018-09-28 | 中国科学院计算技术研究所 | A kind of server energy consumption prediction technique and system based on machine learning |
| CN110209467A (en) * | 2019-05-23 | 2019-09-06 | 华中科技大学 | A kind of flexible resource extended method and system based on machine learning |
| CN110990138A (en) * | 2019-12-04 | 2020-04-10 | 北京三快在线科技有限公司 | Resource scheduling method, device, server and storage medium |
| CN111309539A (en) * | 2020-03-26 | 2020-06-19 | 北京奇艺世纪科技有限公司 | Abnormity monitoring method and device and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| CMS:A Continuous Machine-Learning and Serving Platform for Industrial Big Data;KeDi Li等;《future internet》;20200610;第12卷(第6期);第1-15页 * |
| 面向集群服务器系统的监控平台综述;胡鹤等;《科研信息化技术与应用》;20150120;第9卷(第1期);第79-88页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112346860A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111092852B (en) | Network security monitoring method, device, equipment and storage medium based on big data | |
| EP2741211B1 (en) | Method and device for monitoring power consumption of android terminal application | |
| CN103746829A (en) | Cluster-based fault perception system and method thereof | |
| CN110287081A (en) | A kind of service monitoring system and method | |
| CN104899130A (en) | Application program management method and system | |
| CN103064390A (en) | Room environment remote monitoring system and monitoring method thereof based on Internet of Things | |
| CN110891283A (en) | Small base station monitoring device and method based on edge calculation model | |
| WO2014187304A1 (en) | User state determination method and device | |
| CN110417873B (en) | Network information extraction system for realizing recording webpage interactive operation | |
| CN102497292A (en) | Computer cluster monitoring method and system thereof | |
| CN110502366A (en) | Case executes method, apparatus, equipment and computer readable storage medium | |
| CN111756560A (en) | Data processing method, device and storage medium | |
| CN102541885A (en) | Method and device for detecting database blockage | |
| CN111970151A (en) | Flow fault positioning method and system for virtual and container network | |
| CN112346860B (en) | Method and system for elastically deploying service based on machine learning | |
| CN115085994A (en) | Information centralized safety management system of internal physical machine of cloud computing | |
| CN107463490A (en) | A kind of cluster daily record centralized collection method being applied in platform development | |
| TW201628384A (en) | Traffic prediction and resource scheduling automatic analyzing system of mobile network base station and method thereof | |
| CN106506212A (en) | Abnormal information acquisition methods and user terminal | |
| CN116913350B (en) | Test method and test platform based on modern standby of solid state disk | |
| CN113949652A (en) | User abnormal behavior detection method and device based on artificial intelligence and related equipment | |
| CN116567127B (en) | Smart phone with fault monitoring function | |
| CN110337081A (en) | The monitoring method of local fault, device and storage medium in indoor distributed system | |
| CN115525392A (en) | Container monitoring method and device, electronic equipment and storage medium | |
| CN110175083A (en) | The monitoring method and device of operating system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |