CN112084040B - Container resource planning system and method based on application mirror image data identification - Google Patents
Container resource planning system and method based on application mirror image data identification Download PDFInfo
- Publication number
- CN112084040B CN112084040B CN202011038544.9A CN202011038544A CN112084040B CN 112084040 B CN112084040 B CN 112084040B CN 202011038544 A CN202011038544 A CN 202011038544A CN 112084040 B CN112084040 B CN 112084040B
- Authority
- CN
- China
- Prior art keywords
- application
- container
- resource
- mirror image
- probe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
Abstract
The invention discloses a system and a method for planning container resources based on application mirror image data identification, wherein the system comprises a container management platform, a container resource planning module and one or more applications, each application comprises two forms of an application mirror image and an application container, and one or more application mirror images and one or more application containers are arranged; an application information probe is deployed on the container management platform; a resource model library, a resource model calculation engine and a resource planning controller are deployed on the container resource planning module; the method comprises the steps of 1-7, the invention takes the integral container mirror image data as the basis of the identification of the resource consumption type of the application container, comprehensively analyzes the resource consumption type of the application container through abundant application information probes, applies a machine learning technology to the analysis and identification of the resource consumption type of the application container, and greatly improves the accuracy of the identification of the application type and the accuracy of the resource allocation.
Description
Technical Field
The invention relates to resource planning of applications on a container platform, in particular to a system and a method for planning container resources based on application mirror image data identification.
Background
The container is widely applied due to the characteristics of second-level starting, light weight, safety isolation and the like, in the process of using a container platform, resource management needs to be carried out on an application, the application comprises a CPU (central processing unit), a memory, a network, a process, a storage I/O (input/output) and the like used by the application container, a limit value and a reserved value are usually set for each resource, the limit value is an upper limit of the resource which can be used by the container, the reserved value is a lower limit of the resource which can be used by the container, namely the system is the resource share which can be easily reserved, and by setting the limit value and the reserved value, performance isolation and high-efficiency use of the resource are realized, resource waste is prevented, and the predictability of the service rate is ensured; when the container is started, the server checks whether enough unreserved resources are available, and only when enough resources are available, the container is allowed to be started, however, in a real enterprise scene, often when a user configures the amount of the container resources, a conservative large value is set according to experience due to risk consideration and is not a test value obtained through testing, so that the number of applications deployed by one host is small, and the waste of the resources is caused; there are currently some container platform solutions related to optimizing container resource planning, including:
1. some container platforms perform simulation operation tests on each container before deployment to obtain a reasonable allocation value of resources, but the method usually introduces corresponding test work on each container mirror image, which greatly increases the workload of users and the use cost of the container platforms, so that the method is not suitable when the number of containers operating on the container platforms is large;
2. some container platforms can set a unified default container resource recommendation value, that is, if a user does not set a resource for a container when deploying the container, the system can set a resource limit value and a reserved value for the container by using the unified default value, so that the difference between the containers is ignored, and the requirements of different types of application programs for the resource may be quite different, for example, the requirements of the application programs adopting Python and Golang programming languages for the resource are lower, and the application programs adopting Java programming language relatively need to occupy more resources such as CPUs, memories and the like, so when the types of the application programs running on the container platforms are more various, the method is not applicable;
3. some container platforms may classify containers according to container mirror information (such as mirror name, mirror type, mirror version), and preset resource configuration values for different container types, such as any container whose mirror name includes "tomcat", no matter tomcat: 6.0 or app-tomcat: 7.0, automatically setting the resource configuration of the corresponding container according to the preset value; however, the amount of information that can be carried by the mirror image information is very limited, and if only the identification and classification are performed through the mirror image information, the resource configuration is often very inaccurate, for example, the mirror image is also a tomcat container mirror image, some containers have very large internal applications and can only need 8G memories to operate, some containers have relatively small internal applications and can only need 1G memories to operate, and in addition, in this kind of method, the resource preset values of the platform are often statically configured in the platform database, and the capability of continuous optimization according to the actual container operation condition is also lacked;
4. some container platforms can pre-judge the future resource requirements of the container according to the historical data of the container operation, and adjust the placement strategy and resources of the container in real time according to the pre-judgment, but the method only focuses on the resource adjustment in the container operation process, does not solve the resource planning problem in the container deployment, and does not intelligently optimize the resource planning strategy in the container deployment according to the historical data of the container operation;
the chinese patent application CN108829494A discloses a load prediction-based intelligent resource optimization method for a container cloud platform, which does not perform intelligent resource allocation before container deployment, but only migrates the operating container to other nodes with appropriate loads to improve the resource utilization rate of the whole cluster, belonging to the resource allocation in operation, and no resource request value and limit value are set for the container, while the resource usage of the container is often fluctuating and even unpredictable in the internet scenario, the problem that may be caused by such a scheduling manner is that a series of operations such as container packaging into mirror images, re-deployment, and deletion of the original container may be frequently performed due to traffic fluctuation, and more additional resource consumption is caused to start and terminate the container, which may even affect fixed IP requirements, data mounting, log consistency, and the like of user services. According to the method, resource allocation optimization is preposed to the initial deployment of the container application, namely, resource usage prediction and allocation are performed before the container application is online, continuous optimization is performed during operation, and resource models of different images are continuously accumulated through a built-in resource model library, so that resources consumed by resource model prediction and optimization repeated calculation of the same image starting container are saved.
The chinese patent application CN109656717A discloses a containerized cloud resource allocation method, which needs a pre-test link for allocating resources to applications, that is, after the applications are respectively deployed in different resource types, the applications are analyzed and evaluated to obtain the resource types that should be configured, and this is exactly the test work that increases the user workload and the use cost of the container platform before the container deployment, which is to be avoided by this patent.
The chinese patent application CN111327655A discloses a method, an apparatus, and an electronic device for predicting resource quotas of multi-tenant containers, where a prediction algorithm for container resource usage is still based on historical data of container operation tests, that is, a preceding test link is required to generate required historical data, and this is also a test work of increasing user workload and use cost of a container platform before container deployment, which is to be avoided by this patent.
The number of containers on a container platform may be huge especially in the micro-service scenario, so the workload of setting by a manual method is very large and is repetitive mechanical, and errors are very easy to occur, and considering that the demands of the same type of container on resources are similar in the same enterprise, a container resource planning system and method based on application mirroring data identification are needed.
Disclosure of Invention
The invention provides a container resource planning system based on application mirror image data identification, which aims to solve the technical problems that the number of containers on a container platform is possibly huge especially in the micro-service scene, so that the workload for setting by a manual mode is very huge and is repeated mechanically, and errors are easy to occur; the method can automatically identify and classify the container types to be deployed on the container platform according to the container mirror image data, and give the recommended values of the corresponding container resources based on strategies, thereby greatly improving the efficiency and accuracy; the whole process adopts an automatic mode, and a large number of application resource consumption models based on best practice are preset, so that the workload and threshold for manually configuring container resources are greatly reduced, the accuracy of the container resource configuration is improved, and the method is very suitable for the current increasingly wide containerized application deployment scene; the application container resource model can be continuously optimized according to the collection and analysis of the real-time running data of the application, so that the scientificity of subsequent application resource planning is further improved through continuous self-learning, and the defects caused by the prior art are overcome.
The invention also provides a container resource planning method based on application mirror image data identification.
In order to solve the technical problems, the invention provides the following technical scheme:
a container resource planning system based on application mirror image data identification comprises a container management platform, a container resource planning module and one or more applications, wherein each application comprises two forms of an application mirror image and an application container, and one or more application mirror images and one or more application containers are arranged;
an application information probe is deployed on the container management platform;
a resource model library, a resource model calculation engine and a resource planning controller are deployed on the container resource planning module;
the application information probe is used for carrying out container type detection scanning on the application mirror image to acquire container mirror image data and is also used for acquiring resource consumption data of an application container in the application during running;
the resource model calculation engine is used for acquiring the container mirror image data and the container runtime resource consumption data; the resource model library is also used for calculating the resource consumption mode of the container mirror image transmitted to the resource model library; the resource model library is also used for generating a resource model transmitted to the resource model library; the system is also used for data verification and algorithm parameter optimization; further for detecting whether the application image is identified;
the resource planning controller is used for mapping the resource model into the container resource setting data and transmitting the container resource setting data back to the container management platform, recovering and executing the deployment behavior of the application container, and completing the deployment of the application container;
the resource model library is used for storing the container resource setting data, the resource model and the resource consumption mode for the resource planning controller to extract.
The system for planning container resources based on application mirror image data identification comprises one or more application information probes, an application middleware probe, an application data source probe, an application algorithm probe, an application access probe, a CPU probe, a memory probe, a network probe, a storage probe, and an I/O probe, wherein the application information probe may have a difference between the use demand of resources in actual operation and the resource model calculated according to experience, and in order to obtain the optimal resource configuration, the application information probe continuously reads the actually consumed resource value from the application container in operation and feeds the actually consumed resource value back to the resource model calculation engine for data verification and algorithm parameter optimization, so that on one hand, the algorithm is continuously optimized according to actual data, and on the other hand, the optimized resource configuration value is synchronized into the resource model base in real time, and applying the application container to the running application container according to the strategy as required;
the application language probe is used for detecting the writing language of the application according to the container mirror image data, such as Java, Python, Golang, Ruby and the like;
the application middleware probe is used for detecting Web middleware used by the application and configuration information thereof, such as Tomcat, flash and other Web middleware and configuration thereof according to the container mirror image data;
the application data source probe is used for detecting the data source of the application according to the container mirror image data;
the application algorithm probe is used for detecting the applied algorithm and the complexity thereof according to the container mirror image data;
the application access probe is used for detecting whether the application needs external access, internal access and access types and frequency according to the container mirror image data;
the CPU probe is used for detecting a CPU actually consumed by the operation of the application container;
the memory probe is used for detecting the memory actually consumed by the operation of the application container;
the network probe is used for detecting network resources actually consumed by the operation of the application container;
the storage probe is used for detecting the storage resources actually consumed by the operation of the application container;
the I/O probe is used for detecting I/O actually occupied by the operation of the application container;
the probes referred to above are suitable for use in both single cluster and multi-cluster environments, each cluster maintaining the deployment architecture described above.
The system for planning container resources based on application mirror image data identification is described above, wherein a mirror image warehouse is built in the container management platform;
the mirror repository is used for storing application mirrors.
The above system for planning container resources based on application mirror image data identification is provided, wherein the data verification is that the actual resource consumption data value is compared with the predicted data when the container runs.
In the above system for planning container resources based on application mirror image data identification, the actual resource consumption data value is compared with the predicted data through classification analysis and cluster analysis.
In the above system for planning container resources based on application mirror image data identification, a corresponding relationship list between the application mirror image and the resource model is stored in the resource model library.
In a second aspect, a method for planning container resources based on application image data identification includes the following steps:
step 1: the resource planning controller monitors and captures the application deployment behavior of the container management platform in real time;
step 2: the resource planning controller captures an application deployment behavior and then intercepts the application deployment behavior and triggers an application information probe to collect information for the application, for example, the resource planning controller can capture and intercept the application deployment behavior through an Admission Control mechanism (Admission Control), and the technical scheme is not limited to use of the Admission Control mechanism (Admission Control) as an interception technology;
and step 3: the application information probe acquires container mirror image data in an application mirror image in application and transmits the container mirror image data to a resource model calculation engine for calculation and analysis of an application resource consumption model;
and 4, step 4: a resource model calculation engine receives container mirror image data and detects whether application mirror image is identified or not;
if the application mirror image is marked as the identified mirror image, the resource model base stores the resource consumption mode of the application mirror image, and the resource planning controller directly calls the resource consumption mode of the application mirror image from the resource model base;
if the application mirror image is not marked as the identified mirror image, the resource model calculation engine calculates a resource model of the application mirror image through a preset application resource consumption model algorithm, adds the resource model into a resource model library, simultaneously marks the application mirror image as the identified mirror image, updates the corresponding relation between the application mirror image and the resource model by the resource model library, and transmits the resource model to the resource planning controller;
and 5: the resource planning controller maps the received resource model into container resource setting data, updates the application container resource configuration in the application deployment behaviors intercepted in the step 2 according to the container resource setting data, and recovers the application deployment behaviors of the container management platform to complete application deployment;
step 6: after the application container runs, the application information probe continuously reads the actual consumption data value of the resource from the running application container and feeds the actual consumption data value back to the resource model calculation engine for continuous algorithm optimization and resource allocation optimization to generate an optimized resource consumption mode;
and 7: and synchronously updating the optimized resource consumption mode into a resource model library, and simultaneously applying the container resource setting data mapped by the optimized resource consumption mode into the running application container.
In the above method for planning container resources based on application image data identification, step 4 further needs to store the application image marked as the identified image into an image warehouse in the container management platform;
the mirror repository is used for storing application mirrors.
In a third aspect, a processor for container resource planning based on application image data identification includes at least one processor and a memory coupled to the at least one processor, the memory storing executable instructions that when executed by the at least one processor cause the method described above to be implemented.
In a fourth aspect, a chip, comprises: a processor for calling and running the computer program from the memory so that the device in which the chip is installed performs: the method described above.
The container technology is that the application and the running environment thereof are packaged together through a combined file system with a layer structure, the packaged product is an application mirror image which is a base stone for constructing a container, the application mirror image is a construction part of a containerization application life cycle, is a static representation and data storage format and can be used as a source code of an application container; when the program packaged in the application image is run by the container engine, the application image becomes an application container; the containerized applications are issued through application images, and resource consumption modes of application containers run by the same application image are similar, so that the system can calculate the resource consumption modes of the applications according to information collected in application image data, namely, the application image is taken as a unit to identify and compare application resource models, meanwhile, a resource model library is arranged in the system, the best practices of common application resource models are stored in advance, and a corresponding relation list of the application image and the resource models is maintained.
According to the technical scheme provided by the system and the method for planning the container resources based on application mirror image data identification, the technical effects are as follows:
the method has the advantages that the whole container mirror image data is used as the basis for identifying the resource consumption types of the application containers, the resource consumption types of the application containers are comprehensively analyzed through abundant application information probes, and the machine learning technology is applied to the analysis and identification of the resource consumption types of the application containers, so that the accuracy of the application type identification and the accuracy of resource allocation are greatly improved;
the method can automatically identify and classify the container types to be deployed on the container platform according to the container mirror image data, and give the recommended values of the corresponding container resources based on strategies, thereby greatly improving the efficiency and accuracy;
the whole process adopts an automatic mode, and a large number of application resource consumption models based on best practice are preset, so that the workload and threshold for manually configuring container resources are greatly reduced, the accuracy of the container resource configuration is improved, and the method is very suitable for the current increasingly wide containerized application deployment scene;
the application container resource model can be continuously optimized according to the collection and analysis of the real-time running data of the application, so that the scientificity of subsequent application resource planning is further improved through continuous self-learning;
compared with the Chinese patent application CN108829494A, the invention discloses a method for optimizing the intelligent resources of a container cloud platform based on load prediction, which provides a limited range section for the use of container resources in operation through the setting of a request value and a limit value of the container resources, can ensure that extra resources are not consumed along with fluctuation to be frequently adjusted within a certain time when the actually used resources of the container fluctuate, and can ensure that the service is still available even in the face of sudden scenes with over-high or over-low flow, and other services and even clusters are not available due to the fact that excessive host resources are not occupied;
compared with the Chinese patent application CN109656717A, the invention discloses a containerization cloud resource allocation method, which pre-arranges resource allocation optimization to the primary deployment of container application, namely, resource usage prediction and allocation are carried out before the container application is on line, thereby greatly reducing the use cost of a container platform, saving the workload of a user, continuously optimizing during the operation, and continuously accumulating resource models of different images through a built-in resource model library so as to save the resource consumed by resource model prediction and optimization repeated calculation of the same image starting container;
compared with the Chinese patent application CN111327655A, the invention discloses a method, a device and an electronic device for predicting resource quota of a multi-tenant container, which leads resource allocation optimization to the initial deployment of container application, namely, the resource usage prediction and allocation are carried out through a static mirror image of the container before the container application is on line, thereby greatly reducing the use cost of a container platform, saving the workload of a user, simultaneously continuously optimizing during the operation, and continuously accumulating resource models of different mirror images through a built-in resource model library so as to save the resource consumed by resource model prediction and optimization repeated calculation of the same mirror image starting container.
Drawings
FIG. 1 is a schematic structural diagram of a system for planning container resources based on application mirroring data identification according to the present invention;
FIG. 2 is a schematic structural diagram of a cluster of application information probes in a container resource planning system based on application mirror image data identification according to the present invention;
FIG. 3 is a schematic structural diagram of another cluster of application information probes in the system for planning container resources based on application mirror image data identification according to the present invention;
fig. 4 is a schematic structural diagram of a container resource planning system based on application mirroring data identification according to an embodiment of the present invention.
Detailed Description
In order to make the technical means, the inventive features, the objectives and the effects of the invention easily understood and appreciated, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific drawings, and it is obvious that the described embodiments are a part of the embodiments of the present invention, but not all of the embodiments.
All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be understood that the structures, ratios, sizes, and the like shown in the drawings and described in the specification are only used for matching with the disclosure of the specification, so as to be understood and read by those skilled in the art, and are not used to limit the conditions under which the present invention can be implemented, so that the present invention has no technical significance, and any structural modification, ratio relationship change, or size adjustment should still fall within the scope of the present invention without affecting the efficacy and the achievable purpose of the present invention.
In addition, the terms "upper", "lower", "left", "right", "middle" and "one" used in the present specification are for clarity of description, and are not intended to limit the scope of the present invention, and the relative relationship between the terms and the terms is not to be construed as a scope of the present invention.
The invention provides a system and a method for planning container resources based on application mirror image data identification, aiming at taking the integral container mirror image data as the basis for identifying the resource consumption type of an application container, comprehensively analyzing the resource consumption type of the application container through abundant application information probes, applying a machine learning technology to the analysis and identification of the resource consumption type of the application container, and greatly improving the accuracy of the application type identification and the accuracy of resource allocation; the method can automatically identify and classify the container types to be deployed on the container platform according to the container mirror image data, and give the recommended values of the corresponding container resources based on strategies, thereby greatly improving the efficiency and accuracy; the whole process adopts an automatic mode, and a large number of application resource consumption models based on best practice are preset, so that the workload and threshold for manually configuring container resources are greatly reduced, the accuracy of the container resource configuration is improved, and the method is very suitable for the current increasingly wide containerized application deployment scene; the application container resource model can be continuously optimized according to the collection and analysis of the real-time running data of the application, so that the scientificity of subsequent application resource planning is further improved through continuous self-learning.
As shown in fig. 1, a system for planning container resources based on application image data identification includes a container management platform, a container resource planning module, and one or more applications, where each application includes two forms of an application image and an application container, the application image and the application container are both provided with one or more application images and application containers, and the container management platform runs on a container running host;
an application information probe is deployed on the container management platform;
a resource model library, a resource model calculation engine and a resource planning controller are deployed on the container resource planning module;
the application information probe is used for carrying out container type detection scanning on the application mirror image to acquire container mirror image data and is also used for acquiring resource consumption data of an application container in the application during running;
the resource model calculation engine is used for acquiring container mirror image data and container runtime resource consumption data; the resource model library is also used for calculating the resource consumption mode of the container mirror image transmitted to the resource model library; the resource model library is also used for generating a resource model transmitted to the resource model library; the system is also used for data verification and algorithm parameter optimization; the application image is also used for detecting whether the application image is identified;
the resource planning controller is used for mapping the resource model into container resource setting data and transmitting the container resource setting data back to the container management platform, recovering and executing the deployment behavior of the application container, and completing the deployment of the application container;
the resource model library is used for storing the container resource setting data, the resource model and the resource consumption mode for the resource planning controller to extract.
As shown in fig. 2-3, the application information probe includes one or more of a probe application language probe, an application middleware probe, an application data source probe, an application algorithm probe, an application access probe, a CPU probe, a memory probe, a network probe, a storage probe, and an I/O probe, the usage requirement of the resource in the actual operation of the application may be different from the resource model calculated according to experience, in order to obtain the optimal resource allocation, the application information probe continuously reads the actually consumed resource value from the running application container and feeds the actually consumed resource value back to the resource model calculation engine for data verification and algorithm parameter optimization, so that on one hand, the algorithm is continuously optimized according to the actual data, on the other hand, the optimized resource allocation value is synchronized into the resource model library in real time and is applied to the running application container according to the strategy as required;
the application language probe is used for detecting the programming language of the application according to the container mirror image data, such as Java, Python, Golang, Ruby and the like;
the application middleware probe is used for detecting Web middleware used by the application and configuration information thereof, such as Tomcat, flash and other Web middleware and configuration thereof according to the container mirror image data;
the application data source probe is used for detecting the data source of the application according to the container mirror image data;
the application algorithm probe is used for detecting an applied algorithm and the complexity thereof according to container mirror image data;
the application access probe is used for detecting whether the application needs external access, internal access and access types and frequency according to the container mirror image data;
the CPU probe is used for detecting the CPU actually consumed by the operation of the application container;
the memory probe is used for detecting the memory actually consumed by the operation of the application container;
the network probe is used for detecting network resources actually consumed by the operation of the application container;
the storage probe is used for detecting the storage resources actually consumed by the operation of the application container;
the I/O probe is used for detecting the I/O actually occupied by the operation of the application container;
the probes referred to above are suitable for use in both single cluster and multi-cluster environments, each cluster maintaining the deployment architecture described above.
The container management platform is internally provided with a mirror image warehouse;
the mirror repository is used for storing application mirrors.
And the data verification is that the actual resource consumption data value is compared with the predicted data when the container runs.
And comparing the actual resource consumption data value with the predicted data through classification analysis and cluster analysis.
The resource model library stores a corresponding relation list of the application mirror image and the resource model.
In a second aspect, a method for planning container resources based on application image data identification includes the following steps:
step 1: the resource planning controller monitors and captures the application deployment behavior of the container management platform in real time;
step 2: the resource planning controller captures an application deployment behavior and then intercepts the application deployment behavior and triggers an application information probe to collect information for the application, for example, the resource planning controller can capture and intercept the application deployment behavior through an Admission Control mechanism (Admission Control), and the technical scheme is not limited to use of the Admission Control mechanism (Admission Control) as an interception technology;
and step 3: the application information probe acquires container mirror image data in an application mirror image in application and transmits the container mirror image data to a resource model calculation engine for calculation and analysis of an application resource consumption model;
and 4, step 4: a resource model calculation engine receives container mirror image data and detects whether application mirror image is identified or not;
if the application mirror image is marked as the identified mirror image, the resource model base stores the resource consumption mode of the application mirror image, and the resource planning controller directly calls the resource consumption mode of the application mirror image from the resource model base;
if the application mirror image is not marked as the identified mirror image, the resource model calculation engine calculates a resource model of the application mirror image through a preset application resource consumption model algorithm, adds the resource model into a resource model library, simultaneously marks the application mirror image as the identified mirror image, updates the corresponding relation between the application mirror image and the resource model by the resource model library, and transmits the resource model to the resource planning controller;
and 5: the resource planning controller maps the received resource model into container resource setting data, updates the application container resource configuration in the application deployment behaviors intercepted in the step 2 according to the container resource setting data, and recovers the application deployment behaviors of the container management platform to complete application deployment;
step 6: after the application container runs, the application information probe continuously reads the actual consumption data value of the resource from the running application container and feeds the actual consumption data value back to the resource model calculation engine for continuous algorithm optimization and resource allocation optimization to generate an optimized resource consumption mode;
and 7: and synchronously updating the optimized resource consumption mode into a resource model library, and simultaneously applying the container resource setting data mapped by the optimized resource consumption mode into the running application container.
Wherein, the application mirror image marked as the identified mirror image is stored in a mirror image warehouse in the container management platform in the step 4;
the mirror repository is used for storing application mirrors.
As shown in fig. 4, the resource consumption behavior of an application container started from the same application image is associated with the resource consumption pattern indicated by the same application image, and the figure only relates to a single cluster, which is the same mechanism in a multi-cluster environment, and each cluster maintains the deployment architecture; the container technology is that the application and the running environment thereof are packaged together through a combined file system with a layer structure, the packaged product is an application mirror image which is a base stone for constructing a container, the application mirror image is a construction part of a containerization application life cycle, is a static representation and data storage format and can be used as a source code of an application container; when the program packaged in the application image is run by the container engine, the application image becomes an application container; the containerized applications are issued through application images, and resource consumption modes of application containers run by the same application image are similar, so that the system can calculate the resource consumption modes of the applications according to information collected in application image data, namely, the application image is taken as a unit to identify and compare application resource models, meanwhile, a resource model library is arranged in the system, the best practices of common application resource models are stored in advance, and a corresponding relation list of the application image and the resource models is maintained.
In a third aspect, a processor for container resource planning based on application image data identification includes at least one processor and a memory coupled to the at least one processor, the memory storing executable instructions that when executed by the at least one processor cause the method described above to be implemented.
In a fourth aspect, a chip, comprises: a processor for calling and running the computer program from the memory so that the device in which the chip is installed performs: the method described above.
Those of skill in the art would understand that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of software and electronic hardware;
whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution;
skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments of the present application, the disclosed system, apparatus and method may be implemented in other ways;
for example, the division of a unit or a module is only one logic function division, and there may be another division manner in actual implementation;
for example, a plurality of units or modules or components may be combined or may be integrated into another system;
in addition, functional units or modules in the embodiments of the present application may be integrated into one processing unit or module, or may exist separately and physically.
It should be understood that, in the various embodiments of the present application, the sequence numbers of the processes do not mean the execution sequence, and the execution sequence of the processes should be determined by the functions and the inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a machine-readable storage medium;
therefore, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a machine-readable storage medium and may include several instructions to cause an electronic device to execute all or part of the processes of the technical solution described in the embodiments of the present application;
the storage medium may include various media that can store program codes, such as ROM, RAM, a removable disk, a hard disk, a magnetic disk, or an optical disk.
In conclusion, the system and the method for planning the container resources based on the application mirror image data identification can comprehensively analyze the resource consumption types of the application containers through abundant application information probes by taking the whole container mirror image data as the basis for identifying the resource consumption types of the application containers, and apply a machine learning technology to the analysis and identification of the resource consumption types of the application containers, thereby greatly improving the accuracy of the application type identification and the accuracy of the resource allocation; the method can automatically identify and classify the container types to be deployed on the container platform according to the container mirror image data, and give the recommended values of the corresponding container resources based on strategies, thereby greatly improving the efficiency and accuracy; the whole process adopts an automatic mode, and a large number of application resource consumption models based on best practice are preset, so that the workload and threshold for manually configuring container resources are greatly reduced, the accuracy of the container resource configuration is improved, and the method is very suitable for the current increasingly wide containerized application deployment scene; the application container resource model can be continuously optimized according to the collection and analysis of the real-time running data of the application, so that the scientificity of subsequent application resource planning is further improved through continuous self-learning.
Specific embodiments of the invention have been described above. It is to be understood that the invention is not limited to the particular embodiments described above, in that devices and structures not described in detail are understood to be implemented in a manner common in the art; various changes or modifications may be made by one skilled in the art within the scope of the claims without departing from the spirit of the invention, and without affecting the spirit of the invention.
Claims (10)
1. A container resource planning system based on application mirror image data identification is characterized by comprising a container management platform, a container resource planning module and one or more applications, wherein each application comprises two forms of an application mirror image and an application container, and one or more application mirror images and one or more application containers are arranged;
an application information probe is deployed on the container management platform;
a resource model library, a resource model calculation engine and a resource planning controller are deployed on the container resource planning module;
the application information probe is used for carrying out container type detection scanning on the application mirror image to acquire container mirror image data and is also used for acquiring resource consumption data of an application container in the application during running;
the resource model calculation engine is used for acquiring the container mirror image data and the container runtime resource consumption data; the resource model library is also used for calculating the resource consumption mode of the container mirror image transmitted to the resource model library; the resource model library is also used for generating a resource model transmitted to the resource model library; the system is also used for data verification and algorithm parameter optimization; further for detecting whether the application image is identified;
the resource planning controller is used for mapping the resource model into the container resource setting data and transmitting the container resource setting data back to the container management platform, recovering and executing the deployment behavior of the application container, and completing the deployment of the application container;
the resource model library is used for storing the container resource setting data, the resource model and the resource consumption mode for the resource planning controller to extract.
2. The system for planning container resources based on application mirroring data identification according to claim 1, wherein the application information probe comprises one or more of a probe application language probe, an application middleware probe, an application data source probe, an application algorithm probe, an application access probe, a CPU probe, a memory probe, a network probe, a storage probe, and an I/O probe;
the application language probe is used for detecting the writing language of the application according to the container mirror image data;
the application middleware probe is used for detecting Web middleware used by the application and configuration information thereof according to the container mirror image data;
the application data source probe is used for detecting the data source of the application according to the container mirror image data;
the application algorithm probe is used for detecting the applied algorithm and the complexity thereof according to the container mirror image data;
the application access probe is used for detecting whether the application needs external access, internal access and access types and frequency according to the container mirror image data;
the CPU probe is used for detecting a CPU actually consumed by the operation of the application container;
the memory probe is used for detecting the memory actually consumed by the operation of the application container;
the network probe is used for detecting network resources actually consumed by the operation of the application container;
the storage probe is used for detecting the storage resources actually consumed by the operation of the application container;
the I/O probe is used for detecting I/O actually occupied by the operation of the application container.
3. The system for planning container resources based on application image data identification as claimed in claim 1 or 2, wherein the container management platform is provided with an image warehouse inside;
the mirror repository is used for storing application mirrors.
4. The system for planning container resources based on identification of application mirroring data as claimed in claim 3 wherein the data validation compares actual resource consumption data values with predicted data for container runtime.
5. The system for planning container resources based on application mirroring data identification as claimed in claim 4 wherein the actual resource consumption data value is compared with the predicted data by classification analysis, clustering analysis.
6. The system for planning container resources based on application image data identification as claimed in claim 1, 2, 4 or 5, wherein the resource model library stores a corresponding relationship list between the application image and the resource model.
7. A method for implementing the system for planning container resources based on application mirroring data identification according to any one of claims 1 to 6, comprising the steps of:
step 1: the resource planning controller monitors and captures the application deployment behavior of the container management platform in real time;
step 2: the resource planning controller intercepts and triggers an application information probe to collect information of the application after capturing application deployment behaviors;
and step 3: the application information probe acquires container mirror image data in an application mirror image in application and transmits the container mirror image data to a resource model calculation engine for calculation and analysis of an application resource consumption model;
and 4, step 4: the resource model calculation engine receives container mirror image data and detects whether an application mirror image is identified or not;
if the application mirror image is marked as the identified mirror image, the resource model base stores the resource consumption mode of the application mirror image, and the resource planning controller directly calls the resource consumption mode of the application mirror image from the resource model base;
if the application mirror image is not marked as the identified mirror image, the resource model calculation engine calculates a resource model of the application mirror image through a preset application resource consumption model algorithm, adds the resource model into a resource model library, simultaneously marks the application mirror image as the identified mirror image, updates the corresponding relation between the application mirror image and the resource model by the resource model library, and transmits the resource model to the resource planning controller;
and 5: the resource planning controller maps the received resource model into container resource setting data, updates the application container resource configuration in the application deployment behaviors intercepted in the step 2 according to the container resource setting data, recovers the application deployment behaviors of the container management platform and completes application deployment;
step 6: after the application container runs, the application information probe continuously reads the actual consumption data value of the resource from the running application container and feeds the actual consumption data value back to the resource model calculation engine for continuous algorithm optimization and resource allocation optimization to generate an optimized resource consumption mode;
and 7: and synchronously updating the optimized resource consumption mode into a resource model library, and simultaneously applying the container resource setting data mapped by the optimized resource consumption mode into the running application container.
8. The method of claim 7, wherein step 4 further comprises storing the application image marked as the identified image in an image warehouse of the container management platform;
the mirror repository is used for storing application mirrors.
9. A processor for container resource planning based on application image data identification, comprising at least one processor and a memory coupled to the at least one processor, the memory storing executable instructions that when executed by the at least one processor cause the method according to any one of claims 7 to 8 to be carried out.
10. A chip, comprising: a processor for calling and running the computer program from the memory so that the device in which the chip is installed performs: the method of any one of claims 7 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011038544.9A CN112084040B (en) | 2020-09-28 | 2020-09-28 | Container resource planning system and method based on application mirror image data identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011038544.9A CN112084040B (en) | 2020-09-28 | 2020-09-28 | Container resource planning system and method based on application mirror image data identification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112084040A CN112084040A (en) | 2020-12-15 |
| CN112084040B true CN112084040B (en) | 2021-07-20 |
Family
ID=73739788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011038544.9A Active CN112084040B (en) | 2020-09-28 | 2020-09-28 | Container resource planning system and method based on application mirror image data identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112084040B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113067818B (en) * | 2021-03-18 | 2022-07-01 | 中电运行(北京)信息技术有限公司 | Probe distribution method and device based on network asset checking |
| CN113467941A (en) * | 2021-06-25 | 2021-10-01 | 北京汇钧科技有限公司 | Method and device for sharing information |
| CN113992525A (en) * | 2021-10-12 | 2022-01-28 | 支付宝(杭州)信息技术有限公司 | Method and device for adjusting number of applied containers |
| CN117194049B (en) * | 2023-10-16 | 2024-02-27 | 上海佑瞻智能科技有限公司 | Cloud host intelligent behavior analysis method and system based on machine learning algorithm |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776005A (en) * | 2016-11-23 | 2017-05-31 | 华中科技大学 | A kind of resource management system and method towards containerization application |
| CN106843945A (en) * | 2017-01-04 | 2017-06-13 | 厦门亿力吉奥信息科技有限公司 | GIS applications dispositions method and system based on PaaS |
| CN108833197A (en) * | 2018-04-10 | 2018-11-16 | 中国科学院信息工程研究所 | A kind of active probe method based on cloud and test platform |
| CN110109686A (en) * | 2019-04-25 | 2019-08-09 | 中电科嘉兴新型智慧城市科技发展有限公司 | It is a kind of that O&M method and system is applied based on Container Management engine |
| CN110333877A (en) * | 2019-07-09 | 2019-10-15 | 西安点告网络科技有限公司 | Visualized container configuring management method, apparatus and system based on application |
| CN110704277A (en) * | 2019-09-27 | 2020-01-17 | 中电万维信息技术有限责任公司 | Method for monitoring application performance, related equipment and storage medium |
| CN110764786A (en) * | 2019-12-27 | 2020-02-07 | 凯泰铭科技（北京）有限公司 | Optimized deployment resource and software delivery platform in cloud computing environment |
| WO2020081134A1 (en) * | 2018-10-16 | 2020-04-23 | Ebay Inc. | User interface resource file optimization |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102404381A (en) * | 2011-09-02 | 2012-04-04 | 西安交通大学 | Software deployment system and deployment method based on workflow in cloud computing environment |
| US9804880B2 (en) * | 2015-06-16 | 2017-10-31 | Vmware, Inc. | Reservation for a multi-machine application |
| US10838750B2 (en) * | 2019-01-10 | 2020-11-17 | Red Hat, Inc. | Combining ahead-of-time compilation and just-in-time compilation to improve application deployment |
-
2020
- 2020-09-28 CN CN202011038544.9A patent/CN112084040B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776005A (en) * | 2016-11-23 | 2017-05-31 | 华中科技大学 | A kind of resource management system and method towards containerization application |
| CN106843945A (en) * | 2017-01-04 | 2017-06-13 | 厦门亿力吉奥信息科技有限公司 | GIS applications dispositions method and system based on PaaS |
| CN108833197A (en) * | 2018-04-10 | 2018-11-16 | 中国科学院信息工程研究所 | A kind of active probe method based on cloud and test platform |
| WO2020081134A1 (en) * | 2018-10-16 | 2020-04-23 | Ebay Inc. | User interface resource file optimization |
| CN110109686A (en) * | 2019-04-25 | 2019-08-09 | 中电科嘉兴新型智慧城市科技发展有限公司 | It is a kind of that O&M method and system is applied based on Container Management engine |
| CN110333877A (en) * | 2019-07-09 | 2019-10-15 | 西安点告网络科技有限公司 | Visualized container configuring management method, apparatus and system based on application |
| CN110704277A (en) * | 2019-09-27 | 2020-01-17 | 中电万维信息技术有限责任公司 | Method for monitoring application performance, related equipment and storage medium |
| CN110764786A (en) * | 2019-12-27 | 2020-02-07 | 凯泰铭科技（北京）有限公司 | Optimized deployment resource and software delivery platform in cloud computing environment |
Non-Patent Citations (2)
| Title |
|---|
| 基于Kubernetes的容器云平台设计与实践;翁湦元等;《铁路计算机应用》;20191231;第28卷(第12期);第49-53页 * |
| 基于容器引擎的云平台设计与实现;朱小亮;《中国优秀硕士学位论文全文数据库信息科技辑》;20200315;第2020年卷(第3期);I139-154 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112084040A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112084040B (en) | Container resource planning system and method based on application mirror image data identification | |
| US8108456B2 (en) | Method and apparatus for migrating the system environment on which the applications depend | |
| CN100451989C (en) | Software testing system and testing method | |
| CN111355606B (en) | Web application-oriented container cluster self-adaptive expansion and contraction system and method | |
| CN112199150A (en) | Online application dynamic capacity expansion and contraction method based on micro-service calling dependency perception | |
| CN111125444A (en) | Big data task scheduling management method, device, equipment and storage medium | |
| CN105122212A (en) | Periodicity optimization in an automated tracing system | |
| CN111181773B (en) | Delay prediction method for multi-component application of heterogeneous border cloud collaborative intelligent system | |
| CN108229799B (en) | Multi-source heterogeneous power grid operation real-time data access system and method | |
| CN106970880A (en) | A kind of distributed automatization method for testing software and system | |
| CN116127899B (en) | Chip design system, method, electronic device, and storage medium | |
| US20200133707A1 (en) | Power efficient workload placement and scheduling in a virtualized computing environment | |
| US20230145025A1 (en) | Modeling cloud inefficiencies using domain-specific templates | |
| CN114967887A (en) | Energy consumption management method, device and equipment based on industrial intelligence and storage medium | |
| CN116594913B (en) | Intelligent software automatic test method | |
| CN111367632B (en) | Container cloud scheduling method based on periodic characteristics | |
| CN112506791A (en) | Application program testing method and device, computer equipment and storage medium | |
| CN114138501B (en) | Processing method and device for edge intelligent service for field safety monitoring | |
| CN110716875A (en) | Concurrency test method based on feedback mechanism in domestic office environment | |
| CN115525392A (en) | Container monitoring method and device, electronic equipment and storage medium | |
| KR102418892B1 (en) | Method of saving energy based on confidence interval and apparatus using the same | |
| CN109660392A (en) | Hardware unitizes adaptive management dispositions method and system under a kind of linux system | |
| CN113703394A (en) | Cutter monitoring and managing method and system based on edge calculation | |
| CN112214278A (en) | Method for identifying and processing private cloud zombie virtual machine | |
| CN114443069B (en) | Method and system for constructing operating system mirror image by dynamically expanding cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 200433 floor 7, building 6, No. 99, jiangwancheng Road, Yangpu District, Shanghai Patentee after: Shanghai Daoke Network Technology Co.,Ltd. Address before: Room 1305-12, No.6 Weide Road, Yangpu District, Shanghai 200433 Patentee before: Shanghai Daoke Network Technology Co.,Ltd. |