CN117435333A - Component resource optimization method and device - Google Patents
Component resource optimization method and device Download PDFInfo
- Publication number
- CN117435333A CN117435333A CN202310872005.2A CN202310872005A CN117435333A CN 117435333 A CN117435333 A CN 117435333A CN 202310872005 A CN202310872005 A CN 202310872005A CN 117435333 A CN117435333 A CN 117435333A
- Authority
- CN
- China
- Prior art keywords
- component
- field
- resource allocation
- resource
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 70
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000013468 resource allocation Methods 0.000 claims abstract description 104
- 238000003860 storage Methods 0.000 claims description 36
- 238000012545 processing Methods 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 22
- 230000008569 process Effects 0.000 claims description 20
- 238000012544 monitoring process Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 10
- 239000002699 waste material Substances 0.000 abstract description 5
- 230000008859 change Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 8
- 241000475481 Nebula Species 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000010354 integration Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 2
- 230000019771 cognition Effects 0.000 description 2
- 238000004883 computer application Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 229910002804 graphite Inorganic materials 0.000 description 1
- 239000010439 graphite Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000003607 modifier Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a component resource optimization method and device. The method comprises the following steps: acquiring metadata of a component, wherein the metadata at least comprises a public field and a characteristic field; extracting the public field, and obtaining component type information corresponding to the component according to the public field; extracting the characteristic field, and determining corresponding component state information according to the content of the characteristic field; determining a corresponding target resource allocation model according to the component type information and the component state information; and optimizing the running resources of the component by using the target resource allocation model. By adopting the method, the resource allocation of the components can be reasonable, so that the resource waste of the components is reduced.
Description
Technical Field
The present disclosure relates to the field of computer applications, and in particular, to a method and an apparatus for optimizing component resources.
Background
With the development of computer application technology, component application technology has emerged. The component is a reusable, stand-alone, software module with interfaces and interactions, and component application techniques facilitate multiplexing, flexibility and maintainability of the software. In the process of applying the component, the component interacts and communicates with the underlying resources to obtain the required computing power, storage space, network connection and other resources, thereby completing the required functions and tasks.
However, because of the many types of components involved in the big data processing, different types of components have different characteristics, resulting in the same demands for various resources. In addition, different users can know the same component to different degrees, and the users often cannot clearly grasp the resource requirement condition of the component. Therefore, the user is highly likely to waste resources of the components during the process of using the components.
Disclosure of Invention
Based on this, it is necessary to provide a component resource optimizing method and device capable of making component resource allocation reasonable, thereby reducing component resource waste.
In a first aspect, the present application provides a component resource optimization method. The method comprises the following steps:
acquiring metadata of a component, wherein the metadata at least comprises a public field and a characteristic field;
extracting the public field, and obtaining component type information corresponding to the component according to the public field;
extracting the characteristic field, and determining corresponding component state information according to the content of the characteristic field;
determining a corresponding target resource allocation model according to the component type information and the component state information;
And optimizing the running resources of the component by using the target resource allocation model.
In one embodiment, the extracting the feature field and determining the corresponding component status information according to the content of the feature field includes:
selecting the corresponding characteristic field according to the component type information;
periodically monitoring the content of the feature field;
and determining the corresponding component state according to the content of the characteristic field so as to obtain the component state information.
In one embodiment, the determining the corresponding target resource allocation model according to the component type information and the component state information includes:
determining a resource optimization direction of a corresponding component according to the component type information;
acquiring historical data of a plurality of preset candidate resource allocation models based on the resource optimization direction;
and selecting a target resource allocation model according to the component state information and the historical data.
In one embodiment, the component type information is used for indicating a function type of the component, the function type of the component includes a storage type and a calculation type, and the selecting the corresponding feature field according to the component type information includes:
When the function type of the component is a storage type, selecting any one or more of an access frequency field, an access time field and a file number field as a characteristic field;
when the function type of the component is a calculation type, any one or more of a CPU utilization rate field, a memory utilization rate field, a Flink back pressure condition field and a historical traffic track field are selected as characteristic fields.
In one embodiment, after determining the corresponding target resource allocation model according to the component type information and the component state information, the method further includes: and generating and outputting resource allocation suggestions according to the resource allocation model.
In one embodiment, the resource allocation proposal includes any one or more of expanding memory space, increasing concurrent processes, adjusting file processing priority, and processing data with access frequency lower than a preset threshold.
In one embodiment, the optimizing the operating resources of the component using the target resource deployment model further includes:
and counting the resource use condition in the component resource pool corresponding to each component, and displaying the resource use condition.
In a second aspect, the present application further provides a component resource optimization device. The device comprises:
the metadata acquisition module is used for acquiring metadata of the component, wherein the metadata at least comprises a public field and a characteristic field;
the public information extraction module is used for extracting the public field and obtaining component type information corresponding to the component according to the public field;
the feature extraction module is used for extracting the feature field and determining corresponding component state information according to the content of the feature field;
the resource allocation module is used for determining a corresponding target resource allocation model according to the component type information and the component state information;
and the process optimization module is used for optimizing the operation resources of the component by using the target resource allocation model.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring metadata of a component, wherein the metadata at least comprises a public field and a characteristic field;
Extracting the public field, and obtaining component type information corresponding to the component according to the public field;
extracting the characteristic field, and determining corresponding component state information according to the content of the characteristic field;
determining a corresponding target resource allocation model according to the component type information and the component state information;
and optimizing the running resources of the component by using the target resource allocation model.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring metadata of a component, wherein the metadata at least comprises a public field and a characteristic field;
extracting the public field, and obtaining component type information corresponding to the component according to the public field;
extracting the characteristic field, and determining corresponding component state information according to the content of the characteristic field;
determining a corresponding target resource allocation model according to the component type information and the component state information;
and optimizing the running resources of the component by using the target resource allocation model.
According to the component resource optimization method and device, the corresponding public fields and the characteristic fields are extracted through the acquisition of the metadata of each component, the components are classified by utilizing the public fields, the optimization directions of different components can be distinguished, the current state of the corresponding component can be clarified by utilizing the content corresponding to the characteristic fields and the characteristic fields, and the component type information, the component state information and the resource allocation model are correspondingly related, so that the resource allocation model applicable to the current state of the component in the current optimization direction can be clarified. Therefore, in the process, through identifying the public fields and the characteristic fields in the component metadata, the subjective cognition of a user can be replaced, and a proper resource allocation model can be determined more objectively, so that the component operation resources can be allocated and utilized more reasonably, the component operation resources are optimized, and the resource waste of the component is reduced.
Drawings
FIG. 1 is an application environment diagram of a component resource optimization method in one embodiment;
FIG. 2 is a flow diagram of a component resource optimization method in one embodiment;
FIG. 3 is a flow chart of step S206 in one embodiment;
FIG. 4 is a flowchart of step S208 in another embodiment;
FIG. 5 is a block diagram of the components resource optimization device in one embodiment;
FIG. 6 is an internal block diagram of a computer device in one embodiment;
fig. 7 is an internal structural view of a computer device in another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The component resource optimization method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. Common fields, characteristic fields corresponding to different components can be stored in the data storage system, and a resource allocation model for optimizing component resources is used. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc., and the terminal 102 may be used to display optimization suggestions and optimization results of component resources. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
In one embodiment, as shown in fig. 1, a component resource optimization method is provided, where the method is applied to a terminal to illustrate the method, it is understood that the method may also be applied to a server, and may also be applied to a system including the terminal and the server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the steps of:
in step S202, metadata of the component is obtained, where the metadata includes at least a public field and a feature field.
Where a component is a reusable software module or part that has specific functions and interfaces that can interact with other components. Components are independent, relatively independent software units that can be developed, tested, deployed, and maintained individually, with well-defined functions and boundaries, that can be used in diverse contexts, and that can be combined and co-operated with other components. Thus, a component is a reusable building block that can be used multiple times in different applications or systems. By encapsulating the functions in components, repeated writing of code can be avoided, and development efficiency and code quality can be improved.
Illustratively, the above-described assembly includes: a data lake table, which refers to structured data stored in a data lake, which can be queried and analyzed in a manner similar to a relational database; HBase, a distributed array-oriented non-relational database (Not only Structured Query Language, noSQL) database, typically a Hadoop-based distributed file system (Hadoop Distributed File System, HDFS) stores data; kafka, a high throughput distributed message queue system for the collection, storage and processing of real-time data streams; nebula, an open-source distributed graph database, focused on storing and processing large-scale graph structure data; neo4j, another popular graph database for storing and querying graph data and providing graph algorithm and relationship analysis functions; and a resource manager in the Yarn, hadoop ecosystem for allocating and managing computing resources in the cluster. The method can be used for scheduling and executing computing tasks such as Flink, spark and the like; kubernetes (K8 s): an open-source container orchestration platform for automated deployment, extension, and management of containerized applications. The method can be used for deploying and managing computing tasks such as Flink, spark and the like; apache Flink, a streaming data processing framework supporting high throughput, low latency real-time data processing and batch processing tasks; apache Spark is a general big data processing framework, and supports various computing modes such as batch processing, interactive query and stream processing.
Wherein the metadata is descriptive information about the data. It can provide information about the attributes, features and structure of the data to aid in understanding, managing and organizing the data. Metadata is important for tasks such as data management, data integration, data querying, and data analysis. It provides the basis for understanding, organizing, and utilizing data. Metadata plays a critical role in data driven applications, database systems, data warehouses, and big data environments. The metadata may contain information on: data attribute information such as data type, length, format, unit, range, etc.; data sources, such as information of creator of data, acquisition time, location of data source, etc.; data structures, such as data tables, fields, relationships, constraints, and the like; the data semantics are used for providing semantic information of the data, describing meaning, application and business rules of the data and facilitating correct interpretation and use of the data; data rights and security, including information related to data rights and security, specifying who can access the data and access levels, rights and safeguards; the data change history is used for recording the change history of the data and comprises information such as the creation time, the modification time, the modifier and the like of the data. This facilitates data versioning and auditing.
Illustratively, metadata may originate from the Hive Meta Store, which is a component of the Hive data Store that stores table structures, partition information, and other metadata. By collecting and storing metadata, understanding and managing the data may be facilitated.
Step S204, extracting the public field, and obtaining the component type information corresponding to the component according to the public field.
The common field refers to a metadata field shared by a certain type of component, for example, a component name field, an identification information field, and the like, and metadata of different types of components include different common fields, so that the components can be classified by extracting the common fields, and the component type information is obtained.
The component type information is used for indicating the function type of the component, and the function type of the component comprises a storage type, and the component of the storage type is responsible for the persistence and storage of data, such as a relational database component, a NoSQL database component, a file system component and the like; a computation type, components of the computation type are used to perform data processing, computation, and analysis tasks, such as batch processing components, stream processing components, big data processing components, and the like; a communication type, components of which provide communication and messaging capabilities between different systems or modules, such as a message queue component, a message bus component, a remote procedure call component, etc.; network type, components of network type for handling network communications and protocols, such as network protocol library, hypertext transfer protocol/hypertext transfer security protocol (Hypertext Transfer Protocol/Hypertext Transfer Protocol Secure, HTTP/HTTPs) server component, webSocket component, etc.; a security type, components of the security type providing security protection of data and systems, such as encryption libraries, authentication and authorization components, firewall components, and the like; user interface types, components of which are used to build user interfaces and interactions, such as graphical user interface components, front end framework components, mobile application interface components, and the like; an integration type, in which components of the integration type are used for integration and collaboration of different systems, services or modules, such as an integrated development environment component, a data integration component, an interface management component, etc.; the system comprises a log and a monitoring type, wherein a component of the log and monitoring type is used for recording and monitoring the running state and performance of the system, such as a log library, a monitoring tool, a performance analysis component and the like; task scheduling types, components of task scheduling types are used to manage and schedule execution of tasks, such as task scheduler components, job scheduling components, timed task components, and the like.
Illustratively, the common fields of the storage type component may include: the method comprises the steps of information such as a component name, a cluster name, a unique identifier, extension information, creation time, a responsible person, last operation time, the number of data pieces, storage size, validity and update time; common fields of the compute type component may include: component name, cluster name, unique identification, extension information, creation time, responsible person, task status (e.g., running, failed, completed), occupied resources (e.g., CPU, memory), tuning advice, etc. This information is used to describe the attributes, state, and resource occupancy of the computing component.
Illustratively, as can be seen from the description of the components above, the storage types may include: data lake, hbase, nebula, neo j, kafka; the components of the computing type may include: flink, spark.
Step S206, extracting the characteristic field and determining the corresponding component state information according to the content of the characteristic field.
The characteristic field is a field capable of representing the state of a corresponding component, and the content of the characteristic field can reflect the current running state of the component in real time, so that the resource utilization condition of the component can be reflected.
Illustratively, the feature fields of the compute type component may include: CPU utilization, memory utilization, flink backpressure conditions, historical traffic trajectories, and the like. These features are used to monitor the performance and operation of the computing components. The feature fields of the storage type component may include: the method comprises the steps of accessing frequency, accessing time, first file number and the like, wherein the first file refers to a file with the memory value occupied by the file smaller than a threshold value. These features are used to monitor the usage and performance of the storage component.
Step S208, determining a corresponding target resource allocation model according to the component type information and the component state information.
Wherein different types of components correspond to different optimization directions. For example, for a storage type component and a computing type component, the optimization directions of the components are different, a policy for optimizing storage resources should be adopted for the storage type component, and a resource allocation model conforming to the policy is selected so as to perform operations such as expanding storage space; for the components of the computing type, a strategy for optimizing computing resources is adopted, and a resource allocation model conforming to the strategy is selected, so that operations such as adjusting file processing priority, adding concurrent processes and the like are performed. Therefore, a proper resource optimization direction is selected according to the component type information, and a proper resource allocation model is determined according to the component state information, so that an optimization model matched with the current component to be optimized can be obtained.
Step S210, optimizing the running resources of the component by using the target resource allocation model.
The specific operations of optimizing component execution resources may be, for example, any one or more of expanding memory space, increasing concurrent processes, adjusting file processing priority, processing data with access frequency below a preset threshold.
According to the component resource optimization method, the corresponding public fields and the characteristic fields are extracted through the acquisition of the metadata of each component, the components are classified by utilizing the public fields, the optimization directions of different components can be distinguished, the current state of the corresponding component can be definitely determined by utilizing the content corresponding to the characteristic fields and the characteristic fields, and the component type information, the component state information and the resource allocation model are correspondingly determined, so that the resource allocation model applicable to the current state of the component in the current optimization direction can be definitely determined. Therefore, in the process, through identifying the public fields and the characteristic fields in the component metadata, the subjective cognition of a user can be replaced, and a proper resource allocation model can be determined more objectively, so that the component operation resources can be allocated and utilized more reasonably, the component operation resources are optimized, and the resource waste of the component is reduced.
Further, in an embodiment, step S208 may further include: generating and outputting resource allocation suggestions according to the resource allocation model.
The resource allocation suggestion may include any one or more of expanding a memory space, increasing concurrent processes, adjusting file processing priority, and processing data with an access frequency below a preset threshold. When the resource optimization suggestion is output, the resource optimization suggestion can be displayed on the terminal 102, so that a user can decide how to optimize the operation resource of the corresponding component according to the resource optimization suggestion, and can further know and master the component.
Further, in an embodiment, step S210 may further include: and counting the resource use condition in the component resource pool corresponding to each component, and displaying the resource use condition.
In the running process of the components, more resources are required to be supported, each component forms a corresponding resource pool, after the obtained target resource allocation model is used for optimizing the running resources of the components, the total amount of resources in the resource pool can be changed, then a report of the use condition of the resources can be generated, the use condition of the resources is displayed, the effect of resource optimization can be definitely made by a user, and other subsequent operations can be carried out.
In one embodiment, as shown in fig. 3, step S206, extracting the feature field, and determining the corresponding component status information according to the content of the feature field includes:
step S302, selecting the corresponding characteristic field according to the component type information.
Illustratively, when the function type of the component is a storage type, any one or more of an access frequency field, an access time field, and a file number field are selected as the feature fields; when the function type of the component is a calculation type, any one or more of a CPU utilization rate field, a memory utilization rate field, a Flink back pressure condition field and a historical traffic track field are selected as characteristic fields.
In step S304, the content of the feature field is periodically monitored.
The periodically monitored data sources may be, for example, a Flink Metric, spark Pattern, kafka JMX or Nebula DashBoard. Wherein, the Flink Metric is a Metric index system provided by Flink for collecting and reporting runtime indexes of the Flink application. By using the Flink Metric, various metrics of the Flink application, such as task throughput, latency, and status information, etc., may be collected and monitored. Spark graph is a monitoring tool used to collect and store time series data. Here, spark uses Graphite to collect computation metrics such as execution time of tasks, memory usage, etc. Kafka JMX: java Management Extensions (JMX) is a standard for monitoring and managing Java applications. Here, kafka uses JMX to provide data about its runtime metrics, such as the production and consumption rate of messages, the offset of partitions, etc. The Nebula Dashboard is a visual management tool of the Nebula graph database, and is used for monitoring and managing the running condition of the database. Through the Nebula Dashboard, various indexes and state information of the Nebula database can be collected and displayed. These components and schemes provide a source and method of collecting and monitoring data that can monitor and analyze the performance and operation of the system.
Illustratively, the feature fields of the compute type component may include: CPU utilization, memory utilization, flink backpressure conditions, historical traffic trajectories, and the like. These features are used to monitor the performance and operation of the computing components. The feature fields of the storage type component may include: the method comprises the steps of accessing frequency, accessing time, first file number and the like, wherein the first file refers to a file with the memory value occupied by the file smaller than a threshold value. These features are used to monitor the usage and performance of the storage component.
The content of the characteristic field is periodically monitored, the change condition of the content of the characteristic field can be timely and comprehensively obtained, and the change rule is summarized, so that a more proper resource allocation model is determined.
Step S306, corresponding component states are determined according to the content of the characteristic fields, so as to obtain component state information.
Illustratively, in an Apache Flink component, backpressure refers to the condition where the input rate of a task exceeds the processing rate as the data stream passes through the stream processing task, resulting in the backlog of large amounts of unprocessed data in the internal queues of the task, leading to reduced system performance or failure. When the rate of the input data exceeds the processing speed of the task, the task cannot process all the input data in time, so that data accumulation in the internal queue is increased. Such backlog of data may cause problems such as increased memory consumption, increased latency, reduced system performance, etc. The Flink component provides a backpressure mechanism to address this problem. When backlogged data in the internal queue of the task exceeds a certain threshold, the task can send a back pressure signal to an upstream data source to inform the data source to slow down the data generation rate. The data source can adapt to the processing capacity of the task by reducing the generation rate of the data, so that the balance of the whole system is realized. According to the Flink back pressure condition of the calculation type component, whether input data is overloaded or not in the running process of the current component is known, so that a proper resource allocation model is selected, and operations such as more parallel processes are executed, so that the running resources of the component are optimized.
In this embodiment, by periodically monitoring the feature field of the component metadata, the change of the running state of the component can be timely obtained, so as to capture the change of the use condition of the resource.
In one embodiment, as shown in fig. 4, step S208 of determining a corresponding target resource allocation model according to the component type information and the component status information includes:
step S402, determining the resource optimization direction of the corresponding component according to the component type information.
Step S404, based on the resource optimization direction, acquiring historical data of a plurality of preset candidate resource allocation models.
When determining the corresponding resource allocation model, one or more target resource allocation models need to be selected from a plurality of different candidate resource allocation models, so that the current optimization direction requirement is met, and the resource allocation model can be matched with the resource state of the current component. Wherein, the historical data is data about the performance and effect of different resource allocation models in the past. The data may include data on performance, optimization effect, resource utilization, etc. of different resource deployment models under different scenarios. By analyzing the historical data, the performance and the advantages of different resource allocation models in the past application can be known.
Therefore, the historical data of the resource allocation model is recorded and acquired, and the function information of each resource allocation model can be clarified, so that the most suitable target resource allocation model is clarified.
Illustratively, the historical data may include: the resource utilization rate historical data comprises CPU utilization rate, memory utilization rate, network bandwidth utilization rate and the like, and helps to know resource utilization conditions of each model under different loads and scenes; task execution performance historical data, recording historical data of performance indexes such as execution time, delay and throughput of tasks when different resource allocation models are used in the past, and providing information about the influence of different models on task execution performance; the resource allocation history records record the resource allocation conditions when different resource allocation models are applied in the past, including allocation proportion, parallelism setting, task priority and the like of different resources, and help to know the change and effect of the resource allocation strategy under each model; resource adjustment history data, recording past history records of adjustment and optimization of a resource allocation model, including resource allocation before and after adjustment, performance improvement conditions, problems, challenges and the like, and providing effects and experiences on different adjustment strategies and optimization measures; workload history data, which collects past workload data, including data volume, request volume, task type, data traffic, etc., helps to understand the adaptability and performance of different resource allocation models under different workloads.
Step S406, selecting a target resource allocation model according to the component state information and the historical data.
For example, if the current component state information indicates that computing power of a computing type component needs to be optimized, the following types of historical data may be obtained: calculating the parallelism historical data of the task, collecting task execution time and performance data when different parallelism settings are used in the past, and comparing task execution conditions under different parallelism to know the influence of the parallelism on the computing capacity and determine the optimal parallelism setting; the resource utilization rate historical data is used for collecting the resource utilization rate data when different resource allocation models are used in the past, wherein the resource utilization rate data comprises CPU utilization rate, memory utilization rate and the like. The efficiency and the optimization degree of different models on the resource utilization can be known by analyzing the change of the resource utilization rate; task delay historical data records the delay condition of tasks when different resource allocation models are used in the past, and the influence of the models on computing capacity and task response time can be evaluated by comparing task delays under different models; a resource adjustment history record, which records the history record of the adjustment and optimization of the resource allocation model in the past, including the resource allocation before and after adjustment, the performance improvement situation, and the like, which can provide effects and experience about different adjustment strategies and optimization measures; workload history data, collecting past workload data, including data volume, request volume, task type, data traffic, etc., may help to understand the adaptability and performance of different resource allocation models under different workloads.
By analyzing the historical data, information about the influence of the resource allocation model on the computing capacity and the optimizing effect can be obtained, indexes such as task execution time, resource utilization rate, task delay and the like under different models are compared, and the most suitable resource allocation model is selected to optimize the computing capacity of the computing type component, such as increasing parallelism, adjusting resource allocation strategies and the like.
The embodiment of the application also provides another component resource optimization method. The component resource optimization method comprises the following steps: first, metadata of the target component Apache Flink is obtained by Hive Meta Store. First extracting its common fields: component name, cluster name, unique identification, extension information, creation time, responsible person, task status (e.g., running, failed, completed), occupied resources (e.g., CPU, memory), tuning advice, etc. Subsequently, since the component type is a calculation type, the feature field is selected: CPU utilization, memory utilization, flink backpressure conditions, historical traffic trajectories, and the like. And then, continuously and periodically monitoring the contents of the characteristic fields, wherein the periodically monitored data is from a Flink Metric, so that the change condition of the contents of the characteristic fields is timely and comprehensively obtained, and the change rule is summarized, thereby determining a more suitable resource allocation model.
Then, a corresponding component state is determined according to the contents of the feature field to obtain component state information. According to the Flink back pressure condition of the Apache Flink component, whether input data is overloaded or not in the running process of the current component is known, so that a proper resource allocation model is selected, and operations such as more parallel processes are executed, so that the running resources of the component are optimized.
When the component state information indicates that the component is overloaded, determining an optimization direction as an optimized computing resource according to the component of the computing type, and selecting one or more target resource allocation models from a plurality of different candidate resource allocation models based on the optimization direction to realize the addition of parallel processes, thereby meeting the current optimization direction requirement. The task selected by the model is completed by acquiring historical data, wherein the historical data comprises: the method comprises the steps of calculating parallelism historical data of tasks, resource utilization historical data, task delay historical data, resource adjustment historical record and workload historical data. And selecting a target resource allocation model according to the component state information and the historical data.
Finally, an optimization proposal for adding the parallel process by using the selected resource allocation model is generated, and the optimization proposal is output to the terminal 102, so that a user can select to manually complete the optimization or automatically complete the optimization by default, and the running resource of the component is optimized by using the target resource allocation model, thereby completing the task of adding the parallel process. Each component forms a corresponding resource pool, after the obtained target resource allocation model is used for optimizing the running resources of the component, the total amount of resources in the resource pool is changed, a report of the use condition of the resources can be generated immediately, the use condition of the resources is displayed, the effect of resource optimization can be clarified by a user, and other subsequent operations are performed.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a component resource optimization device for realizing the above related component resource optimization method. The implementation scheme of the device for solving the problem is similar to the implementation scheme described in the above method, so the specific limitation in the embodiment of the device for optimizing component resources provided below can be referred to the limitation of the method for optimizing component resources in the above description, and will not be repeated here.
In one embodiment, as shown in fig. 5, there is provided a component resource optimizing apparatus, including: a metadata acquisition module 502, a public information extraction module 504, a feature extraction module 506, a resource allocation module 508, and a process optimization module 510, wherein:
a metadata obtaining module 502, configured to obtain metadata of the component, where the metadata includes at least a public field and a feature field;
a common information extraction module 504, configured to extract a common field, and obtain component type information corresponding to a component according to the common field;
the feature extraction module 506 is configured to extract a feature field, and determine corresponding component status information according to the content of the feature field;
the resource allocation module 508 is configured to determine a corresponding target resource allocation model according to the component type information and the component state information;
the process optimization module 510 is configured to optimize the running resources of the component using the target resource allocation model.
In one embodiment, feature extraction module 506 includes a field extraction unit, a field monitoring unit, and a state acquisition unit, wherein:
the field extraction unit is used for selecting a corresponding characteristic field according to the component type information;
the field monitoring unit is used for periodically monitoring the content of the characteristic field;
And the state acquisition unit is used for determining the corresponding component state according to the content of the characteristic field so as to acquire component state information.
In one embodiment, the resource allocation module 508 includes a direction determination unit, a historical data acquisition unit, and a model matching unit, wherein:
the direction determining unit is used for determining the resource optimization direction of the corresponding component according to the component type information;
the historical data acquisition unit is used for acquiring historical data of a plurality of preset candidate resource allocation models based on the resource optimization direction;
and the model matching unit is used for selecting a target resource allocation model according to the component state information and the historical data.
In one embodiment, the field extraction unit comprises a storage field selection subunit and a computation field selection subunit, wherein:
a storage field selection subunit, configured to select any one or more of an access frequency field, an access time field, and a file number field as a feature field when the function type of the component is a storage type;
and the calculation field selection subunit is used for selecting any one or more of a CPU utilization rate field, a memory utilization rate field, a Flink back pressure condition field and a historical traffic track field as a characteristic field when the function type of the component is the calculation type.
In one embodiment, the component resource optimization apparatus further comprises a suggestion output module for generating and outputting resource allocation suggestions according to the resource allocation model.
In one embodiment, the resource allocation proposal includes any one or more of expanding memory space, increasing concurrent processes, adjusting file processing priority, processing data with access frequency below a preset threshold.
In one embodiment, the component resource optimizing device further includes a display module, configured to count a resource usage situation in a component resource pool corresponding to each component, and display the resource usage situation.
The respective modules in the above-described component resource optimization apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing component metadata, resource allocation model historical data and the like. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of component resource optimization.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a component resource optimization method.
It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:
acquiring metadata of the component, wherein the metadata at least comprises a public field and a characteristic field;
extracting a public field, and obtaining component type information corresponding to the component according to the public field;
extracting a characteristic field, and determining corresponding component state information according to the content of the characteristic field;
determining a corresponding target resource allocation model according to the component type information and the component state information;
and optimizing the running resources of the component by using the target resource allocation model.
In one embodiment, the processor when executing the computer program further performs the steps of:
selecting a corresponding characteristic field according to the component type information;
Periodically monitoring the content of the feature field;
and determining the corresponding component state according to the content of the characteristic field to obtain component state information.
In one embodiment, the processor when executing the computer program further performs the steps of:
determining a resource optimization direction of a corresponding component according to the component type information;
acquiring historical data of a plurality of preset candidate resource allocation models based on a resource optimization direction;
and selecting a target resource allocation model according to the component state information and the historical data.
In one embodiment, the processor when executing the computer program further performs the steps of:
when the function type of the component is a storage type, selecting any one or more of an access frequency field, an access time field and a file number field as a characteristic field;
when the function type of the component is a calculation type, any one or more of a CPU utilization rate field, a memory utilization rate field, a Flink back pressure condition field and a historical traffic track field are selected as characteristic fields.
In one embodiment, the processor when executing the computer program further performs the steps of: generating and outputting resource allocation suggestions according to the resource allocation model.
In one embodiment, the processor when executing the computer program further performs the steps of: and counting the resource use condition in the component resource pool corresponding to each component, and displaying the resource use condition.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method for optimizing component resources, the method comprising:
acquiring metadata of a component, wherein the metadata at least comprises a public field and a characteristic field;
extracting the public field, and obtaining component type information corresponding to the component according to the public field;
extracting the characteristic field, and determining corresponding component state information according to the content of the characteristic field;
Determining a corresponding target resource allocation model according to the component type information and the component state information;
and optimizing the running resources of the component by using the target resource allocation model.
2. The component resource optimization method of claim 1, wherein the extracting the feature field and determining the corresponding component status information according to the content of the feature field comprises:
selecting the corresponding characteristic field according to the component type information;
periodically monitoring the content of the feature field;
and determining the corresponding component state according to the content of the characteristic field so as to obtain the component state information.
3. The component resource optimization method of claim 2, wherein the determining a corresponding target resource allocation model according to the component type information and the component state information comprises:
determining a resource optimization direction of a corresponding component according to the component type information;
acquiring historical data of a plurality of preset candidate resource allocation models based on the resource optimization direction;
and selecting a target resource allocation model according to the component state information and the historical data.
4. The component resource optimization method according to claim 2, wherein the component type information is used to indicate a function type of the component, and the function type of the component includes a storage type and a calculation type; the selecting the corresponding feature field according to the component type information includes:
When the function type of the component is a storage type, selecting any one or more of an access frequency field, an access time field and a file number field as a characteristic field;
when the function type of the component is a calculation type, any one or more of a CPU utilization rate field, a memory utilization rate field, a Flink back pressure condition field and a historical traffic track field are selected as characteristic fields.
5. The component resource optimization method according to claim 1, further comprising, after said determining a corresponding target resource allocation model according to the component type information and the component state information: and generating and outputting resource allocation suggestions according to the resource allocation model.
6. The component resource optimization method of claim 5, wherein the resource allocation proposal includes any one or more of expanding memory space, increasing concurrent processes, adjusting file processing priority, processing data with access frequency below a preset threshold.
7. The component resource optimization method of any one of claims 1-6, wherein the optimizing the operating resources of the component using the target resource allocation model further comprises:
And counting the resource use condition in the component resource pool corresponding to each component, and displaying the resource use condition.
8. A component resource optimization apparatus, the apparatus comprising:
the metadata acquisition module is used for acquiring metadata of the component, wherein the metadata at least comprises a public field and a characteristic field;
the public information extraction module is used for extracting the public field and obtaining component type information corresponding to the component according to the public field;
the feature extraction module is used for extracting the feature field and determining corresponding component state information according to the content of the feature field;
the resource allocation module is used for determining a corresponding target resource allocation model according to the component type information and the component state information;
and the process optimization module is used for optimizing the operation resources of the component by using the target resource allocation model.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310872005.2A CN117435333A (en) | 2023-07-17 | 2023-07-17 | Component resource optimization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310872005.2A CN117435333A (en) | 2023-07-17 | 2023-07-17 | Component resource optimization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117435333A true CN117435333A (en) | 2024-01-23 |
Family
ID=89545136
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310872005.2A Pending CN117435333A (en) | 2023-07-17 | 2023-07-17 | Component resource optimization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117435333A (en) |
-
2023
- 2023-07-17 CN CN202310872005.2A patent/CN117435333A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11243704B2 (en) | Data pipeline architecture for analytics processing stack | |
| US11086687B2 (en) | Managing resource allocation in a stream processing framework | |
| CN107577805B (en) | Business service system for log big data analysis | |
| US20230004434A1 (en) | Automated reconfiguration of real time data stream processing | |
| CN109074377B (en) | Managed function execution for real-time processing of data streams | |
| US9842000B2 (en) | Managing processing of long tail task sequences in a stream processing framework | |
| US9191325B2 (en) | Method and system for processing network traffic flow data | |
| US9990385B2 (en) | Method and system for collecting and analyzing time-series data | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| US10116534B2 (en) | Systems and methods for WebSphere MQ performance metrics analysis | |
| US20150120959A1 (en) | Method and system for monitoring and analysis of network traffic flows | |
| US20170083368A1 (en) | Maintaining throughput of a stream processing framework while increasing processing load | |
| CN113010565B (en) | Server real-time data processing method and system based on server cluster | |
| US20150295970A1 (en) | Method and device for augmenting and releasing capacity of computing resources in real-time stream computing system | |
| CN109885624A (en) | Data processing method, device, computer equipment and storage medium | |
| CN107515784B (en) | Method and equipment for calculating resources in distributed system | |
| CA3168857A1 (en) | Method and apparatus for managing and controlling resource, device and storage medium | |
| CN114090580A (en) | Data processing method, device, equipment, storage medium and product | |
| CN103713935A (en) | Method and device for managing Hadoop cluster resources in online manner | |
| CN112953993A (en) | Resource scheduling method, device, network system and storage medium | |
| JPWO2016084327A1 (en) | Resource prediction apparatus, resource prediction method, resource prediction program, and distributed processing system | |
| CN117435335A (en) | Computing power dispatching method, computing power dispatching device, computer equipment and storage medium | |
| CN117435333A (en) | Component resource optimization method and device | |
| Jehangiri et al. | Distributed predictive performance anomaly detection for virtualised platforms | |
| CN111782688A (en) | Request processing method, device and equipment based on big data analysis and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |