CN110928750A

CN110928750A - Data processing method, device and equipment

Info

Publication number: CN110928750A
Application number: CN201811095539.4A
Authority: CN
Inventors: 孙相征; 何万青; 贺荣徽; 李临川; 田永军; 余洋
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2018-09-19
Filing date: 2018-09-19
Publication date: 2020-03-27
Anticipated expiration: 2038-09-19
Also published as: CN110928750B

Abstract

The embodiment of the application provides a data processing method, a device and equipment, wherein the method comprises the following steps: collecting system performance index data of each computing node in a cluster, wherein the computing nodes run a target application program, and application performance index data generated by the target application program when the computing nodes run; determining a system performance of each compute node running the target application based on the system performance indicator data for each compute node; determining the application performance of each computing node corresponding to the target application program based on application performance index data generated by the target application program in operation of each computing node; and comprehensively processing the application performance of the target application program in each computing node and the system performance of each computing node. The embodiment of the application improves the comprehensiveness of node analysis.

Description

Data processing method, device and equipment

Technical Field

The embodiment of the application relates to the technical field of computer application, in particular to a data processing method, device and equipment.

Background

An application may operate on a computing node, which may refer to individual computing devices in a cluster, which may refer to computing systems and environments that use many computing nodes to provide large-scale, high-speed computing or processing services. In order to obtain the usage of the computing nodes, the computing nodes are generally monitored.

In the prior art, a monitoring tool may be installed on a computing node, where the monitoring tool defines each monitoring index to acquire a CPU (Central Processing Unit) occupancy rate, a bandwidth occupancy rate, and/or a memory occupancy of the computing node, so as to obtain a use performance of the computing node.

When the above method is adopted for monitoring, monitoring can only be performed on each computing node in the cluster, and monitoring information consisting of monitoring data of monitoring indexes of a single computing node can only be obtained, so that the obtained monitoring information is single and not accurate enough.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device and data processing equipment, which are used for solving the technical problems that monitoring information obtained in the prior art is single and not accurate enough.

In a first aspect, an embodiment of the present application provides a data processing method, including:

collecting system performance index data of each computing node in a cluster, wherein the computing nodes run a target application program, and application performance index data generated by the target application program when the computing nodes run;

determining a system performance of each compute node running the target application based on the system performance indicator data for each compute node;

determining the application performance of each computing node corresponding to the target application program based on application performance index data generated by the target application program in operation of each computing node;

and comprehensively processing the application performance of the target application program in each computing node and the system performance of each computing node.

In a second aspect, a data processing apparatus is provided, including:

the data acquisition module is used for acquiring system performance index data of each computing node in a cluster, which runs a target application program, and application performance index data generated by the running of the target application program at each computing node at the same acquisition time;

a first determination module for determining system performance of each compute node running the target application based on the system performance indicator data for each compute node;

the second determination module is used for determining the application performance of each computing node corresponding to the target application program based on the application performance index data generated by the target application program running on each computing node;

and the comprehensive processing module is used for comprehensively processing the application performance of the target application program on each computing node and the system performance of each computing node.

In a third aspect, a data processing apparatus is provided, including: a storage component and a processing component; the storage component is used for storing one or more computer instructions, and the one or more computer instructions are called and executed by the processing component;

the processing component is to:

collecting system performance index data of each computing node in a cluster, wherein the computing nodes run a target application program, and application performance index data generated by the target application program when the computing nodes run; determining a system performance of each compute node running the target application based on the system performance indicator data for each compute node; determining the application performance of each computing node corresponding to the target application program based on application performance index data generated by the target application program in operation of each computing node; and comprehensively processing the application performance of the target application program in each computing node and the system performance of each computing node.

In the embodiment of the application, the system performance index of each computing node running the target application program in the cluster and the application performance index data generated by the target application program running at each computing node are collected at the same collection time, that is, the data of the system performance index of the computing node and the data of the application performance index of the target application program are collected at the same collection time, and then the collected data of the system performance index can be analyzed to obtain the system performance index of each computing node, and the collected data of the application performance index is analyzed to obtain the application performance of the target application program corresponding to each computing node. The application performance of the application program and the node performance of the computing node and other performances of two different layer index levels are obtained simultaneously, further comprehensive processing can be performed according to the application performance and the system performance, comprehensive analysis processing such as unified acquisition, analysis and summarization is achieved, comprehensive analysis with more comprehensive contents is completed, and the accuracy and the effectiveness of analysis are improved.

These and other aspects of the present application will be more readily apparent from the following description of the embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 illustrates a flow diagram of one embodiment of a data processing method provided herein;

FIG. 2 illustrates a flow diagram of yet another embodiment of a data processing method provided herein;

FIG. 3 illustrates an example graph of a cluster performance presentation provided herein;

FIG. 4 illustrates an exemplary graph of a composite performance presentation provided herein;

FIG. 5 illustrates an example graph of a node performance presentation provided herein;

FIG. 6 illustrates an example diagram of a process performance presentation provided herein;

FIG. 7 illustrates an example graph of a functional performance presentation provided herein;

FIG. 8 is a block diagram illustrating an embodiment of a data processing apparatus provided herein;

FIG. 9 is a block diagram illustrating an embodiment of a data processing apparatus provided herein;

FIG. 10 is a block diagram illustrating one embodiment of a data processing system provided herein.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

In some of the flows described in the specification and claims of this application and in the above-described figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, the number of operations, e.g., 101, 102, etc., merely being used to distinguish between various operations, and the number itself does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.

The technical scheme of the embodiment of the application can be applied to performance analysis scenes of devices such as clusters and nodes, and more comprehensive and systematic performance analysis can be realized by acquiring data of indexes of different levels.

In the prior art, a single node may be analyzed, and a manner of configuring a collection tool in a computing node may be generally adopted, so as to set a monitoring index by inputting, selecting, and the like on the collection tool, and display data collected by the monitoring index to a user. The monitoring of the performance index of a single node usually has the problem of monitoring one-sidedness. Generally, the acquisition tool can only acquire index data for certain set specific monitoring indexes in the running process of the application program, the index data are usually fixedly set, only performance indexes of certain specific characteristics in the running process of the application program can be adopted, only relatively fixed performance of the application program in running can be reflected, and the monitoring indexes are not necessarily key factors influencing the running performance of the program. In addition, the problem of poor correlation of monitoring indexes also exists in monitoring of performance indexes of a single node, and after index data of some monitoring indexes are detected, the index data of the indexes are directly output in a data form, so that the change process of the index data cannot be obtained, and comprehensive display of the indexes is not facilitated. In addition, there is a problem of lack of detailed analysis for monitoring performance indexes of a single node, and generally, outputting index data is only to output data in the form of a monitoring result, and does not perform detailed analysis on an entire node, an entire program, or a key function, and a user can only confirm a performance index with a problem through experience, and lacks of index correlation analysis.

The inventor thinks whether the index sampling of different levels can be carried out aiming at each computing node in the cluster so as to obtain the index data of different levels, and then the correlation analysis is carried out so as to obtain the analysis of multi-level cascade, thereby improving the comprehensiveness and pertinence of the analysis and helping the user to carry out the abnormal positioning quickly.

In the embodiment of the application, system performance index data of each computing node running the target application program in the cluster and application performance index data generated by the running of the target application program at each computing node can be collected at the same collection time. The data acquisition aiming at the performance indexes of different levels is realized, and the acquired performance indexes are more comprehensive. Determining system performance of each compute node running the target application based on the system performance indicator data for each compute node; and determining the application performance of each computing node corresponding to the target application program based on the application performance index data generated by the target application program running on each computing node. The performance of two levels of system performance and application performance can be determined by collecting performance indexes of different levels, the actual use performance of the node can be obtained for the system performance, and the monitoring result of the application level can be analyzed in detail for the application performance to obtain a detailed analysis result. And then, when the application performance of each computing node and the system performance of each computing node of the target application program are comprehensively processed, the association analysis of each performance can be realized, the detailed analysis of different levels such as a node level and an application level is realized, and the accuracy and the effectiveness of the analysis are improved.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Fig. 1 is a flowchart of an embodiment of a data processing method according to an embodiment of the present invention, where the method may include the following steps:

101: and at the same acquisition time, acquiring system performance index data of each computing node in the cluster, which runs the target application program, and application performance index data generated by the target application program running at each computing node.

The data processing method can be applied to servers in a cluster or intelligent equipment with computing processing capacity, such as any computing node. Each computing node can be provided with a collection tool, and the collection tool can collect the system performance index data of each computing node and the application performance index data of a target application program in the operating production line of each computing node. The collection tool may store the collected system performance index data and application performance index data into a performance feature library for subsequent use and review.

Optionally, after the collection tool collects the system performance index data and the application performance index data, the system performance index data and the application performance index data may be uploaded to a performance feature library and stored in the feature library. The intelligent device can read the system performance index data and the application performance index data in the performance characteristic library at any time through the access control interface of the performance characteristic library.

In order to obtain the continuous change condition of the index data of each system performance, the system performance index data of each computing node running the target application program in the cluster and the application performance index data generated by the target application program running at each computing node can be collected for multiple times at the same collection time to obtain the continuously changed system performance index data and application performance index data, and continuous analysis is performed to monitor the continuous change condition of each performance index. The same acquisition time may mean that the acquisition times of the system performance index data and the application performance index data are the same.

In order to obtain the real-time data of each system performance index, the system performance index data of each computing node of a target application program running in a real-time cluster and the application performance index data generated by the target application program running in each computing node can be acquired at the same acquisition time, so that the real-time changed system performance index data and the application performance index data are obtained, and real-time analysis is performed to monitor the real-time change condition of each performance index.

In order to clarify the system performance index data of the computing node corresponding to each target application program, a node identifier may be set for the computing node, and the system performance index data of the computing node and the application performance index data corresponding to the target application program run by the node are obtained through the node identifier of the node and identified.

Optionally, the target application program of each computing node may include multiple target application programs, that is, the target application program may simultaneously obtain application performance index data corresponding to different application programs running in each node, and perform application performance analysis on the application program according to the application performance index data of each application program.

102: determining system performance for each compute node running the target application based on the system performance indicator data for each compute node.

For each computing node's system performance indicator data, the system performance of each computing node may be determined. The system performance of each node may include analyzing the change rule of the key performance index of the node during the running of the target application program from the perspective of the node. The change rule of the key performance index of the node during the running period of the target application program can be obtained by analyzing the system performance index data of the computing node.

103: and determining the application performance of each computing node corresponding to the target application program based on the application performance index data generated by the target application program running on each computing node.

The application performance of the target application program corresponding to each computing node can be determined according to the application performance index data generated by the target application program running on each computing node. The application performance of the target application program corresponding to each computing node may refer to the actual operation performance of the target application program when each computing node operates.

104: and comprehensively processing the application performance of the target application program in each computing node and the system performance of each computing node.

The step of comprehensively processing the application performance of the target application program at each computing node and the system performance of each computing node may specifically refer to establishing an association relationship between the application performance of the target application program at each computing node and the system performance of each computing node, and may specifically establish a corresponding relationship between the application performance and the system performance and other indexes.

Optionally, the application performance of the target application program at each computing node and the system performance of each computing node may be comprehensively processed, and specifically, the application performance of the target application program at each computing node and the system performance of each computing node may also be comprehensively shown. In practical application, the application performance of the target application program in each calculation and the system performance of each calculation node can be comprehensively shown in the form of a chart and the like.

The method and the device aim at collecting the system performance index data and the application performance index data at the same collecting time, and can avoid the condition that the running logics of application programs are different and the obstructed indexes cannot be associated due to the non-uniform collecting time, so that the collected system performance index data and the collected application performance index data can be subjected to associated analysis, and no analysis error in time is generated.

In the embodiment of the invention, the data acquisition aiming at the performance indexes of different levels is realized, and the acquired performance indexes are more comprehensive. The system performance of each computing node can be further determined based on the system performance index data; and determining the application performance of each computing node corresponding to the target application program based on the application performance index data. The system performance and the application performance can be determined by collecting the performance indexes of different levels, the actual use performance of the nodes can be obtained for the system performance, and the monitoring result of the application level can be analyzed in detail for the application performance to obtain a detailed analysis result. And then, when the application performance of each computing node and the system performance of each computing node of the target application program are comprehensively processed, the association analysis of each performance can be realized, the detailed analysis of different levels such as a node level and an application level is realized, and the accuracy and the effectiveness of the analysis are improved.

As shown in fig. 2, a schematic structural diagram of another embodiment of a data processing method according to an embodiment of the present invention is provided, where the method includes the following steps:

201: and collecting system performance index data of each computing node in the cluster, which runs the target application program, and application performance index data generated by the target application program running at each computing node at the same collecting time.

202: determining system performance for each computing node running the target application based on the system performance indicator data for each computing node.

203: and determining the application performance of each computing node corresponding to the target application program based on the application performance index data generated by the target application program running on each computing node.

204: and summarizing the system performance of each computing node to obtain the system performance of the cluster.

The system performance of the cluster is obtained by summarizing the system performance of each computing node, so that the overall use condition of the system resources of the whole cluster is obtained from the perspective of the cluster. The system performance of the cluster may include system saturation of the cluster, so as to ensure that abnormal use conditions occurring when the system resource is saturated in use are monitored in real time.

As a possible implementation manner, the system performance of the cluster mainly may include a system memory usage rate, a system processor occupancy rate, a network bandwidth condition, and/or a system storage usage rate, and by monitoring the system performance of the system cluster, a situation that a use exception occurs, such as a write operation is executed again to a fully written memory, can be avoided.

In some embodiments, system analysis may be performed on a plurality of system clusters to obtain use conditions of different system clusters, and then scheduling between the clusters is performed to improve service effects of the clusters. As shown in fig. 3, the system performance of two clusters, namely a cluster 301 and a cluster 302, is shown at the same time, where the system performance of each cluster is mainly composed of bar charts corresponding to performance parameter data such as CPU utilization, memory utilization, and network total bandwidth.

205: and comprehensively processing the application performance of each computing node corresponding to the target application program, the system performance of each computing node and the system performance of the cluster.

In the embodiment of the invention, the system performance and the application performance are obtained by analysis after the index data of two levels of the system performance index data and the application performance index data are collected. The system performance of the whole cluster is obtained by summarizing the system performance of each computing node, so that the system performance of the computing nodes, the application performance of the application program on the corresponding computing node and the system performance of the cluster can be summarized, hierarchical progressive analysis can be realized, comprehensive system analysis of the whole cluster can be obtained, effectiveness and accuracy are improved, and more effective guidance can be provided for a user.

When different performances are comprehensively processed, the performance association can be carried out according to the association relation among different clusters, nodes and application programs, so that the index change among different performance indexes is associated, and the problem of poor association of the performance indexes is solved by the interaction and change rule of the different performance indexes in the operation process.

Therefore, as an embodiment, the comprehensively processing the application performance of the target application program at each computing node, the system performance of each computing node, and the system performance of the cluster may include:

and establishing an association relation among the system performance of the cluster, the system performance of each computing node and the application performance of the target application program in each computing node according to the association relation among the cluster, each computing node and the target application program.

In some embodiments, after establishing the association relationship among the system performance of the cluster, the system performance of each computing node, and the application performance of each computing node of the target application according to the association relationship among the cluster, each computing node, and the target application, the method further includes:

and displaying the association relation among the system performance of the cluster, the system performance of each computing node and the application performance of the target application program in each computing node.

The association relationship between the system performance of the cluster, the system performance of each computing node and the application performance of the target application program on each computing node can be shown in the form of different types of graphs such as a graph, a flame diagram, a histogram and the like.

The association relationship among the cluster, each computing node and the target application program may specifically mean that each computing node may form in the cluster, and the target application program runs in each computing node, and any computing node may be queried through the cluster; any target application may be queried by any computing node.

As a possible display manner, each performance may be displayed in a corresponding interface, for example, the system performance of the cluster may be displayed in a first interface, the system performance of each computing node may be displayed in a second interface, and the process performance and the function performance of the application performance of the target application program in each computing node may be analyzed and displayed in a corresponding third interface and a corresponding fourth interface, respectively. In practical application, the first display interface, the second display interface, and the third display interface may all be integrated into the same main display interface, and a plurality of display frames may be displayed in the main display interface, and each display frame may form a display interface, as shown in fig. 4, for an exemplary diagram of integrating a plurality of display interfaces into one main display interface provided in the embodiment of the present application, the main display interface may include: a first interface 401, a second interface 402, a third interface 403, a fourth interface 404, etc. Only any selected interface can be displayed in the main display interface, and the selected interface in fig. 3 is the fourth interface 404 corresponding to the function performance analysis.

Optionally, the system performance of each computing node in a cluster may be exposed. During displaying, the change curve of the system performance can be drawn by the system performance acquired for multiple times, and the continuous change of the system performance can be displayed. The system performance of each node may specifically include: CPU utilization, i.e. CPU occupancy; the memory usage rate, i.e., the memory occupancy rate; the total bandwidth of the network, i.e. the network bandwidth. As shown in fig. 5, a system performance graph 501-504 of 4 compute nodes is presented.

When the system performance of the cluster is displayed, all performance parameter data of the cluster can be displayed in a form of a graph; when the system performance of the computing node is displayed, the system performance index data of the computing node can be displayed in a chart form after being processed uniformly; the application performance of each computing node displaying the target application program can display the target application program in a graph form after the application performance index data actually run by the computing node is uniformly processed.

To obtain more detailed application performance indicator data, as yet another embodiment, the application performance indicator data may include: process level performance index data and function level performance index data; the application performance comprises the following steps: process performance and function performance;

the determining the application performance of the target application program corresponding to each computing node based on the application performance index data generated by the target application program running on each computing node comprises:

determining the process performance of each computing node corresponding to the target application program based on the process-level performance index data generated by the target application program in each computing node;

and determining the function performance of each computing node corresponding to the target application program based on the function level performance index data generated by the target application program running on each computing node.

As a possible design, the process-level performance indicators may actually consist of multiple performance indicators to collect more comprehensive process-level performance indicator data, including process identification, process communication information, and/or process application information.

The determining, based on process-level performance indicator data generated by the target application running on each computing node, the process performance of each computing node corresponding to the target application includes:

and determining the process performance of the process corresponding to the process identification at each computing node based on the process communication information and/or the process application information generated by the target application program running at each computing node.

From the process perspective, process communication information may be obtained, which may include message or data transmission information between an application running in a process and an application running in another process. The executed application information may be obtained, and may include utilization of CPU, memory, and/or network bandwidth by an application running in the process.

Each target application program may include at least one process, and may obtain performance of each process performed separately.

The process performance of the corresponding process at each compute node may be exposed for the process identification. When exposing, the process performance exposed by each process may include CPU occupancy, memory occupancy, process ID, and process name. As shown in fig. 6, a plurality of computer processes included in one computing node are shown, each process shows information such as CPU occupancy, memory occupancy, process ID, and process name of the process, that is, processes 601 to 605, each process corresponds to one performance profile, and processes 601 to 605 correspond to 606 to 610 in the profiles respectively, and the operating condition of each process can be visually observed through the intuitive performance profile.

As another possible design, the function-level performance index may actually consist of multiple performance indexes to acquire more comprehensive function-level performance index data, where the function-level performance index data includes a function name, function call information, and/or function running time;

the determining the function performance of the target application program corresponding to each computing node comprises:

and determining the function performance of the function corresponding to the function name at each computing node based on function call information and/or function running duration generated by the target application program running at each computing node.

From the function perspective, the call information of the function can be acquired, and the call information of the function can be directly displayed to a user as a function performance, so that the user can conveniently acquire the call information of each function, for example, in practical application, the call stack information of each function can be displayed. The running time of the function can be obtained, the time consumption distribution of the function can be analyzed by utilizing the running time of the function, so that the function with longer time consumption can be obtained, a user can conveniently check the function, and the function with longer time consumption is optimized.

In practical applications, the function performance of the function may be shown in a graph form, for example, the fourth interface 404 in fig. 4 is a display of the function running performance of one function.

Alternatively, the functional performance may be shown by two graphs, one for TOP-DOWN and one for BOTTOM-UP, each in the form of a flame diagram. For TOP-DOWN, starting from the first layer, the upper layer of each layer is the calling function corresponding to the function of each layer and the running time of the main body function of the function, and the horizontal column length represents the running time of each function, and the running time of each function includes the running time of the sub-function of the function, so as to facilitate the user to determine the calling relationship of the processing logic and the function and the total time consumption of each function. For the BOTTOM-UP, starting from the first layer, the last layer of each layer is the call function for each layer of function and the running duration of the main function of the function, that is, the horizontal column length represents the running duration of each function, but the running duration does not include the running duration of the sub-function of the function, and the running duration is independent for each function, so that a user can conveniently obtain the actual running duration of each function. As shown in fig. 4, which is a function performance diagram shown in a form of BOTTOM-UP, wherein the flame diagram 405 is a performance analysis result of the function stage (not including the rated operation time of the sub-function), the call information of each function and the overall operation time can be obtained. As shown in fig. 7, which is a functional performance diagram shown in a TOP-DOWN form, wherein a flame diagram 701 is a performance analysis result of a function stage (including the operation time length of a sub-function), the call information of each function and the independent operation time length can be obtained.

The various performances are displayed in the forms of charts and the like, the visualization processing of the various performances is realized, and the correlation and the change rule of different indexes in the same chart are intuitively reflected.

In order to perform a targeted analysis for different function-level performance indexes to obtain diverse function performances and obtain more detailed analysis results, in some embodiments, the determining, based on function call information and/or function running time length generated by the target application running at each computing node, the function performance of the function corresponding to the function name at each computing node includes:

determining a hot point calling function based on function calling information generated by the target application program in each computing node;

determining the calling performance of the hotspot calling function in the function calling process of each computing node according to the running duration of the hotspot calling function;

and/or determining a function time-consuming ranking based on the function running time generated by the target application program running on each computing node;

determining a hot spot operation function according to the function time-consuming ranking;

acquiring stack information called by the hotspot running function at each computing node to determine the running performance of the hotspot running function in running.

To obtain metric data for detailed system performance metrics, as yet another embodiment, the system performance metrics include: CPU occupation index, memory utilization index, network bandwidth index and/or CPU micro-architecture index;

the determining system performance of each computing node based on the system performance indicator data for each computing node running the target application comprises:

and comprehensively determining the system performance of each computing node based on the CPU occupancy rate, the memory utilization rate, the network bandwidth and/or the CPU microarchitecture data of each computing node running the target application program.

The CPU microarchitectural data may include: floating point operation efficiency, memory bandwidth, vectorization proportion and the like. CPU occupancy may refer to user, system, idle, iowait (CPU free but outstanding read and write requests), etc.

As shown in fig. 8, a schematic structural diagram of an embodiment of a data processing apparatus provided in the present application is shown, where the apparatus includes:

the data acquisition module 801: the system performance index data acquisition device is used for acquiring system performance index data of each computing node in a cluster, wherein a target application program runs on each computing node, and application performance index data generated by the target application program running on each computing node;

a first determining module 802, configured to determine a system performance of each computing node running the target application based on the system performance indicator data of each computing node;

a second determining module 803, configured to determine, based on application performance indicator data generated by the target application program running on each computing node, application performance of each computing node corresponding to the target application program;

and the comprehensive processing module 804 is used for comprehensively processing the application performance of the target application program at each computing node and the system performance of each computing node.

In the embodiment of the invention, aiming at the data acquisition of the performance indexes of different levels, the acquired performance indexes are more comprehensive. The system performance of each computing node can be further determined based on the system performance index data; and determining the application performance of each computing node corresponding to the target application program based on the application performance index data. The system performance and the application performance can be determined by collecting the performance indexes of different levels, the actual use performance of the nodes can be obtained for the system performance, and the monitoring result of the application level can be analyzed in detail for the application performance to obtain a detailed analysis result. And then, when the application performance of each computing node and the system performance of each computing node of the target application program are comprehensively processed, the association analysis of each performance can be realized, the detailed analysis of different levels such as a node level and an application level is realized, and the accuracy and the effectiveness of the analysis are improved.

As an embodiment, the apparatus further comprises:

the performance summarizing module is used for summarizing the system performance of each computing node to obtain the system performance of the cluster;

the integrated processing module comprises:

and the comprehensive processing unit is used for comprehensively processing the application performance of the target application program on each computing node, the system performance of each computing node and the system performance of the cluster.

As a possible implementation manner, the system performance of the cluster mainly may include a system memory usage rate, a system processor occupancy rate, a network bandwidth condition, and/or a system storage usage rate, and by monitoring the system performance of the system cluster, a situation that a use exception occurs, such as a write operation is executed again to a fully written memory, can be avoided. In some embodiments, system analysis may be performed on a plurality of system clusters to obtain use conditions of different system clusters, and then scheduling between the clusters is performed to improve service effects of the clusters.

In the embodiment of the invention, the system performance of the whole cluster is obtained by summarizing the system performance of each computing node, so that the system performance of the computing nodes, the application performance of an application program on the corresponding computing node and the system performance of the cluster can be summarized, hierarchical progressive analysis can be realized, comprehensive system analysis of the whole cluster can be obtained, the effectiveness and the accuracy are improved, and more effective guidance can be provided for a user.

Thus, as an embodiment, the integrated processing unit comprises:

and the association subunit is used for establishing an association relationship among the system performance of the cluster, the system performance of each computing node and the application performance of the target application program in each computing node according to the association relationship among the cluster, each computing node and the target application program.

The integrated processing unit further includes:

and the display subunit is used for displaying the association relationship among the system performance of the cluster, the system performance of each computing node and the application performance of the target application program in each computing node.

To obtain more detailed application performance indicator data, as yet another embodiment, the application performance indicator data comprises: process level performance index data and function level performance index data; the application performance comprises the following steps: process performance and function performance;

the second determining module includes:

the first determining unit is used for determining the process performance of each computing node corresponding to the target application program based on the process-level performance index data generated by the target application program running on each computing node;

and the second determining unit is used for determining the function performance of each computing node corresponding to the target application program based on the function level performance index data generated by the target application program running on each computing node.

The process level performance index data comprises process identification, process communication information and/or process application information;

the first determination unit includes:

and the first determining subunit is used for determining the process performance of the process corresponding to the process identifier at each computing node based on the process communication information and/or the process application information generated by the target application program running at each computing node.

The process performance of the corresponding process at each compute node may be exposed for the process identification.

To obtain more detailed application performance indicator metric data, as yet another embodiment, the function level performance indicator data includes a function name, function call information, and/or a function run time length;

the second determining unit includes:

and the second determining subunit is used for determining the function performance of the function corresponding to the function name at each computing node based on the function call information and/or the function running time length generated by the target application program running at each computing node.

From the function perspective, the call information of the function can be acquired, and the call information of the function can be directly displayed to a user as a function performance, so that the user can conveniently acquire the call information of each function, for example, in practical application, the call stack information of each function can be displayed.

Alternatively, the functional performance may be shown by two graphs, one for TOP-DOWN and one for BOTTOM-UP, each in the form of a flame diagram.

In order to perform a targeted analysis on different function-level performance indicators to obtain diverse function performances and obtain more detailed analysis results, in some embodiments, the second determining subunit is specifically configured to:

the first determining module includes:

and the third determining unit is used for comprehensively determining the system performance of each computing node based on the CPU occupancy rate, the memory utilization rate, the network bandwidth and/or the CPU micro-architecture data of each computing node running the target application program.

The data processing apparatus described above may execute the steps in the data processing steps described in the above embodiments, and the execution content and technical effects are not described again, and the specific manner of the operations executed by each step in the data processing method in the above embodiments has been described in detail in the embodiments related to the apparatus, and will not be described in detail here.

In one possible design, the embodiment shown in fig. 8 may be implemented as a data processing apparatus, which may include, as shown in fig. 9: a storage component 901 and a processing component 902; the storage component 901 is configured to store one or more computer instructions, which are configured to be invoked and executed by the processing component 902;

the processing component 902 is configured to:

The processing component is further to:

summarizing the system performance of each computing node to obtain the system performance of the cluster;

the application performance of the target application program on each computing node and the system performance of each computing node are comprehensively processed specifically as follows:

and comprehensively processing the application performance of the target application program on each computing node, the system performance of each computing node and the system performance of the cluster.

Therefore, as an embodiment, the step of comprehensively processing the application performance of the target application program at each computing node, the system performance of each computing node, and the system performance of the cluster by the processing component specifically includes:

After the processing component establishes an association relationship among the system performance of the cluster, the system performance of each computing node, and the application performance of each computing node of the target application program according to the association relationship among the cluster, each computing node, and the target application program, the processing component further includes:

To obtain more detailed application performance indicator data, as yet another embodiment, the application performance indicator data comprises: process level performance index data and function level performance index data; the application performance comprises the following steps: process performance and function performance; the processing component determines, based on application performance index data generated by the target application program running on each computing node, that the application performance of each computing node corresponding to the target application program is specifically:

the processing component determines, based on process-level performance index data generated by the target application program running on each computing node, that the process performance of each computing node corresponding to the target application program is specifically:

the processing component target application program runs and generates function level performance index data on each computing node, and the step of determining the function performance of each computing node corresponding to the target application program comprises the following steps:

In order to perform a targeted analysis on different function-level performance indexes to obtain diverse function performances and obtain more detailed analysis results, in some embodiments, the processing component determines, based on function call information and/or function running time length generated by running the target application program on each computing node, the function performance of the function corresponding to the function name at each computing node specifically is:

the processing component determines, based on the system performance indicator data of each computing node running the target application, that the system performance of each computing node is specifically:

The data processing device described above may execute the steps in the data processing steps described in the above embodiments, and the execution content and technical effects are not described again, and the specific manner of the operations executed by each step in the data processing method in the above embodiments has been described in detail in the embodiments related to the device, and will not be described in detail here.

Fig. 10 is a schematic structural diagram of an embodiment of a data processing system according to the present application, where the system may include:

at least one computing node 1001, a performance characteristic library storage device 1002, and a server 1003.

At least one computing node 1001 collects system performance index data of each computing node running a target application program in a cluster and application performance index data generated by the running of the target application program on each computing node at the same collection time, and uploads the system performance index data and the application performance index data to the performance feature library storage device 1002;

then, the performance characteristic library storage device 1002 stores the corresponding system performance index data and the application performance index data, respectively, according to the node identifier of the at least one computing node 1001.

Thereafter, the server 1003 may acquire, at any time, system performance index data of each computing node in the cluster running the target application program and application performance index data generated by the target application program running on each computing node at the same collection time. At this time, the server may determine the system performance of each computing node based on the system performance index data of each computing node running the target application; determining the application performance of each computing node corresponding to the target application program based on application performance index data generated by the target application program in operation of each computing node; and comprehensively processing the application performance of the target application program in each computing node and the system performance of each computing node.

The data processing system can acquire the system performance index of each computing node in a cluster running a target application program and the application performance index data generated by the target application program running at each computing node at the same acquisition time, namely simultaneously acquire the data of the system performance index of the computing node and the data of the application performance index of the target application program at the same acquisition time, and then analyze the acquired system performance index data to acquire the system performance index of each computing node and analyze the acquired application performance index data to acquire the application performance of each computing node corresponding to the target application program. The application performance of the application program and the node performance of the computing node and other performances of two different layer index levels are obtained simultaneously, further comprehensive processing can be performed according to the application performance and the system performance, comprehensive analysis processing such as unified acquisition, analysis and summarization is achieved, comprehensive analysis with more comprehensive contents is completed, and the accuracy and the effectiveness of analysis are improved.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A data processing method, comprising:

2. The method of claim 1, further comprising:

the comprehensive processing of the application performance of the target application program at each computing node and the system performance of each computing node comprises:

3. The method of claim 2, wherein the synthetically processing the application performance of the target application at each computing node, the system performance of each computing node, and the system performance of the cluster comprises:

4. The method of claim 3, wherein after establishing the association relationship among the system performance of the cluster, the system performance of each computing node, and the application performance of each computing node of the target application according to the association relationship among the cluster, each computing node, and the target application, further comprises:

5. The method of claim 1, wherein applying performance indicator data comprises: process level performance index data and function level performance index data; the application performance comprises the following steps: process performance and function performance;

6. The method of claim 1, wherein the system performance indicators comprise: CPU occupation index, memory utilization index, network bandwidth index and/or CPU micro-architecture index;

7. The method of claim 5, wherein the process level performance indicator data comprises process identification, process communication information, and/or process application information;

8. The method of claim 5, wherein the function-level performance indicator data comprises a function name, function call information, and/or a function run-time;

9. The method according to claim 8, wherein the determining the function performance of the function corresponding to the function name at each computing node based on the function call information and/or the function running time length generated by the target application program running at each computing node comprises:

10. A data processing apparatus, comprising:

11. A data processing apparatus, characterized by comprising: a storage component and a processing component; the storage component is used for storing one or more computer instructions, and the one or more computer instructions are called and executed by the processing component;

the processing component is to: