CN116561103A

CN116561103A - Data migration performance monitoring method, device, computer equipment and storage medium

Info

Publication number: CN116561103A
Application number: CN202310542682.8A
Authority: CN
Inventors: 冯子杰; 梁海燕; 易照埔; 蒋子豪
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-05-15
Filing date: 2023-05-15
Publication date: 2023-08-08

Abstract

The application relates to a data migration performance monitoring method, a data migration performance monitoring device, computer equipment and a storage medium. Relates to the technical field of computers, and can be used in the field of financial science and technology or other related fields. The method comprises the following steps: and in the data migration process, acquiring an uploading time stamp and a downloading time stamp of each message data, determining a time delay index and a performance index of the message data according to the uploading time stamp and the downloading time stamp of each message data, if the performance index exceeds a preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold, and outputting an alarm notification according to the updated data migration concurrency number and/or the performance index. The method can be used for carrying out visual monitoring on the performance index of the data migration tool in real time, and can automatically adjust concurrent data when the data migration performance is poor so as to improve the efficiency of data migration.

Description

Data migration performance monitoring method, device, computer equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and apparatus for monitoring data migration performance, a computer device, and a storage medium.

Background

In disaster recovery scenarios, it is often necessary to perform data migration work with a data migration tool, which may be, for example, a cross-cluster synchronization tool (MirrorMaker) for data migration between different Kafka clusters, and thus, it is often necessary to verify the disaster-resistant capability of the data migration tool by performance indicators during data migration.

However, the current data migration tool does not have a visual performance monitoring function, so that a user cannot intuitively know the performance of data migration (such as data migration delay, success rate, migration rate and the like).

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data migration performance monitoring method, apparatus, computer device, and storage medium capable of implementing visual monitoring of data migration performance.

In a first aspect, the present application provides a method for monitoring data migration performance. The method comprises the following steps:

in the data migration process, obtaining an uploading time stamp and a downloading time stamp of each message data;

determining a time delay index and a performance index of the message data according to the uploading time stamp and the downloading time stamp of each message data;

If the performance index exceeds the preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold;

and outputting an alarm notification according to the updated data migration concurrency number and/or the performance index.

In one embodiment, if the performance index exceeds a preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold includes:

if the performance index exceeds the preset performance threshold, determining the newly added concurrency number required by each message data according to the time delay index and the time delay threshold;

and updating the data migration concurrency number according to the newly added concurrency number.

In one embodiment, if the performance index exceeds a preset performance threshold, determining the number of newly added concurrency required for each message data according to the time delay index and the time delay threshold includes:

if the performance index exceeds a preset performance threshold, acquiring the current utilization rate of the local processor, the current utilization rate of the local memory and the current data migration concurrency number;

determining the temporary concurrency number supported by the processor according to the current utilization rate of the local processor and a preset processor utilization rate threshold;

determining the temporary concurrency number of memory support according to the current utilization rate of the local memory and a preset memory utilization rate threshold;

Determining the temporary concurrency number supported by the message according to the data migration concurrency number and the concurrency number threshold;

determining the number of temporary concurrency required by the message according to the time delay index, the time delay threshold value and the time delay change value between the current moment and the historical moment;

if the temporary concurrency number supported by the processor, the temporary concurrency number supported by the memory, the temporary concurrency number supported by the message and the temporary concurrency number required by the message are all greater than zero, determining the temporary concurrency number required by the message as the newly added concurrency number required by each message data.

In one embodiment, determining the delay index and the performance index of the message data according to the uploading timestamp and the downloading timestamp of each message data includes:

determining a time delay index in a preset period according to the uploading time stamp and the downloading time stamp of each message data;

and determining the performance index according to the time delay index, the total number of message data and the successful migration number of the message data in the preset time period.

In one embodiment, determining the delay indicator in the preset period according to the uploading timestamp and the downloading timestamp of each message data includes:

determining message data which is successfully migrated within a preset period according to the uploading time stamp and the downloading time stamp of each message data;

And determining a time delay index in a preset period according to the uploading time stamp and the downloading time stamp of the message data which are successfully migrated.

In one embodiment, the performance index is determined according to the time delay index, the total number of message data and the successful number of message data migration in the preset time period, and the performance index comprises at least one of the following steps:

determining a time delay average value in the performance index according to the time delay index in the preset time period and the total number of message data in the preset time period;

determining a time delay peak value in the performance index according to the time delay index in the preset time period;

determining migration rate in the performance index according to the total number of message data in a preset period and a preset duration;

and determining the migration success rate in the performance index according to the message data migration success number and the total message data number in the preset time period. In one of the embodiments of the present invention,

if the performance index exceeds the preset performance threshold and the duration exceeding the performance threshold meets the preset duration, updating the data migration concurrency number according to the time delay index and the time delay threshold.

In one embodiment, the method further comprises:

if the performance index does not exceed the preset performance threshold, judging whether the current data migration concurrency number is updated;

if the performance index is updated, judging whether the performance index is lower than the average value of the performance index in the historical preset period;

and if the data is lower than the data migration concurrency number, updating the data migration concurrency number according to the historical update data of the data migration concurrency number.

In a second aspect, the present application further provides a data migration performance monitoring apparatus. The device comprises:

the time acquisition module is used for acquiring an uploading time stamp and a downloading time stamp of each message data in the data migration process;

the index determining module is used for determining the time delay index and the performance index of the message data according to the uploading time stamp and the downloading time stamp of each message data;

the concurrency updating module is used for updating the data migration concurrency number according to the time delay index and the time delay threshold value if the performance index exceeds the preset performance threshold value;

and the alarm module is used for outputting alarm notification according to the updated data migration concurrency number and/or the performance index.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

According to the data migration performance monitoring method, the device, the computer equipment and the storage medium, the uploading time stamp and the downloading time stamp of the message data in the data migration process are obtained, the time delay index and the performance index are calculated based on the obtained uploading time stamp and the downloading time stamp, whether the performance of the data migration tool is abnormal or not is judged through setting the performance threshold, after the performance index is abnormal, the data migration concurrency number is adjusted through the time delay index and the time delay threshold, and meanwhile, an alarm notification is generated and sent to a user, so that the user can visually monitor the performance index of the data migration tool through the alarm notification in real time, and the problem that the user cannot intuitively know the performance index of the data migration tool is solved.

Drawings

Fig. 1 is an application environment diagram of a data migration performance monitoring method provided in this embodiment;

fig. 2 is a flow chart of a first data migration performance monitoring method according to the present embodiment;

FIG. 3 is a flow chart of updating concurrency count provided in the present embodiment;

fig. 4 is a schematic flow chart of determining a time delay index and a performance index according to the present embodiment;

FIG. 5 is a flow chart for monitoring data migration performance according to the present embodiment;

FIG. 6 is a flowchart illustrating a second method for monitoring data migration performance according to the present embodiment;

FIG. 7 is a block diagram of a first data migration performance monitoring apparatus according to the present embodiment;

FIG. 8 is a block diagram of a second data migration performance monitoring apparatus according to the present embodiment;

FIG. 9 is a block diagram of a third data migration performance monitoring apparatus according to the present embodiment;

FIG. 10 is a block diagram of a fourth data migration performance monitoring apparatus according to the present embodiment;

fig. 11 is an internal structure diagram of a computer device according to the present embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The data migration performance monitoring method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in FIG. 1. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data such as message data, time stamps, time delay indexes, performance indexes, concurrence numbers, alarm notices and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection to transmit an alarm notification to the external terminal. The computer program, when executed by a processor, implements a data migration performance monitoring method.

The embodiment provides a data migration performance monitoring method, in the data migration process, an uploading time stamp and a downloading time stamp of each message data are obtained, a time delay index and a performance index of the message data are determined according to the uploading time stamp and the downloading time stamp of each message data, if the performance index exceeds a preset performance threshold, the data migration concurrency number is updated according to the time delay index and the time delay threshold, and an alarm notification is output according to the updated data migration concurrency number and/or the performance index.

In one embodiment, as shown in fig. 2, a method for monitoring data migration performance is provided, which is described by taking an example that the method is applied to the server in fig. 1, and the method for monitoring data migration performance provided in this embodiment may be specifically applied to a data migration tool installed in the server, that is, the data migration tool installed in the server is used to execute the method for monitoring data migration performance provided in this embodiment, and includes the following steps:

s201, in the data migration process, uploading time stamps and downloading time stamps of each message data are obtained.

The data migration may be an operation of migrating data. Alternatively, the data migration may be a process of migrating data between different cluster devices (such as servers), or may be a process of migrating data between different folders in the same cluster device, which is not limited in this application.

The message data may be data that needs to be migrated. Alternatively, the message data may be all message data in the cluster device that needs to perform data migration, or part of message data set manually.

The uploading time stamp may be a time stamp generated in the process of uploading the message data to the target cluster device (i.e., the cluster device receiving the message data). Alternatively, the upload time stamp may include an upload start time stamp (i.e., a time at which the upload of the message data is started) and an upload completion time stamp (i.e., a time at which the upload of the message data is completed).

The download timestamp may be a timestamp generated during the process of downloading the message data from the source cluster device (i.e., the cluster device that needs to perform migration of the message data). Alternatively, the download timestamp may include a download start timestamp (i.e., a time at which the message data starts to be downloaded) and a download completion timestamp (i.e., a time at which the message data is completed).

Optionally, in the process of performing data migration by using the data migration tool, the interceptor in the data migration tool may obtain, in real time, a download timestamp, that is, a download start timestamp and a download completion timestamp, generated in the process of downloading each message data from the source cluster device, and meanwhile, the interceptor may also obtain, in real time, an upload timestamp, that is, an upload start timestamp and an upload completion timestamp, generated in the process of uploading the message data to the target cluster device.

It should be noted that, in the process of performing data migration by using the data migration tool, the data migration tool may also set an initial data migration concurrency number, so that message data may be migrated by using the data migration tool according to the initial data migration concurrency number to perform migration of multiple message data at the same time, so as to improve the efficiency of performing data migration by using the data migration tool. Wherein the concurrent number of data migration may characterize the number of data migration that may be performed on the message data simultaneously.

S202, determining the time delay index and the performance index of the message data according to the uploading time stamp and the downloading time stamp of each message data.

The delay indicator may be an indicator that characterizes a delay time of the message data during the data migration process. Optionally, the delay index may include an upload delay index, a download delay index, and a migration delay index. The uploading delay index may be a delay index generated in a process of uploading the message data to the target cluster device, the downloading delay index may be a delay index generated in a process of downloading the message data from the source cluster device, and the migration delay index may be a delay index generated in a process of migrating the message data from the source cluster device to the target cluster device.

Wherein the performance index may be an index characterizing the performance of the data migration tool to migrate message data. Optionally, the performance metrics may include a delay mean, a delay peak, a migration rate, and a migration success rate. The time delay average value may be an average value of all time delay indexes generated in a data migration process in a preset period, the time delay peak value may be a maximum value of all time delay indexes generated in the data migration process in the preset period, the migration rate may be efficiency of migrating message data in the data migration process in the preset period, and the migration success rate may be success rate of completing data migration in the preset period.

Optionally, according to the uploading time stamp and the downloading time stamp of each message data, determining the time delay index and the optional mode of the performance index of the message data may be that according to the uploading time stamp and the downloading time stamp of each message data, calculating the difference between the uploading time stamp and the downloading time stamp, taking the difference as the time delay index of the message data, and simultaneously searching the performance index corresponding to the time delay index according to the time delay index through a preset performance index table; the uploading time stamp and the downloading time stamp of each message data can be input into the index model through a pre-trained index model, the index model can calculate the uploading time stamp and the downloading time stamp of the message data, and finally the time delay index and the performance index corresponding to the message data are obtained.

And S203, if the performance index exceeds a preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold.

Optionally, if the obtained performance index exceeds the performance threshold, it is proved that the time delay of message data migration is longer in the data migration process, and the data migration speed of the message data needs to be improved, so that the data migration concurrency number at the current moment is calculated according to the time delay index and the time delay threshold through preset concurrency number calculation logic, the update of the data migration concurrency number is completed according to the data migration concurrency number at the current moment, and the migration of the message data is continuously completed through the updated data migration concurrency number.

Preferably, when the performance index temporarily exceeds the performance threshold and returns to the normal level, the data migration tool determines that the performance index exceeds the performance threshold, so that the data migration tool still updates the data migration concurrency number at this time, but after the data migration concurrency number is updated, excessive data migration concurrency number may be caused to cause more resource waste, so, in order to reduce the resource waste, the embodiment may introduce a preset duration to avoid the data migration concurrency number update caused by the temporary exceeding of the performance index by the performance threshold. Specifically, if the performance index exceeds a preset performance threshold and the duration exceeding the performance threshold meets the preset duration, updating the data migration concurrency number according to the time delay index and the time delay threshold. If the performance index exceeds the preset performance threshold, starting a timer to judge whether the performance index in the preset time period continuously exceeds the performance threshold, if not, not updating the data migration concurrency number, and if so, updating the data migration concurrency number according to the time delay index and the time delay threshold.

S204, outputting alarm notification according to the updated data migration concurrency number and/or the performance index.

Optionally, when the performance index temporarily exceeds the performance threshold, but the duration exceeding the performance threshold does not meet the preset duration, the performance index is considered to temporarily exceed the performance threshold and does not affect the efficiency of data migration, so that an alarm notification can be generated only according to the performance index exceeding the performance threshold and output through the terminal; when the performance index temporarily exceeds the performance threshold and the duration exceeding the performance threshold meets the preset duration, the performance index is considered to exceed the performance threshold and the efficiency of data migration is influenced to a certain extent, so that an alarm notification can be generated only according to the updated data migration concurrency number and output through the terminal, the alarm notification can also be generated according to the updated data migration concurrency number and the performance index exceeding the performance threshold, and the alarm notification is output through the terminal, and the specific required data for generating the alarm notification can be determined according to the setting of a user.

FIG. 3 is a flow chart of updating concurrency numbers in one embodiment. In this embodiment, when the performance index exceeds the performance threshold, it is proved that the time delay of message data migration is longer in the data migration process, that is, the data migration speed is slower, so that it is crucial to improve the data migration speed, and in this embodiment, an update to the concurrency number is introduced, so that more message data can be migrated at the same time, and further improve the data migration speed, so that this embodiment provides an alternative way for updating the concurrency number, which includes the following steps:

and S301, if the performance index exceeds a preset performance threshold, determining the newly added concurrency number required by each message data according to the time delay index and the time delay threshold.

The newly added concurrency number may be the concurrency number which needs to be added according to the message data.

Optionally, if the performance index exceeds a preset performance threshold, the current usage rate of the local processor, the current usage rate of the local memory and the current data migration concurrency number are obtained, the temporary concurrency number supported by the processor is determined according to the current usage rate of the local processor and the preset processor usage rate threshold, the temporary concurrency number supported by the memory is determined according to the current usage rate of the local memory and the preset memory usage rate threshold, the temporary concurrency number supported by the message is determined according to the data migration concurrency number and the concurrency number threshold, the temporary concurrency number required by the message is determined according to the time delay index, the time delay threshold and the time delay change value between the current time and the historical time, and if the temporary concurrency number supported by the processor, the temporary concurrency number supported by the memory, the temporary concurrency number supported by the message and the temporary concurrency number required by the message are all greater than zero, the temporary concurrency number required by the message is determined as the new concurrency number required by each message data.

The local processor may be a processor of a server where the data migration tool is located. The processor support temporary concurrency number may be the maximum value of the concurrency number that the processor can newly increase at the current time. The local memory may be the memory of the server where the data migration tool resides. The temporary concurrency number supported by the memory may be the maximum value of the concurrency number that can be newly added at the current time. The message support temporary concurrency number may be the maximum value of the concurrency number that message data can be newly added at the current time. The number of temporary concurrency required for the message may be the number of concurrency required to be newly added at the current time.

Specifically, if the performance index exceeds a preset performance threshold, acquiring the current utilization rate of the local processor, the current utilization rate of the local memory and the current data migration concurrency number, simultaneously calculating a processor difference value between the current utilization rate of the local processor and the preset processor utilization rate threshold, then calculating a processor quotient value between the processor difference value and the local processor utilization rate required by single message data, and taking the processor quotient value as the determined processor support temporary concurrency number; calculating a memory difference value between the current utilization rate of the local memory and a preset memory utilization rate threshold value, then calculating a memory quotient value between the memory difference value and the local memory utilization rate required by single message data, and taking the memory quotient value as the determined temporary concurrency number of the memory support; calculating a concurrency number difference value between the migration concurrency number at the current moment and a concurrency number threshold value, and taking the concurrency number difference value as a temporary concurrency number supported by the message; calculating a time delay difference value between a time delay index and a time delay threshold value, simultaneously calculating a time delay change value between the time delay index at the current moment and a time delay index at a preset historical moment, finally calculating a demand quotient value between the time delay difference value and the time delay change value, and taking the demand quotient value as a temporary concurrency number required by a message; based on the obtained processor support temporary concurrency number, memory support temporary concurrency number, message support temporary concurrency number and message required temporary concurrency number, judging whether the processor support temporary concurrency number, memory support temporary concurrency number, message support temporary concurrency number and message required temporary concurrency number are all larger than zero, if not, not allowing updating of the data migration concurrency number, and if so, taking the message required temporary concurrency number as the newly added concurrency number required by each message data.

For example, the calculation manner of the temporary concurrency number supported by the processor may be that the temporary concurrency number supported by the processor= (processor usage threshold-current usage of local processor)/local processor usage required by single message data; the calculation mode of the temporary concurrency number of the memory support may be that the temporary concurrency number of the memory support= (memory usage threshold value-current usage of the local memory)/the local memory usage required by single message data; the calculation mode of the temporary concurrency number of the message support may be that the temporary concurrency number of the message support=the threshold value of the concurrency number of the message support-the migration concurrency number at the current moment; the calculation method of the temporary concurrency number required by the message may be that the temporary concurrency number required by the message= (delay threshold value-delay index)/(delay index at the current time-delay index at the historical time).

S302, updating the data migration concurrency number according to the newly added concurrency number.

Optionally, according to the obtained newly added concurrency number, calculating a sum value between the newly added concurrency number and the initial concurrency number, and taking the sum value as the data migration concurrency number to finish updating the data migration concurrency number.

According to the method for updating the concurrency number, if the performance index exceeds the preset performance threshold, the efficiency of data migration at the current moment is low, so that the newly-increased concurrency number required by each message data can be determined according to the time delay index and the time delay threshold, and the data migration concurrency number is updated according to the newly-increased concurrency number, namely, more message data can be subjected to data migration at the same time in a mode of increasing the concurrency number, and the efficiency of data migration is further improved.

Fig. 4 is a flow chart illustrating determining a delay indicator and a performance indicator according to an embodiment. In order to ensure the efficiency and accuracy of data migration, how to accurately acquire the delay index and the performance index is important, so on the basis of the above embodiment, the present embodiment provides an optional way of determining the delay index and the performance index, which includes the following steps:

s401, determining a time delay index in a preset period according to the uploading time stamp and the downloading time stamp of each message data.

Optionally, there are various ways to determine the delay indicator in the preset period according to the uploading timestamp and the downloading timestamp of each message data, which is not limited in this application.

One alternative implementation manner may be to calculate a difference between the uploading timestamp and the downloading timestamp according to the uploading timestamp and the downloading timestamp of each message data, and use the difference as a delay indicator of the message data.

Another alternative implementation manner may be to determine message data that is migrated successfully in a preset period according to an upload timestamp and a download timestamp of each message data, and determine a delay indicator in the preset period according to the upload timestamp and the download timestamp of the message data that is migrated successfully. Specifically, according to the uploading time stamp and the downloading time stamp of each message data, whether the uploading time stamp and the downloading time stamp of the message data are complete or not is determined, namely the message data should carry four time stamps, namely, the uploading starting time stamp, the uploading finishing time stamp, the downloading starting time stamp and the downloading finishing time stamp. If yes, the message data migration is proved to be successful, and if not, the message data migration is proved to be unsuccessful. And determines the number of message data that have been successfully migrated for a predetermined period of time (e.g., 1 minute). Meanwhile, according to the acquired message data with successful migration, calculating an uploading difference value between an uploading starting time stamp and an uploading finishing time stamp of the message data, and taking the uploading difference value as an uploading time delay index in the time delay indexes; calculating a downloading difference value between a downloading starting time stamp and a downloading finishing time stamp of the message data, and taking the downloading difference value as a downloading delay index in the delay indexes; and calculating a migration difference value between the uploading starting time stamp and the downloading finishing time stamp of the message data, and taking the migration difference value as a migration time delay index in the time delay indexes.

For example, the calculation manner of the uploading time delay index may be that uploading time delay index=uploading completion time stamp-uploading start time stamp; the calculation mode of the downloading time delay index may be that the downloading time delay index=downloading completion time stamp-downloading start time stamp; the migration delay indicator may be calculated by migration delay indicator=download completion timestamp-upload start timestamp.

S402, determining a performance index according to the time delay index, the total number of message data and the successful number of message data migration in a preset period.

The total number of message data may be the number of all message data for performing the data migration operation in the preset period.

The number of successful message data migration may be the number of message data that performs data migration operation and is successful in migration within a preset period.

Optionally, the performance index is determined in various ways according to the time delay index, the total number of message data and the successful number of message data migration in the preset period, which is not limited in this application.

One of the alternative implementation manners may be that a pre-trained performance index model is used to input a time delay index, a total number of message data and a successful number of message data migration in a preset period into the performance index model, and the performance index model analyzes the time delay index, the total number of message data and the successful number of message data migration in the preset period to obtain a performance index.

Another alternative implementation manner may be to determine a time delay average value in the performance index according to the time delay index in the preset time period and the total number of message data in the preset time period; and determining a time delay peak value in the performance index according to the time delay index in the preset time period, determining the migration rate in the performance index according to the total number of message data in the preset time period and the preset time length, and determining the migration success rate in the performance index according to the successful migration number of the message data in the preset time period and the total number of message data. Specifically, calculating a time delay index in a preset time period, and taking a time delay quotient value between the time delay index and the total number of message data in the preset time period as a time delay average value in the performance index; selecting the maximum value from all the time delay indexes within a preset period as a time delay peak value in the performance indexes; calculating a rate quotient between the total number of message data in a preset time period and a preset time period (such as 60 seconds), and taking the rate quotient as a migration rate in a performance index; and calculating a success rate quotient between the message data migration success number and the total message data number in a preset period, and taking the success rate quotient as the migration success rate in the performance index.

For example, the delay average value may be calculated by the delay average value=delay index/total message data; the migration rate may be calculated by migration rate=total message data/60; the migration success rate may be calculated by migration success rate=the number of migration success message data/the total number of message data.

According to the method for determining the time delay index and the performance index, the time delay index in the preset time period can be determined more accurately according to the uploading time stamp and the downloading time stamp of each message data, the performance index can be determined more reasonably by considering the influence of various factors on the performance index according to the time delay index, the total number of the message data and the successful number of message data migration in the preset time period, so that the user can monitor the data migration performance conveniently, and meanwhile, data support and guarantee are provided for judging whether the data migration concurrency number needs to be updated or not later.

FIG. 5 is a flow diagram of monitoring data migration performance in one embodiment. In this embodiment, after determining the performance index of data migration, according to the relationship between the performance index and the performance threshold, it is determined how to adjust the concurrency number of data migration, so as to ensure the efficiency and accuracy of data migration, and meanwhile, make the loss of local equipment smaller.

S501, in the data migration process, an uploading time stamp and a downloading time stamp of each message data are obtained.

S502, determining the time delay index and the performance index of the message data according to the uploading time stamp and the downloading time stamp of each message data.

S503 judges whether the performance index exceeds the preset performance threshold, if yes, execute step S504, if no, execute step S505.

And S504, if the performance index exceeds a preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold, and outputting an alarm notification according to the updated data migration concurrency number and/or the performance index.

If the performance index does not exceed the preset performance threshold, the step S505 is to judge whether the current data migration concurrency number is updated, if yes, the step S506 is executed, and if not, the step S506 is returned to execute the operation of obtaining the uploading time stamp and the downloading time stamp of each message data.

Optionally, when the performance index does not exceed a preset performance threshold, the current data migration efficiency is proved to be higher, at this time, whether the data migration concurrency number is updated or not is judged, if not, the data migration concurrency number is proved not to be increased, the data migration concurrency number is in a reasonable range, the possibility of resource waste is avoided, and the operation of obtaining the uploading time stamp and the downloading time stamp of each message data is performed in a return mode; if yes, the data migration concurrency number is proved to be updated, that is, the data migration concurrency number at the current moment is obtained by adding the newly added concurrency number on the basis of the initial concurrency number, so that resource waste possibly exists, and the step S506 is needed to be continuously executed so as to judge whether the data migration concurrency number needs to be adjusted.

If the update is performed in S506, it is determined whether the performance index is lower than the average value of the performance indexes within the historical preset period, if yes, step S507 is executed, and if not, the operation of obtaining the uploading time stamp and the downloading time stamp of each message data is returned to be executed.

Optionally, if the data migration concurrency number is updated, it is proved that the data migration concurrency number at the current moment may cause resource waste, so it is further required to determine whether the performance index at the current moment is lower than the average value of the performance indexes in the historical preset period, if not, it is proved that the data migration concurrency number at the current moment can meet the requirement that the data migration is kept high in efficiency, and no more resource waste is caused, and therefore, the operation of obtaining the uploading timestamp and the downloading timestamp of each message data is performed in a return mode; if so, it is proved that the performance index at the current moment is lower than the average value of the historical performance indexes, that is, the data migration efficiency at the current moment is high, but a certain resource waste is caused, so that in order to reduce the resource waste, the concurrent number of the data migration should be adjusted, that is, step S507 needs to be executed.

And if the data migration concurrency number is lower than S507, updating the data migration concurrency number according to the historical updating data of the data migration concurrency number.

The historical update data may be the number of data migration complications increased or decreased when the number of data migration complications is updated historically.

Optionally, if the performance index is lower than the average value of the performance indexes in the historical preset period, it is proved that the data migration at the current moment causes a certain resource waste, so that the number of the concurrent data migration needs to be reduced, that is, the difference between the concurrent data migration at the current moment and the historical updated data is calculated, and the difference is used as the concurrent data migration at the current moment to finish updating the concurrent data migration.

According to the method for monitoring the data migration performance, if the performance index does not exceed the preset performance threshold, it is proved that the data migration efficiency at the current moment meets the preset requirement, whether the data migration is wasted or not needs to be further judged, namely whether the current data migration concurrency number is updated or not is judged, if the current data migration concurrency number is updated, the data migration concurrency number at the current moment is proved to be subjected to increasing operation, so that the possibility of resource wasting exists, whether the data migration at the current moment is determined to be wasted or not is further judged, namely whether the performance index is lower than the average value of the performance index in the historical preset period or not is judged, if the performance index is lower than the average value of the performance index in the historical preset period, it is proved that the data migration at the current moment is really wasted, and therefore, the data migration concurrency number is updated according to the historical update data of the data migration concurrency number, namely, the current data migration concurrency number is reduced, and more resource wasting is reduced on the basis that the efficiency of the data migration is not reduced.

In one embodiment, this embodiment provides an alternative way of monitoring data migration performance, and the method is used for a server to be described as an example. As shown in fig. 6, the method includes the steps of:

s601, in the data migration process, an uploading time stamp and a downloading time stamp of each message data are obtained.

S602, according to the uploading time stamp and the downloading time stamp of each message data, message data which is successfully migrated in a preset period is determined.

S603, determining a time delay index in a preset period according to the uploading time stamp and the downloading time stamp of the message data which are successfully migrated.

S604, determining a time delay average value in the performance index according to the time delay index in the preset time period and the total number of message data in the preset time period.

S605, determining a time delay peak value in the performance index according to the time delay index in the preset time period.

S606, determining the migration rate in the performance index according to the total number of message data in the preset time period and the preset time length.

S607, determining the migration success rate in the performance index according to the message data migration success number and the total message data number in the preset time period.

S608 determines whether the performance index exceeds the performance threshold, if so, step S609 is executed, and if not, step S617 is executed.

S609, if the performance index exceeds a preset performance threshold, the current utilization rate of the local processor, the current utilization rate of the local memory and the current concurrent number of data migration are obtained.

S610, determining that the processor supports temporary concurrency according to the current utilization rate of the local processor and a preset processor utilization rate threshold.

S611 determines the temporary concurrency number of the memory support according to the current utilization rate of the local memory and a preset memory utilization rate threshold.

S612 determines that the message supports the temporary concurrency number according to the data migration concurrency number and the concurrency number threshold.

S613, determining the number of temporary concurrency required by the message according to the time delay index, the time delay threshold value and the time delay change value between the current time and the historical time.

S614, if the processor supports the temporary concurrency number, the memory supports the temporary concurrency number, the message supports the temporary concurrency number and the temporary concurrency number required by the message are all greater than zero, determining the temporary concurrency number required by the message as the newly added concurrency number required by each message data.

S615, updating the data migration concurrency number according to the newly added concurrency number.

S616, according to the updated data migration concurrency number and/or the performance index, an alarm notification is output.

S617, if the performance index does not exceed the preset performance threshold, judging whether the current data migration concurrency number is updated, if yes, executing step S618, and if not, returning to execute the operation of acquiring the uploading time stamp and the downloading time stamp of each message data.

If the update is performed at S618, it is determined whether the performance index is lower than the average value of the performance indexes within the historical preset period, if yes, step S619 is executed, and if not, the operation of obtaining the uploading time stamp and the downloading time stamp of each message data is returned to be executed.

And if the data migration concurrency number is lower than the data migration concurrency number, updating the data migration concurrency number according to the historical updating data of the data migration concurrency number.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data migration performance monitoring device for implementing the above related data migration performance monitoring method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the data migration performance monitoring device or devices provided below may be referred to the limitation of the data migration performance monitoring method hereinabove, and will not be described herein.

In one embodiment, as shown in fig. 7, there is provided a data migration performance monitoring apparatus 1, including: a time acquisition module 10, an index determination module 11, a concurrency update module 12, and an alarm module 13, wherein:

and the time acquisition module 10 is used for acquiring the uploading time stamp and the downloading time stamp of each message data in the data migration process.

The index determining module 11 is configured to determine a time delay index and a performance index of the message data according to the uploading timestamp and the downloading timestamp of each message data.

And the concurrency updating module 12 is configured to update the data migration concurrency number according to the time delay index and the time delay threshold if the performance index exceeds the preset performance threshold.

And the alarm module 13 is used for outputting alarm notification according to the updated data migration concurrency number and/or the performance index.

In one embodiment, as shown in fig. 8, the concurrent update module 12 in fig. 7 includes:

the new determining unit 120 is configured to determine, if the performance index exceeds a preset performance threshold, a new number of concurrency required for each message data according to the delay index and the delay threshold.

And a concurrency updating unit 121, configured to update the data migration concurrency number according to the newly added concurrency number.

In one embodiment, the new addition determining unit 120 in fig. 7 includes:

the data acquisition subunit is used for acquiring the current utilization rate of the local processor, the current utilization rate of the local memory and the current concurrent number of data migration if the performance index exceeds a preset performance threshold.

And the processor concurrency number determining subunit determines that the processor supports the temporary concurrency number according to the current utilization rate of the local processor and a preset processor utilization rate threshold value.

And the memory concurrency number determining subunit determines the temporary concurrency number supported by the memory according to the current utilization rate of the local memory and a preset memory utilization rate threshold value.

And the message concurrency number determining subunit determines that the message supports the temporary concurrency number according to the data migration concurrency number and the concurrency number threshold.

And the required concurrency number determining subunit determines the required temporary concurrency number of the message according to the time delay index, the time delay threshold value and the time delay change value between the current moment and the historical moment.

And the newly added concurrency number determining subunit is used for determining the temporary concurrency number required by the message as the newly added concurrency number required by each message data if the temporary concurrency number supported by the processor, the temporary concurrency number supported by the memory, the temporary concurrency number supported by the message and the temporary concurrency number required by the message are all greater than zero.

In one embodiment, as shown in fig. 9, the index determination module 11 in fig. 7 includes:

the delay determining unit 110 is configured to determine a delay indicator in a preset period according to the uploading timestamp and the downloading timestamp of each message data.

The performance determining unit 111 is configured to determine a performance index according to the time delay index, the total number of message data and the number of message data migration successes within a preset period.

In one embodiment, the delay determining unit 110 in fig. 9 includes:

and the migration success subunit is used for determining message data which are successfully migrated within a preset period according to the uploading time stamp and the downloading time stamp of each message data.

And the time delay determining subunit is used for determining a time delay index in a preset period according to the uploading time stamp and the downloading time stamp of the message data which are successfully migrated.

In one embodiment, the performance determining unit 111 in fig. 9 includes:

and the average value determining subunit is used for determining the average value of the time delays in the performance indexes according to the time delay indexes in the preset time period and the total number of the message data in the preset time period.

And the peak value determining subunit is used for determining the time delay peak value in the performance index according to the time delay index in the preset time period.

And the rate determining subunit is used for determining the migration rate in the performance index according to the total number of the message data in the preset time period and the preset time length.

And the success rate determining subunit is used for determining the migration success rate in the performance index according to the message data migration success number and the message data total number in the preset time period.

In one embodiment, the concurrency updating module 12 in fig. 7 is further configured to update the data migration concurrency count according to the latency index and the latency threshold if the performance index exceeds a preset performance threshold and the duration of exceeding the performance threshold meets the preset duration.

In one embodiment, as shown in fig. 10, the data migration performance monitoring apparatus 1 in fig. 7 further includes:

the update determination module 14 is configured to determine whether the current data migration concurrency count is updated if the performance index does not exceed the preset performance threshold.

The index comparison module 15 is configured to determine whether the performance index is lower than the average value of the performance index in the historical preset period if the performance index is updated.

And the concurrency adjustment module 16 is configured to update the data migration concurrency number according to the historical update data of the data migration concurrency number if the data migration concurrency number is lower than the historical update data.

The modules in the data migration performance monitoring apparatus may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 11. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program, when executed by a processor, implements a data migration performance monitoring method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 11 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:

In one embodiment, the processor when executing the computer program further performs the steps of:

and determining the migration success rate in the performance index according to the message data migration success number and the total message data number in the preset time period.

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:

In one embodiment, the computer program when executed by the processor further performs the steps of:

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:

The data (including, but not limited to, data for analysis, data stored, data displayed, etc.) referred to in this application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method for monitoring data migration performance, the method comprising:

if the performance index exceeds a preset performance threshold, updating the data migration concurrency number according to the time delay index and the time delay threshold;

2. The method of claim 1, wherein updating the data migration concurrency count according to the latency index and the latency threshold if the performance index exceeds a preset performance threshold comprises:

if the performance index exceeds a preset performance threshold, determining the newly added concurrency number required by each message data according to the time delay index and the time delay threshold;

3. The method of claim 2, wherein determining the number of additional concurrency required for each message data based on the latency indicator and the latency threshold if the performance indicator exceeds a preset performance threshold comprises:

determining the number of temporary concurrency required by the message according to the time delay index, the time delay threshold and the time delay change value between the current moment and the historical moment;

and if the temporary concurrency number supported by the processor, the temporary concurrency number supported by the memory, the temporary concurrency number supported by the message and the temporary concurrency number required by the message are all greater than zero, determining the temporary concurrency number required by the message as the newly added concurrency number required by each message data.

4. The method according to claim 1, wherein determining the delay index and the performance index of the message data according to the uploading timestamp and the downloading timestamp of each message data comprises:

5. The method of claim 4, wherein determining the delay indicator for the predetermined period based on the uploading timestamp and the downloading timestamp of each message data comprises:

Determining message data which is successfully migrated in a preset period according to the uploading time stamp and the downloading time stamp of each message data;

6. The method of claim 4, wherein determining the performance index based on the time delay index, the total number of message data and the number of message data migration successes within the predetermined period of time comprises at least one of:

7. The method of claim 1, wherein updating the data migration concurrency count according to the latency index and the latency threshold if the performance index exceeds a preset performance threshold comprises:

And if the performance index exceeds a preset performance threshold and the duration exceeding the performance threshold meets the preset duration, updating the data migration concurrency number according to the time delay index and the time delay threshold.

8. The method according to any one of claims 1 to 7, further comprising:

if the performance index does not exceed a preset performance threshold, judging whether the current data migration concurrency number is updated;

if the performance index is updated, judging whether the performance index is lower than the average value of the performance indexes in the historical preset period;

9. A data migration performance monitoring apparatus, the apparatus comprising:

the concurrency updating module is used for updating the data migration concurrency number according to the time delay index and the time delay threshold value if the performance index exceeds a preset performance threshold value;

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 8 when the computer program is executed.

11. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.