CN116974871A

CN116974871A - System performance evaluation method, device, equipment and storage medium

Info

Publication number: CN116974871A
Application number: CN202310971242.4A
Authority: CN
Inventors: 赵磊; 王晓垚; 杨晨; 张寒
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-08-03
Filing date: 2023-08-03
Publication date: 2023-10-31

Abstract

The present disclosure provides a system performance evaluation method, apparatus, device, and storage medium, which can be applied to the fields of system evaluation and financial science and technology or other related fields. The method comprises the following steps: acquiring operation information of a system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal; for each problem period, determining a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period; determining a system impact duration of the system during the problem period based on the first transaction cumulative value and the plurality of second transaction cumulative values; the deployment failure rate is determined based on the plurality of version deployment times and the patch release times in the deployment data.

Description

System performance evaluation method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of system evaluation and financial technology or other related fields, and more particularly, to a system performance evaluation method, apparatus, device, and storage medium.

Background

As project requirements increase, so does the iteration frequency of the versions. Under the high-frequency version production background, the version package is required to be produced and implemented according to the requirements of the project, so that the application system is continuously updated, updated and brought online.

In the process of implementing the disclosed concept, the inventor finds that at least the following problems exist in the related art: high-frequency version production can cause unstable operation of an application system, and because a plurality of operation indexes exist in the operation process of the application system, the performance of the application system is difficult to accurately evaluate, so that the overall operation risk of the application system is caused.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a system performance evaluation method, apparatus, device, medium, and program product.

According to a first aspect of the present disclosure, there is provided a system performance evaluation method, including: acquiring operation information of a system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal; determining, for each problem period, a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period; determining a system influence duration of the system in the problem time period based on the first transaction cumulative value and a plurality of the second transaction cumulative values; determining a deployment failure rate based on a plurality of version deployment times and patch release times in the deployment data; and determining a risk value of the system in the preset period according to a plurality of system influence time periods, the accumulated alarm time periods and the deployment failure rate, so as to evaluate the performance of the system based on the risk value.

According to an embodiment of the present disclosure, before determining the first transaction cumulative value of the problem period and the second transaction cumulative values of the plurality of history periods corresponding to the problem period, the method further includes: for each of the problem time periods, determining a problem occurrence date corresponding to the problem time period; determining a plurality of history dates according to the occurrence date of the problems; a history period corresponding to the problem period in each of the history periods is determined.

According to an embodiment of the present disclosure, the determining a system influence duration of the system in the problem period based on the first transaction cumulative value and a plurality of the second transaction cumulative values includes: calculating a service loss rate based on the first transaction cumulative value and a plurality of the second transaction cumulative values; and integrating the service loss rate and the problem duration corresponding to the problem time period to obtain the system influence duration.

According to an embodiment of the present disclosure, the calculating a traffic loss rate based on the first transaction cumulative value and a plurality of the second transaction cumulative values includes: making a difference between each of the second transaction accumulated values and the first transaction accumulated value to obtain a plurality of intermediate difference values; the second transaction accumulated value corresponding to each intermediate difference value is made a quotient to obtain a plurality of loss rates; and setting the maximum value of the loss rates as the service loss rate.

According to an embodiment of the present disclosure, the determining a deployment failure rate based on the plurality of version deployment times and the patch release times in the deployment data includes: determining a deployment state at each of the version deployment times based on a plurality of the version deployment times and the patch release times, the deployment state including a success state and a failure state; determining the total number of failures of which the deployment state is a failure state in a plurality of version deployment times in the preset period; determining the total deployment times in the preset period based on a plurality of version deployment times; and (5) the total failure times and the total deployment times are used as a quotient to obtain the deployment failure rate.

According to an embodiment of the present disclosure, the determining a deployment status at each of the version deployment times based on a plurality of the version deployment times and the patch release times includes: determining whether a patch release time which is the same as the version deployment time exists for each version deployment time; when the patch release time which is the same as the version deployment time exists, determining a historical deployment time corresponding to the version deployment time, wherein the historical deployment time is a deployment time which is before the version deployment time and is closest to the version deployment time in the version deployment times.

According to an embodiment of the present disclosure, the determining a risk value of the system in the preset period according to a plurality of the system influence durations, the accumulated alarm durations, and the deployment failure rate includes: summing the system influence time durations in the preset period to obtain a period influence time duration; converting the period influence duration and the accumulated alarm duration into values of a preset time unit to obtain a target influence duration and a target alarm duration; determining a target failure value based on a preset weight and the deployment failure rate; and summing the target influence time, the target alarm time and the target failure value to obtain a risk value of the system in the preset period.

According to an embodiment of the present disclosure, the above method further includes: determining a plurality of risk ranges according to a plurality of index values in a risk level table, wherein each risk range has a corresponding processing scheme for the system; and determining a target range in which the risk value is located from the plurality of risk ranges, so as to process the system based on a processing scheme corresponding to the target range.

A second aspect of the present disclosure provides a system performance evaluation apparatus, comprising: the information acquisition module is used for acquiring operation information of the system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal; a transaction determining module for determining, for each problem period, a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period; an influence determining module, configured to determine a system influence duration of the system in the problem time period based on the first transaction cumulative value and a plurality of the second transaction cumulative values; the failure determining module is used for determining the deployment failure rate based on the deployment time of a plurality of versions and the patch release time in the deployment data; and the risk determining module is used for determining a risk value of the system in the preset period according to a plurality of system influence time periods, the accumulated alarm time periods and the deployment failure rate so as to evaluate the performance of the system based on the risk value.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.

According to the system performance evaluation method, device, equipment, medium and program product provided by the disclosure, the performance of the system is evaluated by taking the operation indexes of the system influence duration, the accumulated alarm duration and the deployment failure rate as the set of evaluation indexes, so that the evaluation result is more accurate. In the process of determining the system influence time length, the transaction accumulated value of the historical time period is combined for determination, so that the determined system influence time length is more accurate and effective, and the accuracy of system performance evaluation is further improved.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a system performance evaluation method, apparatus, device, medium and program product according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a system performance evaluation method according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of a system performance evaluation method according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a block diagram of a system performance evaluation apparatus according to an embodiment of the present disclosure; and

fig. 5 schematically illustrates a block diagram of an electronic device adapted to implement a system performance evaluation method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical scheme of the invention, the related user information (including but not limited to user personal information, user image information, user equipment information, such as position information and the like) and data (including but not limited to data for analysis, stored data, displayed data and the like) are information and data authorized by a user or fully authorized by all parties, and the processing of the related data such as collection, storage, use, processing, transmission, provision, disclosure, application and the like are all conducted according to the related laws and regulations and standards of related countries and regions, necessary security measures are adopted, no prejudice to the public welfare is provided, and corresponding operation inlets are provided for the user to select authorization or rejection.

Under the condition that the existing business demands are rapidly increased, project demands accepted by the financial and technological side are gradually increased. Further, the frequency of version iteration is increased, and the number of version production times is increased. In the high-frequency production background, the probability of the occurrence of problems of the application system is improved. Since the application system has a plurality of operation indexes in the operation process, it is difficult to accurately evaluate the performance of the application system. Without effective management, high frequency project requirements may pose an overall operational risk to the application system, resulting in unstable operation of the application system.

In view of this, embodiments of the present disclosure provide a system performance evaluation method, a system performance evaluation apparatus, an electronic device, a readable storage medium, and a computer program product. The method comprises the following steps: acquiring operation information of a system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal; for each problem period, determining a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period; determining a system impact duration of the system during the problem period based on the first transaction cumulative value and the plurality of second transaction cumulative values; determining a deployment failure rate based on a plurality of version deployment times and patch release times in the deployment data; and determining a risk value of the system in a preset period according to the influence time lengths, the accumulated alarm time lengths and the deployment failure rate of the multiple systems so as to evaluate the performance of the system based on the risk value.

Fig. 1 schematically illustrates an application scenario diagram of a system performance evaluation method, apparatus, device, medium and program product according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 through the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages, etc. Various communication client applications, such as a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that, the system performance evaluation method provided by the embodiment of the present disclosure may be generally performed by the server 105. Accordingly, the system performance evaluation apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The system performance evaluation method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105. Accordingly, the system performance evaluation apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The system performance evaluation method of the disclosed embodiment will be described in detail below by way of fig. 2 and 3 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a system performance evaluation method according to an embodiment of the disclosure.

As shown in fig. 2, the system performance evaluation method of this embodiment includes operations S210 to S250.

In operation S210, operation information of the system in a preset period is acquired, where the operation information includes abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data includes a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal.

In operation S220, for each problem period, a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period are determined.

In operation S230, a system impact duration of the system during the problem period is determined based on the first transaction cumulative value and the plurality of second transaction cumulative values.

In operation S240, a deployment failure rate is determined based on the plurality of version deployment times and the patch release times in the deployment data.

In operation S250, a risk value of the system in a preset period is determined according to the plurality of system influence durations, the accumulated alarm durations and the deployment failure rate, so as to evaluate the performance of the system based on the risk value.

According to the embodiment of the disclosure, the operation information of the system in a preset period is acquired, and the preset period can be set to be one month or one quarter. The operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system. Specifically, for abnormal operation data, monitoring is set in the operation process of the system, and if abnormality occurs in the operation process, the abnormal operation data can be generated. Aiming at deployment data, according to the requirements of projects, version deployment is completed by putting the version package into production. Deployment data can be generated in the version deployment process, and the deployment data specifically comprise version deployment time, patch release time and the like.

According to an embodiment of the present disclosure, the abnormal operation data includes a plurality of problem periods in which the system is abnormal. The system occurrence anomaly may be an anomaly due to a production problem, the problem period being the time from when the problem occurs until the problem is effectively resolved (including fully resolved and otherwise circumvented).

According to the embodiment of the disclosure, the abnormal operation data further comprises accumulated alarm duration of abnormal target transaction indexes in the system. The target transaction index may be set based on different services performed by the system. For example: when the transfer service is executed, the target transaction index is a key index in the execution of the transfer service; when executing login transaction business, the target transaction index is a key index in executing the login transaction business. And monitoring the target transaction index in real time, if related alarm occurs, representing that the target transaction index is abnormal, and recording the duration of each monitoring alarm of the target transaction index in a preset period. And summing the duration times in the preset period to obtain the accumulated alarm duration.

According to an embodiment of the present disclosure, for each problem period, a first transaction cumulative value, i.e., a total amount of transactions during the problem period, is determined. And determining a second transaction accumulation value of a plurality of historical time periods corresponding to the problem time period, namely the transaction total amount in the historical time period, wherein the historical time period and the problem time period are the same time period in different days.

According to the embodiment of the disclosure, the first transaction accumulated value and the second transaction accumulated values are compared one by one, and the system influence duration of the system in the problem time period is determined. The deployment failure rate is determined based on the multiple version deployment times and the patch release times in the deployment data, specifically, if a production patch (without the deployment day) is released between two adjacent regular version deployments, the previous version deployment is regarded as a deployment failure. And determining a risk value of the system in a preset period according to the influence time lengths, the accumulated alarm time lengths and the deployment failure rate of the multiple systems so as to evaluate the performance of the system based on the risk value.

According to the embodiment of the disclosure, the performance of the system is evaluated by taking the operation indexes of the system influence duration, the accumulated alarm duration and the deployment failure rate as the set of evaluation indexes, so that the evaluation result is more accurate. In the process of determining the system influence time length, the transaction accumulated value of the historical time period is combined for determination, so that the determined system influence time length is more accurate and effective, and the accuracy of system performance evaluation is further improved.

In accordance with an embodiment of the present disclosure, before determining the first transaction cumulative value at the problem time period and the second transaction cumulative value at the plurality of history time periods corresponding to the problem time period, the method may further include the operations of:

for each problem time period, determining a problem occurrence date corresponding to the problem time period; determining a plurality of history dates according to the problem occurrence date; a historical time period corresponding to the problem time period in each historical date is determined.

According to the embodiment of the disclosure, the problem occurrence date corresponding to each problem time period is determined, and a plurality of history dates are determined according to the problem occurrence date. A plurality of time distances may be set in determining the history date, for example: one day, two days, seven days, one month and one year. A plurality of history dates is determined based on each of the time distances and the problem occurrence date.

According to the embodiment of the disclosure, a historical time period corresponding to a problem time period in each historical date is determined, wherein the historical time period and the problem time period are the same time period in different dates. For example: the problem occurrence date is A1 year, B1 month and C1 day, and the problem time period is 9:00-10:00; the historical date is A2 years, B2 months and C2 days, and the historical time period is 9:00-10:00. A plurality of historical time periods are determined based on the problem occurrence date, so that the transaction accumulated value is more accurate and reasonable when compared.

According to an embodiment of the present disclosure, calculating a traffic loss rate based on a first transaction cumulative value and a plurality of second transaction cumulative values may include the operations of:

making a difference between each second transaction accumulated value and the first transaction accumulated value to obtain a plurality of intermediate difference values; the second transaction accumulated value corresponding to each intermediate difference value is made a quotient to obtain a plurality of loss rates; the maximum value of the plurality of loss rates is set as the traffic loss rate.

According to the embodiment of the disclosure, each second transaction accumulated value is differenced with the first transaction accumulated value to obtain a plurality of intermediate differences. For example: the first transaction cumulative value is x, and the corresponding plurality of second transaction cumulative values includes x1, x2, x3, x4, x5. And (3) taking the second transaction accumulated value corresponding to each intermediate difference value and the intermediate difference value as a quotient to obtain a plurality of loss rates, and taking the maximum value in the plurality of loss rates as the service loss rate, so that the maximum risk in a reasonable range can be considered when determining the risk value. The specific calculation formula is as follows:

According to an embodiment of the present disclosure, determining a system influence duration of a system over a problem period based on a first transaction cumulative value and a plurality of second transaction cumulative values may include the operations of:

calculating a traffic loss rate based on the first transaction cumulative value and the plurality of second transaction cumulative values; and integrating the service loss rate and the problem duration corresponding to the problem time period to obtain the system influence duration.

According to the embodiment of the disclosure, the first transaction accumulated value and the second transaction accumulated values are compared one by one, and the service loss rate is determined according to the total transaction amount of the problem time period and the total transaction amount of the historical time periods. And integrating the service loss rate and the problem duration corresponding to the problem time period to obtain a transaction accumulated value relative to history, and affecting the duration of the whole system in the problem time period. By the method, the system influence time length is determined, so that the determination result is more accurate and effective, and the accuracy of system performance evaluation is further improved.

According to an embodiment of the present disclosure, determining a deployment failure rate based on a plurality of version deployment times and patch release times in deployment data may include the following operations:

determining a deployment state at each version deployment time based on the plurality of version deployment times and the patch release time, wherein the deployment state comprises a success state and a failure state; determining the total number of failures of which the deployment state is a failure state in a plurality of version deployment times in a preset period; determining the total deployment times in a preset period based on the deployment times of the versions; and (5) the total failure times and the total deployment times are made a quotient to obtain the deployment failure rate.

According to the embodiment of the disclosure, the plurality of versions are put into production in the same day and are only recorded as one version deployment, and only one version deployment time is recorded. The deployment state of each version deployment time is determined so as to know whether the version deployment of the current time is successful or not. Determining the total number of failures when the deployment state is the failure state in the deployment time of a plurality of versions in a preset period, and taking the total number of failures and the total number of deployments as a quotient to obtain the deployment failure rate. And calculating the deployment failure rate through the total failure times and the total deployment times, so that the calculation result is more accurate.

According to an embodiment of the present disclosure, determining a deployment state at each version deployment time based on a plurality of version deployment times and patch release times may include the following operations:

determining whether a patch release time which is the same as the version deployment time exists for each version deployment time; when the patch release time which is the same as the version deployment time exists, determining a historical deployment time corresponding to the version deployment time, wherein the historical deployment time is the deployment time which is before the version deployment time and is closest to the version deployment time in the plurality of version deployment times; the deployment state of the historical deployment time is determined to be a failure state.

According to the embodiment of the disclosure, if a production patch is released (without the deployment day) between two adjacent version deployments, the previous version deployment is regarded as a deployment failure, that is, the deployment state of the version deployment time of the previous version deployment is a failure state. Specifically, whether the patch release time which is the same as the version release time exists is determined, and if the patch release time exists, the fact that not only a new version is deployed but also a production patch is released at the version release time is indicated. The production patch is used to revise the previously deployed version, thus determining that the previous version was deployed as a deployment failure. And determining the historical deployment time corresponding to the previous version deployment based on the current version deployment time. And determining the deployment state of the historical deployment time corresponding to the previous version deployment as a failure state.

According to the embodiment of the disclosure, if the patch release time which is the same as the version deployment time does not exist, the version deployed in the previous time is not corrected, so that the version deployed in the previous time is determined to be successfully deployed. And determining the deployment state of the historical deployment time corresponding to the previous version deployment as a successful state. The deployment state is determined through the version deployment time and the patch release time, so that the determination process is more reasonable, visual and accurate.

According to the embodiment of the disclosure, according to a plurality of system influence durations, accumulated alarm durations and deployment failure rates, determining a risk value of a system in a preset period may include the following operations:

summing a plurality of system influence durations in a preset period to obtain a period influence duration; converting the period influence duration and the accumulated alarm duration into values of a preset time unit to obtain a target influence duration and a target alarm duration; determining a target failure value based on a preset weight and a deployment failure rate; and summing the target influence time, the target alarm time and the target failure value to obtain a risk value of the system in a preset period.

According to the embodiment of the disclosure, a plurality of system influence durations within a preset period are summed to obtain the period influence duration. And converting the period influence duration and the accumulated alarm duration into values of a preset time unit to obtain a target influence duration and a target alarm duration. Wherein the preset time unit may be set as minutes. The target failure value is determined based on a preset weight and a deployment failure rate, wherein the preset weight may be set to 100. Multiplying the preset weight by the failure rate to obtain a target failure value. And unit conversion is carried out on the period influence duration, the accumulated alarm duration and the deployment failure rate, so that the final risk value is calculated conveniently.

According to the embodiment of the disclosure, the target influence time, the target alarm time and the target failure value are summed to obtain a risk value of the system in a preset period. As shown in table 1, assuming that the preset period is three months, risk values of the system x and the system y in the preset period are calculated, respectively. Specifically, by 1 month, the risk value of system x is 40, and the risk value of system y is 3; by 2 months, the risk value of system x is 145 and the risk value of system y is 15; by 3 months, the risk value for system x is 152 and the risk value for system y is 37. Further, in the process of evaluating the system based on the risk value, the greater the risk value, the worse the current performance of the system is, and the current running of the system is unstable.

TABLE 1

System and method for controlling a system	Month of month	Duration of target influence	Target alarm duration	Target failure value
					x	1 month	10	20	10
x	2 months of	30	45	30
					x	3 months of	0	2	5
y	1 month	0	3	0
					y	2 months of	5	5	2
y	3 months of	7	10	5

Fig. 3 schematically illustrates a flow chart of a system performance evaluation method according to another embodiment of the present disclosure.

As shown in fig. 3, the system performance evaluation method of this embodiment includes operations S310 to S340.

In operation S310, a risk value of the system is determined.

In operation S320, a plurality of risk ranges are determined according to a plurality of index values in the risk level table, wherein each risk range has a corresponding processing scheme for the system.

In operation S330, a target range in which the risk value is located is determined from among the plurality of risk ranges.

In operation S340, the system is processed based on the processing scheme corresponding to the target range.

According to the embodiment of the disclosure, the risk level table is set according to the characteristics and the characteristics of the system and the control direction. Specifically, as shown in table 2, the risk level table includes a plurality of risk indexes T1, T2, and T3, the index value corresponding to the risk index T1 is 50, the index value corresponding to the risk index T2 is 100, and the index value corresponding to the risk index T3 is 150. A plurality of risk ranges are determined from a plurality of index values in the risk level table. Specifically, the risk range includes a first range index value of 50 or less, a second range risk value of 50 or less and 100 or less, a third range risk value of 100 or less and 150 or less, and a fourth range risk value of 150 or more.

TABLE 2

According to the embodiment of the disclosure, a target range in which a risk value is located is determined from a plurality of risk ranges. For example: assuming a risk value of 145, the target range is determined to be a third range. The system is processed based on a processing scheme corresponding to the target range. There is a corresponding treatment regime for the system for each risk range. Specifically, the treatment scheme corresponding to the first range is not required to be controlled. The second range corresponds to a processing scheme for reducing the project deployment times. The processing scheme corresponding to the third range is to stop the technical project reception. The processing scheme corresponding to the fourth range is to stop the acceptance of all types of items and enter the rectification cycle of the system.

According to the embodiment of the disclosure, the risk value calculated according to the table 1 is available, and the whole y system operates stably without management and control. And stopping technical project acceptance after 2 months in the x application system, stopping all types of project acceptance after 3 months, controlling project risks, and entering a rectification cycle. By evaluating the system performance based on the risk value, the risk can be effectively managed and controlled in advance, so that the stability of the system is improved.

Based on the system performance evaluation method, the disclosure also provides a system performance evaluation device. The device will be described in detail below in connection with fig. 4.

Fig. 4 schematically shows a block diagram of a system performance evaluation apparatus according to an embodiment of the present disclosure.

As shown in fig. 4, the system performance evaluation apparatus 400 of this embodiment includes an information acquisition module 410, a transaction determination module 420, an influence determination module 430, a failure determination module 440, and a risk determination module 450.

The information obtaining module 410 is configured to obtain operation information of the system in a preset period, where the operation information includes abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data includes a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal. In an embodiment, the information obtaining module 410 may be configured to perform the operation S210 described above, which is not described herein.

The transaction determination module 420 is configured to determine, for each problem period, a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period. In an embodiment, the transaction determining module 420 may be configured to perform the operation S220 described above, which is not described herein.

The impact determination module 430 is configured to determine a system impact duration of the system during the problem period based on the first transaction cumulative value and the plurality of second transaction cumulative values. In an embodiment, the influence determining module 430 may be configured to perform the operation S230 described above, which is not described herein.

The failure determination module 440 is configured to determine a deployment failure rate based on the plurality of version deployment times and the patch release times in the deployment data. In an embodiment, the failure determination module 440 is configured to perform the operation S240 described above, which is not described herein.

The risk determination module 450 is configured to determine a risk value of the system in a preset period according to a plurality of system influence durations, accumulated alarm durations and deployment failure rates, so as to evaluate the performance of the system based on the risk value. In an embodiment, the risk determination module 450 is configured to perform the operation S250 described above, which is not described herein.

According to an embodiment of the present disclosure, the system performance evaluation apparatus 400 further includes a first date determination module, a second date determination module, and a time determination module.

And the first date determining module is used for determining the problem occurrence date corresponding to the problem time period for each problem time period.

And the second date determining module is used for determining a plurality of historical dates according to the problem occurrence date.

And the time determining module is used for determining a historical time period corresponding to the problem time period in each historical date.

According to an embodiment of the present disclosure, the impact determination module 430 further includes an loss calculation sub-module and a duration calculation sub-module.

And the loss calculation sub-module is used for calculating the service loss rate based on the first transaction accumulated value and the plurality of second transaction accumulated values.

And the duration calculation sub-module is used for integrating the service loss rate and the problem duration corresponding to the problem time period to obtain the system influence duration.

According to an embodiment of the present disclosure, the loss calculation submodule includes a subtraction unit, a quotient unit, and a loss determination unit.

The difference making unit is used for making a difference between each second transaction accumulated value and the first transaction accumulated value to obtain a plurality of intermediate difference values;

the business unit is used for making a business for each intermediate difference value and a second transaction accumulated value corresponding to the intermediate difference value to obtain a plurality of loss rates;

and a loss determination unit configured to take the maximum value of the plurality of loss rates as a traffic loss rate.

According to the present embodiment, the failure determination module 840 includes a status determination sub-module, a failure determination sub-module, a total determination sub-module, and a failure calculation sub-module.

The state determination sub-module is used for determining the deployment state at each version deployment time based on a plurality of version deployment times and patch release times, wherein the deployment state comprises a success state and a failure state.

The failure determination submodule is used for determining the total number of failures when the deployment state is the failure state in a plurality of version deployment times in a preset period.

The total number determining sub-module is used for determining the total deployment times in a preset period based on the deployment times of the versions.

And the failure calculation sub-module is used for taking the total failure times and the total deployment times as the quotient to obtain the deployment failure rate.

According to the present embodiment, the state determination submodule includes a release determination unit, a deployment determination unit, and a state determination unit.

And the release determination unit is used for determining whether the patch release time which is the same as the version deployment time exists for each version deployment time.

The deployment determining unit is used for determining the historical deployment time corresponding to the version deployment time when the patch release time same as the version deployment time exists, wherein the historical deployment time is the deployment time which is before the version deployment time and is closest to the version deployment time in the plurality of version deployment times.

And the state determining unit is used for determining the deployment state of the historical deployment time as a failure state.

According to the present embodiment, the risk determination module 850 includes a duration calculation sub-module, a duration conversion sub-module, a target determination sub-module, and a risk calculation sub-module.

And the time length calculation sub-module is used for summing the multiple system influence time lengths in the preset period to obtain the period influence time length.

The time length conversion sub-module is used for converting the period influence time length and the accumulated alarm time length into the numerical value of a preset time unit to obtain a target influence time length and a target alarm time length;

The target determining submodule is used for determining a target failure value based on a preset weight and a deployment failure rate;

and the risk calculation sub-module is used for summing the target influence time length, the target alarm time length and the target failure value to obtain a risk value of the system in a preset period.

According to an embodiment of the present disclosure, the system performance evaluation apparatus 800 further includes a range determination module and a scenario determination module.

The range determining module is used for determining a plurality of risk ranges according to a plurality of index values in the risk level table, wherein each risk range has a corresponding processing scheme for the system.

And the scheme determining module is used for determining a target range in which the risk value is located from the multiple risk ranges so as to process the system based on a processing scheme corresponding to the target range.

Any of the information acquisition module 410, the transaction determination module 420, the impact determination module 430, the failure determination module 440, and the risk determination module 450 may be combined in one module to be implemented, or any of the modules may be split into multiple modules, according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. At least one of the information acquisition module 410, the transaction determination module 420, the impact determination module 430, the failure determination module 440, and the risk determination module 450 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three. Alternatively, at least one of the information acquisition module 410, the transaction determination module 420, the impact determination module 430, the failure determination module 440, and the risk determination module 450 may be implemented at least in part as computer program modules that, when executed, perform the corresponding functions.

As shown in fig. 5, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. The processor 501 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 501 may also include on-board memory for caching purposes. The processor 501 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 503, various programs and data required for the operation of the electronic apparatus 500 are stored. The processor 501, ROM 502, and RAM 503 are connected to each other by a bus 504. The processor 501 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 502 and/or the RAM 503. Note that the program may be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 500 may also include an input/output (I/O) interface 505, the input/output (I/O) interface 505 also being connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to an input/output (I/O) interface 505: an input section 506 including a keyboard, a mouse, and the like; an output portion 507 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The drive 510 is also connected to an input/output (I/O) interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as needed so that a computer program read therefrom is mounted into the storage section 508 as needed.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 502 and/or RAM 503 and/or one or more memories other than ROM 502 and RAM 503 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code, when executed in a computer system, causes the computer system to implement the system performance assessment method provided by embodiments of the present disclosure.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 501. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, and/or installed from a removable medium 511 via the communication portion 509. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509, and/or installed from the removable media 511. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 501. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A system performance evaluation method, comprising:

acquiring operation information of a system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal;

For each problem period, determining a first transaction summary value for the problem period and a second transaction summary value for a plurality of historical periods corresponding to the problem period;

determining a system influence duration of the system during the problem period based on the first transaction cumulative value and a plurality of the second transaction cumulative values;

determining a deployment failure rate based on a plurality of version deployment times and patch release times in the deployment data;

and determining a risk value of the system in the preset period according to a plurality of system influence time periods, the accumulated alarm time periods and the deployment failure rate, so as to evaluate the performance of the system based on the risk value.

2. The method of claim 1, wherein prior to the determining the first transaction summary value over the problem period and the second transaction summary value over the plurality of historical periods corresponding to the problem period, further comprising:

determining a problem occurrence date corresponding to the problem time period for each of the problem time periods;

determining a plurality of history dates according to the problem occurrence date;

and determining a historical time period corresponding to the problem time period in each historical date.

3. The method of claim 1, wherein the determining a system impact duration of the system during the problem period based on the first transaction cumulative value and a plurality of the second transaction cumulative values comprises:

calculating a traffic loss rate based on the first transaction cumulative value and a plurality of the second transaction cumulative values;

and integrating the service loss rate and the problem duration corresponding to the problem time period to obtain the system influence duration.

4. The method of claim 3, wherein the calculating a traffic loss rate based on the first transaction cumulative value and a plurality of the second transaction cumulative values comprises:

making a difference between each second transaction accumulated value and the first transaction accumulated value to obtain a plurality of intermediate difference values;

the second transaction accumulated value corresponding to each intermediate difference value and the intermediate difference value is made a quotient to obtain a plurality of loss rates;

and taking the maximum value of the loss rates as the service loss rate.

5. The method of claim 1, wherein the determining a deployment failure rate based on a plurality of version deployment times and patch release times in the deployment data comprises:

determining a deployment state at each version deployment time based on a plurality of version deployment times and the patch release time, wherein the deployment state comprises a success state and a failure state;

Determining the total number of failures of which the deployment state is a failure state in a plurality of version deployment times in the preset period;

determining the total deployment times in the preset period based on a plurality of version deployment times;

and the total failure times and the total deployment times are used as the quotient to obtain the deployment failure rate.

6. The method of claim 5, wherein the determining a deployment status at each of the version deployment times based on a plurality of the version deployment times and the patch release times comprises:

determining whether the patch release time which is the same as the version deployment time exists for each version deployment time;

determining a historical deployment time corresponding to the version deployment time when the patch release time same as the version deployment time exists, wherein the historical deployment time is a deployment time which is before the version deployment time and is closest to the version deployment time in a plurality of version deployment times;

and determining the deployment state of the historical deployment time as the failure state.

7. The method of claim 1, wherein the determining a risk value for the system over the preset period based on a plurality of the system impact durations, the cumulative alert duration, and the deployment failure rate comprises:

Summing a plurality of system influence durations in the preset period to obtain a period influence duration;

converting the period influence duration and the accumulated alarm duration into values of a preset time unit to obtain a target influence duration and a target alarm duration;

determining a target failure value based on a preset weight and the deployment failure rate;

and summing the target influence time, the target alarm time and the target failure value to obtain a risk value of the system in the preset period.

8. The method of claim 1, further comprising:

determining a plurality of risk ranges according to a plurality of index values in a risk level table, wherein each risk range has a corresponding processing scheme for the system;

and determining a target range in which the risk value is located from a plurality of risk ranges, so as to process the system based on a processing scheme corresponding to the target range.

9. A system performance evaluation device, comprising:

the information acquisition module is used for acquiring operation information of a system in a preset period, wherein the operation information comprises abnormal operation data of the system and deployment data generated by version deployment in the system, and the abnormal operation data comprises a plurality of problem time periods when the system is abnormal and accumulated alarm duration when a target transaction index in the system is abnormal;

A transaction determination module for determining, for each problem period, a first transaction cumulative value for the problem period and a second transaction cumulative value for a plurality of historical periods corresponding to the problem period;

an impact determination module for determining a system impact duration of the system during the problem period based on the first transaction cumulative value and a plurality of the second transaction cumulative values;

the failure determining module is used for determining a deployment failure rate based on a plurality of version deployment times and patch release times in the deployment data;

and the risk determining module is used for determining a risk value of the system in the preset period according to a plurality of system influence time periods, the accumulated alarm time periods and the deployment failure rate so as to evaluate the performance of the system based on the risk value.

10. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.

11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-8.

12. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.