CN114397876A - Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium - Google Patents

Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium Download PDF

Info

Publication number
CN114397876A
CN114397876A CN202210077658.7A CN202210077658A CN114397876A CN 114397876 A CN114397876 A CN 114397876A CN 202210077658 A CN202210077658 A CN 202210077658A CN 114397876 A CN114397876 A CN 114397876A
Authority
CN
China
Prior art keywords
detection unit
domain controller
test sample
performance
planning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210077658.7A
Other languages
Chinese (zh)
Inventor
吕博
李曙方
缪新育
潘峰
程华
胡昌军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Academy of Information and Communications Technology CAICT
Original Assignee
China Academy of Information and Communications Technology CAICT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Academy of Information and Communications Technology CAICT filed Critical China Academy of Information and Communications Technology CAICT
Priority to CN202210077658.7A priority Critical patent/CN114397876A/en
Publication of CN114397876A publication Critical patent/CN114397876A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B23/00—Testing or monitoring of control systems or parts thereof
    • G05B23/02—Electric testing or monitoring
    • G05B23/0205—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults
    • G05B23/0218—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterised by the fault detection method dealing with either existing or incipient faults
    • G05B23/0256—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterised by the fault detection method dealing with either existing or incipient faults injecting test signals and analyzing monitored process response, e.g. injecting the test signal while interrupting the normal operation of the monitored system; superimposing the test signal onto a control signal during normal operation of the monitored system
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/20—Pc systems
    • G05B2219/24—Pc safety
    • G05B2219/24065—Real time diagnostics

Abstract

The application relates to the technical field of domain controller testing, and discloses a method for evaluating the performance of a domain controller, which comprises the following steps: obtaining M.N verification results of the detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved. The application also discloses a device, an electronic device and a storage medium for evaluating the performance of the domain controller.

Description

Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium
Technical Field
The present application relates to the field of domain controller testing technologies, and for example, to a method and an apparatus for evaluating performance of a domain controller, an electronic device, and a storage medium.
Background
At present, with the rapid development of the 5G technology, a planning unit of a domain controller is generally adopted to perform network planning to obtain a planning result, and a detection unit is used to determine whether the logic of a network link in the planning result is contradictory, whether configuration is missed, whether a timing loop exists, and the like, so as to obtain a verification result of the detection unit on the planning result. However, the performance of the domain controller is directly evaluated through the verification result, and since whether the verification result is correct or not is unknown, the performance evaluation result of the domain controller cannot be accurately obtained.
In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art: and the performance of the domain controller is directly evaluated through the verification result, and the accuracy of the obtained performance evaluation result of the domain controller is poor.
Disclosure of Invention
The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview nor is intended to identify key/critical elements or to delineate the scope of such embodiments but rather as a prelude to the more detailed description that is presented later.
The embodiment of the disclosure provides a method and a device for evaluating the performance of a domain controller, an electronic device and a storage medium, so as to improve the accuracy of an obtained performance evaluation result of the domain controller.
In some embodiments, a method for evaluating performance of a domain controller, the domain controller comprising a planning unit and N detection units, N being an integer greater than 1, the method comprising: obtaining M.N verification results of the detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result.
In some embodiments, the apparatus for evaluating performance of a domain controller comprises a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the above-described method for evaluating performance of a domain controller.
In some embodiments, the electronic device comprises the above-mentioned means for evaluating the performance of the domain controller.
In some embodiments, the storage medium stores program instructions that, when executed, perform the above-described method for evaluating domain controller performance.
The method and the device for evaluating the performance of the domain controller, the electronic equipment and the storage medium provided by the embodiment of the disclosure can realize the following technical effects: obtaining M.N verification results of the detection unit; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved.
The foregoing general description and the following description are exemplary and explanatory only and are not restrictive of the application.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the accompanying drawings and not in limitation thereof, in which elements having the same reference numeral designations are shown as like elements and not in limitation thereof, and wherein:
FIG. 1 is a schematic diagram of a method for evaluating the performance of a domain controller according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of another method for evaluating the performance of a domain controller provided by an embodiment of the present disclosure;
FIG. 3 is a schematic structural diagram of an evaluation device provided in the embodiments of the present disclosure;
FIG. 4 is a diagram illustrating a verification result and initial parameters provided by an embodiment of the disclosure;
fig. 5 is a schematic diagram of a verification result and an evaluation result in the case that one test round is 5 according to the embodiment of the present disclosure;
fig. 6 is a schematic diagram of a verification result and an evaluation result in the case that a test round is 100 according to an embodiment of the present disclosure;
fig. 7 is a schematic diagram of a verification result and an evaluation result in the case that one test round is 1000 according to the embodiment of the present disclosure;
fig. 8 is a schematic diagram of an apparatus for evaluating performance of a domain controller according to an embodiment of the present disclosure.
Reference numerals:
100: a test sample set storage module; 101: a network manager/controller; 102: a synchronization network; 103: a planning unit of the software defined synchronous network domain controller; 104: a detection unit of the software defined synchronous network domain controller; 105: an evaluation unit; 106: the software defines a synchronous network domain controller.
Detailed Description
So that the manner in which the features and elements of the disclosed embodiments can be understood in detail, a more particular description of the disclosed embodiments, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. In the following description of the technology, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown in simplified form in order to simplify the drawing.
The terms "including" and "having," and any variations thereof, in the description and claims of embodiments of the present disclosure and the above-described drawings are intended to cover non-exclusive inclusions.
The term "plurality" means two or more unless otherwise specified.
In the embodiment of the present disclosure, the character "/" indicates that the preceding and following objects are in an or relationship. For example, A/B represents: a or B.
The term "correspond" may refer to an association or binding relationship, and a corresponds to B refers to an association or binding relationship between a and B.
The present application is applicable to a system or software for evaluating the performance of a domain controller. Generally, a software-defined synchronous network domain controller comprises a synchronous network planning unit and a synchronous network detection unit, wherein the synchronous network planning unit can automatically plan information such as a synchronous link path and synchronous port configuration priority according to topological connection and resource information of a network; the synchronous network detection unit can detect the integrity of the actual operation or offline planned synchronous operation state of the synchronous network, including the problems of missing configuration, timing loop, synchronous hop number out-of-limit and the like. The performance of the domain controller is evaluated, i.e. the planning unit or the detection unit of the software defined synchronous network domain controller is evaluated.
Meanwhile, the electronic device according to the embodiment of the present invention may include, but is not limited to, a mobile phone, a tablet computer, a personal computer, a palmtop computer, a server, and the like.
With reference to fig. 1, an embodiment of the present disclosure provides a method for evaluating performance of a domain controller, where the domain controller includes a planning unit and N detection units, where N is an integer greater than 1, and the method for evaluating performance of the domain controller includes:
s101, the electronic equipment obtains M.N verification results of a detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1;
s102, the electronic equipment determines the credible reference values of M test samples according to M.N verification results and initial parameters;
step S103, the electronic equipment evaluates the performance of the domain controller according to the M credible reference values and the initial parameters;
and step S104, the electronic equipment outputs an evaluation result.
By adopting the method for evaluating the performance of the domain controller provided by the embodiment of the disclosure, M.N verification results of the detection unit are obtained; determining the credible reference values of the M test samples according to the M.N verification results; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved.
Alternatively, M · N represents M multiplied by N.
In some embodiments, the electronic device receives a verification result sent by each detection unit of the domain controller; or the electronic equipment calls a preset verification result database and acquires a verification result from the preset verification result database; a plurality of verification results are stored in a preset verification result database.
Optionally, the verification result is 0 or 1; 0 is used for representing that the planning result of the planning unit by the detection unit is verified as not passing; 1 is used for representing that the planning result of the detection unit to the planning unit is verification passing.
Optionally, the initial parameters include initial weights corresponding to the N detection units, respectively; determining a credible reference value of the test sample according to the verification result and the initial parameters, wherein the credible reference value comprises the following steps: computing
Figure BDA0003484682940000051
Obtaining a credible reference value of the test sample; wherein R isj'(pk,sj) Is the credible reference value of the jth test sample; w'iIs the initial weight corresponding to the ith detection unit; r (D)i|pk,sj) The verification result of the planning result of the ith test sample of the planning unit by the ith detection unit; "·" denotes a multiplication number. Therefore, the credible reference value of each test sample can be automatically determined through the initial parameters and the verification results of each detection unit, and the efficiency of the performance evaluation result of the obtained domain controller is improved.
In some casesIn the examples, R (D)i|pk,sj) The verification result of the planning result of the ith test sample of the planning unit by the ith detection unit is shown. I.e. s in the test samplejIn case of (2), plan unit pkAccording to test specimen sjPlanning to obtain planning result pk,sj(ii) a Detecting unit DiFor the planning result pk,sjThe verification result of (2) is R (D)i|pk,sj)。
In some embodiments, Rj'(pk,sj) Is the confidence reference value of the jth test sample. I.e. the jth test sample is in the programming unit pkIs determined.
In some embodiments, the domain controller comprises 4 detection units. Determining a planning result of the planning unit according to the test sample 1 as a planning result of the test sample 1; obtaining that the initial weight of the detection unit D1 is 0.25, and the verification result of the detection unit D1 on the planning result of the test sample 1 is 0; obtaining that the initial weight of the detection unit D2 is 0.25, and the verification result of the detection unit D2 on the planning result of the test sample 1 is 1; obtaining that the initial weight of the detection unit D3 is 0.25, and the verification result of the detection unit D3 on the planning result of the test sample 1 is 1; the initial weight of the detecting unit D4 is 0.25, and the verification result of the detecting unit D4 on the planning result of the test sample 1 is 1. Computing
Figure BDA0003484682940000052
Obtaining the weighting result of each detection unit corresponding to the test sample 1, namely: 0.25 × 1+0.25 × 0+0.25 × 1+0.25 × 1 ═ 0.75; since the weighting result 0.75 is greater than 0.5, the confidence reference value of the test sample 1 is 1.
Optionally, evaluating the domain controller performance according to the M trusted benchmark values and the initial parameters includes: respectively determining the type of each verification result according to the verification result corresponding to the test sample and the credible reference value; types include true negative, true positive and false positive; counting the number of true negatives, the number of true positives and the number of false positives in the verification result respectively corresponding to each detection unit; and determining the evaluation value of each detection unit and the corresponding weight of each detection unit according to the number of true negatives, the number of true positives, the number of false positives and the initial parameters. Therefore, the accuracy of the verification result corresponding to each detection unit can be determined by determining the type of the verification result and counting the number of the types of the verification results corresponding to each detection unit. Therefore, the performance of each detection unit of the domain controller can be evaluated quantitatively, and improvement of each detection unit by a user according to the evaluation value of each detection unit is facilitated. Meanwhile, the user can conveniently select a proper detection unit to carry out network planning verification.
Optionally, the trusted reference value is 0 or 1; respectively determining the type of each verification result according to the verification result corresponding to the test sample and the credible reference value, wherein the method comprises the following steps: determining the type of the verification result as true positive under the condition that the credible reference value and the verification result corresponding to the test sample are both 1; determining the type of the verification result as true negative under the condition that the credible reference value corresponding to the test sample and the verification result are both 0; and determining the type of the verification result as false positive under the condition that the credible reference value corresponding to the test sample is 0 and the verification result is 1.
Optionally, the initial parameters further include detection unit initial parameters; determining the evaluation value of the detection unit and the corresponding weight of the detection unit according to the number of true negatives, the number of true positives, the number of false positives and the initial parameter, wherein the method comprises the following steps: computing
Figure BDA0003484682940000061
Obtaining the accuracy of the detection unit; computing
Figure BDA0003484682940000062
Obtaining a detecting unit recall rate; computing
Figure BDA0003484682940000063
Obtaining a detection unit score; computing
Figure BDA0003484682940000064
Obtaining an evaluation value of a detection unit; computing
Figure BDA0003484682940000065
Obtaining the weight corresponding to the detection unit; wherein precision _ rate is the detection unit accuracy, TP (D)i) The number of true positives exists in the verification result corresponding to the ith detection unit; TN (D)i) The number of true negatives existing in the verification result corresponding to the ith detection unit; FP (D)i) The number of false positives existing in the verification result corresponding to the ith detection unit is obtained; the call _ rate is the recall rate of the ith detection unit; f _ score (D)i) Scoring the ith detection unit; f _ score (D)g) Scoring the g-th detection unit; p (D)i) An i-th detection unit evaluation value; p' (D)i) Initial parameters of the ith detection unit; iter is the test run; w is aiThe weight corresponding to the ith detection unit.
Optionally, the test run is greater than or equal to 1.
In some embodiments, the test run is 1; the initial parameter of the detecting unit D3 was 0.5. Determining a planning result of the planning unit according to the test sample 3 as a planning result of the test sample 3; and determining a planning result of the planning unit according to the test sample 4 as a planning result of the test sample 4. Obtaining a credible reference value of 1 of a test sample 3; obtaining a credible reference value of 0 of the test sample 4; obtaining that the verification result a of the test unit D3 on the planning result of the test sample 3 is 1; obtaining a verification result b of the test unit D3 on the planning result of the test sample 4 as 1; obtaining a verification result c of the test unit D4 on the planning result of the test sample 3 as 1; the verification result D of the planning result of the acquisition detecting unit D4 for the test sample 4 is 0. If the verification result a corresponding to the test sample 3 is 1 and the credible reference value is 1, the type of the verification result a is true positive; if the verification result b corresponding to the test sample 3 is 1 and the credible reference value is 1, the type of the verification result b is true positive; if the verification result c corresponding to the test sample 4 is 1 and the credible reference value is 0, the type of the verification result c is false positive; and if the verification result d corresponding to the test sample 4 is 0 and the credible reference value is 0, the type of the verification result d is true negative. The number of true positives corresponding to detection cell D3 is 2; the number of false positives corresponding to detection cell D4 is 1; number of true negatives corresponding to detection cell D4The amount is 1. By calculation of
Figure BDA0003484682940000071
The accuracy of the detection unit D3 is 1; by calculation of
Figure BDA0003484682940000081
Obtaining the recall rate of the detection unit D3 as 1; by calculation of
Figure BDA0003484682940000082
Obtaining a detection unit D3 with a detection unit score of 1; by calculation of
Figure BDA0003484682940000083
The evaluation value of the detection unit D3 was obtained as 0.75.
Optionally, the initial parameters further include planning unit initial parameters; evaluating the performance of the domain controller according to each credible reference value and the initial parameter, and further comprising the following steps: computing
Figure BDA0003484682940000084
Obtaining a planning unit score of a domain controller; computing
Figure BDA0003484682940000085
Obtaining a planning unit evaluation value of a domain controller; wherein, score (p)k) Is the planning unit score; p (P)k) Evaluating the value for the planning unit; p' (P)k) Initial parameters for the planning unit; . In this way, the credible reference value of the test sample is determined by each verification result, that is, the verification result of the programming unit for the test sample is verified or not verified can be determined. And determining the successful planning condition corresponding to the planning unit by calculating the passing condition of the planning unit for the verification of the planning result of the test sample. Therefore, the performance of the planning unit of the domain controller can be evaluated quantitatively, and improvement of each planning unit by a user according to the evaluation value of the planning unit is facilitated. Meanwhile, the user can conveniently select a proper planning unit to carry out network planning.
In some embodiments, the test run is 1; the initial parameter of the planning unit is 1;obtaining a confidence reference value of 5 test samples, for example: the credible reference value of the test sample 1 is 1; the credible reference value of the test sample 2 is 0; the credible reference value of the test sample 3 is 1; the credible reference value of the test sample 4 is 1; the test sample 5 has a confidence reference value of 0. By calculation of
Figure BDA0003484682940000086
Namely: (1+0+1+1+0)/5 ═ 3/5; the plan unit of the domain controller is obtained 3/5. By calculation of
Figure BDA0003484682940000091
The planned cell evaluation value was obtained as 0.8.
Optionally, the initial parameters further include test sample initial parameters; after the performance of the domain controller is evaluated according to the M credible reference values and the initial parameters, the method further comprises the following steps: computing
Figure BDA0003484682940000092
Obtaining a test sample score; computing
Figure BDA0003484682940000093
Obtaining a test sample evaluation value; wherein, score(s)j) Scoring the jth test sample; p(s)j) Evaluating the jth test sample; p'(s)j) Initial parameters for the jth test sample. In this way, each test sample can be quantitatively evaluated, so that a user can determine the difficulty level of each test sample according to the test sample evaluation value.
In some embodiments, the test run is 1; the domain controller includes 4 detection units and 1 planning unit. The initial parameters of the test sample 7 were 20; determining a planning result of the planning unit according to the test sample 7 as a planning result of the test sample 7; obtaining the initial weight of the detection unit D1 as 0.25, and the verification result of the planning result of the test sample 7 as 1; obtaining the initial weight of the detection unit D2 as 0.25, and the verification result of the planning result of the test sample 7 as 1; obtaining the initial weight of the detection unit D3 as 0.25, and the verification result of the planning result of the test sample 7 as 1; obtaining initial weight of detection cell D40.25, and 1 for the verification result of the test sample 7 plan result. Computing
Figure BDA0003484682940000094
Obtaining a test sample 7 score of 0; computing
Figure BDA0003484682940000095
The test sample 7 was evaluated to be 10.
In some embodiments, in the case that the test round is 1, obtaining initial parameters by searching a preset parameter database; the preset parameter database stores initial parameters of the detection units, initial parameters of the planning units, initial parameters of the test samples and initial weights corresponding to the detection units respectively.
Optionally, after the output of the evaluation result, the method further includes: obtaining a test round; displaying the evaluation result to a user under the condition that the test round reaches a preset round; accumulating the test rounds under the condition that the test rounds do not reach the preset rounds; determining an evaluation result as an initial parameter; obtaining M.N verification results of the detection unit again; determining the credible reference values of the M test samples according to the M.N verification results; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, by iterating the evaluation result, the domain controller performance can be evaluated more accurately.
With reference to fig. 2, another method for evaluating performance of a domain controller is provided in an embodiment of the present disclosure, where the domain controller includes a planning unit and N detection units, where N is an integer greater than 1, and the method for evaluating performance of the domain controller includes:
step S201, the electronic equipment obtains M.N verification results of the detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1; then step S202 is performed.
Step S202, the electronic equipment determines the credible reference values of the M test samples according to the M.N verification results and the initial parameters; then step S203 is performed.
Step S203, the electronic equipment evaluates the performance of the domain controller according to the M credible reference values and the initial parameters; then, step S204 is performed.
Step S204, the electronic equipment outputs an evaluation result; then step S205 is performed.
Step S205, the electronic equipment acquires a test round; step S206 is then performed.
Step S206, the electronic equipment judges whether the test round reaches a preset round or not; executing step S208 when the test round reaches the preset round; in case the test round does not reach the preset round, step S207 is executed.
In step S207, the electronic device determines the evaluation result as an initial parameter, and then performs step S201.
And step S208, the electronic equipment displays the evaluation result to the user.
By adopting the method for evaluating the performance of the domain controller, M.N verification results of the detection unit are obtained by obtaining the test round and under the condition that the test round does not reach the preset round; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result, and determining the evaluation result as an initial parameter until the test round reaches a preset round. Therefore, the performance of the domain controller can be more accurately evaluated by performing loop iteration on the evaluation result.
Optionally, determining the evaluation result as an initial parameter includes: determining the weight corresponding to the detection unit as the initial weight corresponding to the detection unit; determining the evaluation value of the detection unit as an initial parameter of the detection unit; determining the evaluation value of the planning unit as an initial parameter of the planning unit; and determining the evaluation value of the test sample as the initial parameter of the test sample.
Optionally, presenting the evaluation result to the user includes: and pushing the evaluation result to a preset client.
Optionally, presenting the evaluation result to the user includes: and sending the evaluation result to a preset display screen, and triggering the display screen to display the evaluation result.
In some embodiments, fig. 3 is a schematic structural diagram of an evaluation device. As shown in fig. 3, the test sample set storage module 100 is connected to a network manager or a controller 101; the network management or controller 101 is connected with the synchronous network 102 through a network management interface or a southbound interface; the synchronous network 102 is connected with a planning unit 103 of the software-defined synchronous network domain controller through a southbound interface, and the planning unit 103 of the software-defined synchronous network domain controller is connected with a detection unit 104 of the software-defined synchronous network domain controller through a northbound interface; the evaluation unit 105 is connected to the software-defined synchronous network domain controller 106. The network management or the controller calls the test samples from the test sample set storage module and loads the test samples into a network management configuration file or an SDN controller configuration file; a network manager or a controller maps configuration information corresponding to a network management configuration file or an SDN controller configuration file to a synchronous network through a network management interface or a southbound interface; the planning unit acquires configuration and operation information from the synchronous network through a southbound interface, and performs network planning to acquire a planning result; the planning unit sends the planning result to the plurality of detection units, and the detection units verify the planning result to obtain a verification result. And the software-defined synchronous network domain controller sends the verification result to the evaluation unit. The evaluation unit, when running, performs a method for evaluating the performance of the domain controller.
In some embodiments, the evaluation unit 105 is connected to the test sample set storage module 100, and the evaluation unit marks the corresponding test sample evaluation value on the test sample in the test sample set storage module.
In some embodiments, the test samples include synchronization device type, bearer device type, number and scale, synchronization time/clock source injection point location, network topology connection, network planning strategy, and the like. Optionally, the synchronization device types include a time server, a frequency synchronization source, a bearer network synchronization unit, a time allocation unit, and the like. The network manager/controller is a network manager or controller of the synchronous device/bearer device, and is used for mapping the configuration information of the test sample to the synchronous network. The synchronous network is an actual physical network or an offline virtual network entity composed of a synchronous time/clock source, a bearing device and the like. The planning unit of the software-defined synchronous network domain controller is a controller unit meeting the requirements specified in YD/T3700-2020 software-defined synchronous network technical requirements, and has the automatic planning capability of a synchronous network; the software-defined synchronous network domain controller-detection unit is a controller unit meeting the requirements specified in YD/T3700-2020 software-defined synchronous network technical requirements, and has the automatic detection capability of a synchronous network.
In some embodiments, domain controller performance is evaluated using 13 test samples, a planning unit of 1 domain controller, and a detection unit of 5 domain controllers. The method comprises the steps of firstly obtaining planning results of a domain controller for 13 test samples respectively, and then verifying the 13 planning results through 5 detection units respectively to obtain 65 verification results. Then, a confidence reference value for the 13 test samples is determined based on the 65 validation results and the initial parameters. And finally, evaluating the performance of the controller according to the credible reference value and the initial parameter, namely acquiring the evaluation value of the detection unit, the weight corresponding to the detection unit, the evaluation value of the planning unit and the evaluation value of the test sample. Fig. 4 is a schematic diagram of the verification result and the initial parameters, and num is the number of the test sample as shown in fig. 4; result _ d1 is the verification result of detecting unit 1; result _ d2 is the verification result of the detection unit 2; result _ d3 is the verification result of the detection unit 3; result _ d4 is the verification result of the detection unit 4; result _ d5 is the verification result of the detection unit 5; result _ hat is a test sample credible reference value; sample _ hard _ score is a test sample evaluation value; f1_ d1 is the detection unit 1 evaluation value; f1_ d2 is the detection unit 2 evaluation value; f1_ d3 is the detection unit 3 evaluation value; f1_ d4 is the detection unit 4 evaluation value; f1_ d5 is the detection unit 5 evaluation value; weight _ d1 is the weight of detection cell 1; weight _ d2 is the weight of detection cell 2; weight _ d3 is the weight of detection cell 3; weight _ d4 is the weight of detection unit 4; weight _ d5 is the weight of detection cell 5. The verification result of the detection unit 1 on the test sample 2 is 0, and the verification result of the detection unit 2 on the test sample 2 is 0; the verification result of the detection unit 3 on the test sample 2 is 0; the verification result of the detection unit 4 on the test sample 2 is 1; the verification result of the detection unit 5 on the test sample 2 is 0; the initial credible reference value of the test sample 2 is 0; the initial test sample evaluation value of test sample 2 is 35.736373; the initial detection unit evaluation values of the detection unit 1, the detection unit 2, the detection unit 3, the detection unit 4 and the detection unit 5 are all 0.2; the initial weights of the detection unit 1, the detection unit 2, the detection unit 3, the detection unit 4, and the detection unit 5 are all 0.2.
In some embodiments, after the electronic device outputs the evaluation result, a test round is obtained, whether the test round reaches a preset round is judged, and the performance of the domain estimation controller is reevaluated under the condition that the test round does not reach the preset round. In the case that the number of test rounds is 5, as shown in fig. 5, the verification result of the test sample 5 by the detecting unit 1 is 1, and the verification result of the test sample 5 by the detecting unit 2 is 1; the verification result of the detection unit 3 on the test sample 5 is 1; the verification result of the detection unit 4 on the test sample 5 is 0; the verification result of the test sample 5 by the detection unit 5 is 1; the credible reference value of the test sample 5 is 1.0; the test sample evaluation value of the test sample 5 is 32.882007; the detection unit 1 evaluates to 0.727273; the detection unit 2 evaluates to 0.833333; the detection unit 3 evaluates to 0.8; the detection unit 4 evaluates to 0.5; the detection unit 5 evaluates to 0.8; detection unit 1 weight 0.184376; detection element 2 has a weight of 0.226845; detection unit 3 has a weight of 0.217727; detection unit 4 is weighted 0.174943; detection unit 5 has a weight of 0.196109. In the case where the number of test rounds is 100, the verification results and the evaluation results are shown in fig. 6. In the case where the number of test rounds is 1000, the verification results and the evaluation results are shown in fig. 7.
In some embodiments, different verification results are used to evaluate the performance of the domain controller in general. For example: acquiring a verification result set A, wherein a plurality of verification results A are stored in the verification result set A; the verification result A is obtained by verifying the planning result of the planning unit according to the test sample set A by the detection unit; a plurality of test samples are stored in the test sample set A; and obtaining an evaluation result A according to the verification result set A. Acquiring a verification result set B, wherein a plurality of verification results B are stored in the verification result set B; the verification result B is obtained by verifying the planning unit according to the planning result of the test sample set B by the detection unit; a plurality of test samples are stored in the test sample set B; and obtaining an evaluation result B according to the verification result set B. And comprehensively evaluating the performance of the domain controller according to the evaluation result A and the evaluation result B. The test samples stored in the test sample set A and the test samples stored in the test sample set B are not identical or completely different.
As shown in fig. 8, an apparatus for evaluating performance of a domain controller according to an embodiment of the present disclosure includes a processor (processor)800 and a memory (memory) 801. Optionally, the apparatus may also include a Communication Interface 802 and a bus 803. The processor 800, the communication interface 802, and the memory 801 may communicate with each other via a bus 803. Communication interface 802 may be used for information transfer. The processor 800 may call logical instructions in the memory 801 to perform the method for evaluating the performance of the domain controller of the above-described embodiments.
By adopting the device for evaluating the performance of the domain controller provided by the embodiment of the disclosure, M.N verification results of the detection unit are obtained; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved.
In addition, the logic instructions in the memory 801 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand-alone product.
The memory 801 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 100 executes functional applications and data processing, i.e., implements the method for evaluating the performance of the domain controller in the above-described embodiments, by executing program instructions/modules stored in the memory 801.
The memory 801 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. In addition, the memory 801 may include a high-speed random access memory, and may also include a nonvolatile memory.
The embodiment of the disclosure provides an electronic device, which comprises the above device for evaluating the performance of the domain controller.
Alternatively, the electronic device comprises a computer or server or the like.
By adopting the electronic equipment provided by the embodiment of the disclosure, M.N verification results of the detection unit are obtained; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved.
The disclosed embodiments provide a storage medium storing program instructions that, when executed, perform the above-described method for evaluating domain controller performance.
The aforementioned storage medium may be a non-transitory storage medium comprising: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes, and may also be a transient storage medium.
The above description and drawings sufficiently illustrate embodiments of the disclosure to enable those skilled in the art to practice them. Other embodiments may incorporate structural, logical, electrical, process, and other changes. The examples merely typify possible variations. Individual components and functions are optional unless explicitly required, and the sequence of operations may vary. Portions and features of some embodiments may be included in or substituted for those of others. Furthermore, the words used in the specification are words of description only and are not intended to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and/or" as used in this application is meant to encompass any and all possible combinations of one or more of the associated listed. Furthermore, the terms "comprises" and/or "comprising," when used in this application, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Without further limitation, an element defined by the phrase "comprising an …" does not exclude the presence of other like elements in a process, method or apparatus that comprises the element. In this document, each embodiment may be described with emphasis on differences from other embodiments, and the same and similar parts between the respective embodiments may be referred to each other. For methods, products, etc. of the embodiment disclosures, reference may be made to the description of the method section for relevance if it corresponds to the method section of the embodiment disclosure.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In the description corresponding to the flowcharts and block diagrams in the figures, operations or steps corresponding to different blocks may also occur in different orders than disclosed in the description, and sometimes there is no specific order between the different operations or steps. For example, two sequential operations or steps may in fact be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. Each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claims (10)

1. A method for evaluating the performance of a domain controller, said domain controller comprising a planning unit and N detection units, N being an integer greater than 1, characterized in that said method comprises:
obtaining M.N verification results of the detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1;
determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters;
evaluating the performance of the domain controller according to the M credible reference values and the initial parameters;
and outputting an evaluation result.
2. The method according to claim 1, wherein the initial parameters include initial weights corresponding to the N detection units respectively; determining a credible reference value of the test sample according to the verification result and the initial parameters, wherein the credible reference value comprises the following steps:
computing
Figure FDA0003484682930000011
Obtaining a credible reference value of the test sample;
wherein R isj'(pk,sj) Is the credible reference value of the jth test sample; w'iIs the initial weight corresponding to the ith detection unit; r (D)i|pk,sj) Is the ith test sheetAnd the meta-pair planning unit verifies the planning result according to the jth test sample.
3. The method of claim 1, wherein evaluating domain controller performance based on the M trusted benchmark values and the initial parameters comprises:
respectively determining the type of each verification result according to the verification result corresponding to the test sample and the credible reference value; the types include true negative, true positive and false positive;
counting the number of true negatives, the number of true positives and the number of false positives in the verification result respectively corresponding to each detection unit;
and determining the evaluation value of each detection unit and the corresponding weight of each detection unit according to the number of true negatives, the number of true positives, the number of false positives and the initial parameters.
4. The method of claim 3, wherein the initial parameters further comprise detection cell initial parameters; determining the evaluation value of the detection unit and the corresponding weight of the detection unit according to the number of true negatives, the number of true positives, the number of false positives and the initial parameter, wherein the method comprises the following steps:
computing
Figure FDA0003484682930000021
Obtaining the accuracy of the detection unit;
computing
Figure FDA0003484682930000022
Obtaining a detecting unit recall rate;
computing
Figure FDA0003484682930000023
Obtaining a detection unit score;
computing
Figure FDA0003484682930000024
Obtaining an evaluation value of a detection unit;
computing
Figure FDA0003484682930000025
Obtaining the weight corresponding to the detection unit;
wherein precision _ rate is the detection unit accuracy, TP (D)i) The number of true positives exists in the verification result corresponding to the ith detection unit; TN (D)i) The number of true negatives existing in the verification result corresponding to the ith detection unit; FP (D)i) The number of false positives existing in the verification result corresponding to the ith detection unit is obtained; the call _ rate is the recall rate of the ith detection unit; f _ score (D)i) Scoring the ith detection unit; f _ score (D)g) Scoring the g-th detection unit; p (D)i) An i-th detection unit evaluation value; p' (D)i) Initial parameters of the ith detection unit; iter is the test run; w is aiThe weight corresponding to the ith detection unit.
5. The method of claim 3, wherein the initial parameters further comprise planning unit initial parameters; evaluating the performance of the domain controller according to each credible reference value and the initial parameter, and further comprising the following steps:
computing
Figure FDA0003484682930000026
Obtaining a planning unit score of a domain controller;
computing
Figure FDA0003484682930000027
Obtaining a planning unit evaluation value of a domain controller;
wherein, score (p)k) Is the planning unit score; p (P)k) Evaluating the value for the planning unit; p' (P)k) Initial parameters are for the planning unit.
6. The method of claim 3, wherein the initial parameters further comprise test sample initial parameters; after the performance of the domain controller is evaluated according to the M credible reference values and the initial parameters, the method further comprises the following steps: computing
Figure FDA0003484682930000031
Obtaining a test sample score;
computing
Figure FDA0003484682930000032
Obtaining a test sample evaluation value;
wherein, score(s)j) Scoring the jth test sample; p(s)j) Evaluating the jth test sample; p'(s)j) Initial parameters for the jth test sample.
7. The method according to any one of claims 1 to 6, wherein after outputting the evaluation result, further comprising:
obtaining a test round;
displaying the evaluation result to a user under the condition that the test round reaches a preset round;
accumulating the test rounds under the condition that the test rounds do not reach the preset rounds; determining an evaluation result as an initial parameter; obtaining M.N verification results of the detection unit again; determining the credible reference values of the M test samples according to the M.N verification results; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result.
8. An apparatus for evaluating performance of a domain controller, comprising a processor and a memory having stored thereon program instructions, characterized in that the processor is configured to perform the method for evaluating performance of a domain controller according to any of claims 1 to 7 when executing said program instructions.
9. An electronic device, characterized in that it comprises means for evaluating the performance of a domain controller according to claim 8.
10. A storage medium storing program instructions, characterized in that said program instructions, when executed, perform the method for evaluating domain controller performance according to any of claims 1 to 7.
CN202210077658.7A 2022-01-24 2022-01-24 Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium Pending CN114397876A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210077658.7A CN114397876A (en) 2022-01-24 2022-01-24 Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210077658.7A CN114397876A (en) 2022-01-24 2022-01-24 Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium

Publications (1)

Publication Number Publication Date
CN114397876A true CN114397876A (en) 2022-04-26

Family

ID=81233390

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210077658.7A Pending CN114397876A (en) 2022-01-24 2022-01-24 Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium

Country Status (1)

Country Link
CN (1) CN114397876A (en)

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011041366A1 (en) * 2009-09-30 2011-04-07 Glasshouse Technologies, Inc. Computer software and hardware evaluation system and device
CN103370716A (en) * 2010-11-03 2013-10-23 维吉尼亚技术知识产权公司 Using power fingerprinting (PFP) to monitor the integrity and enhance security of computer based systems
WO2013182286A1 (en) * 2012-06-07 2013-12-12 Alcatel Lucent Secure data processing
US20150121148A1 (en) * 2012-07-03 2015-04-30 Hitachi, Ltd. Malfunction influence evaluation system and evaluation method
DE102015204301A1 (en) * 2015-03-11 2016-09-15 Volkswagen Aktiengesellschaft A test setup and emulation unit for a cell management controller of a battery module for use in a test setup for testing a battery management controller
US20170308869A1 (en) * 2016-04-21 2017-10-26 Continental Tide Defense Systems, Inc. Workflow, assessment, verification, and evaluation (wave) system and method
US20170366646A1 (en) * 2016-06-17 2017-12-21 Airwatch, Llc Remote providisioning and enrollment of enterprise devices with on-premises domain controllers
US10003555B1 (en) * 2016-05-06 2018-06-19 Amazon Technologies, Inc. Power management of routing tables using vertical scaling
CN108335719A (en) * 2018-02-24 2018-07-27 上海兆芯集成电路有限公司 Capability evaluating device and performance estimating method
CN109857095A (en) * 2019-03-25 2019-06-07 重庆长安新能源汽车科技有限公司 A kind of off-line detection system and method for domain controller
CN110165714A (en) * 2019-05-30 2019-08-23 广州水沐青华科技有限公司 Micro-capacitance sensor integration scheduling and control method, computer readable storage medium based on limit dynamic programming algorithm
CN111949316A (en) * 2020-08-18 2020-11-17 中国信息通信研究院 Method, device, system and computer-readable storage medium for evaluating software development operation maturity
WO2021037957A1 (en) * 2019-08-30 2021-03-04 Jaguar Land Rover Limited Layered electrical architecture for vehicle diagnostics
CN112583725A (en) * 2019-09-27 2021-03-30 中国电信股份有限公司 SDN (software defined network) network route determining method and system and SDN network system
US10985998B1 (en) * 2018-05-21 2021-04-20 Amazon Technologies, Inc. Domain controller configurability for directories
US20210148591A1 (en) * 2019-11-14 2021-05-20 Siemens Industry, Inc. Zone controller and method for identifying a root cause failure
CN112835341A (en) * 2020-12-31 2021-05-25 北京新能源汽车技术创新中心有限公司 Real vehicle test evaluation method of automatic driving area controller
CN113467420A (en) * 2021-06-30 2021-10-01 通号城市轨道交通技术有限公司 Method and device for detecting zone controller fault

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011041366A1 (en) * 2009-09-30 2011-04-07 Glasshouse Technologies, Inc. Computer software and hardware evaluation system and device
CN103370716A (en) * 2010-11-03 2013-10-23 维吉尼亚技术知识产权公司 Using power fingerprinting (PFP) to monitor the integrity and enhance security of computer based systems
WO2013182286A1 (en) * 2012-06-07 2013-12-12 Alcatel Lucent Secure data processing
US20150121148A1 (en) * 2012-07-03 2015-04-30 Hitachi, Ltd. Malfunction influence evaluation system and evaluation method
DE102015204301A1 (en) * 2015-03-11 2016-09-15 Volkswagen Aktiengesellschaft A test setup and emulation unit for a cell management controller of a battery module for use in a test setup for testing a battery management controller
US20170308869A1 (en) * 2016-04-21 2017-10-26 Continental Tide Defense Systems, Inc. Workflow, assessment, verification, and evaluation (wave) system and method
US10003555B1 (en) * 2016-05-06 2018-06-19 Amazon Technologies, Inc. Power management of routing tables using vertical scaling
US20170366646A1 (en) * 2016-06-17 2017-12-21 Airwatch, Llc Remote providisioning and enrollment of enterprise devices with on-premises domain controllers
CN108335719A (en) * 2018-02-24 2018-07-27 上海兆芯集成电路有限公司 Capability evaluating device and performance estimating method
US10985998B1 (en) * 2018-05-21 2021-04-20 Amazon Technologies, Inc. Domain controller configurability for directories
CN109857095A (en) * 2019-03-25 2019-06-07 重庆长安新能源汽车科技有限公司 A kind of off-line detection system and method for domain controller
CN110165714A (en) * 2019-05-30 2019-08-23 广州水沐青华科技有限公司 Micro-capacitance sensor integration scheduling and control method, computer readable storage medium based on limit dynamic programming algorithm
WO2021037957A1 (en) * 2019-08-30 2021-03-04 Jaguar Land Rover Limited Layered electrical architecture for vehicle diagnostics
CN112583725A (en) * 2019-09-27 2021-03-30 中国电信股份有限公司 SDN (software defined network) network route determining method and system and SDN network system
US20210148591A1 (en) * 2019-11-14 2021-05-20 Siemens Industry, Inc. Zone controller and method for identifying a root cause failure
CN111949316A (en) * 2020-08-18 2020-11-17 中国信息通信研究院 Method, device, system and computer-readable storage medium for evaluating software development operation maturity
CN112835341A (en) * 2020-12-31 2021-05-25 北京新能源汽车技术创新中心有限公司 Real vehicle test evaluation method of automatic driving area controller
CN113467420A (en) * 2021-06-30 2021-10-01 通号城市轨道交通技术有限公司 Method and device for detecting zone controller fault

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张平，等: "基于域控制器的硬件架构及FPD-Link Ⅲ关键技术研究", 设计研究, 31 October 2021 (2021-10-31), pages 37 - 42 *

Similar Documents

Publication Publication Date Title
CN106682906B (en) Risk identification and service processing method and equipment
CN111896914A (en) Cooperative positioning method, device, equipment and storage medium
CN107729227B (en) Application program test range determining method, system, server and storage medium
CN108959370B (en) Community discovery method and device based on entity similarity in knowledge graph
CN108388509B (en) Software testing method, computer readable storage medium and terminal equipment
CN106201856A (en) A kind of multi version performance test methods and device
CN110083581B (en) Log tracing method and device, storage medium and computer equipment
WO2019019356A1 (en) Application program test method and apparatus, computer device and storage medium
CN103577660A (en) Gray scale experiment system and method
CN110348215A (en) Exception object recognition methods, device, electronic equipment and medium
CN112214402B (en) Code verification algorithm selection method, device and storage medium
CN111338955B (en) Software graphical interface testing method and device, software development system and server
CN102523286A (en) Method and device for obtaining credit degree of service
CN114397876A (en) Method and apparatus for evaluating performance of domain controller, electronic device, and storage medium
CN115037790B (en) Abnormal registration identification method, device, equipment and storage medium
CN107451063B (en) Script testing method and device
CN114970495A (en) Name disambiguation method and device, electronic equipment and storage medium
US20220046039A1 (en) Method, device, and computer program product for abnormality detection
CN111198986B (en) Information transmission method, device, electronic equipment and storage medium
CN109246297B (en) Method, device and storage medium for determining factory reset time of mobile terminal
CN113935034A (en) Malicious code family classification method and device based on graph neural network and storage medium
CN110297625B (en) Application processing method and device
CN110851344A (en) Big data testing method and device based on computational formula complexity and electronic equipment
CN117220911B (en) Industrial control safety audit system based on protocol depth analysis
CN112837040B (en) Power data management method and system applied to smart grid

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination