CN110245443A - A kind of polymorphic type output reliability of simulation model comprehensive estimation method based on mapping function - Google Patents
A kind of polymorphic type output reliability of simulation model comprehensive estimation method based on mapping function Download PDFInfo
- Publication number
- CN110245443A CN110245443A CN201910541592.0A CN201910541592A CN110245443A CN 110245443 A CN110245443 A CN 110245443A CN 201910541592 A CN201910541592 A CN 201910541592A CN 110245443 A CN110245443 A CN 110245443A
- Authority
- CN
- China
- Prior art keywords
- data
- distance
- simulation model
- error
- credibility
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004088 simulation Methods 0.000 title claims abstract description 121
- 238000013507 mapping Methods 0.000 title claims abstract description 49
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000011156 evaluation Methods 0.000 claims abstract description 72
- 238000005259 measurement Methods 0.000 claims abstract description 46
- 238000004364 calculation method Methods 0.000 claims description 32
- 230000003247 decreasing effect Effects 0.000 claims description 29
- 238000000691 measurement method Methods 0.000 claims description 20
- 238000011524 similarity measure Methods 0.000 claims description 19
- 238000005070 sampling Methods 0.000 claims description 17
- 239000013598 vector Substances 0.000 claims description 10
- 230000014509 gene expression Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 6
- 238000005452 bending Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000012360 testing method Methods 0.000 claims description 4
- 238000012952 Resampling Methods 0.000 claims description 2
- 230000008030 elimination Effects 0.000 claims description 2
- 238000003379 elimination reaction Methods 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 238000004458 analytical method Methods 0.000 description 7
- 238000004445 quantitative analysis Methods 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004451 qualitative analysis Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- Geometry (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to reliability of simulation model evaluation areas, in particular to a kind of confidence level comprehensive estimation method suitable for the complex simulation model with multiple types output characteristics, step includes: to establish reliability of simulation model evaluation index system；Real system and simulation model are tested respectively, obtain quantitative data needed for assessing, or assess qualitative index using expert's qualitative evaluation method；The difference degree of reference data and emulation data is measured, objective difference measurement result is calculated；Difference measurement result is mapped as by confidence level scoring using mapping function；The comprehensive assessment result of reliability of simulation model is calculated using weighted sum method；The present invention is capable of the reliability assessment of the different types of output characteristics of comprehensive simulating model as a result, the quantitative result of simulation model entirety confidence level is calculated, and has many advantages, such as that execution efficiency is high, objectivity is strong, easy to operate, applied widely.
Description
Technical Field
The invention belongs to the field of reliability evaluation of simulation models, and particularly relates to a comprehensive reliability evaluation method suitable for complex simulation models with various output characteristics.
Background
Simulations have become an effective tool for observing and studying real systems when it is difficult to perform a large number of experiments due to the fact that it is too difficult or costly to directly operate such systems. Modeling and simulation are conducted on the basis of establishing an equivalent simulation model through sufficient abstraction and simplification of a real system, and research, analysis and experiments are conducted, so that the real world is explored, known and reconstructed in an economic, efficient, safe, repeatable and nondestructive mode. In recent years, with the rapid development of computers, modeling and simulation technologies, simulation models are widely applied in the fields of military, industry, economy, environment, society and the like. Simulation model-based experiments and analysis play an increasingly important role in scheme evaluation, strategy formulation, problem solving, military training and key decision making.
With the continuous expansion of the application range and the requirements, the simulation model becomes more and more elaborate and complex, and the function and the performance are greatly enhanced. Meanwhile, the simulation model is deeply and widely applied in different fields, people are prompted to pay more and more attention to the reliability problem, and the requirement on the reliability of the simulation model is higher and higher. Simulation results generated by simulation models lacking sufficient confidence may be misleading to decision makers and even cause irreparable military, political and economic losses. Therefore, the credibility assessment of the simulation model plays a crucial role in defining and improving the credibility and the application range of the simulation model and enhancing the confidence that people fully apply the simulation model and the result in important assessment and decision problems, and becomes a key problem in the simulation field. The credibility is an important attribute of the simulation model and provides basis for using, comparing and optimizing various simulation models.
The evaluation of the trustworthiness of complex simulation models typically involves verification and comprehensive evaluation of multiple types of output characteristics. For example, the output characteristics of the missile simulation model generally include a flight trajectory time sequence, the damage area of the missile explosion to a specific target, a series of guidance control behavior characteristics during flight, and the like. In order to obtain reliable reliability evaluation results, it is often insufficient to evaluate only one characteristic of the simulation model, and the evaluation results of multiple characteristics should be aggregated to obtain a comprehensive evaluation result. Currently, different qualitative or quantitative methods are often used for analysis for different types of output characteristics. The time sequence data is mainly analyzed by a distance measurement method or a similarity measurement method, the determined value data is mainly subjected to error analysis, and qualitative indexes are mainly evaluated by an expert subjective evaluation method. However, the evaluation results obtained by different qualitative or quantitative methods are often not in the same scale space, and they need to be mapped to a uniform credibility analysis space for subsequent synthesis, analysis, comparison and optimization. In the traditional mapping method, a domain expert judges whether a qualitative analysis result or a quantitative measurement result is good or bad, and a credibility score is subjectively given. The mapping method relying on the domain expert causes great subjectivity, inconsistency, unreliability and difficulty in repetition, and particularly, when the credibility of a plurality of simulation models is evaluated and compared, the domain expert is required to subjectively map the analysis results of the plurality of models, which causes the domain expert to be tired and has low efficiency.
Disclosure of Invention
The purpose of the invention is: in order to solve the problems that the traditional method verifies and evaluates various output characteristics in a one-sided mode, is single in method, low in efficiency, strong in subjectivity, easy to cause inconsistency, unreliable in evaluation result, difficult to repeat and the like, the multi-type output simulation model reliability comprehensive evaluation method based on the mapping function is provided.
In order to achieve the technical purpose, the technical scheme adopted by the invention is as follows:
a multi-type output simulation model credibility comprehensive evaluation method based on a mapping function comprises the following steps:
step 1: establishing an evaluation index system: according to the expected application purpose of the simulation model, the idea of decomposing from top to bottom and layer by layer is adopted, a user of the simulation model and experts in related fields select a reliability evaluation index, and an evaluation index system is established, wherein the evaluation index system comprises index weight and data required by evaluation, and the index weight is determined by adopting an Analytic Hierarchy Process (AHP);
step 2: acquiring data required for evaluation: respectively testing a real system and a simulation model aiming at a credibility evaluation index capable of obtaining quantitative data to obtain reference data and simulation data, sorting the reference data and the simulation data into determination value data and time sequence data after the reference data and the simulation data are in one-to-one correspondence, and preprocessing the determination value data and the time sequence data; aiming at the credibility evaluation index which can only obtain a qualitative result, directly giving a credibility score by adopting an expert qualitative evaluation method according to the functional characteristics expressed in the actual operation process of the simulation model; determining the value data as an accurate value; the time series data are data collected in time sequence and are used for the condition that the described phenomenon changes along with time; the qualitative result means that the quantization can not be directly realized but needs to be realized through subjective judgment;
and step 3: data difference metric: analyzing the difference degree of the reference data and the simulation data by adopting an error measurement method for the determined value data, and calculating to obtain an objective difference measurement result; analyzing the difference degree of the reference data and the simulation data by adopting a distance measurement method or a similarity measurement method for the time series data, and calculating to obtain an objective difference measurement result;
and 4, step 4: reliability mapping: mapping the difference measurement result into a credibility score by adopting a mapping function; the mapping function comprises a convex decreasing function of the error/distance metric, a concave decreasing function of the error/distance metric, a convex increasing function of the similarity metric, and a concave increasing function of the similarity metric;
and 5: and calculating a comprehensive credibility evaluation result: integrating the reliability evaluation result of each index and the index weight by using a weighted summation method, and calculating to obtain the reliability of the simulation model;
further limiting, the reference data in step 2 is from a test result or historical data of a real system, or may be a theoretical calculation result; and in the step 2, the simulation data come from the operation result of the simulation model, and the operation conditions of the simulation model and the real system are consistent.
Further limiting, the preprocessing in the step 2 includes smoothing filtering, singular value elimination, missing value interpolation, resampling and coordinate transformation.
Further limiting, the expert qualitative assessment method in the step 2 adopts multiple assessors to respectively score, and average scores are taken as assessment results;
further limited, the error measurement method in the step 3 comprises an absolute error and a relative error; the distance measurement method comprises a Minkowski distance, a Jffreys distance, a Camberra distance, a Theil inequality coefficient and a dynamic time bending distance; the similarity measurement method comprises the steps of determining coefficients, cosine of included angle and grey correlation coefficients;
further defined, the specific expressions of the absolute error and the relative error are as follows:
(1) the absolute error e is calculated by the formula:
e＝s-o
wherein, o is reference data, s is simulation data, and the reference data and the simulation data are determined value data; the smaller the absolute value of the absolute error is, the higher the reliability is;
(2) the relative error epsilon is calculated by the formula:
wherein, o is reference data, s is simulation data, and the reference data and the simulation data are determined value data; the smaller the absolute value of the relative error is, the higher the reliability is;
further, the specific expressions of the Minkowski distance, the Jffreys distance, the Camberra distance, the Theil inequality coefficient and the dynamic time warping distance are as follows:
(1) the Minkowski distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points; and when m is respectively 1 and 2, the Manhattan distance and the Euclidean distance are obtained. The calculation of the Manhattan distance has equal consideration to the difference of each element in the vector, and the calculation of the Euclidean distance emphasizes the function of the larger element difference in the distance measurement;
(2) the Jffreys distance calculation formula is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points; the smaller the distance between the simulation data and the reference data is, the higher the reliability is; the Jffreys distance emphasizes the function of small element difference on the basis of the Euclidean distance and is complementary to the Euclidean distance;
(3) the Camberra distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points; element standardization is carried out on the Camberra distance, the influence of different element dimensions is overcome, and the method is suitable for highly biased data;
(4) the calculation formula of Theil inequality coefficients is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points; the closer the Theil inequality coefficient is to 0, the higher the consistency degree of the two time sequences is;
(5) the dynamic time warping distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) For simulation data, n is the number of sampling time points, d can be the square root of the absolute value of the difference, the sum of the squares of the difference, and the goal of dynamic time warping is to find the optimal pairing between the two time series so that the total distance is minimal. Thus, the first on the time series sMapping of data points to the th on the time series oA data point.
Further limiting, the specific expression of the determined coefficient, the cosine of the included angle, and the grey correlation coefficient is as follows:
(1) the coefficient calculation formula is determined as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;andthe mean values of the s and o vectors of the time series, respectively; determining the value range of the coefficient as [0,1]]The reliability is higher when the value is larger;
(2) the cosine of the included angle is calculated as:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points; the cosine value range of the included angle is [0,1]]The reliability is higher when the value is larger;
(3) the grey correlation coefficient calculation formula is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) For simulation data, n is the number of sampling time points, and ρ is taken to be [0,1]]A value of (d) in between;
further, the specific expressions of the convex decreasing function of the error/distance measure, the concave decreasing function of the error/distance measure, the convex increasing function of the similarity measure and the concave increasing function of the similarity measure in step 4 are as follows:
(1) the convex decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, and k (k is more than or equal to 1) is an index representing the descending speed; the larger the k value, the larger the value of xminThe larger the nearby decreasing velocity, at xmaxThe smaller the nearby decreasing velocity;
(2) the concave decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, and k (k is more than or equal to 1) is an index representing the descending speed; the larger the k value, the larger the value of xmaxThe larger the nearby decreasing velocity, at xminThe smaller the nearby decreasing velocity;
(3) the convex increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measurement critical value of full confidence, and k (k is more than or equal to 1) is an index representing the increasing speed; the larger the k value, the larger the value of xmaxThe greater the nearby increasing speed, at xminThe smaller the nearby increasing speed;
(4) the concave increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measurement critical value of full confidence, and k (k is more than or equal to 1) is an index representing the increasing speed; the larger the k value, the larger the value of xminThe greater the nearby increasing speed, at xmaxThe smaller the nearby increasing speed;
the invention has the beneficial effects that:
1. the mapping function-based comprehensive evaluation method for the credibility of the multi-type output simulation model, provided by the invention, can be used for comprehensively evaluating the credibility evaluation results of different types of output characteristics of the simulation model, calculating the credibility by adopting different qualitative or quantitative methods aiming at the determined value data, the time sequence data and the qualitative indexes, and mapping the evaluation results of different types to a uniform credibility scale space, thereby calculating the overall credibility of the simulation model. The method integrates the advantages of different methods aiming at different types of credibility assessment indexes, is suitable for credibility assessment of simulation models with various output characteristics, can comprehensively depict the credibility of the simulation models, and improves the scientificity of assessment results.
2. According to the comprehensive evaluation method for the credibility of the multi-type output simulation model based on the mapping function, after the domain expert or the decision maker determines the parameters of the mapping function, the data difference measurement result is automatically mapped into the credibility score, so that the participation of the domain expert is reduced, and the subjectivity is reduced to the maximum extent. Particularly, when the credibility evaluation and comparison are carried out on a plurality of simulation models, the reliability, repeatability and efficiency of the evaluation process are greatly improved.
3. The invention provides a mapping function-based multi-type output simulation model reliability comprehensive evaluation method, which defines a series of flexible mapping functions with adjustable parameters, and a decision maker can flexibly select different types of mapping functions and suitable parameters according to different application scenes and requirements and automatically map a data difference measurement result into a reliability score. The mapping function defined by the invention has only 3 parameters, needs less subjectivity and less calculation amount and can express different preferences of a decision maker.
Drawings
FIG. 1 is a flow chart of a comprehensive evaluation method for reliability of a multi-type output simulation model based on a mapping function according to the present invention;
FIG. 2 is a convex decreasing function of the present invention for converting error/distance metrics to confidence scores at different values of k;
FIG. 3 is a concave decreasing function for converting error/distance metrics to confidence scores for different values of k according to the present invention;
FIG. 4 is a convex increasing function for converting similarity measurements to confidence scores for different values of k according to the present invention;
FIG. 5 is a concave increasing function for converting similarity measures to confidence scores for different values of k according to the present invention;
Detailed Description
In order that those skilled in the art can better understand the present invention, the following technical solutions are further described with reference to the accompanying drawings and examples.
As shown in fig. 1, an embodiment of the comprehensive evaluation method for the reliability of the multi-type output simulation model based on the mapping function, provided by the invention, for the reliability evaluation of the ballistic missile simulation model includes the following steps:
step 1: establishing an evaluation index system: for reliability evaluation of a ballistic missile simulation model, typical evaluation indexes comprise flight height, damage area, behavior characteristics and the like, and index weight is determined by an Analytic Hierarchy Process (AHP);
step 2: acquiring data required for evaluation: acquiring reference data and simulation data of flight height and damage area, wherein the flight height is represented by time sequence data, the damage area is represented by determined value data, and behavior characteristics are generally qualitatively evaluated;
the reference data and the simulation data of the preprocessed time series of the flying heights are shown in table 1;
TABLE 1 reference data and simulation data for time series of flight heights
| Time (t/s) | Reference data (m) | Simulation data (m) |
| 5 | 208 | 159 |
| 10 | 422 | 388 |
| 15 | 525 | 553 |
| 20 | 593 | 605 |
| 25 | 615 | 623 |
| 30 | 627 | 631 |
| 35 | 618 | 621 |
| 40 | 610 | 613 |
| 45 | 592 | 600 |
| 50 | 581 | 583 |
| 55 | 567 | 565 |
| 60 | 543 | 549 |
| 65 | 507 | 500 |
| 70 | 380 | 459 |
| 75 | 209 | 382 |
| 80 | 195 | 298 |
| 85 | 185 | 227 |
| 90 | 153 | 105 |
The real data of the damage area of the ballistic missile to a specific target is 408m2The simulation result output by the simulation model is 451m2；
Because the missiles have a large number of different behavior characteristics under various conditions, the missiles are difficult to quantitatively express, 3 experts subjectively give credibility scores according to the actual running condition of a simulation model and by virtue of professional knowledge and experience of the 3 experts, the credibility scores are respectively 83, 88 and 76, and the average score is 82.3;
and step 3: data difference metric: analyzing the difference degree of the reference data and the simulation data by adopting an error measurement method for the determined value data; analyzing the difference degree of the reference data and the simulation data by adopting a distance measurement method or a similarity measurement method for the time series data, and calculating to obtain an objective difference measurement result;
the determined value data is a determined value, the reference data is represented as o, and the simulation data is represented as s;
(1) the absolute error e is calculated by the formula:
e＝s-o
the absolute error e can quantitatively describe the magnitude of the difference and reflects the magnitude of the precision, but the absolute error has no scale invariance. For example, the maximum range of action for a missile and rifle is 500km and 0.1km respectively, and their absolute error may both be 0.05 km. Obviously, the absolute error of the same size for both the missile and rifle cannot be considered to be of equal precision.
The absolute error of the damaged area in step 2 is 451-408 ═ 43m2；
(2) The relative error epsilon is calculated by the formula:
in general, relative errors quantify the magnitude of the difference better than absolute errors and reflect the confidence level. The smaller the absolute value of the relative error of the simulation result is, the higher the reliability is.
The relative error of the damaged area in the step 2 is (451 + 408)/408-10.54%;
the time sequence data is a data sequence which can be expressed as an n-dimensional vector (n is the number of sampling time points) after being preprocessed, and the reference data is expressed as o ═ o (o)1,o2,...,on) The simulation data is expressed as s ═ s(s)1,s2,...,sn)；
The distance measurement method comprises a Minkowski distance, a Jffreys distance, a Camberra distance, a Theil inequality coefficient and a dynamic time bending distance;
the smaller the distance between the simulation data and the reference data is, the higher the reliability is;
(1) the Minkowski distance calculation formula is:
and when m is respectively 1 and 2, the Manhattan distance and the Euclidean distance are obtained. The calculation of the Manhattan distance has equal consideration to the difference of each element in the vector, and the calculation of the Euclidean distance emphasizes the function of the larger element difference in the distance measurement;
(2) the Jffreys distance calculation formula is as follows:
the Jffreys distance emphasizes the function of small element difference on the basis of the Euclidean distance and is complementary to the Euclidean distance;
(3) the Camberra distance calculation formula is:
element standardization is carried out on the Camberra distance, the influence of different element dimensions is overcome, and the method is suitable for highly biased data;
(4) the calculation formula of Theil inequality coefficients is as follows:
the closer the Theil inequality coefficient is to 0, the higher the consistency degree of the two time sequences is;
(5) the dynamic time warping distance calculation formula is:
where d may be the root of the absolute difference, the sum of the squares of the difference, and the goal of dynamic time warping is to find the optimal pairing between the two time series so that the total distance is minimized. Thus, the first on the time series sMapping of data points to the th on the time series oA data point.
The results of the distance measurements of the time series of flying heights in step 2 are shown in table 2;
TABLE 2 distance measurement results for time series of flight heights
The similarity measurement method comprises the steps of determining coefficients, cosine of included angle and grey correlation coefficients;
the greater the similarity of the simulation data and the reference data is, the higher the reliability is;
(1) the coefficient calculation formula is determined as follows:
wherein Andthe mean values of the s and o vectors of the time series, respectively; the determination coefficient is a dimensionless index for measuring the linear correlation between two groups of data, and the value range is [0,1]](ii) a The coefficient is determined only by considering the linear correlation of the data, and the proportional relation of the whole data is lack of attention.
(2) The cosine of the included angle is calculated as:
the value range of the cosine of the included angle is [0,1], and the larger the value is, the higher the similarity degree of the two vectors is; the cosine of the included angle only measures the proximity degree of the space vector, does not consider the variation trend among data, and has insufficient response to the integral difference of the data.
(3) The grey correlation coefficient calculation formula is as follows:
wherein rho is a value between [0,1], and the grey correlation analysis is only used for analyzing the similarity of the space curve shapes formed by the two time series without considering the distance between the two curves.
The results of the similarity measurement of the time series of flying heights in step 2 are shown in table 3;
TABLE 3 results of similarity measurements for time series of fly heights
| Similarity measurement method | Similarity measure results |
| Determining coefficients | 0.9095 |
| Cosine of included angle | 0.9940 |
| Grey correlation coefficient (take epsilon as 0.5) | 0.8045 |
And 4, step 4: reliability mapping: mapping the difference measurement result into a credibility score by adopting a mapping function;
the mapping function comprises a convex decreasing function of the error/distance measure, a concave decreasing function of the error/distance measure, a convex increasing function of the similarity measure, and a concave increasing function of the similarity measure;
(1) the convex decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, k (k is more than or equal to 1) is an index representing the descending speed, and the larger the value of k is, the larger the value of x isminThe larger the nearby decreasing velocity, at xmaxThe smaller the nearby decreasing velocity;
the convex decreasing function of the error/distance metric for different k values to convert the error/distance metric to a confidence score is shown in fig. 2;
if the reliability degression is obvious under the smaller error/distance measurement deviation and the reliability degression is more moderate under the larger error/distance measurement deviation, a convex degression function of the error/distance measurement is suitable for being adopted;
(2) the concave decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, k (k is more than or equal to 1) is an index representing the descending speed, and the larger the value of k is, the larger the value of x ismaxThe larger the nearby decreasing velocity, at xminThe smaller the nearby decreasing velocity;
the concave decreasing function of the error/distance metric for different values of k, which converts the error/distance metric to a confidence score, is shown in fig. 3;
if the reliability degression is milder under a smaller error/distance measurement deviation and the reliability degression is obvious under a larger error/distance measurement deviation, a concave degression function of the error/distance measurement is suitable for being adopted;
the relative error of the damaged area calculated in the step 3 is 10.54%, the mapping function is taken as a concave decreasing function of the defined error/distance measurement, and the parameter of the mapping function determined by the domain expert and the decision maker is xmin＝1％,xmaxThe credibility score of the damage area index is calculated to be 91.4093, wherein the k is 50 percent and the k is 1.5;
(3) the convex increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measurement critical value of full confidence, k (k ≧ 1) is an index representing the increasing speed, the larger the value of k, the larger the value of xmaxThe greater the nearby increasing speed, at xminThe smaller the nearby increasing speed;
the convex increasing function of the similarity measure for converting the similarity measure into a confidence score at different values of k is shown in fig. 4;
if the increment of the credibility is milder under the smaller similarity measurement and the decrement of the credibility is more obvious under the larger similarity measurement, a convex increment function of the similarity measurement is suitable to be adopted;
the determination coefficient of the time sequence of the flight heights calculated in the step 3 is 0.9095, the mapping function is taken as a convex increasing function of the similarity measurement, and the parameter of the mapping function determined by the domain expert and the decision maker is xmin＝0.3,xmaxWhen k is 2.0, the calculated credibility score of the flight height index is 82.7557;
(4) the concave increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measurement critical value of full confidence, k (k ≧ 1) is an index representing the increasing speed, the larger the value of k, the larger the value of xminThe greater the nearby increasing speed, at xmaxThe smaller the nearby increasing speed;
a concave increasing function of the similarity measure for different values of k, converting the similarity measure into a confidence score is shown in fig. 5;
if the increment of the reliability is obvious under the smaller similarity measurement and the increment of the reliability is more moderate under the larger similarity measurement, a concave increment function of the similarity measurement is suitable to be adopted;
and 5: and calculating a comprehensive credibility evaluation result: integrating the reliability evaluation result of each index and the index weight by using a weighted summation method, and calculating to obtain the reliability of the simulation model;
the calculation formula of the weighted summation method is as follows:
wherein ,TiIs the confidence score of index i, wiThe weight of the index i is obtained, m is the total number of indexes, and S obtained through calculation is the reliability of the simulation model;
the weight coefficient in the weighted summation method is determined by adopting an Analytic Hierarchy Process (AHP);
the AHP pairwise comparison matrix of the flight height, the damage area and the behavior characteristic 3 indexes is as follows:
the maximum characteristic root of the matrix is 3.0183, the normalized characteristic vector corresponding to the maximum characteristic root is [0.4434,0.1692,0.3874], the AHP consistency index CI is (3.0183-3)/(3-1) 0.0091, the consistency ratio CR is CI/RI 0.0091/0.52 is 0.0176<0.1, and CR meets the consistency requirement; therefore, the weight vector of the flight height, the damage area and the behavior characteristic 3 indexes is [0.4434,0.1692,0.3874 ];
the calculated flight altitude reliability evaluation result, damage area reliability evaluation result, behavior feature reliability evaluation result, and the weights of the 3 indexes are shown in table 4;
TABLE 4 evaluation results and weights of 3 indexes in ballistic missile simulation model credibility evaluation
| Index name | Evaluation results | Weight of |
| Flying height | 82.7557 | 0.4434 |
| Area of damage | 91.4093 | 0.1692 |
| Behavioral characteristics | 82.3 | 0.3874 |
The comprehensive reliability evaluation result of the ballistic missile simulation model calculated by the weighted summation method is 84.0434;
therefore, the comprehensive evaluation of the reliability of the ballistic missile simulation model is completed through the 5 steps;
in conclusion, the mapping function-based comprehensive reliability evaluation method for the multi-type output simulation model can be used for comprehensively evaluating the reliability evaluation results of different types of output characteristics of the simulation model, flexibly measuring the difference degree of the various output characteristics by adopting different qualitative or quantitative methods, and mapping the difference measurement results to a uniform reliability scale space, thereby quantitatively calculating the overall reliability of the simulation model;
the comprehensive reliability evaluation method for the multi-type output simulation model based on the mapping function can be applied to reliability evaluation of simulation models in different fields, such as reliability evaluation of ballistic missile simulation models, reliability evaluation of phased array radar simulation models, reliability evaluation of urban traffic simulation models, reliability evaluation of social and economic system dynamic simulation models and the like;
the above embodiments merely illustrate the method steps of the present invention and its core ideas, but do not limit the present invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A multi-type output simulation model credibility comprehensive evaluation method based on a mapping function is characterized by comprising the following steps: the method comprises the following specific steps:
step 1: establishing an evaluation index system: according to the expected application purpose of the simulation model, the idea of decomposing from top to bottom and layer by layer is adopted, a user of the simulation model and experts in related fields select a reliability evaluation index, and an evaluation index system is established, wherein the evaluation index system comprises index weight and data required by evaluation, and the index weight is determined by adopting an Analytic Hierarchy Process (AHP);
step 2: acquiring data required for evaluation: respectively testing a real system and a simulation model aiming at a credibility evaluation index capable of obtaining quantitative data to obtain reference data and simulation data, sorting the reference data and the simulation data into determination value data and time sequence data after the reference data and the simulation data are in one-to-one correspondence, and preprocessing the determination value data and the time sequence data; aiming at the credibility evaluation index which can only obtain a qualitative result, directly giving a credibility score by adopting an expert qualitative evaluation method according to the functional characteristics expressed in the actual operation process of the simulation model;
and step 3: data difference metric: analyzing the difference degree of the reference data and the simulation data by adopting an error measurement method for the determined value data, and calculating to obtain an objective difference measurement result; analyzing the difference degree of the reference data and the simulation data by adopting a distance measurement method or a similarity measurement method for the time series data, and calculating to obtain an objective difference measurement result;
and 4, step 4: reliability mapping: mapping the difference measurement result into a credibility score by adopting a mapping function; the mapping function comprises a convex decreasing function of the error/distance metric, a concave decreasing function of the error/distance metric, a convex increasing function of the similarity metric, and a concave increasing function of the similarity metric;
and 5: and calculating a comprehensive credibility evaluation result: and integrating the reliability evaluation result of each index and the index weight by using a weighted summation method, and calculating to obtain the reliability of the simulation model.
2. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 1, wherein: the reference data in the step 2 is from a test result or historical data of a real system, or can be a theoretical calculation result; and in the step 2, the simulation data come from the operation result of the simulation model, and the operation conditions of the simulation model and the real system are consistent.
3. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 2, wherein: the preprocessing in the step 2 comprises smooth filtering, singular value elimination, missing value interpolation, resampling and coordinate transformation.
4. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 3, wherein: in the step 2, multiple evaluators are respectively used for scoring, and the average score is taken as an evaluation result.
5. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 1, wherein: the error measurement method in the step 3 comprises an absolute error and a relative error; the distance measurement method comprises a Minkowski distance, a Jffreys distance, a Camberra distance, a Theil inequality coefficient and a dynamic time bending distance; the similarity measurement method comprises the steps of determining coefficients, the cosine of an included angle and grey correlation coefficients.
6. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 5, wherein: the specific expressions of the absolute error and the relative error are as follows:
(1) the absolute error e is calculated by the formula:
e＝s-o
wherein, o is reference data, s is simulation data, and the reference data and the simulation data are determined value data;
(2) the relative error epsilon is calculated by the formula:
wherein, o is reference data, and s is simulation data, both of which are determined value data.
7. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 5, wherein: the specific expressions of the Minkowski distance, the Jffreys distance, the Camberra distance, the Theil inequality coefficient and the dynamic time bending distance are as follows:
(1) the Minkowski distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;
(2) the Jffreys distance calculation formula is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;
(3) the Camberra distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;
(4) the calculation formula of Theil inequality coefficients is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;
(5) the dynamic time warping distance calculation formula is:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) For simulation data, n is the number of sampling time points, d is the root of the absolute value of the difference or the sum of the squares of the difference, the first in time series sMapping of data points to the th on the time series oA data point.
8. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 5, wherein: the specific expressions of the determined coefficients, the cosine of the included angle and the grey correlation coefficients are as follows:
(1) the coefficient calculation formula is determined as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s＝(s1,s2,...,sn) N is the number of sampling time points;andthe mean values of the s and o vectors of the time series, respectively;
(2) the cosine of the included angle is calculated as:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) N is the number of sampling time points;
(3) the grey correlation coefficient calculation formula is as follows:
wherein o ═ o (o)1,o2,...,on) For reference data, s ═ s(s)1,s2,...,sn) For simulation data, n is the number of sampling time points, and ρ is taken to be [0,1]]With a value in between.
9. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 1, wherein: in step 4, the specific expressions of the convex decreasing function of the error/distance measure, the concave decreasing function of the error/distance measure, the convex increasing function of the similarity measure and the concave increasing function of the similarity measure are as follows:
(1) the convex decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, and k (k is more than or equal to 1) is an index representing the descending speed;
(2) the concave decreasing function of the error/distance metric is calculated as:
wherein ,xminIs the error/distance metric threshold, x, of full confidencemaxIs an error/distance measurement critical value without confidence, and k (k is more than or equal to 1) is an index representing the descending speed;
(3) the convex increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measurement critical value of full confidence, and k (k is more than or equal to 1) is an index representing the increasing speed;
(4) the concave increasing function of the similarity measure is calculated by the formula:
wherein ,xminIs a confidence-free similarity metric threshold, xmaxIs a similarity measure threshold value of the full confidence level, and k (k ≧ 1) is an index representing the increasing speed.
10. The method for comprehensively evaluating the credibility of the multi-type output simulation model based on the mapping function as claimed in claim 1, wherein: the calculation formula of the weighted summation method is as follows:
wherein ,TiIs the confidence score of index i, wiAnd m is the total number of indexes i, and S obtained by calculation is the reliability of the simulation model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910541592.0A CN110245443B (en) | 2019-06-21 | 2019-06-21 | Missile simulation model credibility comprehensive evaluation method based on mapping function |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910541592.0A CN110245443B (en) | 2019-06-21 | 2019-06-21 | Missile simulation model credibility comprehensive evaluation method based on mapping function |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110245443A true CN110245443A (en) | 2019-09-17 |
| CN110245443B CN110245443B (en) | 2023-08-18 |
Family
ID=67888722
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910541592.0A Active CN110245443B (en) | 2019-06-21 | 2019-06-21 | Missile simulation model credibility comprehensive evaluation method based on mapping function |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245443B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110889207A (en) * | 2019-11-08 | 2020-03-17 | 江南机电设计研究所 | System combination model credibility intelligent evaluation method based on deep learning |
| CN110955862A (en) * | 2019-11-26 | 2020-04-03 | 新奥数能科技有限公司 | Evaluation method and device for trend similarity of equipment model |
| CN111027190A (en) * | 2019-11-29 | 2020-04-17 | 新奥数能科技有限公司 | Evaluation method and device for numerical similarity of equipment model |
| CN111368456A (en) * | 2020-03-19 | 2020-07-03 | 上海机电工程研究所 | Evidence theory-based reliability assessment method and system for blue square model |
| CN112231786A (en) * | 2020-10-15 | 2021-01-15 | 北京航天发射技术研究所 | General evaluation method and system for evaluating effectiveness of simulation model |
| CN112257332A (en) * | 2020-09-23 | 2021-01-22 | 北京航空航天大学 | Simulation model evaluation method and device |
| CN112528418A (en) * | 2020-12-18 | 2021-03-19 | 北京机电工程研究所 | Evaluation system for semi-physical simulation test under non-reference condition |
| CN114279557A (en) * | 2022-03-02 | 2022-04-05 | 山东卓朗检测股份有限公司 | Distributed acoustic detection equipment and detection method thereof |
| CN117875094A (en) * | 2024-03-13 | 2024-04-12 | 西安航天动力研究所 | Precision evaluation method and device for engine simulation model and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1997002544A1 (en) * | 1995-06-30 | 1997-01-23 | Iterated Systems, Inc. | System and method for modeling discrete data sequences |
| CN107766668A (en) * | 2017-10-31 | 2018-03-06 | 哈尔滨工业大学 | A kind of complex simulation model verification method based on neutral net |
-
2019
- 2019-06-21 CN CN201910541592.0A patent/CN110245443B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1997002544A1 (en) * | 1995-06-30 | 1997-01-23 | Iterated Systems, Inc. | System and method for modeling discrete data sequences |
| CN107766668A (en) * | 2017-10-31 | 2018-03-06 | 哈尔滨工业大学 | A kind of complex simulation model verification method based on neutral net |
Non-Patent Citations (1)
| Title |
|---|
| 杨丽丽;刘华;王华涛;: "仿真系统可信度多层次灰色评估方法研究", 国外电子测量技术, no. 03 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110889207B (en) * | 2019-11-08 | 2023-08-22 | 江南机电设计研究所 | Deep learning-based intelligent assessment method for credibility of system combination model |
| CN110889207A (en) * | 2019-11-08 | 2020-03-17 | 江南机电设计研究所 | System combination model credibility intelligent evaluation method based on deep learning |
| CN110955862A (en) * | 2019-11-26 | 2020-04-03 | 新奥数能科技有限公司 | Evaluation method and device for trend similarity of equipment model |
| CN110955862B (en) * | 2019-11-26 | 2023-10-13 | 新奥数能科技有限公司 | Evaluation method and device for equipment model trend similarity |
| CN111027190A (en) * | 2019-11-29 | 2020-04-17 | 新奥数能科技有限公司 | Evaluation method and device for numerical similarity of equipment model |
| CN111368456A (en) * | 2020-03-19 | 2020-07-03 | 上海机电工程研究所 | Evidence theory-based reliability assessment method and system for blue square model |
| CN112257332A (en) * | 2020-09-23 | 2021-01-22 | 北京航空航天大学 | Simulation model evaluation method and device |
| CN112231786A (en) * | 2020-10-15 | 2021-01-15 | 北京航天发射技术研究所 | General evaluation method and system for evaluating effectiveness of simulation model |
| CN112231786B (en) * | 2020-10-15 | 2024-01-30 | 北京航天发射技术研究所 | General evaluation method and system for evaluating effectiveness of simulation model |
| CN112528418A (en) * | 2020-12-18 | 2021-03-19 | 北京机电工程研究所 | Evaluation system for semi-physical simulation test under non-reference condition |
| CN114279557A (en) * | 2022-03-02 | 2022-04-05 | 山东卓朗检测股份有限公司 | Distributed acoustic detection equipment and detection method thereof |
| CN114279557B (en) * | 2022-03-02 | 2022-05-17 | 山东卓朗检测股份有限公司 | Distributed acoustic detection equipment and detection method thereof |
| CN117875094A (en) * | 2024-03-13 | 2024-04-12 | 西安航天动力研究所 | Precision evaluation method and device for engine simulation model and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110245443B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245443A (en) | A kind of polymorphic type output reliability of simulation model comprehensive estimation method based on mapping function | |
| CN111664365B (en) | Oil and gas pipeline leakage detection method based on improved VMD and 1DCNN | |
| CN110728411B (en) | High-low altitude area combined rainfall prediction method based on convolutional neural network | |
| CN111680870B (en) | Comprehensive evaluation method for quality of target motion trail | |
| CN107766668B (en) | Complex simulation model verification method based on neural network | |
| CN112182720B (en) | Building energy consumption model evaluation method based on building energy management application scene | |
| CN108182500A (en) | Ammunition Storage Reliability Forecasting Methodology based on accelerated life test | |
| CN116431975A (en) | Environment monitoring method and system for data center | |
| CN115374995A (en) | Distributed photovoltaic and small wind power station power prediction method | |
| CN107688863A (en) | The short-term wind speed high accuracy combination forecasting method that adaptive iteration is strengthened | |
| CN112884012A (en) | Building energy consumption prediction method based on support vector machine principle | |
| CN102096633A (en) | Application field oriented software quality standard evaluating method | |
| CN117370766A (en) | Satellite mission planning scheme evaluation method based on deep learning | |
| CN116595411A (en) | Real-time maneuver identification method and device based on early time sequence classification | |
| CN113553708B (en) | Method and device for tracing key influence factors of simulation model | |
| CN113488164B (en) | Data-based knowledge base psychological sand table analysis method, system and equipment | |
| CN114970698A (en) | Metering equipment operation performance prediction method based on improved LWPLS | |
| CN113076661B (en) | Uncertain data driven radar early warning detection modeling method | |
| CN111222736A (en) | Ammunition storage reliability evaluation method based on mixed correlation vector machine model | |
| CN113035363B (en) | Probability density weighted genetic metabolic disease screening data mixed sampling method | |
| CN116956642B (en) | Threat assessment algorithm performance dynamic simulation verification method | |
| CN113534129B (en) | Method and system for evaluating high-speed target detection performance of foundation broadband radar | |
| CN117171681B (en) | Unmanned plane control surface intelligent fault diagnosis method and device under unbalanced small sample | |
| CN115186941B (en) | Variable optimization climate mode method based on multiple space-time indexes and comprehensive sequencing | |
| CN116384465A (en) | Neural network model quantization loss evaluation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |