CN115708376A

CN115708376A - Network evaluation method, device and storage medium

Info

Publication number: CN115708376A
Application number: CN202110959933.3A
Authority: CN
Inventors: 梁双春; 张安兵; 孔样红; 吴拓; 刘一强
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Communications Ltd Research Institute
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Communications Ltd Research Institute
Priority date: 2021-08-20
Filing date: 2021-08-20
Publication date: 2023-02-21

Abstract

The invention discloses a network evaluation method, a device and a storage medium, comprising the following steps: determining indexes to be evaluated in the NFV network; monitoring the index, and acquiring a first index of the index before fault injection; injecting a fault; monitoring the index, and acquiring a second index of the index after fault injection; restoring the NFV network to the first indicator; the NFV network is evaluated according to the second index. By adopting the invention, the technical scheme for evaluating the fault-tolerant capability of the NFV network and the damage resistance degree of the NFV network within the acceptable range of the service index can be provided.

Description

Network evaluation method, device and storage medium

Technical Field

The present invention relates to the field of communications technologies, and in particular, to a network evaluation method, apparatus, and storage medium.

Background

For a Network Function Virtualization (NFV) Network based on a cloud resource pool, a redundant resource load balancing technology of cloud computing and a stateless system architecture are generally utilized to realize high availability of the NFV Network, so as to improve the stability of the whole Network. In the prior art, a service pressure test is mainly adopted, and a system availability index or a data integrity index is used for evaluating the reliability degree of the whole network.

The defects of the prior art are as follows:

the prior art does not relate to how to evaluate the fault tolerance of the NFV network and the network element and how to damage the NFV network and the network element within the acceptable range of the service index.

Disclosure of Invention

The invention provides a network and network element evaluation method, a network and network element evaluation device and a storage medium, which are used for solving the problems that no scheme is used for evaluating the fault tolerance capability of an NFV network and a network element and the damage resistance degree of the NFV network within an acceptable range of a service index.

The invention provides the following technical scheme:

a network evaluation method, comprising:

determining indexes to be evaluated in the NFV network;

monitoring the index, and acquiring a first index of the index before fault injection;

injecting a fault;

monitoring the index, and acquiring a second index of the index after fault injection;

restoring the NFV network to the first index;

the NFV network is evaluated according to the second index.

In an implementation, the method further comprises the following steps:

and when the fault is injected, controlling the fluctuation of the predetermined index of the NFV network to be within a preset range.

In an implementation, the predetermined index is one of the following indexes or a combination thereof:

one or a combination of the following criteria at the 4G base station: a service volume type index, a success rate type index, a time delay type index and an error type index;

one or a combination of the following criteria at the 5G base station: the service volume and capacity type index, the success rate type index, the time delay type index and the error type index;

one or a combination of the following indicators on the 4G core network element:

a traffic type index, a success rate type index, a delay type index and an error type index on the MME;

the service volume, the capacity type index and the success rate type index on the servingGW;

a traffic and capacity type index, a delay type index and an error type index on the PGW;

a service volume and capacity type index, a success rate type index and an error type index on the HSS;

a traffic type index, a success rate type index and an error type index on the PCRF;

one or a combination of the following indicators on the 5G core network element:

the service quantity type index, the success rate type index, the time delay type index and the error type index on the AMF;

a traffic type index, a success rate type index, a delay type index and an error type index on the SMF;

a service type index, a success rate type index and an error type index on the UPF;

success rate type indicator on UDM;

a service volume type index, a success rate type index and an error type index on the PCF;

a traffic type index, a success rate type index, an error type index on the NRF;

success rate type indicator, error type indicator on NSSF.

one or a combination of the following indicators on the 4G base station:

traffic type index: the number of bytes sent and/or received by an eNB Ethernet interface, the number of bytes of service sent and/or received by an eNB S1 interface, the number of bytes of cell user plane and/or control plane uplink and/or downlink PDCP SDU;

success rate type index: E-RAB establishment success rate related to the service, radio access rate related to the service, RRC connection establishment success rate and switching success rate;

the delay type index: RRC connection average and/or maximum establishment duration, E-RAB average and/or maximum establishment duration;

error type index: the method comprises the following steps of (1) wireless call drop rate, TCP service-based wireless call drop rate, uplink and/or downlink PDCP SDU packet loss rate, uplink and/or downlink transmission block error number and paging record discarding number;

one or a combination of the following indicators on the 5G base station:

traffic and capacity type indices: the gNB sends and/or receives the service data volume from the NG interface, the byte number of the cell user plane uplink and/or downlink PDCP PDU, the traffic data volume received and/or sent by the gNB from the S1 interface, the number of uplink and/or downlink transmission TBs, the number of available downlink PDSCH PRBs, the receiving number of paging records, the average and/or maximum number of RRC connections in dual connection, the average and/or maximum ERAB number of NSASCG Split Bear types;

success rate type index: RRC connection establishment success rate, flow establishment success rate, PDSESSION establishment success rate, switching success rate and NG interface UE related logic signaling connection establishment success rate;

the delay type index: the method comprises the following steps of RRC connection average and/or maximum establishing time, RLC downlink data packet average processing time delay, NG switching average time length between gNBs, xn switching average time length between gNBs, epsfallback service switching time delay from 5G to 4G, and RLC downlink data packet average processing time delay of each slice cell;

error type index: flow establishes and/or modifies failure number, up going and/or down going PDCP packet loss number;

on the MME:

traffic type indicator: average and/or maximum number of MME load, number of users in an MME idle state and/or a connection state, average and/or maximum number of attached users;

success rate type index: EPS attachment success rate, authentication success rate, DNS analysis success rate initiated by MME, default and/or special bearing activation success rate, PDN connection establishment success rate, service request success rate, paging success rate, tracking area update success rate and switching success rate;

the delay type index is as follows: average and/or maximum attachment duration, dedicated bearer setup average and/or maximum duration;

error type index: the number of authentication parameter errors, the number of UE authentication failures, the number of EPS attachment failures and the number of tracking area update failures;

on the ServingGW:

traffic and capacity type indicators: average and/or peak utilization rate of SGW bearing capacity, uplink and/or downlink flow of a user plane, uplink and/or downlink average flow of GTP generated by each attached user of the serving GW, average and/or maximum attached user number of the serving GW, average and/or maximum bearing number of the serving GW, and utilization rate of SGW data throughput capacity;

success rate type index: the success rate of establishing the default and/or special bearing of the servingGW;

on the PGW:

traffic and capacity type indicators: the method comprises the following steps that PGW data throughput capacity utilization rate, PGW load capacity average and/or peak utilization rate, PGW S5 and/or S8 interface uplink and/or downlink traffic, PGW average and/or maximum attached user number, PGW average and/or peak load number, and SGi interface receiving and/or sending traffic;

success rate type index: the dedicated bearer establishment success rate and the CDR transmission success rate;

the delay type index is as follows: establishing average and/or maximum duration of special load initiated by a PGW;

error type index: the number of GTP packets discarded by errors of PGW S5 and/or S8 interfaces and the number of IP packets discarded by errors of SGi interfaces;

on the HSS:

traffic and capacity type indicators: HSS number allocation and/or number of active users, HSS authentication capacity utilization rate and HSS static capacity utilization rate;

success rate type index: HSS authentication information inquiry success rate, HSS updating and/or canceling position success rate, HSS inserting and/or deleting user data success rate and HSS UE clearing success rate;

error type index: updating the position failure times;

on the PCRF:

traffic type index: gx session processing capacity average and/or peak utilization;

success rate type index: the strategy control initiates and/or updates and/or finishes the success rate, re-authentication success rate, application session authorization success rate;

error type index: applying the session call loss rate;

on the AMF:

traffic type index: AMF register state and/or idle state user number, paging request times, paging response times;

success rate type index: the success rate of initial registration, the success rate of registration update, the success rate of switching, the success rate of UE CM deregistering, the success rate of establishing the context of the N11 interface session, the success rate of updating the context of the N11 interface session, the success rate of releasing the context of the N11 interface session, the success rate of inquiring the context of the N11 interface session and the success rate of service request;

time delay type index: initial registration average duration;

error type index: the number of errors of the authentication parameters, the number of authentication refusal times, the number of initial registration failure times, the number of registration updating failure times and the number of refused service requests;

on SMF:

traffic type index: average and/or maximum PDU session number, average and/or maximum Qos flow number;

success rate type index: a PDU session establishment success rate, a PDU session modification success rate initiated by SMF, an N7 interface establishment SM strategy success rate, an N10 interface UE context registration success rate, an N7 interface updating SM strategy success rate, an N10 interface UE context de-registration success rate and an N7 interface deletion SM strategy success rate;

the delay type index: the PDU conversation establishes the average duration of the flow;

error type index: PDU conversation establishment failure times and PDU conversation modification failure times initiated by SMF;

on the UPF:

traffic type index: average and/or maximum QoS flow number, GTP packet receiving and/or sending byte number of an N3 interface, GTP packet receiving and/or sending byte number of an N9a interface and GTP packet receiving and/or sending byte number of an N6 interface;

success rate type index: the success rate of the PFCP session establishment and the success rate of the PFCP session modification;

error type index: the number of times of failure of PFCP session establishment and/or modification, the number of GTP packets received by an N3 interface, the number of IP packets discarded by an N6 interface in error and the number of GTP packets received by an N9c interface are/is determined;

on the UDM:

success rate type index: UECM registration success rate initiated by AMF, UECM registration success rate initiated by SMF, registration parameter updating success rate, user data acquisition success rate, user data subscription success rate and user data unsubscribe success rate;

on the PCF:

traffic type index: the average and/or maximum value of the AM strategy association total number and the average and/or maximum value of the SM strategy association total number;

success rate type index: the method comprises the following steps of AM strategy association establishing success rate, AM strategy association updating success rate, AM strategy association deleting success rate, SM strategy association establishing success rate, SM strategy association updating success rate and SM strategy association deleting success rate;

error type index: the SM strategy association establishment failure times and the SM strategy association updating failure times;

on NRF:

traffic type index: storing the number of instances;

success rate type index: the NF initiates an updating success rate and an NF discovery success rate;

error type index: the NF initiates the updating failure times and the NF discovers the failure times;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

In implementation, when a fault is injected, the fault is injected layer by layer from an infrastructure layer to a VNF layer according to the NFV network function architecture; and/or the presence of a gas in the gas,

according to the NFV network function architecture, faults are injected layer by layer from the VNF layer to the infrastructure layer.

In implementation, when the fault is injected, the fault is injected in one of the following aspects or a combination of the following aspects:

a main NFV network service plane, a standby NFV network service plane, a management plane, a network element component plane and an infrastructure.

In implementation, one or a combination of the following faults are injected at the NFV network traffic plane:

injecting a fault into the main NFV network service as a cloud OS database IP address conflict;

when the main NFV network service injection fault is that VIM completely blocks partial service virtual machines of the superimposed network element from being damaged;

deleting an S1 interface route for the data center gateway when the main NFV network service injection fault causes the total blocking of the main NFV network service;

the method comprises the steps that a service injection fault in a main large area is used as a main large area batch virtual machine storage disconnection;

the business injection fault in the main large area is that the read-write time delay of the CloudOS storage component is large;

the service injection fault in the main large area is a CloudOS storage component exchange fault;

the fault of the user plane service injection is the fault of the CloudOS component, which causes the connection between the user plane network element and the control plane network element to be interrupted.

In implementation, the OMU fault of VNF, which is injected in the management plane, cannot be accessed.

In an implementation, the indicator is an NFV network performance indicator of one or a combination of the following indicators:

one or a combination of the following indicators on the 4G base station: a service volume type index, a success rate type index, a time delay type index and an error type index;

one or a combination of the following indicators on the 5G base station: the service volume and capacity type index, the success rate type index, the time delay type index and the error type index;

the service volume and capacity type index, success rate type index and error type index on HSS;

a service volume type index, a success rate type index, a delay type index and an error type index on the SMF;

a traffic type index, a success rate type index, and an error type index on the UPF;

success rate type indicator on UDM;

success type indicator, error type indicator on NSSF.

In implementation, evaluating the NFV network or network element according to the second indicator is to evaluate the capability of the overall network service quality to resist damage when a component in the NFV network or network element fails.

In an implementation, the NFV network is evaluated according to the second index in one or a combination of the following ways:

NFV network element toughness = α (a) ₁ * Network component failure ratio + a ₂ * Calculating/storing component fault proportion + β OMU fault proportion; wherein, the NFV network element toughness is an evaluation index for evaluating the network element service quality damage resistance when a component in the NFV network element fails, alpha represents the user plane weight, beta represents the management plane weight, a _i {i∈[1,3]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

NFV network toughness = α + β (b) NFV network element failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failure fraction) 100%; the NFV network toughness is an evaluation index for evaluating the damage resistance of the whole network service quality when NFV network element, EMS, VNFM and NFVO functional components fail, wherein alpha represents the user plane weight, beta represents the management plane weight, and b represents the management plane weight _i {i∈[1,6]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

4G wireless network element toughness = (alpha (a)) ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR, EOR, network card, high available computing sumThe storage component comprises one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless network element toughness = (alpha (a) = ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Σ 5G base station high availability computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high availability network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault ratio + a ₂ * Σ 4G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component fault ratio + a ₂ * Σ 5G core network element high available compute and storage component failure) + β OMU failure) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α × 4G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (alpha is 5G base station fault ratio + beta (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows that the fault proportion of each high-availability component is summed, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (alpha:sigma4G core network element (MME, servingGW, PGW, HSS, PCRF) fault proportion + beta: (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: the TOR switch (Top of Rack), EOR switch (End of Row), network card, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma represents the summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure fraction + β [ (+ b [ ]) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS highAvailable compute and storage component failure ratio + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machines, virtual machines, databases; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]NFV network element toughness = α (a) ₁ * Network component failure ratio + a ₂ * Calculating/storing component fault proportion + β OMU fault proportion; wherein, the NFV network element toughness is an evaluation index for evaluating the network element service quality damage resistance when a component in the NFV network element fails, alpha represents the user plane weight, beta represents the management plane weight, a _i {i∈[1,3]}∈[0,1]Representing the weight coefficients of the various functional layers of the user plane.

In an implementation, evaluating the NFV network according to the second index is evaluating one or a combination of the following performance of the NFV network:

the toughness of the wireless network element and the toughness of the core network element,

the toughness of the core network element is used for evaluating the proportion weighted sum of the functional layer components of the core network element when faults occur.

In the implementation, the toughness of the wireless network element is used for verifying the cloud expansion capability and the fault tolerance capability of the wireless network element service and the influence of the maximum uncertainty problem on the network element steady state; and/or the presence of a gas in the gas,

the toughness of the core network element is used for verifying the cloud expansion capability and the fault tolerance capability of the core network element service and the influence of the maximum uncertainty problem on the network element steady state.

In implementation, one or a combination of wireless network element toughness, core network element toughness of the NFV network is evaluated according to the second index as follows:

wireless network element toughness = (alpha (a)) ₁ * Network component failure fraction + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0,1]]；

Core network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machines, virtual machines, databases; the value ranges of alpha, beta, a1 and a2 are [0,1]]。

A network evaluation apparatus comprising:

a processor for reading the program in the memory and executing the following processes:

determining indexes to be evaluated in the NFV network;

injecting a fault;

restoring the NFV network to the first indicator;

evaluating the NFV network according to the second index;

a transceiver for receiving and transmitting data under the control of the processor.

In an implementation, the method further comprises the following steps:

In an implementation, the predetermined index is one or a combination of the following:

a service traffic type index, a success rate type index and an error type index on the PCRF;

success rate type indicator on UDM;

success rate type indicator, error type indicator on NSSF.

one or a combination of the following indicators on the 4G base station:

traffic type indicator: the number of bytes sent and/or received by an eNB Ethernet interface, the number of bytes of service sent and/or received by an eNB S1 interface, the number of bytes of cell user plane and/or control plane uplink and/or downlink PDCP SDU;

the delay type index is as follows: RRC connection average and/or maximum establishment duration, E-RAB average and/or maximum establishment duration;

one or a combination of the following criteria at the 5G base station:

traffic and capacity type indices: the gNB sends and/or receives the service data volume from the NG interface, the byte number of the uplink and/or downlink PDCP PDU of the cell user plane, the traffic data volume received and/or sent by the gNB from the S1 interface, the number of TBs (transmission blocks) of the uplink and/or downlink transmission, the number of PRBs (physical resource blocks) available for the downlink PDSCH, the receiving number of paging records, the average and/or maximum number of RRC connections in double connection, the average and/or maximum ERAB number of NSA SCG Split Bear types;

the delay type index is as follows: the method comprises the following steps of RRC connection average and/or maximum establishing time, RLC downlink data packet average processing time delay, NG switching average time length between gNBs, xn switching average time length between gNBs, epsfallback service switching time delay from 5G to 4G, and RLC downlink data packet average processing time delay of each slice cell;

on the MME:

traffic type index: average and/or maximum number of MME load, number of users in an MME idle state and/or a connection state, average and/or maximum number of attached users;

on the ServingGW:

traffic and capacity type indicators: average and/or peak utilization rate of SGW (serving gateway) bearing capacity, uplink and/or downlink traffic of a user plane, GTP (general packet radio service) uplink and/or downlink average traffic generated by each attached user of the serving GW, average and/or maximum attached user number of the serving GW, average and/or maximum bearing number of the serving GW and utilization rate of SGW (serving gateway) data throughput capacity;

on the PGW:

at the HSS:

error type index: updating the position failure times;

on the PCRF:

traffic type indicator: gx session processing capacity average and/or peak utilization;

error type index: applying a session call loss rate;

on the AMF:

traffic type index: AMF registration state and/or idle state user number, paging request times, paging response times for one time and paging response times for two times;

success rate type index: the success rate of initial registration, the success rate of registration update, the success rate of switching, the success rate of UE CM deregistering, the success rate of establishing N11 interface session context, the success rate of updating N11 interface session context, the success rate of releasing N11 interface session context, the success rate of inquiring N11 interface session context and the success rate of service request;

the time delay index is as follows: initial registration average duration;

error type index: the number of errors of the authentication parameters, the number of authentication refusals, the number of initial registration failures, the number of registration updating failures and the number of refused service requests;

on SMF:

the delay type index is as follows: PDU conversation establishing flow average time length;

on the UPF:

traffic type indicator: average and/or maximum QoS flow number, GTP packet receiving and/or sending byte number of an N3 interface, GTP packet receiving and/or sending byte number of an N9a interface and GTP packet receiving and/or sending byte number of an N6 interface;

success rate type index: the success rate of PFCP session establishment and the success rate of PFCP session modification;

on UDM:

on the PCF:

on NRF:

traffic type index: storing the number of instances;

on NSSF:

success rate type index: network slice selection success rate;

error type index: the number of network slice selection failures.

according to the NFV network functional architecture, faults are injected layer by layer from the VNF layer to the infrastructure layer.

main NFV network service plane, standby NFV network service plane, management plane, network element component plane and infrastructure.

In implementation, one or a combination of the following faults are injected in the NFV network traffic plane:

injecting a fault into the main NFV network service to completely block the VIM and damage part of service virtual machines of the superposed network elements;

injecting a fault into the main large area service to store disconnection of the batch virtual machines in the main large area;

In implementation, the OMU fault with VNF injected fault at the management plane is inaccessible.

one or a combination of the following indicators on the 5G base station: service volume and capacity type indexes, success rate type indexes, time delay type indexes and error type indexes;

success rate type indicator on UDM;

success type indicator, error type indicator on NSSF.

In implementation, evaluating the NFV network according to the second indicator is to evaluate an ability of an overall network quality of service to combat damage when a component in the NFV network fails.

NFV network element toughness = α (a) ₁ * Network component failure ratio + a ₂ * Calculating/storing component fault proportion + β OMU fault proportion; wherein, the NFV network element toughness is an evaluation index for evaluating the damage resistance of the network element service quality when a component in the NFV network element fails, alpha represents the user plane weight, beta represents the management plane weight, and a represents the management plane weight _i {i∈[1,3]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

NFV network toughness = α + β (b) NFV network element failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high availability network component failure odds + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failure fraction) 100%; the NFV network toughness is an evaluation index for evaluating the damage resistance of the whole network service quality when NFV network element, EMS, VNFM and NFVO functional components fail, wherein alpha represents the user plane weight, beta represents the management plane weight, and b represents the management plane weight _i {i∈[1,6]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

4G wireless network element toughness = (alpha (a)) ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switches, EOR switches, network cards, high availability computing and storage components including one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless network element toughness = (alpha (a)) ₁ *∑5G base station high available network component fault ratio + a ₂ * Σ 5G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault ratio + a ₂ * Σ 4G core network element high available compute and storage component failure plus β OMU failure) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component fault ratio + a ₂ * Σ 5G core network element high available compute and storage component failure) + β OMU failure) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α x 4G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database;sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (α x 5G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows that the fault proportion of each high-availability component is summed, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (α ∑ 4G core network element (MME, servingGW, PGW, HSS, PCRF) failure fraction + β = (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: the TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma represents the summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure ratio + β × (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high availability network component failure odds + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and memory component fault ratio + -) 100%, where the high available components include one ofOne or a combination thereof: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

In an implementation, evaluating the NFV network according to the second index is evaluating one or a combination of the following performances of the NFV network:

the toughness of the wireless network element is used for evaluating proportion weighted summation of faults of each functional layer component of the base station, and the toughness of the core network element is used for evaluating proportion weighted summation of faults of each functional layer component of the core network element.

In implementation, the toughness of the wireless network element is used for verifying the cloud expansion capability and the fault tolerance capability of the wireless network element service and the influence of the uncertainty problem to the maximum extent on the network element steady state; and/or the presence of a gas in the gas,

In implementation, one or a combination of radio network element toughness, core network element toughness of the NFV network is evaluated according to the second indicator as follows:

wireless network element toughness = (alpha (a)) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0,1]]；

Core network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch, EOR switch, network card; computing storageThe component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0, 1')]。

A network evaluation apparatus comprising:

the determining module is used for determining indexes to be evaluated in the NFV network;

the monitoring module is used for monitoring the index and acquiring a first index of the index before fault injection;

a fault module for injecting a fault;

the monitoring module is also used for monitoring the index and acquiring a second index of the index after fault injection;

the fault module is also used for recovering the NFV network until the index is the first index;

and the evaluation module is used for evaluating the NFV network according to the second index.

In an implementation, the method further comprises the following steps:

and the control module is used for controlling the fluctuation of the preset index of the NFV network to be within a preset range when the fault is injected.

In an implementation, the control module is further configured to determine the predetermined indicator as one or a combination of:

success rate type indicator on UDM;

success rate type indicator, error type indicator on NSSF.

one or a combination of the following criteria at the 4G base station:

one or a combination of the following criteria at the 5G base station:

on the MME:

success rate type index: EPS attachment success rate, authentication success rate, DNS analysis success rate initiated by MME, default and/or special bearer activation success rate, PDN connection establishment success rate, service request success rate, paging success rate, tracking area update success rate and switching success rate;

on the ServingGW:

success rate type index: the success rate of establishing a default and/or special bearing of the servingGW;

on the PGW:

at the HSS:

traffic and capacity type indices: HSS number allocation and/or number of active users, HSS authentication capacity utilization rate and HSS static capacity utilization rate;

error type index: updating the location failure times;

on the PCRF:

error type index: applying the session call loss rate;

on the AMF:

the time delay index is as follows: initial registration average duration;

on SMF:

traffic type indicator: average and/or maximum PDU session number, average and/or maximum Qos flow number;

on the UPF:

on UDM:

on the PCF:

traffic type indicator: the average and/or maximum value of the AM strategy association total number and the average and/or maximum value of the SM strategy association total number;

on NRF:

traffic type indicator: storing the number of instances;

success rate type index: the NF initiates an updating success rate and an NF discovers the success rate;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

In implementation, the fault module is further configured to, when injecting the fault, inject the fault layer by layer from an infrastructure layer to a VNF layer according to the NFV network functional architecture; and/or the presence of a gas in the gas,

In an implementation, the fault module is further configured to, when injecting the fault, inject the fault in one or a combination of the following aspects:

In an implementation, the fault module is further configured to inject one or a combination of the following faults at the NFV network traffic plane:

deleting an S1 interface route for the data center gateway at the main NFV network service injection fault to cause the main NFV network service to be completely blocked;

In an implementation, the failure module is further configured to inject, on the management plane, that the OMU failure of the VNF is inaccessible.

In an implementation, the determining module is further configured to determine that the indicator is an NFV network performance indicator of one or a combination of the following indicators:

a traffic type index, a success rate type index, a time delay type index and an error type index on the MME;

success rate type indicator on UDM;

traffic type index, success rate type index, error type index on NRF;

success type indicator, error type indicator on NSSF.

one or a combination of the following indicators on the 4G base station:

one or a combination of the following indicators on the 5G base station:

traffic and capacity type indices: the gNB sends and/or receives the service data volume from the NG interface, the byte number of the cell user plane uplink and/or downlink PDCP PDU, the traffic data volume received and/or sent by the gNB from the S1 interface, the number of uplink and/or downlink transmission TBs, the number of available downlink PDSCH PRBs, the number of paging record receiving, the average and/or maximum number of RRC connections in dual connection, the average and/or maximum ERAB number of NSA SCG Split Bear type;

the delay type index is as follows: the method comprises the following steps of RRC connection average and/or maximum establishing time, RLC downlink data packet average processing time delay, NG switching average time length among gNBs, xn switching average time length among gNBs, epsfallback service switching time delay from 5G to 4G, and RLC downlink data packet average processing time delay of each slice cell;

on the MME:

on the ServingGW:

on the PGW:

the delay type index is as follows: the average and/or maximum duration of the special bearer initiated by the PGW is established;

at the HSS:

success rate type index: HSS authentication information query success rate, HSS updating and/or canceling position success rate, HSS inserting and/or deleting user data success rate and HSS UE removing success rate;

error type index: updating the position failure times;

on the PCRF:

traffic type index: gx session handling capacity average and/or peak utilization;

error type index: applying the session call loss rate;

on the AMF:

the time delay index is as follows: initial registration average duration;

on SMF:

success rate type index: a PDU session establishment success rate, a PDU session modification success rate initiated by SMF, an SM strategy establishment success rate of an N7 interface, an SM strategy registration success rate of an N10 interface UE context, an SM strategy updating success rate of an N7 interface, an SM strategy de-registration success rate of an N10 interface UE context and an SM strategy deletion success rate of an N7 interface;

the delay type index is as follows: the PDU conversation establishes the average duration of the flow;

on the UPF:

traffic type indicator: average and/or maximum QoS flow number, GTP packet receiving and/or sending byte number of an N3 interface, GTP packet receiving and/or sending byte number of an N9a interface and byte number of an N6 interface;

error type index: PFCP session establishment and/or modification failure times, the number of GTP packets received by an N3 interface in error, the number of IP packets discarded by an N6 interface in error, and the number of GTP packets received by an N9c interface in error;

on the UDM:

on the PCF:

success rate type index: the method comprises the following steps of AM strategy association establishment success rate, AM strategy association updating success rate, AM strategy association deletion success rate, SM strategy association establishment success rate, SM strategy association updating success rate and SM strategy association deletion success rate;

on NRF:

traffic type index: storing the number of instances;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

In an implementation, the evaluating module is further configured to evaluate, according to the second index, that the NFV network is an ability to evaluate an overall network quality of service against damage when a component in the NFV network fails.

In an implementation, the evaluation module is further configured to evaluate the NFV network according to the second criterion in one or a combination of the following ways:

NFV network toughness = α + β (b) NFV network element failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failure fraction) 100%; the NFV network toughness is an evaluation index for evaluating the damage resistance of the whole network service quality when NFV network element, EMS, VNFM and NFVO functional components fail, alpha represents the weight of a user plane, beta represents the weight of a management plane, and b represents the weight of the management plane _i {i∈[1,6]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

4G wireless network element toughness = (alpha (a) = ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switches, EOR switches, network cards, high availability computing and storage components including one or a combination of the following: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless network element toughness = (alpha (a) = ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Σ 5G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault proportion + a ₂ * Σ 4G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component failure percentage + a ₂ * Σ 5G core network element high available compute and storage component failure) + β OMU failure) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α x 4G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, wherein the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the highly available computing and storage components includeOne or a combination thereof: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (α x 5G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machines, virtual machines, databases; sigma shows that the fault proportion of each high-availability component is summed, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (alpha:sigma4G core network element (MME, servingGW, PGW, HSS, PCRF) fault proportion + beta: (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure ratio + β × (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Sigma NFVO high available compute and storage component failure fraction100%, wherein the highly available components include one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

In an implementation, the evaluating module is further configured to evaluate the NFV network according to the second index as one or a combination of the following performances of the NFV network:

In an implementation, the evaluating module is further configured to evaluate one or a combination of the radio network element toughness, the core network element toughness of the NFV network according to the second index as follows:

wireless network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the values of alpha and beta are 0.5, and the suggested values of a1 and a2 are 1;

core network element toughness = (α = (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) + 100%Wherein, the network component comprises one or the combination of the following components: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0,1]]。

A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the above-described network evaluation method.

The invention has the following beneficial effects:

in the technical scheme provided by the embodiment of the invention, the index to be evaluated in the NFV network is determined firstly, then the fault is injected, the NFV network is evaluated according to the index by collecting the index after the fault is injected, and therefore, the technical scheme for evaluating the fault tolerance capability of the NFV network and the damage resistance degree of the NFV network within the acceptable range of the service index is provided.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:

FIG. 1 is a schematic diagram illustrating a flow chart of a network evaluation method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating components of an NFV network in accordance with an embodiment of the present invention;

FIG. 3 is a schematic diagram of a relation of a chaotic engineering experiment system in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a closed loop system of a chaos engineering experiment in an embodiment of the invention;

FIG. 5 is a diagram illustrating an NFV network functional architecture in an embodiment of the invention;

fig. 6 is a schematic structural diagram of a network evaluation apparatus according to an embodiment of the present invention.

Detailed Description

The prior art scheme does not relate to how to evaluate the fault tolerance capability of the NFV network and the damage resistance degree of the NFV network within the acceptable range of the service index. Therefore, the scheme mainly solves the problem of how to more accurately evaluate the fault-tolerant capability, namely the damage resistance degree, of the NFV network, and the embodiment of the invention is also called toughness and specifically comprises two parts, namely the toughness of a wireless network element and the toughness of a wireless network.

The following describes embodiments of the present invention with reference to the drawings.

Fig. 1 is a schematic flow chart of an implementation of a network evaluation method, as shown in the figure, the method may include:

step 101, determining indexes to be evaluated in an NFV network;

102, monitoring the index, and collecting a first index of the index before fault injection;

103, injecting faults;

104, monitoring the index, and acquiring a second index of the index after fault injection;

step 105, restoring the NFV network until the index is a first index;

and step 106, evaluating the NFV network according to the second index.

In the embodiment, a chaos engineering method is adopted, the affected degree of network service indexes is monitored by injecting faults into each layer/module of the NFV network, then the proportion of faulty components in each layer/module is analyzed, and the toughness index of the NFV network is calculated and used for measuring the reliability of the damage resistance of the NFV network. The following embodiments will be mainly described in three parts:

firstly, the following steps: establishing a system framework for testing the reliability of the NFV network by chaotic engineering;

II, secondly: fault is injected into each layer/module, and the degree of influence of the service index is monitored;

thirdly, the steps of: and calculating a system toughness index for measuring the reliability of the system to resist damage.

In the cloud era, with the evolution of large-scale distributed systems and micro-service architecture technologies, the behavior of each single micro-service is reasonable, but the dependency between services is complicated, and the combination of the dependent behaviors between micro-services in a specific scene may cause problems beyond the expectation of the system. Meanwhile, due to the fact that business and technology iteration is fast, the continuous guarantee of the stable health of the system faces a great challenge.

By utilizing a chaotic engineering method, through establishing an implementation plan, defining steady-state indexes of an NFV network, executing fault injection, checking the steady-state indexes of the NFV network, repairing and finding problems, automating continuous verification and other links, the influence of uncertainty problems on the steady state of the NFV network can be reduced to the greatest extent, including verification of fault-tolerant red lines of the NFV network and the expansion capability of cloud services, monitoring of the timeliness of alarms, and whether the emergency capability of positioning and solving the problems is qualified or not, so that the fault-tolerant capability of an NFV network architecture is improved, the fault emergency processing efficiency is improved, the fault recurrence rate is reduced, and the reliability of the NFV network is improved as a whole.

Fig. 2 is a schematic diagram of components of an NFV Network, and as shown in the figure, components of the NFV Network generally include a Management plane and a service plane, where the Management plane includes NFVO (NFV Orchestrator, NFV organiser)/NFVO +, VNFM (VNF Manager, VNF Manager; VNF: virtual Network Function) and EMS (Element Management System); the business side comprises an infrastructure layer (inside a data center), a computing/storage layer, a Cloud Operating System (Cloud Operating System) layer and a Virtual Network Function (VNF) layer; each layer includes an unequal number of key components.

Firstly, the method comprises the following steps: and establishing a system framework for testing the reliability of the NFV network by chaotic engineering.

Fig. 3 is a schematic diagram of a relation of a chaotic engineering experiment system, and as shown in the figure, the chaotic engineering experiment system is firstly established for realizing large-scale fault drilling, and the chaotic engineering experiment system has an orchestration function and can adapt to new services. The main functional architecture comprises a fault injection capability layer, an orchestration management layer and an application layer. The chaotic engineering experiment system is in butt joint with the NFV network and the network management system, and can realize fault injection of each functional module of the NFV network, all-around monitoring of the NFV network and real-time closed-loop management of experiment effect evaluation.

Fig. 4 is a schematic diagram of a closed-loop system of a chaos engineering experiment, and as shown in the figure, the complete chaos engineering experiment is a continuously iterative closed-loop system, which mainly includes:

experiment requirements (iteration), feasibility evaluation, monitoring index design, scene design, fault injection capability arrangement, experiment plan making, experiment execution, environment recovery, result analysis, problem tracking, pipeline integration and experiment requirements (iteration) \8230

Starting from experimental requirements, determining an experimental range through feasibility evaluation, designing appropriate monitoring indexes, experimental scenes and environments, and selecting appropriate fault injection capability to arrange; establishing an experiment plan, fully communicating with a human being of a stem system of an experiment object, further jointly executing an experiment process, and collecting a pre-designed monitoring index; after the experiment is finished, cleaning and recovering the experimental environment, analyzing the experimental result, tracing the root cause and solving the problem; the experimental scenes are automated and are merged into a production line to be executed regularly; new experimental ranges can then be added, with continued iteration and orderly improvement. The feasibility evaluation needs to consider the evaluation of the fault resistance capability of the architecture, the selection of experimental environment (development environment, test environment, pre-production environment and production environment), the minimum explosion radius in a fault scene (the lowest influence degree of the fault on the business in the drilling process) and the evaluation of the recovery capability of the experimental environment.

II, secondly, the method comprises the following steps: and (5) fault injection is carried out on the NFV network, and the influence degree of the service index is monitored.

In implementation, when the fault is injected, the fault is injected layer by layer from an infrastructure layer to a VNF layer according to the NFV network function architecture; and/or the presence of a gas in the gas,

Specifically, fig. 5 is a schematic diagram of an NFV network functional architecture, and as shown in the diagram, according to the NFV network functional architecture, when a fault is injected layer by layer from an infrastructure layer to a VNF layer, a service influence radius is gradually increased, and the exposure problem is more abundant. And injecting faults layer by layer from the VNF layer to the infrastructure layer, wherein the service influence radius is gradually reduced, and the exposure problem is more focused.

In implementation, when the fault is injected, the fault is injected in one or a combination of the following aspects:

And injecting faults from the service plane, the management plane and the network element component plane of the main NFV network and the standby NFV network, and verifying the damage resistance of the NFV network. The main fault scenes include that the main NFV network service is completely blocked, the quality of the main NFV network service is damaged, the user plane service is damaged, and the management plane is in fault.

The specific implementation can be as follows:

all blocking scenarios of main NFV network traffic: the injection fault is a cloud OS database IP address conflict, all the VIMs (Virtual Infrastructure Management) block the service Virtual machines of part of the superimposed network elements and are damaged, and the data center gateway deletes the S1 interface route to cause all the main NFV network services to be blocked. The method is used for verifying the cross-region disaster recovery switching capability of the VNF;

main large area service quality loss scene: the injection fault is the storage disconnection of the main large-area batch virtual machine, the read-write time delay of the CloudOS storage component is large, the fault is exchanged, and the bypass capacity of the storage and the VNF fault positioning and processing capacity are verified;

user plane service quality impairment scenario: the injection fault is the interruption of the connection between the user plane network element and the control plane network element caused by the fault of the CloudOS component, and the fault positioning and processing capability is verified.

Managing a plane fault: and (3) the OMU (operation and Maintenance Unit) with the injected fault being VNF cannot be accessed, and the fault positioning and handling capacity is verified.

In an implementation, the method may further include:

one or a combination of the following criteria at the 4G base station: a traffic type index, a success rate type index, a delay type index and an error type index;

a traffic type index, a success rate type index, a delay type index, and an error type index on an MME (Mobility Management Entity);

the service volume, capacity type index and success rate type index on the Serving Gateway (Serving Gateway);

service volume and capacity type indexes, delay type indexes and error type indexes on a PGW (PDN Gateway; PDN: packet Data Network, packet Data Network);

a service volume and capacity type index, a success rate type index, and an error type index on an HSS (Home Subscriber Server);

a service volume type index, a success rate type index and an error type index on a PCRF (Policy and Charging Rules Function entity);

a traffic type index, a success rate type index, a delay type index, and an error type index on an AMF (Access and Mobility Management Function);

a traffic type index, a success rate type index, a delay type index, and an error type index on an SMF (Session Management Function);

a traffic type index, a success rate type index, and an error type index on a UPF (User Plane Function);

success rate type indicator on UDM (Unified Data Management entity);

a traffic type index, a success rate type index and an error type index on a PCF (Policy Control Function);

a service volume type index, a success rate type index and an error type index on an NRF (Network storage Function);

success rate type indicator and error type indicator on NSSF (Network Slice Selection Function).

In a specific implementation, the predetermined index is one of the following indexes or a combination thereof:

one or a combination of the following criteria at the 4G base station: traffic type index: the number of bytes sent/received by an eNB Ethernet interface, the number of bytes of Service sent/received by an eNB S1 interface, the number of bytes of cell user plane/control plane uplink/downlink PDCP (Packet Data Convergence Protocol) SDU (Service Data Unit) and the like; success rate type index: service-related E-RAB (Evolved Radio Access Bearer) establishment success rate, service-related Radio Access rate, RRC (Radio Resource Control) connection establishment success rate, handover success rate (including intra eNB/inter eNB, common frequency/different frequency, handover in/handover out, intra system/inter system), and the like, and delay type indexes: RRC connection average/maximum setup duration, E-RAB average/maximum setup duration, etc., error type indicator: the wireless call drop rate (including session voice service and session live video stream service), the call drop rate (including real-time game service, non-session buffered video stream service and IMS signaling service) and the wireless call drop rate based on TCP (Transmission Control Protocol) service, the uplink/downlink PDCP SDU packet loss rate, the number of errors of uplink/downlink Transmission blocks, the number of discarded paging records and the like;

one or a combination of the following indicators on the 5G base station: traffic and capacity type indicators: the serving Data volume (only applicable to SA networking) transmitted/received by the gNB from the NG interface, the serving Data volume (only applicable to SA networking) of the Cell user plane uplink/downlink PDCP PDU (Packet Data Unit), the serving Data volume (only applicable to NSA networking) received/transmitted by the gNB from the S1 interface, the Transport Block (Transport Block) number (applicable to NSA & SA networking), the Physical downlink shared channel (Physical downlink shared channel) PRB (Physical resource Block) available number (applicable to NSA & SA networking), the paging record reception number (only applicable to SA networking), the RRC connection average/maximum number in dual connection, and the average/maximum RRC connection index number of NSA (Non-Stand Alone network) type SCG (Secondary Cell Group, cell Group) Split Bear, and the like, and the success rate of the type is as follows: RRC connection establishment success rate (only applicable to SA networking) = RRC connection establishment success number/RRC connection establishment request number, flow establishment success rate (only applicable to SA networking) = Flow establishment success number/Flow establishment request number, PDU session establishment success rate (only applicable to SA networking) = PDU establishment success number/PDU establishment request number, handover success rate (including same frequency/different frequency, gsb internal/gsb internal, handover out/handover in) = handover success number/handover request number, NG interface UE related logical signaling connection establishment success rate (only applicable to SA networking) = NG interface UE related logical signaling connection establishment success number/NG interface UE related logical signaling connection establishment request number, and the like, delay type index: the average/maximum establishment time of RRC connection, the average processing time delay of RLC (Radio Link Control) downlink data packets (only applicable to SA networking), the average time delay of NG switching between gNBs (only applicable to SA networking), the average time delay of Xn switching between gNBs (only applicable to SA networking), the time delay of switching from 5G to 4G of EPS fallback (EPS: evolved Packet System, evolved Packet System) service (only applicable to SA networking), the average processing time delay of RLC downlink data packets per slice cell (only applicable to SA networking), and the like, and error type indexes: flow establishment/modification failure number (only applicable to SA networking), uplink/downlink PDCP packet loss number (applicable to NSA & SA networking), and the like

MME: traffic type index: average/maximum number of MME bearers, number of MME idle state/connected state users, average/maximum number of MME attached users, and the like, success rate type index: EPS attachment success rate, authentication success rate, DNS (Domain Name Server) resolution success rate initiated by MME, default/dedicated bearer activation success rate, PDN connection establishment success rate, service request success rate, paging success rate, tracking area update success rate (within/between MME, within/between system), handover success rate (within/between MME, within/between system), etc., delay type index: average/maximum attachment duration, dedicated bearer setup average/maximum duration, etc., error type indicator: the number of authentication parameter errors, the number of UE authentication failures, the number of EPS attachment failures, the number of tracking area update failures and the like;

servingGW: traffic and capacity type indices: average/peak utilization rate of SGW bearing capacity, user plane uplink/downlink flow, GTP (GPRS Tunneling Protocol) generated by each user attached to the serving GW (General Packet Radio Service), average/downlink flow, average/maximum user number attached to the serving GW, average/maximum bearing number of the serving GW, SGW data throughput capacity utilization rate and the like, and success rate type indexes: a success rate of establishing a default/dedicated bearer of the ServingGW, etc.;

PGW: traffic and capacity type indicators: PGW data throughput capacity utilization, PGW load capacity average/peak utilization, PGW S5/S8 interface uplink/downlink traffic, PGW average/maximum attached user number, PGW average/peak load number, SGi interface receive/transmit traffic, and the like, and success rate type index: dedicated bearer establishment success rate, CDR transmission success rate, and the like, delay type index: the average/maximum duration of the special bearer setup initiated by the PGW, etc., error type index: the number of GTP packets discarded by errors of a PGW S5/S8 interface, the number of IP packets discarded by errors of an SGi interface and the like;

HSS: traffic and capacity type indicators: HSS number allocation/active user number, HSS authentication capacity utilization rate, HSS static capacity utilization rate and the like, and the success rate type index is as follows: HSS authentication information inquiry success rate, HSS updating/canceling position success rate, HSS inserting/deleting user data success rate, HSS UE clearing success rate and the like, error type indexes are as follows: number of location update failures, etc.;

PCRF: traffic type index: average/peak utilization rate of Gx session processing capacity, success rate type index: strategy control initiates/updates/finishes success rate, re-authentication success rate and application session authorization success rate, error type index: application session call loss rate, etc.

AMF: traffic type index: AMF registration state/idle state user number, paging request times, first paging response times, second paging response times and the like, and success rate type indexes are as follows: initial registration success rate = initial registration success number/initial registration request number, registration update success rate = registration update acceptance number/registration update request number, handover success rate (including AMF internal/inter-system, intra-system/inter-system) = handover success number/handover attempt number, UECM (UE Context Management ) deregistration success rate (including AMF initiation and UDM initiation) = UECM deregistration success number/UECM deregistration request number, N11 interface session Context setup success rate = N11 interface session Context setup success number/N11 interface session Context setup request number, N11 interface session Context update success rate = N11 interface session Context update success number/N11 interface session Context update request number, N11 interface session Context release success rate = N11 interface session Context release number/N11 interface session Context release request number, N11 interface session Context query success rate = N11 interface session Context query success number/N11 interface session Context query request number, service request success rate, and the like, and the index class: average time of initial registration, etc., error type index: the number of errors of the authentication parameters, the number of authentication refusals, the number of initial registration failures, the number of registration update failures and the number of refused service requests;

SMF: traffic type index: average/maximum PDU session number and average/maximum Qos (Quality of Service) flow number, etc., success rate type index: PDU Session establishment success rate = PDU Session establishment success rate/PDU Session establishment request rate, SMF-initiated PDU Session modification success rate = SMF-initiated PDU Session modification success rate/SMF-initiated PDU Session modification request rate, N7 interface SM (Session Management) policy success rate = N7 interface SM policy creation success rate/N7 interface SM policy creation request rate, N10 interface UE context registration success rate = N10 interface UE context registration success rate/N10 interface UE context registration request rate, N7 interface SM policy update success rate = N7 interface SM policy update success rate/N7 interface SM policy update request rate, N10 interface UE context de-registration success rate = N10 interface UE context de-registration success rate/N10 interface UE context de-registration request rate, N7 interface SM deletion policy success rate = N7 interface SM deletion success rate/N7 interface SM policy deletion request rate, and the like, the delay type index: PDU conversation establishment flow average duration and the like, error type indexes: PDU session establishment failure times, SMF initiated PDU session modification failure times and the like;

UPF: traffic type index: average/maximum QoS flow number, number of GTP packet bytes received/sent by N3 interface, number of GTP packet bytes received/sent by N9a interface, number of GTP packet bytes received/sent by N6 interface, and the like, and the success rate type index: PFCP session establishment success ratio = PFCP session establishment success number/PFCP session establishment request number, PFCP session modification success ratio = PFCP session modification success number/PFCP session modification request number, and the like, error type index: the number of times of failure of PFCP session establishment/modification, the number of GTP packets received by an N3 interface, the number of IP packets discarded by an N6 interface error and the number of GTP packets received by an N9c interface;

UDM: success rate type index: UECM registration success rate initiated by AMF = UECM registration success rate initiated by AMF/UECM registration request rate initiated by AMF, UECM registration success rate initiated by SMF = UECM registration success rate initiated by SMF/UECM registration request rate initiated by SMF, registration parameter update success rate = registration parameter update success rate/registration parameter update request rate, user data acquisition success rate = user data acquisition success rate/user data acquisition request rate, user data subscription success rate = user data subscription success rate/user data subscription request rate, user data unsubscribe success rate = user data unsubscribe success rate/user data unsubscribe request rate, etc.;

PCF: traffic type indicator: AM (Access and Mobility Management) policy association total average/maximum, SM policy association total average/maximum, and the like, and success rate type index: AM policy association establishment success rate = AM policy association establishment success rate/AM policy association establishment request number, AM policy association update success rate = AM policy association update success rate/AM policy association update request number, AM policy association deletion success rate = AM policy association deletion success rate/AM policy association deletion request number, SM policy association establishment success rate = SM policy association establishment success rate/SM policy association establishment request number, SM policy association update success rate = SM policy association update success rate/SM policy association update request number, SM policy association deletion success rate = SM policy association deletion success rate/SM policy association deletion request number, and the like, and an error type index: the SM strategy is associated with the establishment failure times, and the SM strategy is associated with the update failure times;

NRF: traffic type index: number of stored instances, etc., success rate type indicator: NF (Network Function) update initiation success rate = NF update initiation success times/NF update initiation request times, NF discovery success rate = NF discovery success times/NF discovery request times, etc., and error type index: the NF initiates updating failure times, NF discovers failure times and the like;

NSSF: success rate type index: network slice selection success ratio = network slice selection success times/network slice selection request times, error type index: network slice selection failure times.

When the service quality index of the NFV network is within the allowable fluctuation range, the service may be considered to be less affected. For example: the fluctuation of the success rate index is less than a%, the fluctuation of the service quantity index is less than b%, the fluctuation of the delay index is less than c%, the fluctuation of the error index is less than d%, and the fluctuation threshold value is set according to the service requirement of a specific scene.

Thirdly, the steps of: and calculating the toughness index of the network and/or the network element, and measuring the reliability of the damage resistance of the network and/or the network element.

The technical scheme provided by the embodiment mainly considers the damage resistance of the cloud network, so that the network components mainly consider the internal components of the data center and do not consider the components outside the data center. According to the chaos engineering verification result, defining an NFV network toughness index, representing the capability of resisting damage of the whole network service quality when main components in the NFV network break down, and defining the capability as proportion weighted summation of all functional layer components when the functional layer components break down, wherein the higher the toughness index value is, the stronger the damage resistance of the network is.

In a specific implementation, the NFV network is evaluated according to the second criterion in one or a combination of the following ways:

NFV network toughness = α + β (b) NFV network element failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability meterComputing and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failure fraction) 100%; the NFV network toughness is an evaluation index for evaluating the damage resistance of the whole network service quality when NFV network element, EMS, VNFM and NFVO functional components fail, wherein alpha represents the user plane weight, beta represents the management plane weight, and b represents the management plane weight _i {i∈[1,6]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

5G wireless network element toughness = (alpha (a) = ₁ * Sigma 5G base station high available network component failure fraction + a ₂ * Σ 5G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault proportion + a ₂ * Σ 4G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma expression for each high available module fault ratioAnd, α, β, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component fault ratio + a ₂ * Σ 5G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α x 4G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (α x 5G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, wherein the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows that the fault proportion of each high-availability component is summed, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (alpha:sigma4G core network element (MME, servingGW, PGW, HSS, PCRF) fault proportion + beta: (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high performanceWith computing and storage component failure ratio + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure fraction + β [ (+ b [ ]) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high availability network component failure odds + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

In an implementation, one or a combination of the radio network element toughness and the core network element toughness may be evaluated according to the second indicator as follows:

wireless network element toughness = (alpha (a)) ₁ * Network component failure fraction + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the values of alpha and beta are 0.5, and the suggested values of a1 and a2 are 1;

core network element toughness = (α (a) ₁ * Network componentBarrier to area ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0, 1')]。

In specific implementation, the method can be as follows:

1. and (5) toughness of the wireless network element.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the wireless network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

The index definition: in a base station fault injection scenario, the fluctuation of the key performance index of the base station is within an acceptable range (for example: 1% to 3%), and the toughness of the network element is defined as a proportion weighted sum of faults occurring in functional layer components.

Wherein the key performance indicators include: flow type index, success rate type index and error type.

Calculating the formula: (α: (a) ₁ * Network component failure ratio + a ₂ * The percentage of failures of the computing/storing component) + β OMU percentage of failures) × 100%, where the value ranges of α, β, a1, a2 are [0,1]。

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: and (4) network elements.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

2. And (5) wireless network toughness.

Communication network health dimension: and (6) reliability.

Service requirements: the method is used for verifying the cloud expansion capability and fault tolerance capability of the wireless network service and the influence of the maximum uncertainty problem on the network steady state. It is recommended that fault injection be performed once a month.

The index definition: in each base station fault injection scene, the fluctuation of each key performance index of the wireless network is within an acceptable range (for example: 1% to 3%), and the network toughness is defined as the proportion weighted sum of the faults of the base stations and functional layer components of the EMS functional components.

The key performance indexes include: averaging the average flow type indexes of all network elements in the whole network, taking the maximum flow type index of all network elements, averaging the success rate type indexes of the whole network, and summing the error type indexes of the whole network.

Calculating the formula: (α x 4G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, where the high availability network components include TOR switches, EOR switches, network cards, and the like, and the high availability computing and storage components include physical machines, virtual machines, databases, and the like. Sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]。

The data type is as follows: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

3. Core network element toughness.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and fault tolerance capability of the core network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

And (3) index definition: in a core network element fault injection scene, the fluctuation of key performance indexes of the core network element is within an acceptable range (for example, 1% to 3%), and the toughness of the core network element is defined as the proportion weighted sum of the faults of each functional layer component. Key performance indicators of the core network elements are as follows:

flow type index, success rate type index and error type.

Calculating the formula: (α: (a)) ₁ * Sigma core network element high available network component failure fraction + a ₂ * Sigma core network element high-availability computing and storage component failure rate) + beta OMU failure rate) 100%, wherein the high-availability network components comprise TOR switches, EOR switches, network cards and the like, and the high-availability computing and storage components comprise physical machines, virtual machines, databases and the like. Sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]。

Data type: real numbers.

Data unit: %.

Measurement object: and (4) network elements.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

4. Core network toughness.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the core network service and the influence of the maximum uncertainty problem on the network steady state. It is recommended that fault injection be performed once a month.

The index definition: in a core network fault injection scenario, the fluctuation of key performance indicators of the core network is within an acceptable range (e.g., 1% to 3%), and the core network toughness is defined as a weighted sum of the fractions of faults occurring in network elements of the core network, EMS, VNFM, and NFVO. Network key performance indicators: averaging the average flow type indexes of all network elements in the whole network, taking the maximum flow type index of all network elements, averaging the success rate type indexes of the whole network, and summing the error type indexes of the whole network.

Calculating the formula: (alpha sigma core network element fault proportion + beta (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Sigma NFVO high availability compute and storage component fault preemption100% in the formulation, wherein the highly available components comprise one or a combination of the following components: the TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma represents the summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

5. And 4G wireless network element toughness.

Communication network health dimension: and (6) reliability.

Service requirements: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the 4G wireless network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

The index definition: in a base station fault injection scenario, the fluctuation of a base station key performance index is within an acceptable range (for example: 1% to 3%), and the toughness of a network element is defined as a proportion weighted sum of faults occurring in each functional layer component of the base station. Wherein the key performance indicators include: traffic type indicator: the number of bytes sent/received by an eNB Ethernet interface, the number of bytes of service sent/received by an eNB S1 interface, the number of bytes of cell user plane/control plane uplink/downlink PDCP SDU (packet data convergence protocol) SDU (service data Unit); success rate type index: service-related E-RAB establishment success rate, service-related radio access rate, RRC connection establishment success rate, handover success rate (including eNB interior/between eNBs, same frequency/different frequency, handover in/handover out, system interior/system interior), and the like, and a delay type index: RRC connection average/maximum setup duration, E-RAB average/maximum setup duration, etc., error type indicator: the wireless dropped call rate (including conversation voice service and conversation live broadcast video streaming service), the dropped call rate (including real-time game service, non-conversation buffered video streaming service and IMS signaling service) and the wireless dropped call rate based on TCP service, the uplink/downlink PDCP SDU lost packet rate, the number of uplink/downlink transmission block errors, the number of dropped paging records and the like.

Calculating the formula: (α: (a)) ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high availability computing and storage component failure fraction) + β × OMU failure fraction) × 100%, where the high availability network components include TOR switches, EOR switches, network cards, and the like, and the high availability computing and storage components include physical machines, virtual machines, databases, and the like. The high available component fault ratio calculation method takes a network card as an example, the high available network card fault ratio = number of failed high available network cards/total number of high available network cards, Σ represents the sum of the fault ratios of the high available components, α, β, a ₁ 、a ₂ The value range is [0,1]]。

Data type: real numbers.

Data unit: %.

Measurement object: and (4) network elements.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

6. 4G wireless network toughness.

Communication network health dimension: and (6) reliability.

Service requirements: the cloud expansion capability and the fault tolerance capability of the 4G wireless network service are verified, and the influence of the uncertainty problem on the network steady state is maximized. It is recommended that fault injection be performed once a month.

And (3) index definition: in a base station fault injection scenario, the fluctuation of the overall key performance index of the available base stations in the wireless network is within an acceptable range (e.g., 1% to 3%), and the network toughness is defined as the proportion weighted sum of the faults of the base stations and EMS functional components. Wherein the key performance indicators include: traffic type indicator: sigma eNB Ethernet interface sending/receiving byte number, sigma eNB S1 interface sending/receiving service byte number, sigma cell user plane/control plane uplink/downlink PDCP SDU byte number and the like; success rate type index: AVERAGE (service-related E-RAB establishment success rate), AVERAGE (service-related radio access rate), AVERAGE (RRC connection establishment success rate), AVERAGE [ handover success rate (including intra eNB/inter eNB, same frequency/different frequency, handover in/handover out, intra system/inter system) ], and the like, and delay type indicators: AVERAGE (RRC connection AVERAGE establishment duration), MAX (RRC connection maximum establishment duration), AVERAGE (E-RAB AVERAGE establishment duration), MAX (E-RAB maximum establishment duration), and the like, and the error type index: the method comprises the steps of AVERAGE [ wireless call drop rate (including session voice service and session live video stream service) ], AVERAGE [ call drop rate (including real-time game service, non-session buffered video stream service and IMS signaling service) ], AVERAGE (based on TCP service wireless call drop rate), AVERAGE (uplink/downlink PDCP SDU packet loss rate), ∑ uplink/downlink transmission block error number, Σ paging record discarding number and the like, wherein Σ represents summation of indexes of all base stations, AVERAGE represents averaging of indexes of all base stations, and MAX represents maximum value of indexes of all base stations.

Calculating the formula: (α x 4G base station failure fraction + β x (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, where the high availability network components include TOR switches, EOR switches, network cards, and the like, and the high availability computing and storage components include physical machines, virtual machines, databases, and the like. The high available component fault ratio calculation method takes a network card as an example, the high available network card fault ratio = number of failed high available network cards/total number of high available network cards, Σ represents the sum of the fault ratios of the high available components, α, β, b ₁ And b ₂ The value range is [0,1]]。

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

7. And (5G) toughness of the wireless network element.

Communication network health dimension: and (6) reliability.

Service requirements: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the 5G wireless network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

The index definition: in a base station fault injection scene, the fluctuation of each key performance index of a wireless network element is within an acceptable range (for example, 1 to 3 percent), and the toughness of the network element is defined as the proportion weighted sum of faults occurring in functional layer components of a base station. Wherein the key performance indicators include: traffic and capacity type indicators: the gNB sends/receives service data volume (only suitable for SA networking) from an NG interface, the byte number (suitable for NSA & SA networking) of uplink/downlink PDCP PDU (only suitable for NSA networking) of a cell user plane, the service data volume (only suitable for NSA networking) received/sent by the gNB from an S1 interface, the number (suitable for NSA & SA networking) of uplink/downlink transmission TB, the number (suitable for NSA & SA networking) of downlink PDSCH PRB available, the number (only suitable for SA networking) of paging record receiving, the average/maximum number of RRC connections in double connection, the average/maximum ERAB number of NSA SCG Split Bear types, and the like, and the success rate type index: RRC connection establishment success rate (only applicable to SA networking) = RRC connection establishment success number/RRC connection establishment request number, flow establishment success rate (only applicable to SA networking) = Flow establishment success number/Flow establishment request number, pdussion establishment success rate (only applicable to SA networking) = pdussion establishment success number/pdussion establishment request number, handover success rate (including same frequency/different frequency, between gNB interiors/gnbs, handover out/handover in) = handover success number/handover request number, NG interface UE related logic signaling connection establishment success rate (only applicable to SA networking) = NG interface UE related logic signaling connection establishment success number/NG interface UE related logic signaling connection establishment request number, and the like, delay type index: the average/maximum establishment time of RRC connection, the average processing time delay of RLC downlink data packets (only applicable to SA networking), the average time length of NG switching among gNBs (only applicable to SA networking), the average time length of Xn switching among gNBs (only applicable to SA networking), the time delay of switching an Epsfallback service from 5G to 4G (only applicable to SA networking), the average processing time delay of RLC downlink data packets of each slice cell (only applicable to SA networking), and the like, wherein the error type indexes are as follows: flow setup/modification failure number (applicable only to SA networking), uplink/downlink PDCP packet loss number (applicable to NSA & SA networking), etc.

Calculating the formula: (α: (a) ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Sigma 5G base station high availability computing and storage component fault occupationRatio) + β OMU failure fraction) × 100%, where the high-availability network components include TOR switches, EOR switches, network cards, etc., and the high-availability computing and storage components include physical machines, virtual machines, databases, etc. The high available component fault ratio calculation method takes a network card as an example, the high available network card fault ratio = number of failed high available network cards/total number of high available network cards, Σ represents the sum of the fault ratios of the high available components, α, β, a ₁ 、a ₂ The value range is [0,1]]。

Data type: real numbers.

Data unit: %.

Measurement object: a network element.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

8. 5G wireless network toughness.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and fault tolerance capability of the 5G wireless network service and the influence of the maximum uncertainty problem on the network steady state. It is recommended that fault injection be performed once a month.

And (3) index definition: in each base station fault injection scenario, the overall key performance index fluctuation of the wireless network is within an acceptable range (e.g., 1% to 3%), and the network toughness is defined as the proportion weighted sum of the faults of the available base stations and EMS functional components. Wherein the key performance indicators include: traffic type indicator: Σ gNB sends/receives a service data amount from an NG interface (only applicable to SA networking), ∑ cell user plane uplink/downlink PDCP PDU byte count (applicable to NSA & SA networking), ∑ gNB receives/sends a service data amount from an S1 interface (only applicable to NSA networking), ∑ uplink/downlink transmission TB number (applicable to NSA & SA networking), AVERAGE [ paging record receiving number (only applicable to SA networking) ], AVERAGE [ mean number of RRC connections in dual connection ], MAX [ maximum number of RRC connections in dual connection ], AVERAGE [ mean ERAB number of NSA SCG Split Bear type ], MAX [ maximum ERAB number of NSA SCG Split Bear type ], and the like, and success rate type indexes: AVERAGE [ RRC connection establishment success rate (only applicable to SA networking) = RRC connection establishment success number/RRC connection establishment request number ], AVERAGE [ Flow establishment success rate (only applicable to SA networking) = Flow establishment success number/Flow establishment request number ], AVERAGE [ pdusisson establishment success rate (only applicable to SA networking) = pdusisson establishment success number/pdusision establishment request number ], AVERAGE [ handover success rate (including same frequency/different frequency, gsb internal/gsb, handover out/handover in) = handover success number/handover request number ], AVERAGE [ NG interface UE related signaling connection establishment success rate (only applicable to SA networking) = NG interface UE related logic signaling connection establishment success number/NG interface UE related logic signaling connection establishment request number ], and the like, delay type index: AVERAGE (AVERAGE time length of RRC connection), MAX (maximum setup time length), AVERAGE [ AVERAGE processing time delay of RLC downlink packets (only applicable to SA networking) ], AVERAGE time length of NG handover between AVERAGE [ AVERAGE time length of NG handover between gnbs (only applicable to SA networking) ], AVERAGE time length of Xn handover between AVERAGE [ AVERAGE time length of G nb (only applicable to SA networking) ], AVERAGE [ time delay of 5G to 4G of epsfarlback service (only applicable to SA networking) ], AVERAGE [ AVERAGE processing time delay of RLC downlink packets per slice cell (only applicable to SA networking) ], and the like, error type indexes: sigma Flow establishing/modifying failure number (only suitable for SA networking), sigma uplink/downlink PDCP packet loss number (suitable for NSA & SA networking) and the like, wherein sigma represents the summation of indexes of all base stations, AVERAGE represents the averaging of the indexes of all the base stations, and MAX represents the maximum value of the indexes of all the base stations.

Calculating the formula: (α x 5G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, where the high availability network components include TOR switches, EOR switches, network cards, and the like, and the high availability computing and storage components include physical machines, virtual machines, databases, and the like. The method for calculating the fault proportion of the high-available components takes a network card as an example, the fault proportion of the high-available network card = fault proportion of high-available network cards/total number of high-available network cards, sigma represents the sum of the fault proportions of the high-available components, alpha, beta and b ₁ And b ₂ The value range is [0,1]]。

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

9. 4G core network element toughness.

Communication network health dimension: and (6) reliability.

Service requirements: the method is used for verifying the cloud expansion capability and fault tolerance capability of the 4G core network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

The index definition: in a 4G core network element fault injection scenario, the fluctuation of key performance indexes of the core network element is within an acceptable range (for example, 1% to 3%), and the toughness of the core network element is defined as a proportion weighted sum of faults occurring in each functional layer component of the network element. The key performance indexes of the 4G different core network elements are as follows:

MME: traffic type index: average/maximum number of MME bearers, number of MME idle state/connected state users, average/maximum number of MME attached users, and the like, success rate type index: EPS attachment success rate, authentication success rate, MME initiated DNS resolution success rate, default/dedicated bearer activation success rate, PDN connection establishment success rate, service request success rate, paging success rate, tracking area update success rate (intra-MME/inter-MME, intra-system/inter-system), handover success rate (intra-MME/inter-MME, intra-system/inter-system), and the like, delay type index: average/maximum attachment duration, dedicated bearer setup average/maximum duration, etc., error type index: the number of authentication parameter errors, the number of UE authentication failures, the number of EPS attachment failures, the number of tracking area update failures and the like;

servingGW: traffic and capacity type indicators: average/peak utilization rate of SGW bearer capacity, user plane uplink/downlink traffic, GTP uplink/downlink average traffic generated by each user attached to the ServingGW, average/maximum attached user number of the ServingGW, average/maximum bearer number of the ServingGW, and utilization rate of SGW data throughput capacity, etc., success rate type index: a success rate of establishing a default/dedicated bearer of the ServingGW, etc.;

HSS: traffic and capacity type indicators: HSS number allocation/active user number, HSS authentication capacity utilization rate, HSS static capacity utilization rate and the like, and the success rate type index is as follows: HSS authentication information inquiry success rate, HSS updating/canceling position success rate, HSS inserting/deleting user data success rate, HSS UE removing success rate and the like, wherein the error type indexes are as follows: number of failed updates, etc.;

Calculating the formula: (α × (a 1 × Σ 4G core network element high available network component failure fraction + a2 × Σ 4G core network element high available compute and storage component failure fraction) + β × OMU failure fraction) × 100%, where the high available network components include TOR switches, EOR switches, network cards, and the like, and the high available compute and storage components include physical machines, virtual machines, databases, and the like. The method for calculating the fault proportion of the high-available components takes a network card as an example, the fault proportion of the high-available network card = fault number of the high-available network cards/total number of the high-available network cards, Σ represents the sum of the fault proportions of the high-available components, and the value ranges of α, β, a1, and a2 are [0,1].

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: and (4) network elements.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

10. 4G core network toughness.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the 4G core network service and the influence of the maximum uncertainty problem on the network steady state. It is recommended that fault injection be performed once a month.

The index definition: in a 4G core network fault injection scenario, the fluctuation of the overall key performance index of the core network is within an acceptable range (e.g., 1% to 3%), and the core network toughness is defined as the proportion-weighted sum of the faults of the available core network elements, EMS, VNFM and NFVO functional components. The key performance indexes of the 4G different core network elements are as follows:

MME: traffic type index: AVERAGE (MME bearing AVERAGE), MAX (MME bearing maximum number), AVERAGE (MME idle/connected user number), AVERAGE (MME AVERAGE attached user number), MAX (maximum attached user number), and the like, success rate type index: AVERAGE (EPS attachment success rate), AVERAGE (authentication success rate), AVERAGE (DNS resolution success rate initiated by MME), AVERAGE (default/dedicated bearer activation success rate), AVERAGE (PDN connection establishment success rate), AVERAGE (service request success rate), AVERAGE (paging success rate), AVERAGE [ tracking area update success rate (MME inside/between, intra-system/inter-system) ], AVERAGE [ handover success rate (MME inside/between, intra-system/inter-system) ], and the like, and a delay type index: AVERAGE (AVERAGE attachment duration), MAX (maximum attachment duration), AVERAGE (AVERAGE duration of dedicated bearer establishment), MAX (maximum duration of dedicated bearer establishment), and the like, and the error type indicator: sigma authentication parameter error times, sigma UE authentication failure times, sigma EPS attachment failure times, sigma tracking area update failure times and the like;

servingGW: traffic and capacity type indicators: AVERAGE (AVERAGE utilization rate of SGW bearer capacity), MAX (peak utilization rate of SGW bearer capacity), ∑ (uplink/downlink traffic on user plane), AVERAGE (AVERAGE traffic on GTP generated for each attached user of ServingGW), AVERAGE (AVERAGE number of attached users of ServingGW), MAX (maximum number of attached users of ServingGW), AVERAGE (AVERAGE number of bearers of ServingGW), MAX (maximum number of bearers of ServingGW), AVERAGE (utilization rate of SGW throughput capacity), and the like, success rate type index: AVERAGE (serving GW default/dedicated bearer establishment success rate), etc.;

PGW: traffic and capacity type indices: AVERAGE (PGW) throughput capacity utilization rate, AVERAGE (PGW) bearer capacity utilization rate, MAX (PGW bearer capacity peak utilization rate), [ sigma (PGW S5/S8 interface uplink/downlink traffic), AVERAGE (PGW AVERAGE number of attached users), MAX (PGW maximum number of attached users), AVERAGE (PGW AVERAGE number of bearers), MAX (PGW peak bearer number), [ sigma (SGi interface receive/send traffic), and the like, success rate type index: AVERAGE (dedicated bearer establishment success rate), AVERAGE (CDR transfer success rate), and the like, the delay type indicator: AVERAGE (AVERAGE duration of dedicated bearer setup initiated by PGW), MAX (maximum duration of dedicated bearer setup initiated by PGW), etc., error type index: the number of GTP packets discarded by the sigma PGW S5/S8 interface in error, the number of IP packets discarded by the sigma SGi interface in error and the like;

HSS: traffic and capacity type indicators: Σ HSS number allocation/active user number and AVERAGE (HSS authentication capacity utilization), AVERAGE (HSS static capacity utilization), and the like, success rate type index: AVERAGE (success rate of HSS authentication information query), AVERAGE (success rate of HSS update/cancel location), AVERAGE (success rate of HSS insert/delete user data), AVERAGE (success rate of HSS clear UE), and the like, error type indexes: sigma updating position failure times and the like;

PCRF: capacity type index: AVERAGE (AVERAGE utilization rate of Gx session processing capacity), MAX (peak utilization rate of Gx session processing capacity), and the like, and the success rate type index: AVERAGE (policy control initiation/update/end success rate), AVERAGE (re-authentication success rate), AVERAGE (application session authorization success rate), and the like, error type indexes: AVERAGE (application session call loss rate), and the like, wherein Σ represents the summation of each network element index, AVERAGE represents the averaging of each network element index, and MAX represents the maximum value of each network element index.

Calculating the formula: (alpha sigma 4G core network element (MME, servingGW, PGW, HSS, PCRF) fault proportion + beta (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high availability network component failure fraction+b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failures over + ± 100%, where the high available components include network components (TOR switches, EOR switches, network cards, etc.) and compute and storage components (physical machines, virtual machines, databases, etc.). The method for calculating the fault proportion of the high-available component takes a network card as an example, the fault proportion of the high-available network card = fault proportion of high-available network card/total number of high-available network cards, sigma represents summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

Data type: real numbers.

Data unit: % of the total weight of the composition.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

11. 5G core network element toughness.

Communication network health dimension: and (6) reliability.

Service requirements: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the 5G core network element service and the influence of the maximum uncertainty problem on the network element steady state. It is recommended that fault injection be performed once a month.

And (3) index definition: in a 5G core network element fault injection scene, the fluctuation of key performance indexes of the core network element is within an acceptable range (for example, 1% to 3%), and the toughness of the core network element is defined as the proportion weighted sum of the faults of all functional layer components of the network element. The key performance indexes of the 5G different core network elements are as follows:

AMF: traffic type indicator: AMF registration state/idle state user number, paging request times, first paging response times, second paging response times and the like, and success rate type indexes are as follows: initial registration success rate = initial registration success number/initial registration request number, registration update success rate = registration update acceptance number/registration update request number, handover success rate (including AMF internal/inter-system, intra-system/inter-system) = handover success number/handover attempt number, UECM deregistration success rate (including AMF initiation and UDM initiation) = UECM deregistration success number/UECM deregistration request number, N11 interface session context setup success rate = N11 interface session context setup success number/N11 interface session context setup request number, N11 interface session context update success rate = N11 interface session context update success number/N11 interface session context update request number, N11 interface session context release success rate = N11 interface session context release success number/N11 interface session context release request number, N11 interface session context query success rate = N11 interface session context query success number/N11 interface session context query number, service request number, and the like, the delay class index: average time length of initial registration, etc., error type index: the times of error authentication parameters, the times of authentication refusal, the times of initial registration failure, the times of registration updating failure and the times of refused service requests;

SMF: traffic type index: average/maximum PDU session number, average/maximum Qos flow number, etc., success rate type index: PDU session establishment success rate = PDU session establishment success rate/PDU session establishment request rate, SMF-initiated PDU session modification success rate = SMF-initiated PDU session modification success rate/SMF-initiated PDU session modification request rate, N7 interface SM policy establishment success rate = N7 interface SM policy establishment success rate/N7 interface SM policy establishment request rate, N10 interface UE context registration success rate = N10 interface UE context registration success rate/N10 interface UE context registration request rate, N7 interface SM policy update success rate = N7 interface SM policy update success rate/N7 interface SM policy update request rate, N10 interface UE context de-registration success rate = N10 interface UE context de-registration success rate/N10 interface UE context de-registration request rate, N7 interface SM policy deletion success rate = N7 interface SM policy deletion success rate/N7 interface SM policy deletion request rate, and the like, the delay type index: PDU conversation establishment flow average duration and the like, error type indexes: PDU session establishment failure times, SMF initiated PDU session modification failure times and the like;

and (4) UPF: traffic type index: average/maximum QoS flow number, number of GTP packet bytes received/sent by N3 interface, number of GTP packet bytes received/sent by N9a interface, number of GTP packet bytes received/sent by N6 interface, and the like, and the success rate type index: PFCP session establishment success ratio = PFCP session establishment success times/PFCP session establishment request times, PFCP session modification success ratio = PFCP session modification success times/PFCP session modification request times, etc., error type index: the number of times of failure of PFCP session establishment/modification, the number of GTP packets received by an N3 interface, the number of IP packets discarded by an N6 interface error and the number of GTP packets received by an N9c interface;

PCF: traffic type index: AM strategy association total average/maximum value, SM strategy association total average/maximum value, and the like, and success rate type indexes are as follows: AM policy association establishment success rate = AM policy association establishment success number/AM policy association establishment request number, AM policy association update success rate = AM policy association update success number/AM policy association update request number, AM policy association deletion success rate = AM policy association deletion success number/AM policy association deletion request number, SM policy association establishment success rate = SM policy association establishment success number/SM policy association establishment request number, SM policy association update success rate = SM policy association update success number/SM policy association update request number, SM policy association deletion success rate = SM policy association deletion success number/SM policy association deletion request number, and the like, error type indexes: the SM strategy association establishes failure times, and the SM strategy association updates the failure times;

NRF: traffic type index: number of stored instances, etc., success rate type index: NF initiation update success rate = NF initiation update success times/NF initiation update request times, NF discovery success rate = NF discovery success times/NF discovery request times, and the like, error type indexes: the NF initiates the number of updating failure times, the NF discovers the number of failure times, etc.;

Calculating the formula: (α: (a) ₁ * Sigma 5G core network element high available network component fault ratio + a ₂ * Σ 5G core network element high availability computing and storage component failure fraction) + β × OMU failure fraction) × 100%, where the high availability network components include TOR switches, EOR switches, network cards, and the like, and the high availability computing and storage components include physical machines, virtual machines, databases, and the like. The high available component fault ratio calculation method takes a network card as an example, the high available network card fault ratio = number of failed high available network cards/total number of high available network cards, Σ represents the sum of the fault ratios of the high available components, α, β, a ₁ 、a ₂ The value range is [0,1]]。

The data type is as follows: real numbers.

Data unit: %.

Measurement object: a network element.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

12. 5G core network toughness.

Communication network health dimension: and (6) reliability.

Service requirements are as follows: the method is used for verifying the cloud expansion capability and the fault tolerance capability of the 5G core network service and the influence of the maximum uncertainty problem on the network steady state. It is recommended that fault injection be performed once a month.

The index definition: in a 5G core network fault injection scenario, the fluctuation of key performance indexes of the core network is within an acceptable range (for example, 1% to 3%), and the core network toughness is defined as a proportion weighted sum of faults of available core network elements, EMS, VNFM and NFVO functional components. The key performance indexes of the 5G different core network elements are as follows:

AMF: traffic type index: sigma AMF register state/idle state user number, sigma paging request times, sigma paging response times, etc. success rate type index: AVERAGE (initial registration success ratio = initial registration success number/initial registration request number), AVERAGE (registration update success ratio = registration update acceptance number/registration update request number), AVERAGE [ handover success ratio (including AMF internal/inter-system, intra-system/inter-system) = handover success number/handover attempt number ], AVERAGE [ UECM deregistration success ratio (including AMF initiation and UDM initiation) = UECM deregistration success number/UECM deregistration request number ], AVERAGE (N11 interface session context establishment success ratio = N11 interface session context establishment success number/N11 interface session context establishment request number), AVERAGE (N11 interface session context update success ratio = N11 interface session context update success number/N11 interface session context update request number), AVERAGE (N11 interface session context release = N11 interface session context release success number/N11 interface session context release request number), AVERAGE (N11 interface session context query success ratio = N11 interface session context query success number/N11 interface session context release request number), and the like, AVERAGE (N11 interface session context query success ratio = N11 session context query success ratio, inquiry success ratio query number, service query success ratio query number, and the like): AVERAGE (initial registration AVERAGE duration), etc., error type indicator: sigma authentication parameter error times, sigma authentication rejection times, sigma initial registration failure times, sigma registration updating failure times and sigma service request rejection times;

SMF: traffic type index: AVERAGE (AVERAGE PDU session count), MAX (maximum PDU session count), AVERAGE (AVERAGE Qos flow count), MAX (maximum Qos flow count), and the like, and the success rate type index: AVERAGE (PDU session establishment success rate = PDU session establishment success rate/PDU session establishment request rate), AVERAGE (PDU session modification success rate = SMF initiated PDU session modification success rate/SMF initiated PDU session modification request rate), AVERAGE (N7 interface SM policy creation success rate = N7 interface SM policy creation success rate/N7 interface SM policy creation request rate), AVERAGE (N10 interface UE context registration success rate = N10 interface UE context registration success rate/N10 interface UE context registration request rate), AVERAGE (N7 interface SM policy update success rate = N7 interface SM policy update success rate/N7 interface SM policy update request rate), AVERAGE (N10 interface UE context de-registration success rate = N10 interface UE context de-registration success rate/N10 interface UE context de-registration request rate), AVERAGE (N7 interface SM policy deletion success rate = N7 interface SM policy deletion success rate/N7 interface SM request rate), etc., type delay index: AVERAGE duration of PDU session establishment procedure, error type index: the number of times of PDU session establishment failure, the number of times of PDU session modification failure initiated by sigma SMF and the like;

UPF: traffic type indicator: AVERAGE (AVERAGE QoS flow number), MAX (maximum QoS flow number), SIGMA N3 interface receiving/sending GTP packet byte number, SIGMA N9a interface receiving/sending GTP packet byte number, SIGMA N6 interface receiving/sending byte number, etc, success rate type index: AVERAGE (PFCP session establishment success ratio = PFCP session establishment success number/PFCP session establishment request number), AVERAGE (PFCP session modification success ratio = PFCP session modification success number/PFCP session modification request number), and the like, error type index: the number of times of establishment/modification failure of the sigma PFCP session, the number of GTP packets received by a sigma N3 interface, the number of IP packets discarded by a sigma N6 interface in error, and the number of GTP packets received by a sigma N9c interface;

UDM: success rate type index: AVERAGE (UECM registration success rate initiated by AMF = UECM registration success rate initiated by AMF/UECM registration request rate initiated by AMF), AVERAGE (UECM registration success rate initiated by SMF = UECM registration success rate initiated by SMF/UECM registration request rate initiated by SMF/registration request rate initiated by SMF), AVERAGE (registration parameter update success rate = registration parameter update success rate/registration parameter update request rate), AVERAGE (user data acquisition success rate = user data acquisition success rate/user data acquisition request rate), AVERAGE (user data subscription success rate = user data subscription success rate/user data subscription request rate), AVERAGE (unsubscribe user data success rate = unsubscribe user data subscription success rate/unsubscribe user data subscription request rate), etc.;

PCF: traffic type index: AVERAGE (AVERAGE of AM policy associated total number), MAX (maximum of AM policy associated total number), AVERAGE (AVERAGE of SM policy associated total number), MAX (maximum of SM policy associated total number), and the like, and the success rate type index: AVERAGE (AM policy association establishment success rate = AM policy association establishment success number/AM policy association establishment request number), AVERAGE (AM policy association update success rate = AM policy association update success number/AM policy association update request number), AVERAGE (AM policy association deletion success rate = AM policy association deletion success number/AM policy association deletion request number), AVERAGE (SM policy association establishment success rate = SM policy association establishment success number/SM policy association establishment request number), AVERAGE (SM policy association update success rate = SM policy association update success number/SM policy association update request number), AVERAGE (SM policy association deletion success rate = SM policy association deletion success number/SM policy association deletion request number), etc., error type indexes: the sigma SM strategy is associated with the establishment failure times, and the sigma SM strategy is associated with the update failure times;

NRF: traffic type indicator: the sigma stores the number of the examples and the like, and the success rate type index is as follows: AVERAGE (NF success rate = NF success rate of initiating update/NF number of times of initiating update request), AVERAGE (NF success rate = NF success rate of discovering/number of times of discovering request), and the like, error type index: sigma NF initiates the number of times of failure of updating and sigma NF finds the number of times of failure, etc.;

NSSF: success rate type index: AVERAGE (network slice selection success rate = network slice selection success number/network slice selection request number), error type index: Σ number of network slice selection failures.

Where Σ represents the summation of each network element and/or slice index, AVERAGE represents the averaging of each network element and/or slice index, MAX represents the maximum of each network element and/or slice index

Calculating the formula: (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure fraction + β [ (+ b [ (] b ]) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ *∑VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failures over + ± 100%, where the high available components include network components (TOR switches, EOR switches and network cards, etc.) and compute and storage components (physical machines, virtual machines and databases, etc.). The method for calculating the fault proportion of the high-available component takes a network card as an example, the fault proportion of the high-available network card = fault proportion of high-available network card/total number of high-available network cards, sigma represents summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

Data type: real numbers.

Data unit: %.

Measurement object: a network.

And (3) counting period: half a year

The data source is as follows: and (5) network management statistics.

Based on the same inventive concept, the embodiment of the present invention further provides a network evaluation apparatus and a computer-readable storage medium, and since the principle of solving the problem of these devices is similar to that of the network evaluation method, the implementation of these devices can refer to the implementation of the method, and repeated details are not repeated.

When the technical scheme provided by the embodiment of the invention is implemented, the implementation can be carried out as follows.

Fig. 6 is a schematic structural diagram of a network evaluation device, as shown in the figure, the device includes:

the processor 600, which is used to read the program in the memory 620, executes the following processes:

determining indexes to be evaluated in the NFV network;

injecting a fault;

restoring the NFV network to the first indicator;

evaluating the NFV network according to a second index;

a transceiver 610 for receiving and transmitting data under the control of the processor 600.

In an implementation, the method further comprises the following steps:

success rate type indicator on UDM;

success rate type indicator, error type indicator on NSSF.

one or a combination of the following criteria at the 4G base station:

one or a combination of the following indicators on the 5G base station:

traffic and capacity type indicators: the gNB sends and/or receives the service data volume from the NG interface, the byte number of the uplink and/or downlink PDCP PDU of the cell user plane, the traffic data volume received and/or sent by the gNB from the S1 interface, the number of TBs (transmission blocks) of the uplink and/or downlink transmission, the number of PRBs (physical resource blocks) available for the downlink PDSCH, the receiving number of paging records, the average and/or maximum number of RRC connections in double connection, the average and/or maximum ERAB number of NSA SCG Split Bear types;

on the MME:

on the ServingGW:

traffic and capacity type indices: average and/or peak utilization rate of SGW bearing capacity, uplink and/or downlink flow of a user plane, uplink and/or downlink average flow of GTP generated by each attached user of the serving GW, average and/or maximum attached user number of the serving GW, average and/or maximum bearing number of the serving GW, and utilization rate of SGW data throughput capacity;

on the PGW:

on the HSS:

error type index: updating the position failure times;

on the PCRF:

error type index: applying the session call loss rate;

on the AMF:

time delay type index: initial registration average duration;

on SMF:

on the UPF:

on the UDM:

on the PCF:

on NRF:

traffic type indicator: storing the number of instances;

error type index: the NF initiates updating failure times and NF discovers failure times;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

the fault injected in the user plane service is the fault of the CloudOS component, which causes the connection between the user plane network element and the control plane network element to be interrupted.

In practice, the NFV network is evaluated according to the second criterion in one or a combination of the following ways:

NFV network element toughness = α (a) ₁ * Network component failure fraction + a ₂ * Calculating/storing component fault proportion + β OMU fault proportion; wherein, the NFV network element toughness is an evaluation index for evaluating the network element service quality damage resistance when a component in the NFV network element fails, alpha represents the user plane weight, beta represents the management plane weight, a _i {i∈[1,3]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

4G wireless network element toughness = (alpha (a) = ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switches, EOR switches, network cards, high availability computing and storage components including one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless networkToughness = (α: (a) ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Σ 5G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component failure percentage + a ₂ * Σ 5G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α × 4G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: article (A)The system comprises a processor, a virtual machine and a database; sigma shows the sum of the fraction of highly available component failures, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

4G core network toughness = (α ∑ 4G core network element (MME, servingGW, PGW, HSS, PCRF) failure fraction + β = (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high availability network component failure odds + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure fraction + β [ (+ b [ ]) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio + ]100%, where highThe available components include one or a combination of the following: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

wireless network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the values of alpha and beta are 0.5, and the suggested values of a1 and a2 are 1;

core network element toughness = (α (a) ₁ * Network component failure fraction + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch,EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machines, virtual machines, databases; the value ranges of alpha, beta, a1 and a2 are [0, 1')]。

Where in fig. 6, the bus architecture may include any number of interconnected buses and bridges, with various circuits being linked together, particularly one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 610 may be a number of elements including a transmitter and a receiver that provide a means for communicating with various other apparatus over a transmission medium. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 in performing operations.

The embodiment of the invention also provides a network evaluation device, which comprises:

a fault module for injecting a fault;

the fault module is also used for recovering the NFV network until the index is a first index;

In an implementation, the method further comprises the following steps:

In an implementation, the fault module is further configured to, when injecting the fault, inject the fault layer by layer from an infrastructure layer to a VNF layer according to the NFV network functional architecture; and/or the presence of a gas in the gas,

injecting a fault into the IP address conflict of the CloudOS database in the main NFV network service;

In an implementation, the fault module is further configured to inject, at the management plane, that the OMU fault is the VNF is inaccessible.

success rate type indicator on UDM;

success type indicator, error type indicator on NSSF.

one or a combination of the following indicators on the 4G base station:

one or a combination of the following indicators on the 5G base station:

traffic and capacity type indicators: the gNB sends and/or receives the service data volume from the NG interface, the byte number of the cell user plane uplink and/or downlink PDCP PDU, the traffic data volume received and/or sent by the gNB from the S1 interface, the number of uplink and/or downlink transmission TBs, the number of available downlink PDSCH PRBs, the number of paging record receiving, the average and/or maximum number of RRC connections in dual connection, the average and/or maximum ERAB number of NSA SCG Split Bear type;

on the MME:

on the ServingGW:

on the PGW:

traffic and capacity type indicators: PGW data throughput capacity utilization, PGW bearer capacity average and/or peak utilization, PGW S5 and/or S8 interface uplink and/or downlink traffic, PGW average and/or maximum attached user number, PGW average and/or peak bearer number, and SGi interface receive and/or transmit traffic;

error type index: the number of GTP packets discarded by an error of a PGW S5 interface and/or S8 interface and the number of IP packets discarded by an error of an SGi interface;

at the HSS:

error type index: updating the location failure times;

on the PCRF:

error type index: applying the session call loss rate;

on the AMF:

time delay type index: initial registration average duration;

on SMF:

traffic type indicator: average and/or maximum number of PDU sessions, average and/or maximum number of Qos flows;

on the UPF:

on UDM:

on the PCF:

on NRF:

traffic type index: storing the number of instances;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

4G wireless network element toughness = (alpha (a)) ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high availability computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high availability network components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, high availability computing and storage components including one or a combination of the following: physical machine, virtual machine, numberA database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless network element toughness = (alpha (a)) ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Σ 5G base station high availability computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high availability network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault ratio + a ₂ * Σ 4G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component failure percentage + a ₂ * Σ 5G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machines, virtual machines, databases; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α × 4G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)). 100%, where high availability compute and storage component failure fraction) is calculatedThe network component comprises one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (α x 5G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma represents the fault proportion summation of each high-availability component, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (α ∑ 4G core network element (MME, servingGW, PGW, HSS, PCRF) failure fraction + β = (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high availability network component failure odds + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: the TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure ratio + β × (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability computingAnd storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: the TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machines, virtual machines, databases; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

the toughness of the core network element is used for verifying the cloud expansion capability and the fault tolerance capability of the core network element service and the influence of the uncertainty problem to the maximum extent on the network element steady state.

In an implementation, the evaluating module is further configured to evaluate one or a combination of radio network element toughness, core network element toughness of the NFV network according to the second indicator in the following manner:

wireless network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0,1]]；

Core network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one or a combination of the following components: physical machine, virtual machine, database; the value ranges of alpha, beta, a1 and a2 are [0, 1')]。

For convenience of description, each part of the above-described apparatus is separately described as being functionally divided into various modules or units. Of course, the functionality of the various modules or units may be implemented in the same one or more pieces of software or hardware in the practice of the invention.

The embodiment of the present invention further provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores a computer program for executing the network evaluation method.

The specific implementation can be seen in the implementation of the network evaluation method.

In summary, in the technical solution provided in the embodiment of the present invention, an overall solution of the NFV network chaotic engineering test is provided, which includes a system framework, operation steps, a fault type and service index selection solution;

a definition scheme of the NFV network toughness index reflecting the damage resistance of the NFV network is also provided.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. A network evaluation method, comprising:

determining indexes to be evaluated in a Network Function Virtualization (NFV) network;

injecting a fault;

restoring the NFV network to the first index;

the NFV network is evaluated according to the second index.

2. The method of claim 1, further comprising:

3. The method of claim 2, wherein the predetermined index is an index that is one or a combination of:

a service volume type index, a success rate type index, a time delay type index and an error type index on a mobility management entity MME;

service volume, capacity type index and success rate type index on the serving gateway serving GW;

a service volume and capacity type index, a delay type index and an error type index on a packet data network gateway (PGW);

the method comprises the steps that service volume and capacity type indexes, success rate type indexes and error type indexes are arranged on a Home Subscriber Server (HSS);

a service volume type index, a success rate type index and an error type index on a Policy and Charging Rule Function (PCRF) entity;

service volume type index, success rate type index, time delay type index and error type index on the access and mobile management function AMF;

a service volume type index, a success rate type index and an error type index on a user plane function UPF;

a success rate type index on a unified data management entity UDM;

a service volume type index, a success rate type index and an error type index on a policy control function PCF;

a service volume type index, a success rate type index and an error type index on a network storage function NRF;

and a success rate type index and an error type index on a network slice selection functional entity NSSF.

4. The method of claim 3, wherein the predetermined index is an index that is one or a combination of:

one or a combination of the following criteria at the 4G base station:

traffic type indicator: the method comprises the following steps that the number of bytes sent and/or received by an Ethernet interface of an evolution base station eNB, the number of bytes sent and/or received service bytes by an S1 interface of the eNB, the number of bytes of SDU (service data Unit) of a cell user plane and/or a control plane uplink and/or downlink packet data convergence protocol PDCP (packet data convergence protocol);

success rate type index: the success rate of establishing E-RAB, the radio access rate, the success rate of establishing RRC connection and the success rate of switching are respectively related to the service;

error type index: the method comprises the following steps of (1) wireless call drop rate, TCP service wireless call drop rate based on a transmission control protocol, uplink and/or downlink PDCP SDU packet loss rate, uplink and/or downlink transmission block error number and paging record discarding number;

one or a combination of the following indicators on the 5G base station:

traffic and capacity type indicators: the method comprises the following steps that a gNB sends and/or receives service data volume from an NG interface, the number of bytes of uplink and/or downlink PDCP Packet Data Units (PDU) on a cell user plane, the number of bytes of the service data volume received and/or sent by the gNB from an S1 interface, the number of TBs (transport blocks) of uplink and/or downlink transmission, the number of PRBs (physical resource blocks) of a downlink Physical Downlink Shared Channel (PDSCH), the receiving number of paging records, the average and/or maximum number of RRC (radio resource control) connections in double connection, the average and/or maximum number of E-RABs (enhanced radio access groups) of a Split bearing Split Bear type of a NSA (non-independent networking) auxiliary cell group SCG;

success rate type index: the success rate of RRC connection establishment, the success rate of Flow establishment, the success rate of packet data unit session PDSESSION establishment, the success rate of switching and the success rate of NG interface UE related logic signaling connection establishment;

the delay type index: the method comprises the following steps that average and/or maximum RRC connection establishment time, average processing time delay of Radio Link Control (RLC) downlink data packets, average NG switching time length between gNBs, average Xn switching time length between gNBs, time delay of switching an evolved packet system fallback (EPS) from 5G to 4G, and average processing time delay of the RLC downlink data packets of each slice cell;

on the MME:

traffic type index: the average and/or maximum number of the MME loads, the number of users of the MME in an idle state and/or a connected state, and the average and/or maximum number of users of the MME;

success rate type index: an EPS attachment success rate, an authentication success rate, a DNS resolution success rate of a domain name system initiated by an MME, a default and/or special bearer activation success rate, a PDN connection establishment success rate, a service request success rate, a paging success rate, a tracking area update success rate and a switching success rate;

on the ServingGW:

traffic and capacity type indicators: average and/or peak utilization rate of SGW bearing capacity, uplink and/or downlink flow of a user plane, average GTP uplink and/or downlink flow generated by each attached user of a serving GW, average and/or maximum attached user number of the serving GW, average and/or maximum bearing number of the serving GW, and utilization rate of SGW data throughput capacity;

on the PGW:

success rate type index: the special bearer establishment success rate and the CDR transmission success rate;

the delay type index: establishing average and/or maximum duration of special load initiated by a PGW;

at the HSS:

error type index: updating the position failure times;

on the PCRF:

error type index: applying the session call loss rate;

on the AMF:

success rate type index: the method comprises the following steps of initial registration success rate, registration updating success rate, switching success rate, UE CM deregistering success rate of user equipment context management, N11 interface session context establishment success rate, N11 interface session context updating success rate, N11 interface session context release success rate, N11 interface session context query success rate and service request success rate;

the time delay index is as follows: initial registration average duration;

on SMF:

traffic type indicator: average and/or maximum number of PDU sessions, average and/or maximum number of quality of service Qos flows;

success rate type index: a PDU session establishment success rate, a PDU session modification success rate initiated by SMF, an N7 interface establishment session management SM strategy success rate, an N10 interface UE context registration success rate, an N7 interface updating SM strategy success rate, an N10 interface UE context de-registration success rate and an N7 interface deletion SM strategy success rate;

on the UPF:

on UDM:

on the PCF:

success rate type index: access and mobility AM strategy association establishment success rate, AM strategy association update success rate, AM strategy association deletion success rate, SM strategy association establishment success rate, SM strategy association update success rate and SM strategy association deletion success rate;

on NRF:

traffic type index: storing the number of instances;

success rate type index: the network functional entity NF initiates an updating success rate and an NF discovery success rate;

on NSSF:

success rate type index: network slice selection success rate;

error type index: network slice selection failure times.

5. The method of claim 1, wherein when injecting the fault, the fault is injected layer by layer from an infrastructure layer to a Virtual Network Function (VNF) layer according to an NFV network function architecture; and/or the presence of a gas in the gas,

according to the NFV network function architecture, faults are injected layer by layer from a virtual network function VNF layer to an infrastructure layer.

6. The method of claim 1, wherein when injecting a fault, it is one or a combination of the following:

7. The method according to claim 6, characterized by injecting one or a combination of the following faults in the NFV network traffic plane:

injecting a fault into the main NFV network service to be a cloud operating system cloud OS database IP address conflict;

injecting a fault into the main NFV network service to completely block the VIM from damaging partial service virtual machines of the superposed network elements for virtual infrastructure management;

8. The method of claim 6, wherein the Operation and Maintenance Unit (OMU) fault injected at the management plane as a VNF is inaccessible.

9. The method of any of claims 1 to 8, wherein evaluating the NFV network according to the second index evaluates an overall network quality of service against damage in the event of a failure of a component in the NFV network.

10. The method of claim 9, wherein the NFV network is evaluated according to the second criteria in one or a combination of the following ways:

NFV network element toughness = α (a) ₁ * Network component failure fraction + a ₂ * Calculating/storing component fault proportion + β OMU fault proportion; wherein, the NFV network element toughness is an evaluation index for evaluating the damage resistance of the network element service quality when a component in the NFV network element fails, alpha represents the user plane weight, beta represents the management plane weight, and a represents the management plane weight _i {i∈[1,3]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

NFV network toughness = α NFV network element failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and store component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component failure fraction) 100%; the NFV network toughness is an evaluation index for evaluating the overall network service quality damage resistance when NFV network elements, an element management system EMS, a virtual network function manager VNFM and a network function virtualization technology orchestrator NFVO function components fail, wherein alpha represents a user plane weight, beta represents a management plane weight, and b represents a management plane weight _i {i∈[1,6]}∈[0,1]Weight coefficients representing the functional layers of the user plane;

4G wireless network element toughness = (alpha (a)) ₁ * Sigma 4G base station high available network component failure percentage + a ₂ * Σ 4G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: top of rack TOR switches, end of line EOR switches, network cards, high availability computing and storage components including one or a combination of the following: physical machines, virtual machines, databases; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G wireless network element toughness = (alpha (a)) ₁ * Sigma 5G base station high available network component failure percentage + a ₂ * Σ 5G base station high available computing and storage component failure fraction) + β OMU failure fraction) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machines, virtual machines, databases; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G core network element toughness = (α = (a) ₁ * Sigma 4G core network element high available network component fault proportion + a ₂ * Σ 4G core network element high available computing and storage component failure fraction) + β OMU failure fraction) × 100%, wherein the high available network components comprise one or a combination of the following components: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machines, virtual machines, databases; sigma shows the sum of the fraction of faults in each highly available module, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

5G core network element toughness = (α = (a) ₁ * Sigma 5G core network element high available network component fault ratio + a ₂ * Σ 5G core network element high available compute and storage component failure) + β OMU failure) + 100%, wherein the high available network components comprise one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, a ₁ 、a ₂ The value range is [0,1]]；

4G wireless network toughness = (α × 4G base station failure fraction + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability computing and storage component failure fraction)) 100%, wherein the high availability network components include one or a combination of the following: TOR switch, EOR switch,A network card; the high availability computing and storage component includes one or a combination of the following components: physical machine, virtual machine, database; sigma shows the sum of the fault fractions of the highly available modules, alpha, beta, b ₁ And b ₂ The value range is [0,1]]；

5G wireless network toughness = (α x 5G base station failure ratio + β (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Σ EMS high availability compute and storage component failure fraction)) 100%, where the high availability network components include one or a combination of the following: TOR switch, EOR switch, network card; the high available computing and storage components include one or a combination of the following: physical machine, virtual machine, database; sigma shows that the fault proportion of each high-availability component is summed, and the value ranges of alpha, beta, b1 and b2 are [0,1]]；

4G core network toughness = (α ∑ 4G core network element (MME, servingGW, PGW, HSS, PCRF) failure fraction + β = (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and memory component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machine, virtual machine, database; sigma represents the summation, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]；

5G core network toughness = (α ∑ 5G core network elements (AMF, SMF, UPF, UDM, PCF, NRF, NSSF) failure ratio + β × (b) ₁ * Sigma EMS high available network component failure fraction + b ₂ * Sigma EMS high available compute and storage component failure fraction + b ₃ * Sigma VNMF high available network component failure fraction + b ₄ * Sigma VNFM high availability compute and storage component failure fraction + b ₅ * Sigma NFVO high available network component failure fraction + b ₆ * Σ NFVO high available compute and storage component fault ratio +) + 100%, wherein the high available components comprise one or a combination of the following components: TOR switches, EOR switches, network cards, and computation and storage include one or a combination of the following components: physical machines, virtual machines, databases; sigma denotes the sum, alpha, beta, b ₁ 、b ₂ 、b ₃ 、b ₄ 、b ₅ And b ₆ The value range is [0,1]]。

11. The method according to any of claims 1 to 8, wherein evaluating the NFV network according to the second index is evaluating one or a combination of the following performances of the NFV network:

12. The method of claim 11, wherein the wireless network element toughness is used to verify the cloud scalability and fault tolerance capabilities of wireless network element services, and the impact of maximum uncertainty problems on network element steady state; and/or the presence of a gas in the atmosphere,

13. The method of claim 11, wherein one of radio network element toughness, core network element toughness, or a combination thereof is evaluated according to the second index as follows:

wireless network element toughness = (α (a) ₁ * Network component failure ratio + a ₂ * Compute/store component failure fraction) + β OMU failure fraction) 100%, wherein the network component comprises one or a combination of the following: TOR switch, EOR switch, network card; the computing storage component comprises one of the following components or the computing storage componentCombining: physical machines, virtual machines, databases; the value ranges of alpha, beta, a1 and a2 are [0,1]]；

14. A network evaluation apparatus, comprising:

a processor for reading the program in the memory, performing the following processes:

determining indexes to be evaluated in the NFV network;

injecting a fault;

restoring the NFV network to the first indicator;

evaluating the NFV network according to the second index;

15. A network evaluation apparatus, comprising:

a fault module to inject a fault;

16. A computer-readable storage medium, characterized in that it stores a computer program for executing the method of any one of claims 1 to 13.