CN114124710A

CN114124710A - Service verification method and device

Info

Publication number: CN114124710A
Application number: CN202010884122.7A
Authority: CN
Inventors: 金雁飞; 李延如
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-08-28
Filing date: 2020-08-28
Publication date: 2022-03-01
Also published as: WO2022042402A1

Abstract

The application provides a service verification method and a service verification device, which are used for improving the accuracy of service verification on a network element and relate to the technical field of wireless communication. In the method, the network element to be tested can be controlled to provide services for the plurality of terminal devices, and the plurality of terminal devices are obtained by screening according to the attributes of the terminal devices. The method comprises the steps of obtaining test data generated by a plurality of terminal devices in the process of receiving the service provided by the network element to be tested, and calculating the first service success rate of each service attribute and the first service success rate of each device attribute. And determining whether the network element to be tested is normal or not according to the first service success rate and the second service success rate of each service attribute and each equipment attribute. Based on the scheme, because the plurality of terminal devices are obtained by screening according to the service attributes and the device attributes, the service types and the device types covered by the plurality of terminal devices are relatively comprehensive, and the accuracy rate of service verification on the network element can be improved.

Description

Service verification method and device

Technical Field

The present application relates to the field of wireless communication technologies, and in particular, to a method and an apparatus for service verification.

Background

For the wireless communication technology, it is desired to make service distribution faster, and besides the newly-built network slice or the newly-built Network Function (NF) entity itself needs to support flexible design and distribution of the service, it is also necessary to perform service verification on the newly-built network slice or NF before service distribution, so as to ensure that the newly-built network slice or NF can safely and stably support the existing network service.

At present, the gray level verification method is widely applied to a scene of business verification. The current gray level verification method mainly includes the following two methods: the first method is to adopt a small number of test terminals to carry out manual dial test to carry out service verification. And the second is to adopt a manual gray level verification mode to guide a small amount of current network terminal equipment to carry out service verification.

However, in the two kinds of service verification, the service coverage scene of the terminal device is small, so that the verification effect is difficult to guarantee. Moreover, because of manual verification, misjudgment is easily caused by human ability difference.

Disclosure of Invention

The application provides a service verification method and a service verification device, which are used for verifying the service of a network element.

In a first aspect, an embodiment of the present application provides a service verification apparatus. The method may be performed by a service authentication apparatus. The service verification apparatus may be a separate server, or may be a software program newly added to an existing network element in the core network. The service verification device can control the network element to be tested to provide service for the plurality of terminal devices. The plurality of terminal devices are obtained by screening according to the attributes of the terminal devices. The attributes may include at least one service attribute and at least one device attribute, where the service attribute is used to describe a service executed by the terminal device, and the device attribute is used to describe device information of the terminal device. The service verification device can acquire test data generated by the plurality of terminal devices in the process of receiving the service provided by the network element to be tested, and can calculate the first service success rate of each service attribute and the first service success rate of each device attribute according to the acquired test data. The service verification device can determine whether the network element to be tested is normal according to the first service success rate and the second service success rate of each service attribute and each equipment attribute. Wherein the second service success rate of each service attribute and each device attribute is obtained from history data generated when the plurality of terminal devices perform services in the first network. It should be noted that, the plurality of terminal devices are located in the first network, and the network element to be tested is a newly added network element in the first network or any one of network slices newly added in the first network.

Based on the scheme, when a network element or a network slice is newly added in the existing network architecture, the service verification can be performed on the network element, and the network element to be tested can provide services for a plurality of terminal devices during the service verification. Because the plurality of terminal devices are obtained by screening according to the service attributes and the device attributes, the service types and the device types covered by the plurality of terminal devices are relatively comprehensive, and the accuracy rate of service verification on the network element can be improved.

In one possible implementation, the first service success rate may include at least one of: a first access success rate and a first bearer update success rate. It should be appreciated that the second traffic success rate may also include at least one of: a second access success rate and a second bearer update success rate.

Based on the scheme, whether the network element to be tested is normal or not can be determined according to the access success rate or the bearing updating success rate when the plurality of terminal devices execute the service through the network element to be tested and the access success rate or the bearing updating success rate when the plurality of terminal devices execute the service in the first network.

In a possible implementation manner, the plurality of terminal devices may be obtained by filtering according to the following manner: the service verification device may obtain first history data of each terminal device in the first network, and may classify each terminal device in the first network according to a feature field in the first history data. The characteristic field here can be used to characterize the classification properties and device properties of the terminal device. Each class in the classification result can represent terminal equipment which belongs to one equipment attribute and executes a service corresponding to one service attribute. The service verification device can respectively select at least one terminal device in each class to obtain a plurality of terminal devices.

Based on the above scheme, the terminal devices in the first network may be classified according to the feature fields in the first history data of the terminal devices in the first network, and each class in the classification result may represent a terminal device that belongs to one device attribute and executes a service corresponding to one service attribute, that is, a plurality of user groups may be obtained. Because at least one terminal device can be selected in each user group for service verification, the service types and the device types covered by the terminal devices can be relatively comprehensive, and the accuracy of service verification can be improved.

In a possible implementation manner, when the service verification apparatus selects at least one terminal device in each class, a ratio between the number of terminal devices in each class and the total number of terminal devices in the first network may be calculated. The service verification device may determine the maximum number of terminal devices of each class according to the ratio and the total number of terminal devices allowed to be accessed by the network element to be tested. The maximum terminal equipment number of one class represents the maximum value of the number of terminal equipment of one thunder in the terminal equipment of the network element to be tested for providing service. The service verification device may select a corresponding number of terminal devices in each class respectively for the maximum number of terminal devices in each class.

Based on the scheme, the terminal equipment can be respectively selected from each class according to the number of the terminal equipment of each class in the classification result and the total number of the terminal equipment which is allowed to be accessed by the network element to be tested, so that the condition of the selected terminal equipment can be in accordance with the condition of the terminal equipment in the existing network, and the accuracy rate of service verification can be improved.

In one possible implementation, the feature field may include at least one of: the access type of the terminal equipment, the request service type of the terminal equipment, the access point name of the terminal equipment or the unique identification of the terminal equipment.

Based on the scheme, the terminal devices in the first network can be classified according to the characteristic fields, so that the classification result can accurately represent the service attribute and the device attribute of each user group.

In a possible implementation manner, if a first service success rate and a second service success rate of at least one service attribute or at least one device attribute meet a first condition, it is determined that a network element to be tested is abnormal. And if the first service success rate and the second service success rate of each service attribute and each equipment attribute do not meet the first condition, determining that the network element to be tested is normal. The first condition herein may include: the difference value between the first service success rate and the second service success rate is larger than a specified value. Or the relative difference ratio of the first service success rate and the second service success rate is larger than a preset value. Wherein the relative difference ratio may satisfy the following formula (1):

relative difference ratio | (first service success rate-second service success rate)/second service success rate | formula (1)

It should be noted that the specified values and the preset values may be predetermined according to empirical values, and the present application is not limited specifically.

Based on the scheme, whether the network element to be tested is normal or not can be determined according to the first service success rate and the second service success rate of the plurality of terminal devices, so that service verification of the network element to be tested is realized.

In a possible implementation manner, if the network element to be tested is abnormal, the service verification apparatus may obtain first test data of the terminal device belonging to the service attribute and the device attribute, where the first service success rate and the second service success rate satisfy the first condition. The service verification apparatus may perform Root Cause Analysis (RCA) on the cause of the abnormality of the network element to be tested by using a hotspot algorithm according to the acquired first test data.

Based on the above scheme, the reason for the abnormal network element to be tested can be obtained according to the first test data and the hot spot algorithm, that is, the network element to be tested can not normally provide a service for which type of terminal equipment, so that the network element to be tested can be adjusted in a targeted manner.

In a second aspect, a service verification apparatus is provided, which may comprise various modules/units for performing the first aspect or any one of the possible implementations of the first aspect. Such as processing units and communication units.

In a third aspect, a traffic validation apparatus is provided that includes a processor and a memory. The memory is used for storing computer-executable instructions, and when the controller runs, the processor executes the computer-executable instructions in the memory to perform the operation steps of the method in the first aspect or any one of the possible implementation manners of the first aspect by using hardware resources in the controller.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein instructions, which when executed on a computer, cause the computer to perform the method of the above-described aspects.

In a fifth aspect, the present application provides a computer program product storing instructions that, when run on a computer, cause the computer to perform the method of the above aspects.

In addition, the advantageous effects of the second aspect to the fifth aspect may be as shown in the first aspect, and are not described herein again.

Drawings

Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present application;

fig. 2 is a schematic diagram of a communication system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a communication system provided by an embodiment of the present application;

fig. 4 is a schematic structural diagram of a network device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a network device according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a network device according to an embodiment of the present application;

fig. 7 is an exemplary flowchart of a service verification method provided in an embodiment of the present application;

fig. 8 is an interface schematic diagram of a service verification method provided in the embodiment of the present application;

FIG. 9 is a schematic interface diagram of a barreling algorithm provided in an embodiment of the present application;

fig. 10 is a thermodynamic diagram of a service verification method according to an embodiment of the present application;

fig. 11A is an exemplary flowchart of a service verification method provided in an embodiment of the present application;

FIG. 11B is a schematic interface diagram of an RCA provided in an embodiment of the present application;

fig. 12 is a schematic structural diagram of a service verification apparatus according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a service verification apparatus according to an embodiment of the present application.

Detailed Description

Hereinafter, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

1) The service attribute refers to an attribute of a service executed by the terminal device, and may include an Access Type (Access Type), a quality of service identifier (QCI), an Access Point Name (APN), and the like when the terminal device executes the service.

2) The device attribute refers to hardware information of the terminal device, and may include a brand of the terminal device (e.g., HUAWEI, APPLE, etc.), an operating system type of the terminal device (Android, IOS, etc.), a terminal type (smart machine, function machine, data card, etc.), and a terminal model (e.g., HUAWEI NOVA 3, APPLE iPhone 5, etc.), and may be obtained by an International Mobile Equipment Identity (IMEI), and may uniquely identify an IMEI model.

3) The terms "system" and "network" in the embodiments of the present application may be used interchangeably. "plurality" means two or more, and other terms are analogous. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. Furthermore, for elements (elements) that appear in the singular form "a," an, "and" the, "they are not intended to mean" one or only one "unless the context clearly dictates otherwise, but rather" one or more than one. For example, "a device" means for one or more such devices. Still further, at least one (at least one of a).

The technical scheme of the embodiment of the application can be applied to various communication systems, for example: long Term Evolution (LTE) systems, Worldwide Interoperability for Microwave Access (WiMAX) communication systems, future fifth Generation (5th Generation, 5G) systems, such as new radio access technology (NR), and future communication systems, such as 6G systems.

This application is intended to present various aspects, embodiments or features around a system that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. discussed in connection with the figures. Furthermore, a combination of these schemes may also be used.

The network architecture and the service scenario described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application, and as a person of ordinary skill in the art knows that along with the evolution of the network architecture and the appearance of a new service scenario, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

Some scenarios in the embodiment of the present application are described by taking a scenario of an NR network in a wireless communication network as an example, it should be noted that the scheme in the embodiment of the present application may also be applied to other wireless communication networks, and corresponding names may also be replaced by names of corresponding functions in other wireless communication networks.

For the convenience of understanding the embodiments of the present application, a communication system applied to the embodiments of the present application will be first described in detail by taking the communication system shown in fig. 1 as an example. The communication system may include a Communication Service Management Function (CSMF) network element, a slice management function (NSMF) network element, a sub-slice management function (NSSMF) network element, AN access and mobility management function (AMF) network element, a Session Management Function (SMF) network element, a user equipment (user equipment, UE), AN Access Network (AN) device, and a User Plane Function (UPF) network element. The AMF network element and the terminal equipment can be connected through AN N1 interface, the AMF and the AN equipment can be connected through AN N2 interface, the AN equipment and the UPF can be connected through AN N3 interface, and the SMF and the UPF can be connected through AN N4 interface.

The session management function network element, the access and mobility management function network element and the user plane function network element may form one network slice for carrying a service. The interface name is only an example, and the embodiment of the present application is not particularly limited thereto. It should be understood that the embodiments of the present application are not limited to the communication system shown in fig. 1, and the names of the network elements shown in fig. 1 are only illustrated as an example herein, and are not intended to limit the network elements included in the communication system architecture to which the method of the present application is applicable.

As shown in fig. 1, a new network element or a new network slice may be added to the communication system for carrying traffic, and the following describes two cases respectively.

Case 1: adding new network elements

As shown in fig. 2, a new user plane function UPF network element may be added in the communication system, and the user plane function UPF network element and the original user plane function UPF network element share a session management function SMF network element and a control plane network element such as an access and mobility management function AMF network element. Before the added new user plane function UPF network element carries a service, the service verification needs to be performed on the new user plane function UPF network element.

Case 2: adding new network slices

As shown in fig. 3, a new network slice may be added in the communication system, and the new network slice may be implemented by an access and mobility management function AMF network element, a session management function SMF network element, and a user plane function UPF network element. In other words, if a new network slice is to be added in the communication system, new control plane network elements (such as an access and mobility management function, AMF, network element and a session management function, SMF, network element) and user plane network elements (such as a user plane function, UPF, network element) need to be added in the communication system. Therefore, when service verification is performed, the added access and mobility management function AMF network element, the session management function SMF network element and the user plane function UPF network element can be verified one by one.

The functions of the various network elements or devices in the communication system are described in detail below:

the terminal device, which may also be referred to as a User Equipment (UE), a Mobile Station (MS), a Mobile Terminal (MT), etc., is a device that provides voice and/or data connectivity to a user. For example, the terminal device may include a handheld device, a vehicle-mounted device, and the like having a wireless connection function. Currently, the terminal device may be: a mobile phone (mobile phone), a tablet computer, a notebook computer, a palm top computer, a Mobile Internet Device (MID), a wearable device, a Virtual Reality (VR) device, an Augmented Reality (AR) device, a wireless terminal in industrial control (industrial control), a wireless terminal in self-driving (self-driving), a wireless terminal in remote surgery (remote medical supply), a wireless terminal in smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in city (smart city), a wireless terminal in smart home (smart home), and the like. The terminal device in fig. 1 is shown as a UE, which is only an example and is not limited to the terminal device.

The radio access network may be AN Access Network (AN) shown in fig. 1, and provides a radio access service to the terminal device. The access network device is a device for accessing the terminal device to a wireless network in the communication system. The access network device is a node in a radio access network, which may also be referred to as a base station, and may also be referred to as a Radio Access Network (RAN) node (or device). Currently, some examples of access network devices are: a gbb, a Transmission Reception Point (TRP), an evolved Node B (eNB), a Radio Network Controller (RNC), a Node B (NB), a Base Station Controller (BSC), a Base Transceiver Station (BTS), a home base station (e.g., home evolved Node B, or home Node B, HNB), a Base Band Unit (BBU), or a wireless fidelity (Wifi) Access Point (AP), etc.

For example, the radio access network device in the embodiment of the present application may split the radio access network device into two parts according to the protocol stack function: a Centralized Unit (CU) and a Distributed Unit (DU). Wherein a radio access network device may contain one CU and at least one DU, as shown in fig. 4. A CU is connected to at least one DU and may be used to manage or control the at least one DU. The structure can separate the protocol layer of the wireless access network device in the communication system, wherein part of the protocol layer function is realized in the CU, and the rest or all of the protocol layer function is realized in the DU, and the CU controls the DU in a centralized way. Taking radio access network equipment as an example of a gNB, a protocol layer of the gNB includes a Radio Resource Control (RRC) layer, a Service Data Adaptation Protocol (SDAP) layer, a Packet Data Convergence Protocol (PDCP) layer, a Radio Link Control (RLC) layer, a media access control sublayer (MAC) layer, and a physical layer. For example, the CU may be configured to implement the functions of the RRC layer, the SDAP layer, and the PDCP layer, and the DU may be configured to implement the functions of the RLC layer, the MAC layer, and the physical layer. The embodiment of the present application does not specifically limit the protocol stacks included in the CU and the DU. The CU and the DU may be connected using an F1 interface, the CU and other radio access network devices using an Xn interface, and the CU and the 5G Core network (5G Core, 5GC) using an NG interface, as shown in fig. 5.

For example, a CU in the embodiment of the present application may be further divided into a control plane (CU-CP) network element and at least one user plane (CU-user plane, CU-UP) network element. Wherein, the CU-CP can be used for control plane management, and the CU-UP can be used for user plane data transmission. The interface between the CU-CP and the CU-UP can be the E1 port. The interface between the CU-CP and the DU may be F1-C for transport of control plane signaling. The interface between CU-UP and DU may be F1-U for user plane data transmission. And the CU-UP can be connected through an Xn-U port to carry out user plane data transmission. For example, taking the gbb as an example, the structure of the gbb may be as shown in fig. 6.

The core network is used for accessing the terminal equipment to DN which can realize the service of the terminal equipment. The following describes the functions of each network element in the core network:

the core network access and mobility management function network element AMF may be configured to manage access control and mobility of the terminal device, and in an actual application, the core network access and mobility management function network element AMF includes a mobility management function in a Mobility Management Entity (MME) in a network frame in Long Term Evolution (LTE), adds an access management function, and may be specifically responsible for registration, mobility management, a tracking area update procedure, reachability detection, selection of a session management function network element, mobility state transition management, and the like of the terminal device. For example, in 5G, the core network access and mobility management function network element may be an AMF (access and mobility management function) network element, for example, as shown in fig. 2, and in future communication, for example, in 6G, the core network access and mobility management function network element may still be an AMF network element or have another name, which is not limited in this application. When the core network access and mobility management function network element is an AMF network element, the AMF may provide a Namf service.

The session management function network element SMF may be configured to be responsible for session management (including establishment, modification, and release of a session) of the terminal device, selection and reselection of a user plane function network element, Internet Protocol (IP) address allocation, quality of service (QoS) control, and the like of the terminal device. For example, in 5G, the session management function network element may be an SMF (session management function) network element, for example, as shown in fig. 1, in future communications, for example, in 6G, the session management function network element may still be an SMF network element, or may have another name, which is not limited in this application. When the session management function network element is an SMF network element, the SMF may provide an Nsmf service.

The network storage function network element NRF is configured to register and manage (network function, NF) network functions. For example, in 5G, the network storage function network element may be an NRF (network function relocation function), for example, as shown in fig. 1, in future communication, for example, as in 6G, the network storage function network element may still be an NRF network element, or have another name, which is not limited in this application.

The above network elements in the core network may also be referred to as functional entities, and may be network elements implemented on dedicated hardware, or may be software instances running on dedicated hardware, or may be instances of virtualized functions on a suitable platform, for example, the above virtualized platform may be a cloud platform.

It should be noted that the architecture of the communication system shown in fig. 1 is not limited to include only the network elements shown in the figure, and may also include other devices not shown in the figure, which are not specifically listed here.

It should be noted that the embodiment of the present application does not limit the distribution form of each network element in the core network, and the distribution form shown in fig. 1 is only an example, and the present application is not limited.

If a network slice or a network element in a communication system needs to be added, service verification needs to be performed on the network slice or the network element, and it can be ensured that a newly-built network slice or network element can safely and stably carry service. At present, a method for performing service verification on a newly-built network slice or network element needs a large amount of manual test operation, and the service coverage of terminal equipment during service verification is small, so that the accuracy of a verification result is difficult to ensure.

Based on the above requirements, the embodiment of the present application provides a gray scale verification method. Referring to fig. 7, an exemplary flowchart of a service verification method provided in the embodiment of the present application is shown. It should be understood that, when a network element or a newly added network slice is newly added in a communication system, the service verification method provided in the embodiment of the present application may be used to perform service verification on the newly added network element or the newly added network slice, and further, when a network slice is newly added in the communication system, the method in the embodiment of the present application may be specifically used to perform service verification on each network element in the newly added network slice one by one. As shown in fig. 7, a processing flow chart of a service verification method provided in this embodiment of the present application is shown, and the method may be executed by a service verification apparatus. The service verification apparatus may be a separate server, or may also be a software program newly added in an existing network element in the core network, for example, a software program for executing the service verification process in the embodiment of the present application may be newly added in an existing session management function network element, and the embodiment of the present application is not limited thereto. The method may comprise the steps of:

step 701: and controlling the network element to be tested to provide services for the plurality of terminal devices.

The plurality of terminal devices may here be located in a first network which does not comprise the network elements to be tested. The network element to be tested may be the newly added network element, or the network element to be tested may be any one of the newly added network slices. The plurality of terminal devices may be obtained by the service verification apparatus through screening according to the attribute of the terminal device. The service verification device can control the network element to be tested to provide the service of the relevant service for the plurality of terminal devices when the plurality of terminal devices access the network side to execute the service. The attributes may include service attributes and device attributes. The service attributes may be used to describe the services performed by the terminal device. Such as voice traffic, data traffic, etc. The device attributes are used for describing software and hardware information of the terminal device. Such as information on the brand, model, operating system, etc. of the terminal device. Wherein the service attribute and the device attribute of the plurality of terminal devices are not completely the same.

For example, the network element to be tested may be controlled to provide services for terminal device a, terminal device B, and terminal device C. The device attribute of the terminal device a may be a model a of a brand 1, and the service attribute of the terminal device a is a voice service. The device attribute of the terminal device B may be a model B of a brand 2, and the service attribute of the terminal device B is a normal data service. The device attribute of the terminal device C may be a model a of a brand 1, and the service attribute of the terminal device C may be a general data service. Hereinafter, a method of screening service attributes or device attributes of a plurality of terminal devices will be described.

In the embodiment of the present application, first history data output by each network element in the first network may be obtained. The first history data may be external data representation (XDR) data, such as session establishment data (create session), bearer modification data (modify bearer), bearer establishment data (create bearer), bearer update data (update bearer), and the like. Wherein, the first historical data comprises one or more characteristic fields. The characteristic field here can be used to characterize the service attributes and/or device attributes of the terminal device. Hereinafter, each of the characteristic fields is described separately.

1. Terminal access type (access type), may include Universal Terrestrial Radio Access Network (UTRAN), global system for mobile communications/enhanced data rates for GSM Evolution radio communication network (GERAN), Wireless Local Area Network (WLAN), Generated Access Network (GAN), enhanced high speed packet access technology (high-speed packet access, HSPA Evolution), evolved universal terrestrial radio access network (E-UTRAN), and the like.

2. The quality of service identifier (QCI) is used to measure the packet forwarding behavior (e.g., packet loss rate, packet delay budget, etc.) specifically provided to the service data flow.

3. International Mobile Equipment Identity (IMEI), which uniquely identifies a device model, may read the type of the terminal device (such as a smart machine, a function machine, or a data card), the name of the operating system of the terminal device, the brand name of the terminal device, the model of the terminal device, and the like.

4. An Access Point Name (APN) is a parameter that must be configured when a terminal device accesses the internet, and determines which access method the terminal device accesses the mobile network, and may be used to identify a General Packet Radio Service (GPRS) service type.

5. International Mobile Subscriber Identity (IMSI) is an identity that does not repeat in all cellular networks, used to distinguish between different subscribers in a cellular network.

In an embodiment, the network element having a session management function in the first network may output first history data of a specified number of terminal devices, and the service verification apparatus may obtain the characteristic field. The service verification device can also classify the specified number of terminal devices according to the characteristic fields. For example, terminal devices of the same GPRS service class may be classified into one class according to the APN, or terminal devices of the same brand may also be classified into one class according to the IMEI, and so on. Therefore, the terminal device can be selected in each class, and the network element to be tested is controlled to provide services for the selected terminal devices. When a plurality of terminal devices are selected, the same number of terminal devices may be selected from each class. Alternatively, the ratio of the number of terminal devices in each class to the aforementioned specified number may be calculated, and the terminal device may be selected from each class according to the calculated ratio. For example, first history data of 1000 terminal devices may be obtained, and the 1000 terminal devices may be divided into 8 classes according to different feature fields carried in the first history data of the 1000 terminal devices. The number of the terminal devices in each class is 85 classes A, 125 classes B, 300 classes C, 10 classes D, 80 classes E, 110 classes F, 190 classes G and 100 classes H. The ratio corresponding to each class is calculated, wherein the A class accounts for 8.5%, the B class accounts for 12.5%, the C class accounts for 30%, the D class accounts for 1%, the E class accounts for 8%, the F class accounts for 11%, the G class accounts for 19% and the H class accounts for 10%. For example, if the network element to be tested is to be controlled to provide services for 800 terminal devices, 68 terminal devices of class a, 100 terminal devices of class B, 240 terminal devices of class C, 8 terminal devices of class D, 64 terminal devices of class E, 88 terminal devices of class F, 152 terminal devices of class G, and 80 terminal devices of class H may be selected. Therefore, the service attribute of the selected terminal equipment and the coverage of the equipment attribute are wider, and the obtained test result of the network element to be tested is more accurate. Wherein each class may represent a user group, and the terminal devices in the class may be representative users of the user group.

In another embodiment, the first history data of the specified number of terminal devices may be obtained, and the characteristic field may be obtained from the first history data of the specified number of terminal devices. The service authentication device may first classify a specified number of terminal devices according to the IMEI in the above-mentioned characteristic field. And dividing the terminal equipment in each class into a plurality of subclasses according to the characteristic fields of QCI, terminal equipment access type, APN and the like of the terminal equipment in each class. Thus, a plurality of terminal devices may be selected from each of the sub-classes. When a plurality of terminal devices are selected, the same number of terminal devices may be selected from each of the sub-classes. Or, a certain number of terminal devices may be screened from each of the subsets to access the network element to be tested according to a ratio of the number of the terminal devices in each of the subsets to the specified number. For example, the number of terminal devices in each sub-class is respectively sub-class A-85, sub-class B-125, sub-class C-300, sub-class D-10, sub-class E-80, sub-class F-110, sub-class G-190, and sub-class H-100. Taking the specified number as 1000 as an example, the service verification apparatus may calculate the ratio corresponding to each class, where the subclass a accounts for 8.5%, the subclass B accounts for 12.5%, the subclass C accounts for 30%, the subclass D accounts for 1%, the subclass E accounts for 8%, the subclass F accounts for 11%, the subclass G accounts for 19%, and the subclass H accounts for 10%. For example, if the network element to be tested is to be controlled to provide services for 800 terminal devices, 68 terminal devices of subclass a, 100 terminal devices of subclass B, 240 terminal devices of subclass C, 8 terminal devices of subclass D, 64 terminal devices of subclass E, 88 terminal devices of subclass F, 152 terminal devices of subclass G, and 80 terminal devices of subclass H may be selected. Each subclass herein may be a user group, and the terminal devices in the subclass may be representative users of the user group.

In another embodiment, the first history data of the specified number of terminal devices may be obtained, and the specified number of terminal devices may be clustered according to the plurality of characteristic fields carried in the first history data of the specified number of terminal devices. The clustering result may include a plurality of classes, each class may represent a user group, and the terminal device in each class may represent a representative user in a user group. For example, the weight of each feature field may be set during clustering, and the sum of the weights of all feature fields is 1. It should be understood that the weight of each feature field may be determined empirically. For example, the weight of the terminal access type is set to 0.4, the weight of the QCI is set to 0.3, the weight of the IMEI is set to 0.2, and the weight of the APN is set to 0.1. The weight of each feature field may indicate how important the feature field is in clustering. The service verification device can execute a clustering algorithm on a specified number of terminal devices according to the characteristic fields and the weight of each characteristic field, so as to obtain a clustering result. The clustering algorithm here may be a spatial clustering algorithm. The clustering result may include a plurality of classes, each class may represent a user group, and the terminal device in each class may represent a representative user in a user group. It should be understood that each of the aforementioned specified number of terminal devices may belong uniquely to a class, i.e. a user group.

For example, first history data of 1000 terminal devices may be obtained, and a clustering algorithm may be performed on the 1000 terminal devices according to a plurality of feature fields carried in the first history data of the 1000 terminal devices and a weight of each feature field, and an obtained clustering result may include a plurality of classes, where each class may represent a user behavior group. For example, class 1 may represent a user group with 4G service + normal data service + hua as a terminal, and class 2 may represent a user group with 4G service + normal data service + apple terminal, etc.

It should be noted that the service attributes or the device attributes of the terminal devices in each class are not necessarily the same, for example, the class 1 may include a terminal device whose service attribute is 3G service and normal data service and whose device attribute is huacheng, or may further include a terminal device whose service attribute is 4G service and normal data service and whose device attribute is VIVO, and the like. Since the terminal devices are classified by a clustering algorithm, the traffic behavior of the terminal devices in each class may be similar.

In the embodiment of the application, the service verification is performed on the network element to be tested. Therefore, the terminal device to be tested can provide services for any one terminal device in the specified number of terminal devices, but cannot provide services for other terminal devices except the specified number of terminal devices. Because, the network element to be tested cannot know to which user group the aforementioned other terminal equipment belongs. For example, the network element to be tested may maintain a correspondence between the unique identifier of the terminal device and the class to which the terminal device belongs, so that when a terminal device accesses the network element to be tested, which user group the terminal device belongs to may be determined according to the unique identifier of the terminal device, and whether the network element to be tested may provide a service for the terminal device may also be determined according to the unique identifier.

After the service verification device clusters a specified number of terminal devices according to the characteristic fields and the weight values of each characteristic field, the terminal devices can be selected in each class in the clustering result. Wherein a certain number of terminal devices may be selected from each class. For example, the same number of terminal devices may be selected from each class, and the network element to be tested may be controlled to provide services for the terminal devices of each class. Alternatively, the ratio of the number of terminal devices in each class to the aforementioned specified number may be calculated, and the terminal device may be selected in each class according to the calculated ratio. For example, the network element to be tested may provide services for 1000 terminal devices, and if the ratio of the number of the terminal devices in class 1 to the specified number is 10%, 100 terminal devices may be screened from class 1, and the network element to be tested is controlled to provide services for the 100 terminal devices.

In the embodiment of the application, after the plurality of terminal devices are classified, the service verification device can also add a service verification task of the network element to be tested. As shown in fig. 8, the service verification apparatus may select the network element identifier to be tested, the maximum access number of the terminal device providing the service to the network element to be tested, and the time T1 and T2 for providing the service to the network element to be tested after the service verification task is started. After the service verification task is added and started, the time for the network element to be tested to provide service for the plurality of terminal devices can be controlled. The plurality of terminal devices may be terminal devices obtained by filtering according to the above filtering method.

Since the service verification apparatus cannot control the behavior of the terminal device, but in order to ensure that the network element to be tested can provide service for the terminal device of each class, the service verification apparatus may use a barreling algorithm to reserve a space for the terminal device of each class. As shown in fig. 9, assuming that there are 1000 terminal devices, according to the method for screening terminal devices, it is determined that the network element to be tested can provide services for 5 classes of terminal devices, and thus there may be 5 buckets. Each bucket represents a class, i.e., a user group, the number of terminals to be accessed in bucket 1 is 200, the number of terminals to be accessed in bucket 2 is 220, the number of terminals to be accessed in bucket 3 is 100, the number of terminals to be accessed in bucket 4 is 300, and the number of terminals to be accessed in bucket 5 is 180. When it is determined that there is a class of terminal devices accessing the network side, the service verification apparatus may add 1 to the number of accessed terminal devices in the corresponding bucket, and control the network element to be tested to provide services for the accessed terminal devices, and when the number of accessed terminal devices in any bucket reaches the number of terminal devices to be accessed in any bucket, prohibit the network element to be tested from providing services for terminal devices which belong to the class corresponding to any bucket and have not accessed the network element to be tested. The service verification device can determine which class the terminal equipment of the access network side belongs to according to the unique identifier of the terminal equipment of the access network side and the corresponding relation between the unique identifier and the class to which the terminal equipment belongs. The service verification device may add 1 to the number of the accessed terminal devices of the bucket corresponding to the class according to the determined class of the terminal device, and control the network element to be tested to provide service for the accessed terminal device.

For example, the bucket 3 may represent class 1, and when determining that a class 1 terminal device is accessed to the network side, the service verification apparatus may add 1 to the number of accessed terminal devices of the bucket 3, and control the network element to be tested to provide services for the accessed terminal device, and when the number of accessed terminal devices of the bucket 3 reaches 100, prohibit the network element to be tested from providing services for other terminal devices belonging to class 1 that are not accessed to the network. For another example, the bucket 2 may represent class 2, and when it is determined that the terminal device of class 2 accesses the network side, the number of the accessed terminal devices of the bucket 2 may be added by 1, and the network element to be tested is controlled to provide services for the accessed terminal device, and when the number of the accessed terminal devices of the bucket 2 reaches 220, the network element to be tested is prohibited from providing services for other terminal devices belonging to class 2 that do not access the network.

When a class of terminal device finishes the service through the network element to be tested and leaves the service provided by the network element to be tested, the service verification device can reduce the number of the accessed terminal devices of the bucket corresponding to the class by 1. For example, bucket 2 may represent class 2, and the number of accessed terminal devices of bucket 2 has reached 220, so the traffic verification apparatus may prohibit the network element to be tested from providing services for other terminal devices belonging to the non-accessed network in class 2. At this time, one terminal device in class 2 ends the current service through the network element to be tested and leaves, so the service verification apparatus can reduce the number of the accessed terminal devices of bucket 2 by 1, and at this time, the number of the accessed terminal devices of bucket 2 is changed to 219, so that the network element to be tested can be allowed to continue to provide services for other terminal devices belonging to class 2 that are not accessed to the network.

In the above-mentioned barreling algorithm, the number of the terminal devices to be accessed may be the maximum number value that the service verification apparatus allows the network element to be tested to simultaneously provide services for the terminal devices belonging to one class. By the barreling algorithm, the network element to be tested can be ensured to provide service for each class of terminal equipment.

Step 702: and acquiring test data generated by the plurality of terminal devices in the process of receiving the service provided by the network element to be tested.

In the embodiment of the application, the plurality of terminal devices can be screened by the method, and the network element to be tested is controlled to provide services for the plurality of terminal devices, and the plurality of terminal devices can execute corresponding services through the network element to be tested. And when the starting time of the task reaches the designated time, the service verification task can be ended. The specified time period here may be determined based on an empirical value.

After the service verification task is finished, the service verification apparatus may obtain a plurality of test data of a plurality of terminal devices. Wherein, a test data may be a data generated when a terminal device executes any service through the network element to be tested. The test data may include the number of times, the number of requests, the number of successful requests, etc. that the terminal device accesses the network element to be tested. The service verification device may obtain a plurality of test data of a plurality of terminal devices, where the test data is obtained by the service verification device from log information recorded by a network management side, or the test data is actively reported to the service verification device by the terminal device.

Step 703: and calculating the first service success rate of each service attribute and the first service success rate of each equipment attribute according to the test data.

The plurality of test data may include session establishment data, bearer modification data, bearer establishment data, bearer update data, and the like. The service verification device can respectively calculate the first service success rate of the terminal equipment belonging to each class according to the data. The first service success rate here may include a first access success rate and a first bearer update success rate. For example, the service verification apparatus may obtain the first access success rate of the class by calculation according to session establishment data, bearer establishment data, and the like of the terminal device belonging to each class, or obtain the first bearer update success rate of the class by calculation according to bearer modification data and bearer update data of the terminal device belonging to each class.

In specific implementation, a first service success rate when a plurality of terminal devices execute a service through a network element to be tested can be obtained. In an example, a first service success rate when a terminal device belonging to one device attribute executes a service through a network element to be tested may be obtained, or a first service success rate when a plurality of terminal devices execute a service corresponding to one service attribute through the network element to be tested may be obtained. Here, the terminal device belonging to one device attribute may be one class, and the terminal device executing the service corresponding to one service attribute through the network element to be tested may be another class. For example, terminal devices belonging to class 1, class 2, class 3, and class 4, respectively, perform a service through a network element to be tested. The terminal device of class 1 may be a terminal device of device attribute 1, the terminal device of class 2 may be a terminal device of device attribute 2, the terminal device of class 3 may be a terminal device for executing a service corresponding to service attribute 1, and the terminal device of class 4 may be a terminal device for executing a service corresponding to service attribute 2. Therefore, the first service success rate of the device attribute 1 may be calculated according to the plurality of test data of the terminal device in class 1, and the first service success rate of the device attribute 2 may be calculated according to the plurality of test data of the terminal device in class 2. And calculating the first service success rate of the service attribute 1 according to the plurality of test data of the terminal device in the class 3, and calculating the first service success rate of the service attribute 2 according to the plurality of test data of the terminal device in the class 4. The first service success rate of the device attribute 1 may be a service success rate at which the terminal device of the device attribute 1 executes the service corresponding to the service attribute 1 and the service attribute 2 through the network element to be tested. The first service success rate of the device attribute 2 may be a service success rate at which the terminal device of the device attribute 2 executes the service corresponding to the service attribute 1 and the service attribute 2 through the network element to be tested. The first service success rate of the service attribute 1 may be a service success rate at which the terminal device of the device attribute 1 and the terminal device of the device attribute 2 executes the service corresponding to the service attribute 1 through the network element to be tested. The first service success rate of the service attribute 2 may be a service success rate at which the terminal device of the device attribute 1 and the terminal device of the device attribute 2 executes the service corresponding to the service attribute 2 through the network element to be tested.

In an embodiment, a first service success rate of the terminal device of each device attribute executing the service corresponding to each service attribute through the network element to be tested may be calculated. The first service success rate of the terminal device belonging to one device attribute executing the service of one service attribute may represent the first service success rate of one class. Therefore, the first service success rate of each class can be calculated according to a plurality of test data. The first service success rate of each class may be a service success rate when the terminal device belonging to each class performs a service through the network element to be tested.

For example, the network element to be tested provides services for terminal devices belonging to class 1, class 2, class 3, and class 4, respectively. The terminal device of class 1 belongs to the device attribute 1, and executes the service corresponding to the service attribute 1 through the network element to be tested. The terminal device in class 2 belongs to the device attribute 2, and executes the service corresponding to the service attribute 1 through the network element to be tested. The terminal device in class 3 belongs to the device attribute 1, and executes the service corresponding to the service attribute 2 through the network element to be tested. The terminal device in class 4 belongs to the device attribute 2, and executes the service corresponding to the service attribute 2 through the network element to be tested. The first service success rate of class 1 may be a service success rate at which the terminal device of equipment attribute 1 executes the service corresponding to service attribute 1 through the network element to be tested, the first service success rate of class 2 may be a service success rate at which the terminal device of equipment attribute 2 executes the service corresponding to service attribute 1 through the network element to be tested, class 3 may be a service success rate at which the terminal device of equipment attribute 1 executes the service corresponding to service attribute 2 through the network element to be tested, and the first service success rate of class 3 may be a service success rate at which the terminal device of equipment attribute 2 executes the service corresponding to service attribute 2 through the network element to be tested.

In one embodiment, after acquiring the plurality of test data, a thermodynamic diagram as shown in FIG. 10 may be generated. Each small grid in fig. 10 may represent a class, and the abscissa may be different device attributes and the ordinate may be different service attributes. Since the thermodynamic diagram is a two-dimensional image, but the attribute of each class can be obtained by the device attribute and the service attribute in the corresponding two-dimensional diagram, the thermodynamic diagram can be displayed with the device attribute as the abscissa and the service attribute as the ordinate, but as shown in fig. 10, other attributes of the class, such as the terminal access type, the APN, and the like, can also be displayed when each cell is clicked. In other words, each small lattice may represent a class, and the number in each small lattice may represent the total number of times that the terminal device of the class corresponding to the small lattice accesses the network element to be tested, or the number in each small lattice may also represent the total number of times that the terminal device of the class corresponding to the small lattice fails to access the network element to be tested.

For example, as shown in fig. 10, the number in the lattice corresponding to the device attribute 1-service attribute 2 is 9, so that the total number of times that the terminal device in the class corresponding to the device attribute 1-service attribute 2 fails to access the network element to be tested is 9, or the total number of times that the terminal device in the class corresponding to the device attribute 1-service attribute 2 accesses the network element to be tested is 9.

Step 704: and determining whether the network element to be tested is normal or not according to the first service success rate and the second service success rate of each service attribute and each equipment attribute.

The second service success rate of the plurality of terminal devices when executing the service in the first network may be calculated in advance according to the first history data of the plurality of terminal devices. The first history data here may be XDR data, such as session establishment data, bearer modification data, bearer establishment data, bearer update data, etc. The service verification device may calculate a second service success rate of the terminal device belonging to one class according to the XDR data. The second service success rate may include at least one of a second access success rate and a second bearer update success rate. For example, the service verification apparatus may calculate the second access success rate according to session establishment data, bearer establishment data, and the like of the terminal device belonging to one class. For another example, the service verification apparatus may calculate the second bearer update success rate according to the bearer modification data and the bearer update data of the terminal device belonging to the one class.

The service verification apparatus may compare a first service success rate of the terminal device belonging to a class with a second service success rate determined for the class of terminal device to determine whether the network element to be tested is normal. For example, the terminal devices accessing the network element to be tested belong to class 1, class 2, class 3, and class 4, respectively. The terminal device in class 1 belongs to device attribute 1, and executes the service corresponding to the service attribute 1 through the network element to be tested. The terminal device in class 2 belongs to the device attribute 2, and executes the service corresponding to the service attribute 1 through the network element to be tested. The terminal device in class 3 belongs to the device attribute 1, and executes the service corresponding to the service attribute 2 through the network element to be tested. The terminal device in class 4 belongs to the device attribute 2, and executes the service corresponding to the service attribute 2 through the network element to be tested. The service verification device may obtain the second service success rates of class 1, class 2, class 3, and class 4, which are calculated in advance according to the first history data. And acquiring a plurality of test data of a plurality of terminal devices respectively belonging to class 1, class 2, class 3 and class 4, and calculating to obtain the first service success rate of class 1, class 2, class 3 and class 4. The service verification device may calculate a relative difference ratio between the first service success rate and the second service success rate of each class, respectively. For example, the relative difference ratio for class 1 may be | (Q1-Q2)/Q2 |. Wherein Q1 is the first traffic success rate of class 1 and Q2 is the second traffic success rate of class 1. It should be understood that the service verification apparatus may calculate the relative difference ratio between the first service success rate and the second service success rate of class 2, class 3, and class 4 according to the above formula.

If the relative difference ratio of the first service success rate and the second service success rate of one class is larger than a preset value, the network element to be tested is determined to be abnormal. The preset value here may be predetermined based on an empirical value.

By the method, the first service success rate and the second service success rate of each class can be compared to determine whether the network element to be tested is normal. If any one of the comparison results meets the condition that the ratio of the relative difference between the first service success rate and the second service success rate is greater than a first preset value, the network element to be tested is considered to be abnormal and cannot normally bear services. If any comparison result does not exist, the relative difference ratio between the first service success rate and the second service success rate is larger than a preset value, the network element to be tested is considered to be normal.

Or, if the ratio of the relative difference between the first service success rate and the second service success rate which is greater than or equal to the specified value is greater than the preset value, it may be determined that the network element to be tested is abnormal. For example, assuming that the specified value is 2, the ratio of the relative difference between the first service success rate and the second service success rate of class 1 and class 2 is smaller than the preset value, and the ratio of the relative difference between the first service success rate and the second service success rate of class 3 and class 4 is larger than the preset value, it may be determined that the network element to be tested is abnormal and cannot carry a task.

When the network element to be tested is abnormal, Root Cause Analysis (RCA) can be performed through a hotspot algorithm according to a plurality of test data. The RCA can be performed through a hotspot algorithm according to the obtained multiple test data, so as to determine the characteristics of the terminal equipment which executes the abnormal service through the network element to be tested. Such as the model, brand, operating system of the terminal device or service attributes of the service executed by the terminal device.

In an embodiment, the RCA may be performed by using a hot spot algorithm according to the first test data of the terminal device that satisfies the class in which the ratio of the relative difference between the first service success rate and the second service success rate is greater than the first preset value. For example, the relative difference ratio between the first service success rate of class 1 and the second service success rate is greater than a preset value, and the relative difference ratio between the first service success rate of class 2 and the second service success rate of class 2 is also greater than a preset value. Therefore, the first test data of the terminal device of class 1 and the first test data of the terminal device of class 2 can be acquired, and the RCA is performed through the hotspot algorithm.

Hereinafter, a gradation verifying method in the embodiment of the present application is described by specific embodiments. As shown in fig. 11A, the first network includes AMF1, AMF2, SMF1, SMF2, SMF3, UPF1 and UPF2, and it is now desired to add UPF3 as UPF1 and UPF2 to share the service, so that service verification needs to be performed on UPF 3. At this time, the server 1 of the management plane may acquire XDR data of a specified number of terminal devices in the first network from the SMF1, the SMF2, and the SMF3, and classify the specified number of terminal devices according to the XDR data, resulting in a plurality of classes of terminal devices. For the classification method, reference may be made to the related description in the method embodiment shown in fig. 7, which is not described herein again. The server 1 may also display the second service success rate of each class calculated in advance according to the XDR data through the display interface. And a service verification task can be added to the display interface to perform service verification on the UPF 3. After the service verification task is started, the server 1 may send the corresponding relationships between the unique identifiers of the specified number of terminal devices and the corresponding classes to the SMF1, the SMF2, and the SMF3, respectively, and control the UPF3 to provide services for the specified number of terminal devices. The SMF1, the SMF2, or the SMF3 may obtain the unique identifier of the terminal device that is accessed, and if the unique identifier of the terminal device is in the corresponding relationship, the UPF3 may be selected to provide a service for the terminal device, so as to implement service verification on the UPF 3. After the service verification task is finished, the server 1 may acquire a plurality of test data from the SMF1, the SMF2, and the SMF 3. The server 1 may also calculate the success rate of the first services of the multiple classes according to the multiple test data, and display the success rate through the display interface.

As shown in fig. 11B, the total number of times that the terminal device of each class accesses the network element to be tested may be displayed, or the number of times that the terminal device of each class fails to access the network element to be tested may also be displayed. In addition, the first service success rate and the second service success rate of each class can be displayed through a curve graph, and the characteristics of the class of which the relative difference ratio of the first service success rate and the second service success rate is larger than a first preset value are analyzed through an RCA algorithm.

Similar to the above concept, as shown in fig. 12, an embodiment of the present application further provides a service verification apparatus, which can implement the service verification method described above. The apparatus 1200 may comprise a storage unit 1203 and a processing unit 1202. Optionally, a communication unit 1201 is further included. The processing unit 1202 may be connected to the storage unit 1203 and the communication unit 1201 respectively, and the storage unit 1203 may also be connected to the communication unit 1201. Wherein the processing unit 1202 may be integrated with the storage unit 1203.

The storage unit 1203 is used for storing a computer program;

the processing unit 1202 is configured to control a network element to be tested to provide services for a plurality of terminal devices; for the screening method of the multiple terminal devices, reference may be made to the related description in the method embodiment shown in fig. 7, which is not described herein again.

The processing unit is further configured to acquire test data generated by the plurality of terminal devices in a process of accepting the service provided by the network element to be tested. And calculating the first service success rate of each service attribute and the first service success rate of each equipment attribute according to the test data.

The processing unit is further configured to determine whether the network element to be tested is normal according to the first service success rate and the second service success rate of each service attribute and each device attribute. The description of the first service success rate and the second service success rate may refer to the related description in the method embodiment shown in fig. 7, and is not described herein again.

In one design, when the processing unit filters the plurality of terminal devices according to the attribute of the terminal device, the processing unit is specifically configured to: and acquiring first history data of each terminal device in the first network respectively. Classifying each terminal device in the first network according to the characteristic field in the first historical data; the characteristic field is used for representing the service attribute and the equipment attribute of the terminal equipment; each class in the classification result represents terminal equipment which belongs to one equipment attribute and executes a service corresponding to one service attribute; and respectively selecting at least one terminal device in each class to obtain the plurality of terminal devices. The description of the feature field may refer to the related description in the method embodiment shown in fig. 7, and is not repeated here.

In one design, when the processing unit selects at least one terminal device in each class, the processing unit is specifically configured to: calculating the ratio of the number of terminal devices of each class to the total number of the terminal devices in the first network; determining the maximum terminal equipment number of each class according to the ratio and the total number of the terminal equipment allowed to be accessed by the network element to be tested; the maximum terminal equipment number of one class represents the maximum value of the number of the terminal equipment of the one class in the terminal equipment of the network element to be tested for providing services; and respectively selecting a corresponding number of terminal devices in each class according to the maximum number of the terminal devices in each class.

In one design, when determining whether the network element to be tested is normal according to the first service success rate and the second service success rate of each service attribute and each device attribute, the processing unit is specifically configured to: if the first service success rate and the second service success rate of at least one service attribute or at least one equipment attribute meet a first condition, determining that the network element to be tested is abnormal; and if the first service success rate and the second service success rate of each service attribute and each equipment attribute do not meet the first condition, determining that the network element to be tested is normal. For the description of the first condition, reference may be made to the description in the method embodiment shown in fig. 7, which is not described herein again.

In one design, the processing unit is further configured to, if the network element to be tested is abnormal, obtain first test data of a terminal device belonging to a service attribute or a device attribute where the first service success rate and the second service success rate satisfy a first condition; and carrying out Root Cause Analysis (RCA) on the reason of the abnormity of the network element to be tested through a hotspot algorithm according to the first test data.

The apparatus 1200 may also be a chip, where the communication unit may be an input/output circuit or an interface of the chip, the processing unit may be a logic circuit, and the logic circuit may process the data to be processed according to the steps described in the above method aspect, and acquire the processed data.

The embodiment of the application also provides a service verification device, which is the same as the concept. Fig. 13 is a schematic structural diagram of an apparatus according to an embodiment of the present disclosure. The apparatus 1300 includes at least one processor 1320 configured to implement the functions of the methods provided by the embodiments of the present application. The apparatus 1300 may also include a communication unit 1310. In embodiments of the present application, the communication unit may be a transceiver, circuit, bus, module, or other type of communication unit for communicating with other devices over a transmission medium. For example, the communication unit 1310 is used for the apparatus in the apparatus 1300 to communicate with other devices. The processor 1320 may perform the functions of the processing unit 1202 shown in fig. 12, and the communication unit 1310 may perform the functions of the communication unit 1201 shown in fig. 12.

The communications apparatus 1300 can also include at least one memory 1330 for storing program instructions and/or data. A memory 1330 is coupled to the processor 1320. The coupling in the embodiments of the present application is an indirect coupling or a communication connection between devices, units or modules, and may be an electrical, mechanical or other form for information interaction between the devices, units or modules. The processor 1320 may operate in conjunction with the memory 1330. Processor 1320 may execute program instructions stored in memory 1330. At least one of the at least one memory may be included in the processor.

The embodiment of the present application does not limit the specific connection medium among the communication unit 1310, the processor 1320, and the memory 1330. In the embodiment of the present application, the memory 1330, the processor 1320, and the communication unit 1310 are connected by a bus 1340, which is indicated by a thick line in fig. 13, and the connection manner between other components is merely illustrative and not limited. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 13, but this is not intended to represent only one bus or type of bus.

The device may execute the service verification method described in the embodiment of the present application, and for concrete implementation and technical effects, reference is made to the foregoing description, which is not repeated herein.

As another form of the present embodiment, there is provided a computer readable storage medium having stored thereon instructions that, when executed, perform the method described in the above method embodiment.

As another form of the present embodiment, there is provided a computer program product containing instructions that, when executed, perform the method described in the method embodiments above.

It should be understood that the Processor mentioned in the embodiments of the present invention may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will also be appreciated that the memory referred to in this embodiment of the invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (DDR SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchronous link SDRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

It should be noted that when the processor is a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, the memory (memory module) is integrated in the processor.

It should be noted that the memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of the processes should be determined by their functions and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for service authentication, comprising:

controlling a network element to be tested to provide services for a plurality of terminal devices; the plurality of terminal devices are obtained by screening according to the attributes of the terminal devices; the attributes include at least one service attribute and at least one device attribute; the service attribute is used for describing a service executed by the terminal equipment; the device attribute is used for describing device information of the terminal device;

acquiring test data generated by the plurality of terminal devices in the process of receiving the service provided by the network element to be tested;

calculating the first service success rate of each service attribute and the first service success rate of each equipment attribute according to the test data;

determining whether the network element to be tested is normal or not according to the first service success rate and the second service success rate of each service attribute and each equipment attribute; the second service success rate of each service attribute and each equipment attribute is obtained according to historical data generated when the plurality of terminal equipment execute services in the first network;

the terminal devices are located in the first network, and the network element to be tested is a newly added network element in the first network or any one of newly added network slices in the first network.

2. The method of claim 1, wherein the first traffic success rate comprises at least one of:

a first access success rate and a first bearer update success rate.

3. The method according to claim 1 or 2, wherein the plurality of terminal devices are obtained by filtering according to the attributes of the terminal devices in the following manner:

acquiring first history data of each terminal device in the first network;

classifying each terminal device in the first network according to the characteristic field in the first historical data; the characteristic field is used for representing the service attribute and the equipment attribute of the terminal equipment; each class in the classification result represents terminal equipment which belongs to one equipment attribute and executes a service corresponding to one service attribute;

and respectively selecting at least one terminal device in each class to obtain the plurality of terminal devices.

4. The method of claim 3, wherein the selecting at least one terminal device in each class comprises:

calculating the ratio of the number of terminal devices of each class to the total number of the terminal devices in the first network;

determining the maximum terminal equipment number of each class according to the ratio and the total number of the terminal equipment allowed to be accessed by the network element to be tested; the maximum terminal equipment number of one class represents the maximum value of the number of the terminal equipment of the one class in the terminal equipment of the network element to be tested for providing services;

and respectively selecting a corresponding number of terminal devices in each class according to the maximum number of the terminal devices in each class.

5. The method of claim 3 or 4, wherein the feature field comprises at least one of:

the access type of the terminal equipment, the request service type of the terminal equipment, the access point name of the terminal equipment or the unique identification of the terminal equipment.

6. The method according to any one of claims 1 to 5, wherein the determining whether the network element to be tested is normal according to the first service success rate and the second service success rate of each service attribute and each device attribute comprises:

if the first service success rate and the second service success rate of at least one service attribute or at least one equipment attribute meet a first condition, determining that the network element to be tested is abnormal;

if the first service success rate and the second service success rate of each service attribute and each equipment attribute do not meet the first condition, determining that the network element to be tested is normal;

the first condition includes:

the difference value between the first service success rate and the second service success rate is greater than a specified value; or

The relative difference ratio of the first service success rate and the second service success rate is larger than a preset value; wherein the relative difference ratio satisfies the following formula:

the relative difference ratio | (first service success rate-second service success rate) |/second service success rate.

7. The method of claim 6, further comprising:

if the network element to be tested is abnormal, acquiring first test data of the terminal equipment which belongs to the service attribute or the equipment attribute, wherein the first service success rate and the second service success rate meet a first condition;

and according to the first test data, performing Root Cause Analysis (RCA) on the reason of the abnormity of the network element to be tested by a hotspot algorithm.

8. A transaction verification apparatus, comprising: a processing unit and a communication unit;

the communication unit is used for carrying out information interaction with the network element to be tested;

the processing unit is configured to execute, by the communication unit, the following processing:

9. The apparatus of claim 8, wherein the first traffic success rate comprises at least one of:

a first access success rate and a first bearer update success rate.

10. The apparatus according to claim 8 or 9, wherein the processing unit, when filtering the plurality of terminal devices according to the attribute of the terminal device through the communication unit, is specifically configured to:

acquiring first history data of each terminal device in the first network;

11. The apparatus according to claim 10, wherein the processing unit, when selecting at least one terminal device from each of the classes through the communication unit, is specifically configured to:

12. The apparatus of claim 10 or 11, wherein the feature field comprises at least one of:

13. The apparatus according to any one of claims 8 to 12, wherein the processing unit is configured to, when determining, by the communication unit, whether the network element to be tested is normal according to the first service success rate and the second service success rate of each service attribute and each device attribute, specifically:

the first condition includes:

14. The apparatus of claim 13, wherein the processing unit is further configured to perform, by the communication unit:

15. A transaction verification apparatus, comprising: a processor and a memory, wherein the processor is capable of processing a plurality of data,

the memory for storing computer programs or instructions;

the processor for executing the computer program or instructions stored in the memory to perform the method of any of claims 1-7.

16. A computer-readable storage medium having computer-readable instructions stored thereon which, when invoked by a computer, cause the computer to perform the method of any one of claims 1 to 7.