CN116738326A

CN116738326A - Model training method, service perception identification method, device, equipment and medium

Info

Publication number: CN116738326A
Application number: CN202210193658.3A
Authority: CN
Inventors: 潘越洋; 李益刚; 黄河; 王东强; 冯媛; 杨翌晨
Original assignee: ZTE Corp
Current assignee: ZTE Corp
Priority date: 2022-03-01
Filing date: 2022-03-01
Publication date: 2023-09-12

Abstract

The present disclosure provides a model training method, comprising: acquiring a training data set, wherein the training data set comprises wireless KPI data and KQI data corresponding to the wireless KPI data, and the wireless KPI data comprises various KPIs; calculating a KQI perception baseline according to the wireless KPI data and the KQI data; processing the KQI data according to the KQI perception base line to generate a KQI label which indicates whether the business perception is abnormal; determining KPI feature combinations comprising various types of KPIs according to the wireless KPI data and the KQI data; screening wireless KPI data according to KPI feature combination, and training a service perception initial model by utilizing the screened wireless KPI data and the KQI label to obtain a service perception identification model; the method can realize the rapid and real-time identification of the situation of poor service perception quality caused by the problem of poor wireless performance by using the service perception identification model in a pure wireless scene. The disclosure also provides a model training device, a business perception anomaly identification method, a business perception identification device, computer equipment and a readable medium.

Description

Model training method, service perception identification method, device, equipment and medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a model training method, a service awareness identification method, a device, a computer device, and a readable medium.

Background

Because the DPI (Deep Packet Inspection ) to be deployed for obtaining service-aware data has high cost, and requires operators to input a large amount of resources such as hard probes and decoding servers, the DPI is deployed in only a few operator networks, but not in most operator networks.

There are many main reasons for influencing the business experience, mainly including: 1) Wireless network performance, such as coverage, interference, capacity, parameters, failure, etc., problems on the wireless side; 2) Performance on the core network, SP (Service Provider) side; 3) The quality of the transmission between the network elements; 4) User, terminal problems, where wireless network performance is often the most dominant factor in poor business experience.

In the conventional wireless tool platform, only data on a wireless side, such as wireless performance data, configuration data of a network manager, and the like, are often included, but service perception data is not collected, so that the wireless tool platform cannot evaluate service perception and cannot identify cells with poor service perception.

In a pure wireless scenario where end-to-end service sensing indexes cannot be acquired, a network optimization engineer typically analyzes wireless network performance indexes reported by each cell according to expert experience. The performance indexes of the wireless network can comprise performance indexes such as access, switching, call drop, time delay, speed rate and the like, quality indexes such as error block, retransmission and the like, and basic indexes such as coverage, interference, capacity and alarm. The network optimization expert analyzes whether each cell has the problem of poor network quality by combining the wireless network performance indexes reported by the histories, thereby optimizing the wireless indexes in time and improving the network quality.

In a pure wireless scene, how to identify the problem of abnormal service perception, so that the targeted wireless network optimization is facilitated, and the service experience is improved, which is a great difficulty in the operation and maintenance of the current wireless network.

Disclosure of Invention

The present disclosure provides a model training method, a business perception recognition method, a device, a computer device and a readable medium.

In a first aspect, embodiments of the present disclosure provide a model training method, the method comprising:

acquiring a training data set, wherein the training data set comprises wireless Key Performance Index (KPI) data and Key Quality Index (KQI) data corresponding to the wireless KPI data, and the wireless KPI data comprises various types of KPIs;

Calculating a KQI perception baseline according to the wireless KPI data and the KQI data;

processing the KQI data according to the KQI perception base line to generate a KQI label which indicates whether the business perception is abnormal;

determining a KPI feature combination according to the wireless KPI data and the KQI data, wherein the KPI feature combination comprises a plurality of types of KPIs;

and screening the wireless KPI data according to the KPI feature combination, and training a business perception identification initial model by utilizing the screened wireless KPI data and the KQI label to obtain a business perception identification model.

In yet another aspect, an embodiment of the present disclosure further provides a service awareness identifying method, where the method includes:

acquiring first wireless KPI data of a designated network element and a designated cell;

screening the first wireless KPI data according to KPI feature combinations to obtain second wireless KPI data;

inputting the second wireless KPI data into a business perception recognition model to obtain a KQI label for representing whether business perception is abnormal; the business perception recognition model is obtained by training the model training method.

In yet another aspect, the disclosed embodiments also provide a model training apparatus, including an acquisition module, a KQI perception baseline calculation module, a KQI label generation module, a KPI feature combination determination module, and a model training module,

The acquisition module is used for acquiring a training data set, wherein the training data set comprises wireless Key Performance Index (KPI) data and Key Quality Index (KQI) data corresponding to the wireless KPI data, and the wireless KPI data comprises multiple types of KPIs;

the KQI perception baseline calculation module is used for calculating a KQI perception baseline according to the wireless KPI data and the KQI data;

the KQI label generation module is used for processing the KQI data according to the KQI perception base line and generating a KQI label which indicates whether the service perception is abnormal or not;

the KPI feature combination determining module is used for determining KPI feature combinations according to the wireless KPI data, wherein the KPI feature combinations comprise various types of KPIs;

the model training module is used for screening the wireless KPI data according to the KPI feature combination, and training a business perception recognition initial model by utilizing the screened wireless KPI data and the KQI label to obtain a business perception recognition model.

In yet another aspect, an embodiment of the present disclosure further provides a service awareness identifying device, including an obtaining module, a screening module and an identifying module,

the acquisition module is used for acquiring first wireless KPI data of a designated network element and a designated cell;

The screening module is used for screening the first wireless KPI data according to the KPI feature combination to obtain second wireless KPI data;

the identification module is used for inputting the second wireless KPI data into a service perception identification model to obtain a KQI label used for indicating whether service perception is abnormal; the business perception recognition model is obtained by training the model training method.

In yet another aspect, the disclosed embodiments also provide a computer device, comprising: one or more processors; a storage device having one or more programs stored thereon; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the model training method or the business-aware identification method as described above.

In yet another aspect, the disclosed embodiments also provide a computer readable medium having a computer program stored thereon, wherein the program when executed implements the model training method or the business-aware identification method as described previously.

The embodiment of the disclosure provides a model training method, which comprises the following steps: acquiring a training data set, wherein the training data set comprises wireless KPI data and KQI data corresponding to the wireless KPI data, and the wireless KPI data comprises various KPIs; calculating a KQI perception baseline according to the wireless KPI data and the KQI data; processing the KQI data according to the KQI perception base line to generate a KQI label which indicates whether the business perception is abnormal; determining a KPI feature combination according to the wireless KPI data and the KQI data, wherein the KPI feature combination comprises a plurality of types of KPIs; screening the wireless KPI data according to the KPI feature combination, and training a service perception initial model by utilizing the screened wireless KPI data and the KQI label to obtain a service perception identification model; according to the embodiment of the disclosure, by training the wireless performance data and the service perception data, the service perception identification model capable of identifying whether the service perception index is abnormal is fitted, so that the situation of poor service perception quality caused by the problem of poor wireless performance can be rapidly and real-timely identified by using the service perception identification model in a pure wireless scene, and the working efficiency of network optimization personnel is greatly improved.

Drawings

FIG. 1 is a schematic flow chart of a model training method provided in an embodiment of the disclosure;

FIG. 2 is a schematic flow chart of calculating a KQI perception baseline provided by an embodiment of the present disclosure;

FIG. 3 is a graph of KQI versus KPI provided in an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of another KQI versus KPI graph provided by embodiments of the present disclosure;

fig. 5 is a schematic flow chart of determining KQI data corresponding to inflection points of a corresponding KQI-KPI relationship curve for a type of KPI according to an embodiment of the disclosure;

fig. 6 is a schematic flow chart of determining KQI data corresponding to inflection points of a relationship curve between KQI and KPI according to a jitter point, a continuous jitter frequency, and a cell number duty ratio parameter according to an embodiment of the present disclosure;

fig. 7 is a schematic flow chart of determining n types of KPIs with highest association degrees between KPIs and KQI according to an embodiment of the disclosure;

FIG. 8 is a flow chart of determining KPI feature combinations from wireless KPI data provided by embodiments of the present disclosure;

9 a-9 b are schematic diagrams of importance levels of various types of KPIs provided in embodiments of the present disclosure;

FIG. 10 is a schematic flow chart of calculating an average anomaly accuracy rate according to an embodiment of the present disclosure;

fig. 11 is a flow chart of a service awareness identification method according to an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of a model training apparatus according to an embodiment of the present disclosure;

fig. 13 is a schematic structural diagram of a service awareness identifying device according to an embodiment of the present disclosure.

Detailed Description

Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Embodiments described herein may be described with reference to plan and/or cross-sectional views with the aid of idealized schematic diagrams of the present disclosure. Accordingly, the example illustrations may be modified in accordance with manufacturing techniques and/or tolerances. Thus, the embodiments are not limited to the embodiments shown in the drawings, but include modifications of the configuration formed based on the manufacturing process. Thus, the regions illustrated in the figures have schematic properties and the shapes of the regions illustrated in the figures illustrate the particular shapes of the regions of the elements, but are not intended to be limiting.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The wireless network performance index is not the only factor affecting the user perception, i.e. although the wireless network performance index may cause poor service perception, the poor service perception is not necessarily caused by wireless reasons, and problems in aspects of core network, content provider, transmission, user, terminal, etc. may all bring about the problem of poor service perception. Therefore, the accurate service perception index cannot be obtained by direct reasoning through the pure wireless network performance index, and in practice, under many conditions, the service perception index obtained by the wireless network performance index reasoning has larger difference from the real service perception index. The embodiment of the disclosure mainly solves the problem of poor service perception quality caused by poor performance indexes of a wireless network.

In the wireless network optimization scheme of the related technology, wireless performance data cannot intuitively represent user perception. The problems commonly encountered in network operation and maintenance are: network KPIs (Key Performance Indication, key performance indicators) are very good, but user perception is not good. Therefore, although the related technology can find that the KPI is worse and improve the performance of the wireless network through wireless optimization, the service experience and the user perception are not obviously improved.

To solve the above problems, an embodiment of the present disclosure provides a model training method, as shown in fig. 1, including the following steps:

step 11, a training data set is obtained, wherein the training data set comprises wireless KPI data and KQI data corresponding to the wireless KPI data, and the wireless KPI data comprises various types of KPIs.

In this step, a set of wireless KPI history data at cell level self-busy hours and corresponding traffic perception KQI (Key Quality Indication, key quality index) data are collected as a data set D of the training module. The data set D may also contain time information, network element cell information, etc. The wireless KPI data includes multiple types of KPIs, for example, a cell uplink and downlink RLC (Radio Link Control, radio link layer control protocol) SDU (service data unit) average delay, a cell uplink and downlink (UE Throughput (kbps), a CQI (channel quality indication) goodness ratio, an uplink and downlink HARQ retransmission ratio, an uplink and downlink BLER (Block Error Rate), an uplink and downlink per PRB average Throughput, and the like.

Network performance index data (i.e., wireless KPI data) within a period of time and a certain network element range, and service quality index data (i.e., KQI data) corresponding to the period of time and the network element range are collected. A network element refers to a site, under which there may be multiple cells. The data used for training may be site-level data, cell-level data, or user-level data. If the wireless KPI data is site-level, then the KQI data corresponding thereto also needs to be site-level.

And step 12, calculating a KQI perception baseline according to the wireless KPI data and the KQI data.

In the step, through constructing a relation curve of a target KQI and a KPI, determining the correlation between each type of KPI and the target KQI, determining the critical point position of the target KQI at which sudden degradation or severe jitter occurs in the first n high-correlation wireless KPI data intervals, and obtaining a KQI perception baseline according to the critical point position of each type of KPI. The target KQI herein refers to a KQI type identified by a service awareness identification model, and one service awareness identification model is used to identify a type of KQI performance.

And 13, processing the KQI data according to the KQI perception base line to generate a KQI label which indicates whether the business perception is abnormal.

In this step, binarization processing is performed on the KQI data according to the perception baseline of the KQI, so as to generate a KQI label, where the KQI label is a training label of the dataset D and is used to identify whether the service perception represented by the KQI data corresponding to the current sample (i.e., wireless KPI data) is abnormal.

The binarization processing needs to refer to the degradation direction of the KQI data value, taking KQI as a TCP connection confirmation time delay as an example, and the degradation is more obvious as the value of the TCP connection confirmation time delay is larger, wherein the TCP connection confirmation time delay value larger than the KQI sensing base line is replaced by 1 to represent sensing abnormality; and replacing the TCP connection acknowledgement time delay value which is smaller than or equal to the KQI perception baseline with 0 to represent normal perception.

And 14, determining a KPI feature combination according to the wireless KPI data, wherein the KPI feature combination comprises a plurality of types of KPIs.

In the step, aiming at a target KQI, a KPI set is selected in a self-adaptive mode by taking the optimal test result as a criterion to serve as a characteristic combination, wherein the characteristic combination is a combination of various KPIs.

And step 15, screening wireless KPI data according to KPI feature combinations, and training a service perception recognition initial model by utilizing the screened wireless KPI data and a KQI label to obtain a service perception recognition model.

In this step, training is performed using KQI tags, and a fitting model is generated that fits the quality of service index data based on the wireless performance index data. And (3) screening the training set and the verification set according to the KPI feature combination selected in the step (14), training a service perception recognition initial model, wherein the service perception recognition initial model is a machine learning model M', and the trained model is an abnormal cell recognition model of a target KQI (for example, TCP connection confirmation average time delay).

It should be noted that, in the embodiment of the present disclosure, one service awareness identification model is trained for each type of KQI.

The above steps 11-15 complete the training of an abnormal cell identification model for a target KQI (i.e. a specific type), and the steps of training the service aware identification model are the same for other KQIs. Therefore, the model training process can be packaged, and the automatic training of the abnormal identification model of the target KQI can be completed only by inputting the acquired wireless KPI and KQI historical data. After model training is completed, KPI feature combinations and models can be stored and directly used for subsequent KQI prediction.

In some embodiments, after acquiring the training dataset (i.e., step 11) and before calculating the KQI perceived baseline from the wireless KPI data and the KQI data (i.e., step 12), the model training method may further include the steps of: and carrying out data preprocessing on the training data set. First, only the wireless KPI data and the KQI data of the target KQI (e.g., data of TCP average acknowledgement delay) are retained, and other irrelevant columns are deleted. And deleting the row containing the null value or illegal value in the data set D, and processing the KPI of the non-floating point type data, so as to ensure that all wireless KPI data are of a floating point type.

In some embodiments, as shown in fig. 2, the calculating a KQI perceived baseline (i.e., step 12) from wireless KPI data and the KQI data includes the steps of:

and step 21, respectively forming a KQI and KPI relation curve for each type of KPI.

In the step, wireless KPI data are divided into a plurality of intervals by taking the granularity predefined by KPI as a unit for each type of KPI, and the average value of KQI data is calculated in each interval to form a two-dimensional relationship curve of KQI and KPI.

As shown in fig. 3, taking KQI as a relationship curve of TCP connection acknowledgement average delay and KPI as uplink QPSK (Quadrature Phase Shift Keying ) coding ratio as an example, the abscissa is a value of uplink QPSK coding ratio after being divided by a predefined granularity, the ordinate on the right side is a value of TCP connection acknowledgement average delay, the ordinate on the left side is the number of cells, which can be obtained based on data set D statistics, and the histogram of fig. 3 is the number of cells corresponding to the value of uplink QPSK coding ratio. The curve in fig. 3 represents a two-dimensional relation curve of the average delay of the TCP connection confirmation and the uplink QPSK coding ratio, the black dots marked on the curve are KQI inflection points corresponding to the current KPI, and as can be seen from fig. 3, from the inflection points, the relation curve of KQI and KPI starts to generate severe jitter, and the number of corresponding cells is small.

As shown in fig. 4, taking as an example a relationship curve of KQI as the maximum number of activated UEs in a cell and KPI as the HTTP (Hyper Text Transfer Protocol ) download rate, the abscissa is the value of the maximum number of activated UEs in a cell (i.e. the number of users) divided by a predefined granularity, the ordinate on the right side is the value of the HTTP download rate, and the ordinate on the left side is the number of cells. The histogram of fig. 4 is the number of cells corresponding to the current HTTP download rate, and the curve is the relationship curve between the maximum activated UE number of the cells and the HTTP download rate. The black round dots marked on the curve are KQI inflection points corresponding to the current KPI, the corresponding KQI value is 4Mbps, and the relation curve of the KQI and the KPI is subjected to severe jitter from the inflection points, so that the corresponding number of cells is small.

Step 22, calculating the correlation coefficients of the KPI and the KQI for each type of KPI.

In this step, the correlation coefficient between the KPI of each type and the target KQI is calculated for each KPI of each type.

In some embodiments, the correlation coefficient may include: pearson correlation coefficient, spearman correlation coefficient, or Kendall correlation coefficient.

And step 23, determining n types of KPIs with the highest association degree between the KPIs and the KQIs according to the KQI and KPI relation curve and the correlation coefficient.

In this step, the correlation coefficients corresponding to the KPIs of each type calculated in step 22 are ranked from large to small, the association degree between the KPIs of each type and the target KQI is determined according to the ranking and the relationship curve obtained in step 21, and n KPIs with the highest association degree are selected, where n is a natural number greater than 2.

And step 24, respectively determining the KQI data corresponding to the inflection points of the corresponding KQI and KPI relation curves for the n types of KPIs.

In this step, for the n types of KPIs selected in step 23, the inflection point of the KQI-KPI relationship curve corresponding to each type of KPI is determined, and the KQI data of the inflection point position is determined.

And 25, calculating the mean value of the KQI data corresponding to each inflection point, and determining a KQI perception baseline according to the mean value of the KQI data.

In the step, calculating the average value of KQI data of inflection point positions of the relation curves corresponding to the n types of KPIs, wherein the average value is the KQI perception base line.

In some embodiments, the wireless KPI data and KQI data are cell-level data.

The process of determining the KQI data corresponding to the inflection point of the corresponding KQI-KPI relationship for one type of KPI is described below in conjunction with fig. 5. As shown in fig. 5, determining KQI data corresponding to inflection points of a corresponding KQI and KPI relationship for one type of KPI includes the steps of:

And step 51, according to the wireless performance degradation direction of the wireless KPI data value of the type KPI, sequencing the wireless KPI data to obtain a first sequence, and according to the first sequence, obtaining a second sequence of corresponding KQI data.

Different types of KPIs, the wireless performance degradation directions of the wireless KPI data values are different, if the wireless performance degradation is more obvious when the wireless KPI data value is larger, the wireless KPI data in the data set D is ordered from small to large to obtain a first sequence, and a second sequence { y of the KQI data in the data set D is determined according to the first sequence ₁ ,y ₂ ,...,y _m }。

Assuming a total of m samples, the current dataset D is { (x) ₁ ,y ₁ ),(x ₂ ,y ₂ ),...,(x _m ,y _m )}，x _i Representing the value of wireless KPI data in a sample i, y _i Representing the value of KQI data in a sample i, and comparing x according to the wireless performance degradation direction of the value of the wireless KPI data _i And sequencing to obtain a first sequence.

Step 52, calculating the difference between two adjacent KQI data in the second sequence to obtain a difference sequence.

In this step, the difference delta, delta between two adjacent KQI data in the second sequence is calculated ₁ ＝y ₂ -y ₁ ，Δ ₂ ＝y ₃ -y ₂ Similarly, a difference sequence diff_list= { Δ is generated ₁ ,Δ ₂ ,...,Δ _m-1 }. I.e. from y ₂ The absolute value of the difference between each KQI data and the previous KQI data is calculated to obtain the above-mentioned difference sequence.

And 53, traversing the difference sequence, determining jitter points of a KQI and KPI relation curve according to a preset threshold value, and recording continuous jitter times.

In some embodiments, the preset threshold value may be a mean value of the sequence of differences. In this step, the traversal starts from the first element in the difference sequence diff_list, compares whether the value (i.e., delta value) of each element in diff_list is greater than a threshold, and if so, considers that jitter is present here, and the continuous jitter count is incremented by 1. For example, delta ₃ ＝y ₄ -y ₃ If delta ₃ >Threshold value, then consider y ₄ Is the shaking point. It should be noted that if the continuous jitter frequency is less than the preset first threshold value, the jitter is no longer continuous, and the continuous jitter frequency is cleared.

And step 54, determining the KQI data corresponding to the inflection point of the KQI and KPI relation curve according to the continuous jitter times.

As shown in fig. 6, the determining KQI data corresponding to the inflection point of the KQI-KPI relationship curve according to the continuous jitter frequency (i.e. step 54) includes the following steps:

step 541, determining whether the number of continuous dithering is greater than a preset first threshold, if so, executing step 542, otherwise, executing step 53.

In some embodiments, the first threshold may be 5. In this step, if the number of continuous dithering is greater than the first threshold, then continuing to determine whether the cell number duty ratio parameter is less than a preset second threshold (i.e., executing step 542); if the number of continuous dithering is less than or equal to the first threshold, step 53 is executed, i.e. the difference sequence is continuously traversed, the dithering points of the relationship curve between KQI and KPI are determined according to the preset threshold, and the number of continuous dithering is recorded, so as to continuously find the inflection point.

Step 542, calculating the cell number duty cycle parameter.

The number of cells corresponding to the wireless KPI data may be obtained by statistics according to the collected data set D, so as to form a corresponding relationship (for example, the bar graphs in fig. 3 and fig. 4) between the wireless KPI data and the number of cells.

In this step, in response to the number of continuous jitters being greater than a preset first threshold, determining a stationary point immediately before a first jitter point among a plurality of jitter points for continuous jitters, the cell number ratio parameter being the stationary pointThe ratio of the number of cells accumulated after a point to the total number of cells. For example, if y ₄ For a first dither point of a plurality of dither points having a consecutive number of dithers greater than a first threshold, y ₃ I.e. the previous plateau thereof. Cell number duty cycle parameter=n1/N2, N2 is the total number of cells corresponding to the sampled wireless KPI data, and N1 is the sum of all the cell numbers after the stationary point. Taking fig. 3 as an example, a black point in a KQI and KPI relationship curve is a stable point, and the sum of the numbers of cells corresponding to the black frame range in the KQI and KPI relationship curve is N1, i.e. the number of cells corresponding to the black frame range in the histogram.

Step 543, judging whether the cell number duty ratio parameter is smaller than a preset second threshold, if yes, executing step 544; otherwise, step 545 is performed.

In step 544, the previous stationary point of the first jitter point is the inflection point of the relationship curve between KQI and KPI, and the KQI data corresponding to the inflection point is the KQI data corresponding to the previous stationary point of the first jitter point.

If the number of continuous dithering is greater than the first threshold and the cell number duty cycle parameter is less than the second threshold, the inflection point is found, and the inflection point is the stationary point before the first dithering point in the plurality of continuous dithering points, i.e., the stationary point determined in step 54, of the plurality of continuous dithering points, where the number of continuous dithering is greater than the first threshold.

In step 545, the number of consecutive jitters is zero.

If the number of continuous dithering is greater than the first threshold and the cell number duty ratio parameter is greater than or equal to the second threshold, indicating that no inflection point is found yet, then the number of continuous dithering is zero, and step 53 is executed, namely, the sequence of difference values is continuously traversed, the dithering point of the relationship curve between KQI and KPI is determined according to a preset threshold, and the number of continuous dithering is recorded, so as to continuously find the inflection point.

In some embodiments, as shown in fig. 7, the determining n types of KPIs with highest association degree between KPIs and KQI according to the relationship between KQI and KPI and the correlation coefficient (i.e. step 23) includes the following steps:

and 231, sequencing the correlation coefficients from large to small to obtain a correlation coefficient sequence.

In the step, the correlation coefficients corresponding to all the KPIs are ordered to obtain a correlation coefficient sequence.

And 232, determining the association degree of each type of KPI and KQI according to the correlation coefficient sequence and each KQI and KPI relation curve, and sequencing each association degree from high to low to obtain an association degree sequence.

In the step, the relevance scores of the KPIs and the target KQIs of all types are calculated respectively, all the relevance scores are ordered to obtain a relevance sequence, and in the relevance sequence, the relevance score is high in the front.

Step 233, determining KPIs corresponding to the first n relevancy values in the relevancy sequence.

In the step, the first n relevance scores in the relevance sequence are selected, and KPIs corresponding to the n relevance scores are determined.

In some embodiments, as shown in fig. 8, the determining KPI feature combinations from wireless KPI data and KQI data (i.e., step 14) includes the steps of:

and step 141, inputting the wireless KPI data into a first model to obtain the importance of each type of KPI, wherein the first model is generated according to the wireless KPI data and the KQI data in a training way, and the type of the first model is the same as that of the business perception initial model.

A machine learning model M (i.e., a first model), such as a LightGBM model (Light Gradient Boosting Machine, lightweight gradient lifting model), is pre-trained to obtain importance scores for all types of KPIs. 20% of the data from data set D was selected to be the validation set, and the other 80% of the data was used to train the first model. The first model M is trained based on the current training set and validation set using binary labels of all types of KPIs and target KQI (e.g., TCP connection acknowledgement delays).

In this step, importance scores are made for the KPIs of each type using the first model M. As presented in fig. 9a, the first model M scores the top 23 of the importance ranks of KPIs of each type, where the abscissa is the importance score and the ordinate is the column identity corresponding to KPIs of each type. Fig. 9b is a corresponding relationship between column identifiers corresponding to KPIs of various types and KPIs of various types.

It should be noted that, the types of the first model and the service awareness initial model may be any classifier, and by way of example, may be any one of the following: SVM classifier, random forest, decision tree.

Step 142, sorting the KPIs of each type from high to low according to the importance of the KPI of each type to obtain a first list.

In the step, the importance scores of all the types of KPIs are ranked from high to low according to the first model M, a first list L is generated, and the KPIs of all the types in the first list L are ranked from high to low according to the importance scores.

Step 143, a second list is generated according to the association sequence.

In the step, a plurality of KPIs with high relevance in the relevance sequence are selected, and a second list is generated.

Step 144, calculating average abnormal accuracy corresponding to each feature quantity according to the preset feature quantities, the first list and the second list.

In this step, a plurality of feature numbers are set as KPI number screening thresholds, and the first list L is traversed in turn according to the feature numbers, so as to find KPIs and the number of KPIs that optimize the training result of the first model.

And 145, determining KPI feature combinations according to the average abnormal accuracy corresponding to the feature numbers.

In the step, the KPI feature combination corresponding to the highest average abnormal accuracy is selected as the final KPI feature combination.

The process of calculating the average abnormality precision for one feature quantity will be described below with reference to fig. 10. As shown in fig. 10, the average anomaly accuracy is calculated for one feature quantity, including the steps of:

and step 10, acquiring KPIs of the front feature quantity in the first list, and acquiring a union set with the second list to obtain a first KPI feature combination.

In this step, if the current feature quantity is t, a feature combination of t before ranking is selected from the first list L and a union set is obtained from the second list P, so as to obtain a first feature combination F corresponding to the feature quantity t.

And step 20, calculating the average abnormal accuracy by adopting a cross-validation mode according to the first KPI feature combination.

In this step, the data set D is equally divided into 5 parts, and traversal is not repeated, 4 parts of the data set D are selected as training sets each time, and the remaining one part is used as verification set. Under the first feature combination F corresponding to the feature quantity t, training is performed by using the first model M in a cross-validation mode until each piece of wireless KPI data is used as a validated set. And finally, calculating the average abnormal accuracy rate of the cross verification, wherein the average abnormal accuracy rate is the average abnormal accuracy rate corresponding to the feature quantity t.

In some embodiments, the determining KPI feature combinations (i.e. step 145) according to the average anomaly accuracy rate for each feature quantity includes the steps of: and determining the maximum value in each average abnormal accuracy rate corresponding to each feature quantity, and determining a second KPI feature combination corresponding to the maximum value, wherein the KPI feature combination is the second KPI feature combination. That is, after each average abnormal accuracy corresponding to each feature number is calculated in steps 10-20, the maximum average abnormal accuracy is found out, and the KPI feature combination corresponding to the maximum average abnormal accuracy is used as the final KPI feature combination.

In order to clearly illustrate the technical solution of the embodiments of the present disclosure, the following describes the service awareness identification model training process in detail with reference to fig. 4 and a specific example. In this particular example, the target KQI is the HTTP download rate.

The first step, a group of cell-level self-busy hour wireless KPI historical data and corresponding HTTP download rate data are collected, and data alignment is carried out according to time and network elements to form a data set.

And secondly, data preprocessing is carried out on the data set. First, only the wireless KPI data and HTTP download rate are reserved, and other irrelevant columns are deleted. Second, rows in the dataset that contain null or illegal values are deleted. And the characteristics of non-floating point type data are processed, so that all wireless KPI data are ensured to be of a floating point type.

And thirdly, calculating a perception baseline of the HTTP download rate.

(1) The relevance of the KPIs and the HTTP download rate is calculated, wherein the relevance calculation mode can be a Spearman relevance coefficient calculation mode. The Spearman correlation coefficient is relatively stable and reliable compared with the Pearson correlation coefficient, and under the condition that abnormal values or long tail distribution exists in data, the Spearman correlation coefficient is calculated. (2) And carrying out KQI inflection point analysis on n KPIs with high correlation with the HTTP download rate. Fig. 4 presents a plot of the maximum number of active UEs (KPIs) in a cell versus the HTTP download rate (target KQI). Wherein the abscissa is the value of the maximum activated UE number (i.e. the number of users) of the cell after being divided at a predefined granularity, the ordinate on the right side is the value of the HTTP download rate, and the ordinate on the left side is the number of cells. The histogram of fig. 4 is the number of cells corresponding to the current HTTP download rate. The black dots marked on the relation curve are KQI inflection points corresponding to the current type KPI, the corresponding KQI value is 4Mbps, and the KQI value is severely dithered from the inflection points, so that the corresponding cell number is small. And finally, averaging the KQI inflection points of the n types of KPIs to obtain an HTTP download rate sensing base line.

And fourthly, performing binarization processing on the value of the HTTP download rate according to the HTTP download rate perceived baseline obtained by the calculation in the third step, and manufacturing a training label of the data set. Replacing the value superior to the HTTP download rate sensing baseline with 0 to represent normal sensing; values inferior to the HTTP download rate aware baseline are replaced with 1's representing perceptual anomalies.

Fifth, KPIs for training are adaptively selected. First, a machine learning model is pre-trained from existing datasets, substituted into all types of KPIs, where SVM classifiers may be employed. And obtaining importance scores of the machine learning model on all types of KPIs, and generating a first KPI list corresponding to the high score to the low score. Secondly, setting a plurality of feature quantity screening thresholds, and traversing the first list in sequence. If the current feature quantity is t, selecting a feature combination of the top t rank from the first list L, and taking a union set with a second list P to obtain a first feature combination F corresponding to the feature quantity t, wherein the second list P is generated according to a plurality of types of KPIs with high association degree in the association degree sequence. And similarly, calculating the cross-validation abnormal accuracy rate corresponding to the first feature combination F under each feature quantity screening threshold, and selecting a second feature combination with the highest accuracy rate as a KPI feature combination finally used for training a model.

And sixthly, training an abnormal cell identification model of the HTTP download rate by using an SVM classifier according to the KPI feature combination obtained in the fifth step.

The above steps complete the training of an abnormal cell identification model (i.e., a traffic aware identification model) for the HTTP download rate indicator. After the KPI feature combination and the model training result are stored, only the wireless side KPI data of the appointed network element and the cell is required to be input in the follow-up, and whether the HTTP download rate index of the cell is abnormal or not can be identified through the service perception identification model.

According to the embodiment of the disclosure, under a pure wireless scene that the end-to-end service perception index cannot be acquired, user perception is difficult to visually represent through wireless performance data, so that network optimization personnel spend a great deal of time and effort to optimize wireless network performance, but service experience and user perception are not obviously improved. In the embodiment of the disclosure, the training and prediction processes of the service perception recognition model do not need experience intervention of an expert in the wireless communication field or an expert in the artificial intelligence field, so that manpower can be effectively saved, and great efficiency is improved for subsequent network optimization work.

The embodiment of the disclosure also provides a service perception identification method, as shown in fig. 11, which comprises the following steps:

Step 110, first wireless KPI data is acquired.

The first wireless KPI data is cell level data.

And 120, screening the first wireless KPI data according to the KPI feature combination to obtain second wireless KPI data.

And 130, inputting the second wireless KPI data into a service perception identification model to obtain a KQI label for indicating whether the service perception is abnormal. The business perception recognition model is obtained by training the model training method.

After the first wireless KPI data of the designated network element and the designated cell are input into a fitting model (namely a service perception recognition model), outputting a recognition result of whether a target KQI of the network element is better than a KQI perception baseline, and if the degradation direction of the KQI data is from small to large and the value of the KQI data is greater than the KQI perception baseline, the KQI label output through the fitting model is 1, so that the service experience corresponding to the target KQI is poor, namely the service perception is poor; if the degradation direction of the KQI data is from small to large and the value of the KQI data is smaller than or equal to the KQI perception baseline, the KQI label output by the fitting model is 0, which indicates that the service experience corresponding to the target KQI is better, namely the service perception is better.

The service perception identification method of the embodiment of the disclosure comprises the following steps: acquiring first wireless KPI data of a designated network element and a designated cell; screening the first wireless KPI data according to the KPI feature combination to obtain second wireless KPI data; and inputting the second wireless KPI data into a service perception recognition model to obtain a KQI label for indicating whether the service perception is abnormal. According to the embodiment of the disclosure, by training the wireless performance data and the service perception data, the service perception identification model capable of identifying whether the service perception index is abnormal is fitted, so that the situation of poor service perception quality caused by the problem of poor wireless performance can be rapidly and real-timely identified by using the service perception identification model in a pure wireless scene, and the working efficiency of network optimization personnel is greatly improved.

Based on the same technical concept, the embodiment of the disclosure further provides a model training device, as shown in fig. 12, where the model training device includes an acquisition module 101, a KQI sensing baseline calculation module 102, a KQI label generation module 103, a KPI feature combination determination module 104, and a model training module 105, where the acquisition module 101 is configured to acquire a training data set, where the training data set includes wireless key performance index KPI data and key quality index KQI data corresponding to the wireless KPI data, and the wireless KPI data includes multiple types of KPIs.

The KQI sensing baseline calculation module 102 is configured to calculate a KQI sensing baseline according to the wireless KPI data and the KQI data.

The KQI tag generation module 103 is configured to process the KQI data according to the KQI sensing baseline, and generate a KQI tag that indicates whether service sensing is abnormal.

The KPI feature combination determination module 104 is configured to determine a KPI feature combination from the wireless KPI data, the KPI feature combination including multiple types of KPIs.

The model training module 105 is configured to screen the wireless KPI data according to the KPI feature combination, and train a service perception recognition initial model by using the screened wireless KPI data and the KQI label to obtain a service perception recognition model.

In some embodiments, the KQI sensing baseline calculation module 102 is configured to form a KQI-KPI relationship curve for each type of KPI; calculating the correlation coefficients of KPI and KQI for each type of KPI; according to the KQI-KPI relation curve and the correlation coefficient, determining n types of KPIs with the highest association degree between the KPIs and the KQI; determining KQI data corresponding to inflection points of corresponding KQI and KPI relation curves respectively for the n types of KPIs; and calculating the average value of the KQI data corresponding to each inflection point, and determining a KQI perception baseline according to the average value of the KQI data.

In some embodiments, the wireless KPI data and the KQI data are cell-level data. The KQI perceived baseline calculation module 102 is configured to determine KQI data corresponding to inflection points of a corresponding KQI and KPI relationship curve for one type of KPI in the following manner: according to the wireless performance degradation direction of the wireless KPI data value of the type KPI, sequencing the wireless KPI data to obtain a first sequence, and obtaining a second sequence of corresponding KQI data according to the first sequence; calculating the difference value of two adjacent KQI data in the second sequence to obtain a difference value sequence; traversing the difference sequence, determining jitter points of the KQI and KPI relation curve according to a preset threshold value, and recording continuous jitter times; and determining the KQI data corresponding to the inflection point of the KQI and KPI relation curve according to the continuous shaking times.

In some embodiments, the KQI-aware baseline calculation module 102 is configured to, in response to the number of continuous jitters being greater than a preset first threshold and the number of cells being less than a preset second threshold, make a previous stationary point of the first jitter point an inflection point of the KQI-KPI relationship curve, and make KQI data corresponding to the inflection point a KQI data corresponding to the previous stationary point of the first jitter point.

In some embodiments, KQI aware baseline calculation module 102 is further configured to, in response to the number of consecutive jitters being greater than the first threshold and the cell number duty cycle parameter being greater than or equal to the second threshold, zero the number of consecutive jitters and continue traversing the sequence of differences.

In some embodiments, the KQI perceptual baseline calculation module 102 is configured to sort the correlation coefficients from large to small to obtain a correlation coefficient sequence; determining the association degree of each type of KPI and KQI according to the correlation coefficient sequence and each KQI and KPI relation curve, and sequencing each association degree from high to low to obtain an association degree sequence; and determining KPIs corresponding to the first n relevancy levels in the relevancy level sequence.

In some embodiments, the KPI feature combination determination module 104 is configured to input the wireless KPI data into a first model trained in advance, to obtain importance of KPIs of each type, where the first model is the same type as the service awareness initial model; sorting all types of KPIs from high to low according to the importance of all types of KPIs to obtain a first list; generating a second list according to the association sequence; according to a plurality of preset feature quantities, the first list and the second list, respectively calculating average abnormal accuracy corresponding to each feature quantity; and determining KPI feature combinations according to the average abnormal accuracy corresponding to the feature numbers.

In some embodiments, the KPI feature combination determination module 104 is configured to calculate, for a feature quantity, an average anomaly accuracy rate by: acquiring KPIs of the feature quantity in the first list, and acquiring a union set with the second list to obtain a first KPI feature combination; and calculating the average abnormal accuracy rate by adopting a cross-validation mode according to the first KPI feature combination.

In some embodiments, the KPI feature combination determination module 104 is configured to determine a maximum value in each of the average abnormal precision rates corresponding to each of the feature amounts, and determine a second KPI feature combination corresponding to the maximum value, where the KPI feature combination is the second KPI feature combination.

Based on the same concept, the embodiment of the present disclosure further provides a service awareness identifying device, as shown in fig. 13, where the service awareness identifying device includes an obtaining module 201, a screening module 202, and an identifying module 203, where the obtaining module 201 is configured to obtain first wireless KPI data of a specified network element and a specified cell.

The screening module 202 is configured to screen the first wireless KPI data according to KPI feature combinations, to obtain second wireless KPI data.

The identification module 203 is configured to input the second wireless KPI data into a service perception identification model, to obtain a KQI label for indicating whether service perception is abnormal; the business perception recognition model is obtained by training the model training method.

The model training device of the embodiment of the disclosure can be deployed in a network in which DPI is deployed and service perception data can be obtained regularly, so that training data can be updated regularly and enriched, and the accuracy and universality of the model are continuously improved. The service perception recognition device of the embodiment of the disclosure can be deployed in a pure wireless scene in which end-to-end service perception indexes cannot be acquired.

The disclosed embodiments also provide a computer device comprising: one or more processors and a storage device; the storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the model training method or the service perception recognition method provided in the foregoing embodiments.

The disclosed embodiments also provide a computer readable medium having a computer program stored thereon, wherein the computer program when executed implements the model training method or the business-aware identification method as provided by the foregoing embodiments.

Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, functional modules/units in the apparatus disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purpose of limitation. In some instances, it will be apparent to one skilled in the art that features, characteristics, and/or elements described in connection with a particular embodiment may be used alone or in combination with other embodiments unless explicitly stated otherwise. It will therefore be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the scope of the present invention as set forth in the following claims.

Claims

1. A method of model training, the method comprising:

2. The method of claim 1, wherein said calculating a KQI perceived baseline from said wireless KPI data and said KQI data comprises:

forming a KQI and KPI relation curve respectively aiming at each type of KPI;

calculating the correlation coefficients of KPI and KQI for each type of KPI;

according to the KQI-KPI relation curve and the correlation coefficient, determining n types of KPIs with the highest association degree between the KPIs and the KQI;

determining KQI data corresponding to inflection points of corresponding KQI and KPI relation curves respectively for the n types of KPIs;

and calculating the average value of the KQI data corresponding to each inflection point, and determining a KQI perception baseline according to the average value of the KQI data.

3. The method of claim 2, wherein KQI data corresponding to inflection points of a KPI relationship for a respective KQI is determined for one type of KPI by:

According to the wireless performance degradation direction of the wireless KPI data value of the type KPI, sequencing the wireless KPI data to obtain a first sequence, and obtaining a second sequence of corresponding KQI data according to the first sequence;

calculating the difference value of two adjacent KQI data in the second sequence to obtain a difference value sequence;

traversing the difference sequence, determining jitter points of the KQI and KPI relation curve according to a preset threshold value, and recording continuous jitter times;

and determining the KQI data corresponding to the inflection point of the KQI and KPI relation curve according to the continuous shaking times.

4. The method of claim 3, wherein said determining KQI data corresponding to inflection points of the KQI and KPI relationship from the continuous jitter times comprises:

and responding to the continuous shaking times being larger than a preset first threshold value, wherein the cell quantity ratio parameter is smaller than a preset second threshold value, the previous stable point of the first shaking point is the inflection point of the KQI-KPI relation curve, and the KQI data corresponding to the inflection point is the KQI data corresponding to the previous stable point of the first shaking point.

5. The method of claim 4, wherein said determining KQI data corresponding to an inflection point of the KQI and KPI relationship from the continuous jitter times further comprises:

And responding to the continuous dithering frequency being larger than the first threshold value and the cell quantity duty ratio parameter being larger than or equal to the second threshold value, requesting the continuous dithering frequency to be zero, and continuing traversing the difference value sequence.

6. The method of claim 2, wherein said determining n types of KPIs with highest association of KPIs with KQI based on said KQI to KPI relationship and said correlation coefficient comprises:

sequencing the correlation coefficients from large to small to obtain a correlation coefficient sequence;

determining the association degree of each type of KPI and KQI according to the correlation coefficient sequence and each KQI and KPI relation curve, and sequencing each association degree from high to low to obtain an association degree sequence;

and determining KPIs corresponding to the first n relevancy levels in the relevancy level sequence.

7. The method of claim 6, wherein the determining KPI feature combinations from the wireless KPI data and the KQI data comprises:

inputting the wireless KPI data into a first model to obtain importance of each type of KPI, wherein the first model is generated according to the wireless KPI data and the KQI data in a training way and has the same type as the business perception initial model;

Sorting all types of KPIs from high to low according to the importance of all types of KPIs to obtain a first list;

generating a second list according to the association sequence;

according to a plurality of preset feature quantities, the first list and the second list, respectively calculating average abnormal accuracy corresponding to each feature quantity;

and determining KPI feature combinations according to the average abnormal accuracy corresponding to the feature numbers.

8. The method of claim 7, wherein for a feature quantity, the average anomaly accuracy is calculated by:

acquiring KPIs of the feature quantity in the first list, and acquiring a union set with the second list to obtain a first KPI feature combination;

and calculating the average abnormal accuracy rate by adopting a cross-validation mode according to the first KPI feature combination.

9. The method of claim 8, wherein said determining KPI feature combinations from the average anomaly accuracy rates for each of the feature quantities comprises:

and determining the maximum value in the average abnormal accuracy rate corresponding to the feature quantity, and determining a second KPI feature combination corresponding to the maximum value, wherein the KPI feature combination is the second KPI feature combination.

10. A business awareness identification method, comprising:

acquiring first wireless KPI data;

inputting the second wireless KPI data into a business perception recognition model to obtain a KQI label for representing whether business perception is abnormal; the business perception recognition model is trained by the model training method according to any one of claims 1-9.

11. The model training device is characterized by comprising an acquisition module, a KQI perception baseline calculation module, a KQI label generation module, a KPI feature combination determination module and a model training module,

The KPI feature combination determining module is used for determining KPI feature combinations according to the wireless KPI data and the KQI data, wherein the KPI feature combinations comprise various types of KPIs;

12. A service perception recognition device is characterized by comprising an acquisition module, a screening module and a recognition module,

the identification module is used for inputting the second wireless KPI data into a service perception identification model to obtain a KQI label used for indicating whether service perception is abnormal; the business perception recognition model is trained by the model training method according to any one of claims 1-9.

13. A computer device, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the model training method of any of claims 1-9, or the business-aware identification method of claim 10.

14. A computer readable medium having stored thereon a computer program, wherein the program when executed implements the model training method of any of claims 1-9 or the business awareness identification method of claim 10.