RU2780710C1 - Method, system, and apparatus for processing sample data - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to the field of data processing. The described result is achieved by receiving alarm data, fault data, and configuration data associated with at least one target network for multiple times from the database and, accordingly, converting said data into the corresponding vectors, and constructing a multi-level multidimensional space for at least one target network, containing an alarm data level, a fault data level, and a configuration data level of each target network, wherein each level constitutes a two-dimensional matrix of the same size and is used to store the vectors resulting from the conversion in rows in accordance with the sequence of times.

EFFECT: possibility of processing sample data capable of converting configuration data, alarm data, and fault data, increasing the efficiency of storing sample data and ensuring the achievement of inter-domain interaction of sample data and exchange thereof between different manufacturers.

6 cl, 13 dwg, 1 tbl

Description

Technical field

The present invention relates to the field of communication, in particular to a method, system and apparatus for processing sample data.

State of the art

If the current performance of the optical network equipment exceeds the limit, or if some potential performance degradation occurs, a series of alarm data is generated and transmitted to the network management platform. In the event of a failure of the optical network equipment, both alarm data and fault data are generated and transmitted. At present, the operation and maintenance specialists are locating the fault and implementing the troubleshooting strategy by analyzing the alarm and fault data through the administration platform and the management platform, and if necessary, activate the protection switching to ensure the normal operation of the optical network.

With the continuous growth in the scale of optical networks and the increase in the number of optical network equipment, the amount of alarm data and fault data generated by optical networks is growing. Locating and troubleshooting network problems is becoming more and more complex and time consuming. The traditional fault handling mode is fraught with enormous challenges and is difficult to meet current needs. In particular, with the rapid development of communication services and the continuous development and change of communication technologies, the general trend is to move from the traditional hard connection and hard network architecture to the soft connection and flexible cloud-based network architecture. The lower layer of the cloud network implements data forwarding through the optical network equipment, and the middle and upper layers implement the administration and management of resources and services through the management platform, administrative platform, and orchestration platform. The operation and maintenance process of the system is more complex, which is necessary to realize the network data federation view and efficiently extract data operations and calculations, so as to solve the problem of the difficulty of quickly recovering the cloud network after a failure occurs.

The use of artificial intelligence technology to analyze and eliminate network faults is an effective solution to these problems. However, on the one hand, all of the existing alarm data, fault data, and configuration data targeted at O&M professionals contain different types of fields, such as alarm start time, network element type, and number of central processor cores (central processing unit, CPU). These heterogeneous data cannot be directly used as machine learning focused sample data; on the other hand, due to the huge amount of network data, differences between different types of networks, and so on, the required sample data cannot be flexibly extracted according to the needs of machine learning. Therefore, there is an urgent need for flexible and efficient ways to process sample data in order to meet the testing and analysis needs of a machine learning model.

Disclosure of the essence of the invention

The object of the present invention is to provide a sample data processing method, system and apparatus that converts configuration data, alarm data and fault data oriented to operations and maintenance specialists into machine learning oriented sample data, improves would be the efficiency of storing sample data and would ensure the achievement of cross-domain interaction of sample data and their exchange between different producers.

In a first aspect, in an embodiment of the present invention, a method for processing sample data is provided, comprising the steps of:

receiving from the database alarm data, fault data and configuration data of at least one target network for a plurality of time points and accordingly convert them into the corresponding vectors; and

building for at least one target network a multi-level multi-dimensional space containing an alarm data level, a fault data level and a configuration data level of each target network, each level is a two-dimensional matrix having the same size, and is configured to save vectors obtained as a result of the transformation, in rows according to the sequence of time points.

In combination with the first aspect, in the first alternative embodiment, the method further includes extracting the machine learning model input vector from the alarm data layer and the fault data layer, and extracting the machine learning model output vector from the configuration data layer.

In the first alternative embodiment of the first aspect, sequentially performing mode expansion and string vector expansion on the alarm data layer and the fault data layer to obtain an input vector; and successively performing mode expansion and string vector expansion at the configuration data level to obtain an output vector.

In a second alternative embodiment according to the first aspect, a subspace containing target data is extracted from a multi-level multi-dimensional space, said sub-space containing at least one sub-matrix of an alarm data level, a fault data level, and a configuration data level; and

sequentially performing mode expansion and string vector expansion on the specified subspace to obtain an input vector and/or an output vector.

In an optional embodiment, the specified subspace is obtained after segmenting the multilevel multidimensional space by means of a subspace segmentation operator;

performing a string vector expansion on the mode expansion matrix by means of the first mode expansion operator to obtain an input vector; and/or performing a string vector expansion on the mode expansion matrix via the second mode expansion operator to obtain an output vector.

In combination with the third alternative embodiment of the first aspect, storing the vectors resulting from the transformation in the rows of a two-dimensional empty matrix in accordance with the sequence of specified times so as to obtain an alarm matrix, a fault matrix and a configuration matrix of each respective target network;

respectively storing the alarm matrix, the fault matrix, and the configuration matrix in the alarm data layer, the fault data layer, and the configuration data layer of the multi-level multi-dimensional space, and filling the matrix elements having no stored values at each level with zero values.

In combination with the fourth alternative, the vector resulting from the transformation is the Hadamard product of the main vector and the dimensional extension vector, with each element of the main vector representing the value of a field in the alarm data, fault data, or configuration data at some point in time. , and each element of the dimensional expansion vector is the result of a multiple expansion or contraction of the corresponding element of the main vector.

In a second aspect, as an embodiment of the present invention, there is provided a sample data processing system comprising:

a conversion module that is configured to receive alarm data, fault data, and configuration data of at least one target network over a plurality of time points from the database and convert them appropriately into appropriate vectors;

a space building module that is configured to build, for at least one target network, a multi-level multi-dimensional space containing an alarm data layer, a fault data layer, and a configuration data layer of each target network, each layer being a two-dimensional matrix having one and the same size, and is configured to store the vectors obtained as a result of the transformation in rows in accordance with the sequence of time points.

In combination with the second aspect, in the first alternative embodiment, the system further comprises:

an extractor that is configured to extract the input vector of the machine learning model from the alarm data layer and the fault data layer, and extract the output vector of the machine learning model from the configuration data layer.

In the first alternative embodiment according to the second aspect, the extraction module comprises;

a sub-matrix processing unit that is configured to retrieve a sub-space containing target data from a multi-level multi-dimensional space, said sub-space containing at least one sub-matrix of an alarm data level, a fault data level, and a configuration data level;

a matrix calculation unit that is configured to sequentially perform mode expansion and string vector expansion at the alarm data layer and the fault data layer to obtain an input vector; sequentially performing mode expansion and string vector expansion at the configuration data level to obtain an output vector; and sequentially performing mode expansion and string vector expansion on the specified subspace to obtain an input vector and/or output vector.

In combination with the second aspect, in a second alternative embodiment, the build module comprises:

a matrix pre-processing unit that is configured to store the vectors obtained as a result of the transformation with the same data source in the rows of a two-dimensional empty matrix in accordance with a specified sequence of time points so as to obtain an alarm matrix, a fault matrix and a configuration matrix of each target network respectively;

a matrix building unit that is configured to store an alarm matrix, a fault matrix, and a configuration matrix, respectively, at the alarm data level, the fault data level, and the configuration data level of the multi-level multi-dimensional space and fill the matrix elements that do not have stored values at each level with zeros values.

In a third aspect, an embodiment of the present invention provides a network fault processing apparatus based on a machine learning model, comprising a fault diagnosis system and a sample data processing system described in the second aspect;

said troubleshooting system comprises a model building module and an I/O control module; the model building module is configured to obtain an input vector and an output vector from the processing system for building a machine learning model, wherein the input vector is extracted from the alarm data layer and the fault data layer, and the output vector is extracted from the configuration data layer; and

the input/output control module is configured to receive alarm data and fault data of the target network, convert them appropriately into appropriate vectors, input these vectors to the model building module, and transmit the output vector of the model building module to the target network.

Compared with the prior art, in an embodiment of the present invention, alarm data, fault data, and configuration data of at least one target network at a plurality of time points are obtained from a database, respectively converted into corresponding vectors, and a layered multi-dimensional space is constructed for of at least one target network, thereby providing sample data for a machine learning model for processing optical network faults and converting configuration data, alarm data, and fault data oriented to operations and maintenance specialists into sample data oriented to machine learning , improve the efficiency of saving sample data and ensure the achievement of cross-domain interaction and data exchange between different producers.

Description of drawings

In order to better illustrate the technical solution in the embodiments of the present invention, the following schematically presents the drawings necessary to describe the embodiments, and it will be obvious to those skilled in the art that the drawings in the following description relate only to some embodiments of the present invention, and based on these drawings other drawings can also be obtained without any creative work.

On FIG. 1 is a schematic drawing of a cloud network architecture;

in FIG. 2 is a flowchart of a sample data processing method in an embodiment of the present invention;

in FIG. 3 is a schematic diagram of a process for getting data from a database and performing vectorization and matrixing on it;

in FIG. 4 is a flowchart of a method for processing sample data in yet another embodiment of the present invention;

in FIG. 5 shows an embodiment of a multi-level multi-dimensional space;

in FIG. 6 is a schematic diagram of performing mode expansion and string vector expansion on a multi-level multi-dimensional space according to a sample data processing method in yet another embodiment of the present invention;

in FIG. 7 is a flowchart showing the execution of mode expansion and string vector expansion on a subspace of a layered multidimensional space according to a sample data processing method in yet another embodiment of the present invention;

in FIG. 8 is a schematic drawing of embodiments of subspace segmentation in a multi-level multi-dimensional space;

in FIG. 9 is a schematic drawing of yet another embodiment of subspace segmentation on a multi-level multi-dimensional space;

in FIG. 10 is a schematic drawing of performing mode expansion and string vector expansion on subspace;

in FIG. 11 is a block diagram of mode deployment and sub-space segmentation of a multi-level multi-dimensional space;

in FIG. 12 is a schematic drawing of a sample data processing system in an embodiment of the present invention; and

in FIG. 13 is a schematic drawing of a network fault processing apparatus in an embodiment of the present invention.

Implementation of the invention

The technical solutions of the present invention will be clearly and fully described with reference to the accompanying drawings of the embodiments of the present invention, and it is clear that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments of the present invention, obtained by those skilled in the art based on the above embodiments without the use of creative work, should fall within the protection scope of the present invention.

An embodiment of the present invention provides a sample data processing method that provides sample data for a machine learning model for optical network fault processing, converts configuration data, alarm data, and optical network fault data targeting O&M personnel into sample data. machine learning-oriented data, and thus improve the efficiency of sample data storage, facilitate sample data extraction and comparison, and other operations, and facilitate efficient machine learning model building.

The optical networks referred to in the embodiments of the present invention may be not only conventional optical data networks such as optical transport networks (optical transport network, OTN), packet transport networks (packet transport network, PTN) and packet optical transport networks ( packet optical transport network, POTN), but they can also be cloud networks.

As an example, in FIG. 1 is a schematic drawing of a cloud network architecture. At the bottom left of Fig. 1 shows a cloud network base station containing an active antenna unit (AAU), a centralized unit (CU), and a distributed unit (DU). Of these, the CU supports the non-real-time high-level wireless protocol and part of the core network data sink functions and edge equipment functions, and the DU supports the physical layer functions and real-time functions. At the bottom of Fig. 1 shows the cloud network access ring, convergence ring, and main ring. The alarm data, fault data, and configuration data of the network equipment in these ring networks are respectively transmitted to the border data center, the regional data center, and the main data center at the top of FIG. 1 through a network administration platform or a management platform. The alarm data, fault data and configuration data of the base station and border equipment are transmitted to the border data center through the local platform. The 5G core network functions are divided into user plane (UP) functions and control plane (CP) functions. On the one hand, these data collection and processing centers perform the functions of administration, routing and management of the cloud network; on the other hand, these data centers deploy an intelligent cloud network platform, build a knowledge base of cloud network operation and maintenance management based on large amounts of network data and high computing power, and serve as the brain of the cloud network.

In an embodiment of the present invention, each target network may be an access ring, a convergence ring, or the main ring of FIG. 1, however, it is not limited to this. In other embodiments, each target network may also be an access network, a convergence network, or a core network in conventional optical networks (OTN, PTN, or POTN).

In an embodiment of the present invention, the optical network equipment transmits alarm data and corresponding fault data to the network administration platform, and also transmits them to the data collection and processing center through the network administration platform. The alarm signaling generated by the optical network equipment includes root cause signaling and derived signaling, and there is a correlation between root cause signaling and derived signaling. In the event of a failure of the optical network equipment, both alarm data and fault data are generated and transmitted, and the fault must be resolved using distributed configuration data.

Since the optical network alarm dataset contains a large amount of duplicate, redundant, incomplete, and inconsistent data, the data center first cleans the data, removes duplicate, redundant, and low-quality data to obtain high-quality alarm data sets, fault data sets and configuration data sets and stores these data sets in the database accordingly.

On FIG. 2 shows a flowchart of a sample data processing method in an embodiment of the present invention, including the following steps:

S110: Receive alarm data, fault data and configuration data of at least one target network for a plurality of time points from the database and convert them into corresponding vectors accordingly.

S120: for at least the target network, a multi-level multi-dimensional space is built, each level of which is a two-dimensional matrix having the same size, and is configured to store the vectors obtained as a result of the transformation in rows in accordance with the sequence of time points so, to respectively obtain the alarm data level, fault data level and configuration data level of each target network.

In step S110, based on the correlation between the time generated using the alarm data, fault data, and configuration data, all of the alarm data, fault data, and configuration data can be obtained within a set time period from the database, or all of alarm data, fault data and configuration data can be retrieved from the database periodically on a daily, weekly or monthly basis. The set period of time or cycle comprises alarm data for a plurality of time points, fault data for a plurality of time points, and configuration data for a plurality of time points.

Alarm data, fault data, and configuration data are not only heterogeneous data, and these data include fields of various types, with different fields having different sizes. The method of vector representation of heterogeneous data with different sizes includes the following steps:

, где каждый элемент основного вектора

представляет собой значение поля в каждом блоке данных аварийной сигнализации, данных о неисправностях или конфигурационных данных.First, each piece of alarm data, fault data, and configuration data is converted into a main vector

, where each element of the main vector

represents the field value in each block of alarm data, fault data, or configuration data.

For example, the sample data set consisting of all received alarm data contains M _a blocks of alarm data, of which the alarm data generated at a point in time may be one or more blocks, and each block of alarm data contains N _a fields.

For example, the alarm data block shown in FIG. 3 contains eight fields, namely: sequence number (Seq.No.), address (Addr.), line number (Line), alarm type (AlarmType), alarm start time (BeginTime), alarm end time ( EndTime), board type (BoardType), and NE type (NetType) of the alarm data, with BeginTime and EndTime specified in seconds, AlarmType is a string of characters, and NetType is an integer value.

The meanings of all alarm data fields shown in FIG. 3 are converted to real numbers and thus expressed as elements of a vector. During the vectorization of the alarm data, the integer values of these fields are represented in the vector as element values. The minimum value of all the BeginTime and EndTime fields may be mapped to a value of 1, and the number of seconds between other times and the minimum time may be added to the value of 1 to obtain the respective values of the BeginTime and EndTime fields, respectively. For example, if BeginTime is 10 seconds longer than the minimum time, then BeginTime corresponds to the value 11, in which case these two fields are arranged in lexicographic order and then numbered starting from 1 to convert the string to values as vector elements.

выполняют преобразование по размерам, и полученный в результате преобразования вектор

будет представлять собой адамарово произведение основного вектора

и вектора

размерного расширения, то есть

. Элемент вектора

размерного расширения представляет собой результат кратного расширения или сжатия соответствующего элемента основного вектора

; например, при расширении десятичной кратной единицы ширины полосы с мега (М) до гига (G) элемент вектора

размерного расширения будет равен 1024.Then on the main vector

perform a size transformation, and the resulting vector

will be the Hadamard product of the main vector

and vector

dimensional expansion, that is

. vector element

dimensional expansion is the result of a multiple expansion or contraction of the corresponding element of the main vector

; for example, when expanding a decimal multiple of a unit of bandwidth from mega (M) to giga (G), the vector element

dimensional expansion will be equal to 1024.

According to the training requirements of the machine learning model, the corresponding element of the main vector can be multiplied by the corresponding element of the dimensional expansion vector to generate sample data suitable for the training requirements. Similarly, the configuration data at the bottom left of FIG. 3 are also converted to the corresponding vectors. The configuration data contains Num_CPUs: 4, which is the number of CPU cores. For example, the vector group at the bottom of FIG. 3 shows two vectors that are converted using alarm data and configuration data, respectively.

In the case of fault data and configuration data stored in a partially structured XML document in an optical network, the above method can also be used to construct a database vector and a dimensional extension vector, with the number of key/value pairs (Key/Value) in the XML corresponding to the size vector, and the value of the vector element corresponds to the value in the XML document.

In step S110, three vector group pairs are built for the target network, which are respectively an alarm data vector group and a dimensional extension vector group; a fault data vector group and a dimensional expansion vector group; and a configuration data vector group and a dimension extension vector group. The resulting alarm data vector group contains M _a alarm data vectors resulting from conversion from M _a alarm data blocks, and each alarm data vector has N _a elements. The fault data vector group contains M _f fault data vectors resulting from the conversion from the M _f fault data blocks, and each fault data vector has N _f elements. The configuration data vector group contains M _c configuration data vectors resulting from the transformation from M _c configuration data blocks, and each configuration data vector has N _c elements.

After performing vector representation on heterogeneous data with different sizes, in step S120, matrix representation is performed. The number of rows of the two-dimensional matrix in the multi-level multi-dimensional space is the maximum of the total number of vectors of the alarm data vector group, the fault data vector group, and the configuration data vector group, that is, the maximum value of M _a , M _f and M _c . The number of columns of the two-dimensional matrix in the multi-level multi-dimensional space is the maximum of the total number of vector elements of the vector alarm data group, the vector fault data group, and the vector configuration data group, that is, the maximum value of N _a , N _f and N _c .

According to the time sequence M _a of the alarm data vectors, the groups of alarm data vectors are stored in rows in the first layer of the multi-level multidimensional space to obtain the alarm data layer. Similarly, a fault data layer and a configuration data layer can be obtained.

The multilevel multidimensional space is represented as D=R(I ₁ , I ₂ , I ₃ ), where R represents the value of the multilevel multidimensional space in the field of real numbers, I ₁ represents the number of levels, I ₂ represents the number of rows of the level matrix, and I ₃ represents the number columns of the level matrix. For example, R(I ₁ , :, :) represents the matrix element at level I ₁ , R(I ₁ , I ₂ , :) represents all elements of the vectors at level I ₁ and in row I ₂ , R(2, :, : ) represents the matrix element at level 2, and R(2, 3, :) represents all elements of the vectors at level 2 and in row 3.

In an embodiment of the present invention, the number of levels, rows and columns of a multi-level multi-dimensional space is determined, that is, the values of I ₁ , I ₂ and I ₃ in D=R(I ₁ , I ₂ , I ₃ ) and the location of matrices, vectors and elements are determined. In the subsequent construction and analysis of the machine learning model, it is convenient to calculate the intersection and difference sets of alarm data, fault data, and configuration data, and perform operations such as correlation analysis and processing.

Another embodiment of step S110 of the above sample data processing method is shown as step S110'.

In step S110', it is also possible to store the vectors resulting from the transformation in the rows of a two-dimensional empty matrix according to the sequence of times so as to obtain an alarm matrix, a fault matrix, and a configuration matrix of each target network. The use of a matrix to store the vector resulting from the transformation allows saving storage space using the classic sparse matrix storage method. For example, data in a matrix can be stored based on a compressed row storage (CRS) storage method to save storage space.

More specifically, the vectors of all alarm data are converted in the same manner as in the above step S110 and stored in a two-dimensional empty matrix such as the two-dimensional matrix in the lower right of FIG. 3. With M _a =7000 alarm data blocks, an alarm matrix is formed with 7000 rows and 8 columns. Similarly, a fault matrix and a configuration matrix can also be constructed.

Another embodiment of step S120 is shown as step S120', in which the maximum number of rows and columns of the alarm matrix, the fault matrix and the configuration matrix are obtained, and this maximum number of rows and columns is determined as the number of rows and columns of each level in a multi-level multi-dimensional model .

Then, the alarm matrix, the fault matrix, and the configuration matrix are stored in the alarm data layer, the fault data layer, and the configuration data layer of the multi-level multidimensional space, and the matrix elements having no stored values are filled with zero values at each level.

After obtaining a multi-level multi-dimensional space, it is possible to save storage space using the classic sparse matrix storage method. For example, a multi-level multi-dimensional space may be stored based on the CRS method.

A layered multidimensional space is used to uniformly represent alarm data, fault data, and configuration data, which facilitates subsequent construction of the input and output variables of a deep learning neural network model. The input and output data of a deep learning neural network model are represented as vectors. The multi-level multi-dimensional space built on the basis of an embodiment of the present invention allows convenient and accurate transformation of the alarm data level and the fault data level into input vectors, thereby training a deep learning neural network and determining the correlation law between alarm data, alarm data faults and configuration data.

In general, the timing of alarm data, fault data, and configuration data over the optical network are different. In other embodiments, the alarm data layer, the fault data layer, and the configuration data layer may all contain the same point in time. For each level, if there is a vector group at a particular point in time, then the corresponding string vector of that level will be stored; if there is no data at a certain point in time, then the corresponding string vector will be filled with zero values.

As mentioned above, the optical network contains a plurality of target networks such as the access ring, the convergence ring, and the main ring of FIG. 1, which respectively use a source domain and a target domain to represent an access network, a convergence network, and a data center network. For example, it is possible to define the access network as the source domain and the convergence network as the target domain, or the core network as the source domain and the data center network as the target domain, without limitation.

In yet another embodiment, a multi-level, multi-dimensional space is built for a source domain and a target domain, as shown in FIG. 4, and the sample data processing method includes the following steps:

S200: Data acquisition and pre-processing are carried out. More specifically, this includes the following steps:

S201: Upload alarm data, fault data, and optical network configuration data to three types of data collection and processing centers with a network administration platform or a management platform.

S202: due to the presence of a large amount of repeated, redundant, incomplete and inconsistent data in the alarm data, fault data and configuration data over a wide network, first perform the cleaning of these data using the specified three types of data collection and processing centers to delete the specified duplicate, redundant, incomplete, and poor quality data, and obtaining high quality sets of alarm data, fault data, and configuration data, and then storing them in a source domain database and a target domain database, respectively.

S210: Then, methods of vector and matrix representation of heterogeneous data with different sizes are used, respectively, converting the alarm data, fault data, and configuration data in the source domain and the target domain into a one-dimensional vector, and then present them as a corresponding two-dimensional matrix. More specifically, this includes the following steps:

S211: One-dimensional vector construction process is carried out.

S212: executing a two-dimensional matrix building process.

More specifically, a 2D alarm matrix, a 2D fault matrix, and a 2D configuration matrix are respectively constructed according to the alarm data, fault data, and source domain configuration data, and a 2D alarm matrix, a 2D fault matrix, and a 2D configuration matrix are respectively constructed according to with alarm data, fault data, and configuration data of the target domain. The method for constructing a one-dimensional vector and a two-dimensional matrix is similar to the above embodiment and will not be re-examined in this document.

S220: A multi-level multi-dimensional space is built to implement a unified view of alarm data, fault data, and configuration data in a source domain and a target domain. The number of rows and columns of the matrix obtained in step S212 may be different, for example, assuming that the number of rows and columns of the two-dimensional matrix obtained after the matrix representation of the alarm data, fault data, and configuration data in the source domain and the target domain is as shown in Table 1.

Table 1. Examples of the number of rows and columns of a two-dimensional matrix in the source domain and target domain

Matrix type Number of rows and columns in the alarm matrix Number of rows and columns in the fault matrix Number of rows and columns in the configuration matrix Source domain 5000×12 7000×18 3000×32 Target domain 3000×8 5000×12 2000×35

Calculate the maximum number of rows and the maximum number of columns for all alarm matrices, fault matrices and configuration matrices and determine the maximum number of rows and the maximum number of columns as the number of rows and columns of each level of a two-dimensional matrix in a multi-level multi-dimensional space. For example, in Table 1, the number of rows and columns of each level of a two-dimensional matrix in a multi-level multi-dimensional space is 7000 and 35, respectively. Of these, the number of rows of 7000 means that the maximum number of rows in the six matrices is the number of rows in the source domain fault matrix, and the number of columns of 35 means that the maximum number of columns in the six matrices is the number of columns in the configuration matrix target domain.

After obtaining the maximum number of rows of 7000 and the maximum number of columns of 35, build a six-level multidimensional space representation model based on the six matrices in Table 1 above, generate six matrices with 7000 rows and 35 columns, and then copy the data in these six matrices into newly generated empty matrices and fill matrix elements with no stored data with null elements.

On FIG. 5 specifically shows the multi-level multi-dimensional space constructed for the source domain and the target domain, that is, the six-level multi-dimensional space D=R(K ₁ , K ₂ , K ₃ ). The first to third levels respectively represent the alarm data layer, the fault data layer, and the configuration data layer in the source domain corresponding to the alarm matrix, the fault matrix, and the configuration matrix in the source domain, and the fourth to sixth levels respectively represent the data layer alarm, fault data layer, and configuration data layer in the target domain corresponding to the alarm matrix, fault matrix, and configuration matrix in the target domain. The three-level multidimensional space in the source domain can also be expressed in terms of D _s =R (I ₁ , I ₂ , I ₃ ), and the three-level multidimensional space in the target domain can also be expressed in terms of D _t =R (J ₁ , J ₂ , J ₃ ).

According to the method of the above embodiment, a multi-layered multi-dimensional space can also be built for a plurality of target networks such as an access network, a convergence network, a core network, and a data center network without limitation.

With this embodiment of the present invention, using methods for vector and matrix representation of heterogeneous data with different sizes, it is possible to transform structured and semi-structured data with different sizes related to the optical network into vectors and matrices, and due to the filling of a large number of zero elements and the fact the fact that the multilevel multidimensional space is a sparse matrix, it is possible to use the classic sparse matrix storage method so as to save storage space during the storage process. At the same time, the construction of a multi-level multi-dimensional space not only achieves a unified representation of sample data in the source domain and the target domain, but also provides the possibility of cross-domain interaction of sample data and their exchange between different manufacturers, and also removes the obstacle in the form of an information island for subsequent machine learning.

Based on the above embodiment, in yet another embodiment, the sample data processing method further includes extracting a machine learning model input vector from the alarm data layer and the fault data layer, and extracting a machine learning model output vector from the configuration data layer.

In one embodiment, the alarm data layer and the fault data layer sequentially perform mode expansion and string vector expansion to obtain an input vector, and the configuration data layer sequentially perform mode expansion and string vector expansion to obtain an output vector.

As shown in FIG. 6, the sample data processing method includes the following steps.

S310: Perform mode deployment on the multi-level multi-dimensional space to obtain a mode deployment matrix.

S320: perform string vector expansion on the mode expansion matrix to obtain an output vector.

For the layered multidimensional space (which may be a target network or a plurality of target networks) in the above embodiment, such as the layered multidimensional space D=R(K ₁ , K ₂ , K ₃ ) shown in FIG. 5 use a mode expansion operator to expand each level of space D=R(K ₁ , K ₂ , K ₃ ) from top to bottom. The top, H-th level of the multi-level multi-dimensional space is placed at the leftmost position, and the bottom, first level is placed at the rightmost position, to form the mode deployment matrix of FIG. 6, and then string vector expansion is performed on the mode expansion matrix to obtain an input vector/output vector on the right side of FIG. 6. The first row vector of the H-level matrix is placed in the first place, then the second row vector of the H-level matrix is placed, and finally the last row vector of the first level is placed.

For example, for a multi-level multi-dimensional space D _s =R (I ₁ , I ₂ , I ₃ ) in the source domain, use the first operator f _si : D _s →V _ysi of the mode expansion to transform the alarm data layer and the fault data layer in the multilayer multidimensional space D _s =R(I ₁ , I ₂ , I ₃ ) of the original domain into the input vector V _ysi . The second mode deployment operator f _so : D _s →V _yso is used to convert the level of configuration data in the layered multidimensional space D _s =R(I ₁ , I ₂ , I ₃ ) into an output vector V _yso .

In yet another embodiment, an input vector and/or an output vector is extracted from a subspace of a multilevel multidimensional space, as shown in FIG. 7, and the sample data processing method includes the following steps.

S410: extracting a subspace containing target data from a multilevel multidimensional space, said subspace containing at least one submatrix of an alarm data level, a fault data level, and a configuration data level.

The sub-matrix may be a sub-matrix at one level of the multi-level multi-dimensional space; and the submatrix may also be two or more levels of the multilevel multidimensional space, where each level of the submatrix is a submatrix of one level in the multilevel multidimensional space.

S420: Mode expansion is performed on the specified subspace to obtain the expansion matrix, and string vector expansion is performed on the mode expansion matrix to obtain an input vector and/or an output vector.

On FIG. 8 is a schematic drawing of embodiments for deriving a subspace from a multi-level multi-dimensional space, with the left side of FIG. 8 shows an H-level multi-level multidimensional space built in the above embodiment (as shown in FIG. 5).

In the first embodiment, which may be as shown in phantom lines in FIG. 8, a sub-matrix is extracted from the first level and the H-th level of the multi-level multi-dimensional space, respectively. Similarly, a submatrix is extracted from the second level (for simplicity, the submatrix extraction from the second level is not shown in FIG. 8) to obtain a subspace in the box shown with a dashed line on the right side of FIG. 8. This subspace contains three submatrices, at least one of which contains the target data.

The second embodiment may be as shown in dashed lines in FIG. 8. First, the first level and the second level are extracted from the multi-level multi-dimensional space, and then one sub-matrix is extracted from the first level and the second level, respectively, to obtain a sub-space in the box shown in dashed lines on the right side of FIG. 8. This subspace contains two submatrices, and at least one of these submatrices contains the target data. Similarly, the submatrix can also be extracted from the H-th level to obtain another subspace.

The two above embodiments provide the ability to extract a sub-array from anywhere in each level of a multi-level multi-dimensional space, and both embodiments can be arbitrarily combined to obtain a sub-space containing the target data.

In the third embodiment shown in FIG. 9, a subspace is obtained after extracting a submatrix from the same target level location in a multilevel multidimensional space. The target level may be one level, or two or more levels, without limitation.

The operation of the subspace segmentation operator, defined as g: D→D′, is to extract data in a multilevel multidimensional space and construct a subspace D′.

Taking as an example the multi-level multi-dimensional space shown in FIG. 5, then the multilevel multidimensional space in the target domain is D _s =R(I ₁ , I ₂ , I ₃ ), and the subspace segmentation operator, defined as g _s : D _s →D _s ′, extracts data in the multilevel multidimensional space of the source domain and builds a subspace D _s '=R(I ₁ ′, I ₂ ′, I ₃ ′). The multilevel multidimensional space in the target domain is D _t =R(J ₁ , J ₂ , J ₃ ), and the subspace segmentation operator, defined as g _t : D _t →D _t ′, retrieves data in the multilevel multidimensional space of the target domain and constructs subspace D _t ′=R(J ₁ ′, J ₂ ′, J ₃ ′).

After obtaining the mode expansion matrix, the string vector is expanded on this mode expansion matrix by the first mode expansion operator to obtain the input vector and/or the string vector is expanded on the specified mode expansion matrix by the second mode expansion operator to obtain the output vector.

As an example, in FIG. 10 is a schematic drawing of performing mode expansion and string vector expansion on subspace. On the left side of Fig. 10 shows a subspace derived from the multilevel multidimensional space of the original domain. This subspace can be expressed in terms of D _s ′=R(I ₁ ′=2, I ₂ ′=3, I ₃ ′=3), which is a representation model with two levels, three rows and three columns, where R(1, :, :) represents the submatrix in the alarm data layer of the source domain, R(2, :, :) represents the submatrix in the fault data layer of the source domain, R(1, 1, :) represents the first row vector of the alarm data layer submatrix, and R(1, 1, :)=(6, 5, 13). Similarly, R(2, :, 3) represents the third column vector of the fault data layer submatrix, and R(2, :, 3)=(12, 7B, 6). On FIG. 10, each element in the specified subspace can be accurately characterized in terms of location by specifying the coordinates of each row and each column of each level. For example, R(1, 1, 3) represents the value of the element in the first row and third column of the first level, which is 13, and R(2, 3, 2) represents the value of the element in the third row and second column of the first level, which is 21.

Mode deployment is performed on the specified sub-space, the top fault data level sub-matrix is placed on the left, and the alarm data level bottom sub-matrix is placed on the right to obtain the mode deployment matrix shown at the top right in FIG. ten.

The string vector is expanded on the mode expansion matrix by the first mode expansion operator to obtain an input vector, that is, containing 18 elements of a one-dimensional vector at the bottom right of FIG. 10, with the first row vector of the top left matrix of FIG. 10 is placed on the leftmost side of the input vector, then the first row vector of the top right matrix of FIG. 10 and finally place the third row vector of the upper right matrix.

On FIG. 6 shows performing mode expansion on a layered multidimensional space by means of a mode expansion operator to obtain a mode expansion matrix, and performing string vector expansion on a mode expansion matrix to obtain an input vector and an output vector. On FIG. 10 shows performing a mode expansion on a specified subspace of a layered multidimensional space by means of a mode expansion operator to obtain a mode expansion matrix, and performing a string vector expansion on the mode expansion matrix to obtain an input vector and an output vector.

The use of new technologies such as transfer learning to implement intelligent correlation analysis of network alarms and automatic location and automatic troubleshooting has become the focus of ongoing research for operators and manufacturers. The key operation of the optical network self-recovery transfer learning is to formally describe the data in the source domain and the data in the target domain in order to convert alarm data, fault data, and configuration data over the optical network targeted at O&M personnel into sample data. oriented to machine learning and flexible manipulation of the specified sample data, which includes the definition of intersecting and difference sets of sample data in the source domain and the target domain and the exact extraction of a part of the sample data for testing and analysis. The sample data processing method in the embodiment of the present invention enables a more efficient sample data model to be built and facilitates operations such as sample data extraction and comparison, and facilitates efficient transfer learning model building.

As an example, in FIG. 11 shows a block diagram of subspace segmentation, mode expansion, and string vector expansion of a layered multidimensional space for transfer learning.

On the left side of Fig. 11 shows the multi-level multi-dimensional space D _s of the original domain. An input vector and an output vector of the source domain are constructed, respectively, using the first mode deployment operator f _si and the second mode deployment operator f _so , and the source domain subspace D _s ′ is constructed using the first subspace segmentation operator g _s .

On the right side of Fig. 11 shows the multi-level multi-dimensional space of the target domain. The transfer learning input vector and the target domain output vector are constructed respectively by means of the third mode deployment operator f _ti and the fourth mode deployment operator f _to , and the target domain subspace D _t ′ is constructed by the second subspace segmentation operator g _t .

The input vector V _ysi of the source domain and the input vector V _yti of the target domain together form the input transfer learning vector for intelligent self-troubleshooting of the optical network. The output vector V _yso of the source domain and the output vector V _yto of the target domain together form a transfer learning output vector for intelligent self-troubleshooting of the optical network.

In the process of sample data processing, according to specific actual application scenarios, first extract the subspace containing the most critical attribute data as target data from the multi-level multidimensional model, and then build an input vector and an output vector based on the specified subspace. The complex function f _si og _si : D _s →V _{ysi is used} to construct the input vector of the initial domain, where the operational symbol o indicates that the operation g _si is performed first, and then the operation f _si . The complex function f _so og _so : D _s →V _{yso is used} to construct the output vector of the initial domain. For target domain data at optical network failure, the embodiment of the present invention uses the complex function f _ti og _ti : D _t →V _yti to construct the target domain input vector, and uses the complex function f _to og _to : D _y →V _yto to construct the output vector of the target domain.

As shown in FIG. 12, in an embodiment according to the present invention, a sample data processing system is further provided that is capable of implementing the methods of the above embodiments. The sample data processing system includes a conversion module 101 and a space construction module 102 .

The space builder 101 is configured to receive alarm data, fault data, and configuration data of at least one target network over a plurality of time points from a database and convert them accordingly into appropriate vectors.

The space building module 102 is configured to build, for at least one target network, a multi-level multi-dimensional space containing an alarm data layer, a fault data layer, and a configuration data layer of each target network, each layer being a two-dimensional matrix having one and the same size, and is configured to store the vectors obtained as a result of the transformation, in accordance with the sequence of time points.

In other embodiments, the sample data processing system further comprises an extractor 103 that is configured to extract an input machine learning model vector from the alarm data layer and the fault data layer and extract an output machine learning model vector from the configuration data layer.

More specifically, module 103 includes a sub-matrix processing unit 1031 and a matrix calculation unit 1032.

The sub-matrix processing unit 1031 is configured to extract a sub-space containing target data from a multi-level multi-dimensional space, said sub-space containing at least one sub-matrix of an alarm data level, a fault data level, and a configuration data level.

The matrix calculator 1032 is configured to sequentially perform mode expansion and string vector expansion at the alarm data layer and the fault data layer to obtain an input vector; sequentially performing mode expansion and string vector expansion at the configuration data level to obtain an output vector; and sequentially performing mode expansion and string vector expansion on the specified subspace to obtain an input vector and/or output vector.

More specifically, the space builder 102 includes a pre-matrix processing unit 1021 and a matrix builder 1022.

The matrix pre-processing unit 1021 is configured to store the vectors resulting from the transformation with the same data source in the rows of a two-dimensional empty matrix according to the sequence of time points so as to obtain an alarm matrix, a fault matrix, and a configuration matrix of each target network. respectively.

The matrix builder 1022 is configured to appropriately store the alarm matrix, the fault matrix, and the configuration matrix in the alarm data layer, the fault data layer, and the configuration data layer of the multi-level multi-dimensional space, and populate the matrix elements having no stored values at each level with zero values. .

As shown in FIG. 13, in an embodiment according to the present invention, a network fault handling apparatus based on a machine learning model is further provided. The network fault processing apparatus includes the fault diagnosis system 200 and the sample data processing system 100 of the above embodiments.

The fault diagnosis system 200 includes a model building module 201 and an I/O control module 202 . The model building module 201 is configured to receive an input vector and an output vector from a processing system for building a machine learning model, wherein the input vector is derived from the alarm data layer and the fault data layer, and the output vector is derived from the configuration data layer.

The I/O control module 202 is configured to receive alarm data and fault data of the target network, convert them accordingly into appropriate vectors, input these vectors to the model building module 201, and transmit the output vector of the model building module 201 to the target network.

In the above embodiments, this may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented by software, it may be implemented in whole or in part as a computer program product. The computer program product contains one or more computer instructions. By downloading and executing computer program instructions to a computer, processes or functions are generated in whole or in part in accordance with embodiments of the present invention. The computer may be a general purpose computer, a specialized computer, a computer network, or other programmable devices. The computer instructions may be stored on a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center using wired means (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless media (eg infrared, microwave, etc.). Computer-readable storage media can be any available media that can be read by a computer or storage device, such as a server or data center, combined with one or more available media. The available media can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)) or semiconductor media (e.g., solid-state disk (SSD). )), and so on.

The present invention is not limited to the above embodiments. Various improvements and modifications can be made by those skilled in the art without deviating from the principles of the present invention, and these improvements and modifications are also considered to be within the protection scope of the present invention. The content that is not described in detail in the present description, refers to the prior art, well known to experts in this field of technology.

Claims (18)

1. An apparatus for processing network faults based on a machine learning model, comprising a fault diagnosis system and a sample data processing system; while the sample data processing system contains: a conversion module that is configured to receive alarm data, fault data and configuration data of at least one target network for a plurality of time points from the database and convert them appropriately into appropriate vectors, and a space building module that is configured to build, for at least one target network, a multi-level multi-dimensional space containing an alarm data layer, a fault data layer, and a configuration data layer of each target network, each layer being a two-dimensional matrix having one and the same size, and is configured to store the vectors obtained as a result of the transformation in rows in accordance with the sequence of time points; wherein said fault diagnosis system comprises a model building module and an I/O control module, the model building module is configured to receive an input vector and an output vector from the processing system for building a machine learning model, wherein the input vector is extracted from the alarm data layer and the o data layer. faults, and the output vector is extracted from the configuration data layer; and wherein the input/output control module is configured to receive alarm data and fault data of the target network, convert them appropriately into appropriate vectors, input these vectors to the model building module, and transmit the output vector of the model building module to the target network. 2. The network fault handling device based on the machine learning model of claim 1, wherein the sample data processing system also comprises: an extractor that is configured to extract the input vector of the machine learning model from the alarm data layer and the fault data layer, and extract the output vector of the machine learning model from the configuration data layer. 3. The network fault handling device based on the machine learning model of claim 2, wherein the input vector is obtained by sequentially performing mode expansion and string vector expansion at the alarm data layer and the fault data layer, and the output vector is obtained by sequentially performing mode expansion and string vector expansion at the configuration data layer. 4. The device for processing network faults based on the machine learning model according to claim 1, in which the extraction module contains: a sub-matrix processing unit that is configured to retrieve a sub-space containing target data from a multi-level multi-dimensional space, said sub-space containing at least one sub-matrix of an alarm data level, a fault data level, and a configuration data level; a matrix calculation unit that is configured to sequentially perform mode expansion and string vector expansion in the alarm data layer and fault data layer to obtain an input vector, sequentially perform mode expansion, and string vector expansion in the configuration data layer to obtain an output vector and sequentially execute unfolding the mode; and unfolding the string vector on the specified subspace to obtain an input vector and/or an output vector. 5. The device for processing network faults based on the machine learning model according to claim 1, in which the model building module comprises: a matrix pre-processing unit that is configured to store the vectors resulting from the transformation with the same data source in the rows of a two-dimensional empty matrix in accordance with the sequence of time points so as to obtain an alarm matrix, a fault matrix and a configuration matrix of each target networks respectively; a matrix building unit that is configured to store an alarm matrix, a fault matrix, and a configuration matrix, respectively, at the alarm data level, the fault data level, and the configuration data level of the multi-level multi-dimensional space and fill the matrix elements that do not have stored values at each level with zeros values. 6. The network fault handling device based on the machine learning model of claim 1, wherein the vector resulting from the transformation is the Hadamard product of the main vector and the size extension vector, where each element of the main vector represents the value of a field in the alarm data, fault data, or configuration data at some point in time, and the specified element of the size extension vector represents is the result of a multiple expansion or contraction of the corresponding element of the main vector.

Images