CN114095333A - Network troubleshooting method, device, equipment and readable storage medium - Google Patents
Network troubleshooting method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN114095333A CN114095333A CN202111396187.8A CN202111396187A CN114095333A CN 114095333 A CN114095333 A CN 114095333A CN 202111396187 A CN202111396187 A CN 202111396187A CN 114095333 A CN114095333 A CN 114095333A
- Authority
- CN
- China
- Prior art keywords
- data
- network
- log data
- log
- fault
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000013024 troubleshooting Methods 0.000 title claims abstract description 43
- 238000004458 analytical method Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000004140 cleaning Methods 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 28
- 239000000523 sample Substances 0.000 claims description 27
- 230000000007 visual effect Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 2
- 238000005406 washing Methods 0.000 claims 1
- 238000007405 data analysis Methods 0.000 abstract description 15
- 230000009467 reduction Effects 0.000 abstract description 7
- 239000000243 solution Substances 0.000 description 30
- 238000012544 monitoring process Methods 0.000 description 18
- 230000008569 process Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 8
- 238000012423 maintenance Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 239000008186 active pharmaceutical agent Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 238000012806 monitoring device Methods 0.000 description 3
- 238000012827 research and development Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
Abstract
The application discloses a network troubleshooting method, a network troubleshooting device, network troubleshooting equipment and a readable storage medium. The method and the device can acquire the log data of each network node, clean the log data and store the cleaned log data in a database corresponding to the data type of the log data; inputting the log data stored in each database into a preset network fault analysis model to obtain network fault data; after confirming that network fault data exists in the log data, a network node with a network fault can be determined based on the network fault data. The log data of each network node is reported to the data platform in a unified mode for processing, all the network nodes are effectively formed into a full link for data analysis, and further the data platform is adopted for cleaning, dimension reduction and analysis of the data, so that the difficulty of data analysis is effectively reduced, the accuracy of real-time data analysis is improved, and the efficiency of network fault identification is improved.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a method, an apparatus, a device, and a readable storage medium for network troubleshooting.
Background
With the development of the digital era, many digital living companies accelerate the construction of digital services. In particular, in the digital service construction of digital companies, the system scale of many digital companies in video monitoring applications is increasing, the whole monitoring system is usually constructed on different module sets, each sub-module uses different programming languages by different teams and is respectively deployed on a plurality of servers of a plurality of computer rooms, and each request may form a complex distributed call link. Therefore, when a certain network node fails, it is difficult to quickly locate the failed network node. At present, a device, equipment or a method for analyzing the abnormity of a user scene system link, and intelligently removing the fault of a resource pool is absent, and research and development and operation and maintenance personnel often have no lead and no trouble in the face of the problems of abnormal cloud review, video online playing card pause, equipment firmware upgrading and the like of video monitoring application fed back by a user. Therefore, a network troubleshooting scheme is needed to quickly locate a network node with a network failure and quickly provide a solution to the failure problem when the system has a network failure.
Disclosure of Invention
In view of the above, the present application provides a network troubleshooting method, apparatus, device and readable storage medium, which are used to quickly locate a network node with a network failure and quickly provide a solution to the failure problem when a system fails.
A method of network troubleshooting, comprising:
acquiring log data acquired by probes arranged on each network node, wherein the probe of each network node is matched with the equipment type of the network node;
cleaning the log data;
saving the cleaned log data to a database corresponding to the data type of the log data;
inputting the log data stored in each database into a preset network fault analysis model to obtain network fault data, wherein the network fault analysis model is obtained by taking log training data of various types as training samples and taking a label of whether the log training data is the network fault data as a sample label for training;
and determining the network node with the network fault based on the network fault data.
Preferably, after saving the cleaned log data to a database corresponding to the data type of the log data, the method further includes:
dividing, compressing and backing up the log data stored in each database;
and updating the log data stored in each database at regular time based on preset updating conditions.
Preferably, the performing data cleansing on the log data includes:
and carrying out duplicate removal and debugging processing on the log data so as to remove invalid and repeated log data.
Preferably, the method further comprises:
setting a solution of the network fault aiming at the network fault data in advance;
and determining the solution of the network fault corresponding to the network fault data in a preset network fault solution table so as to provide the solution of the network fault for the user.
Preferably, the method further comprises:
and setting a visual interface for a user to inquire the network fault data in real time.
A network troubleshooting device comprising:
a log data acquisition unit for acquiring log data of each network node;
the data cleaning unit is used for cleaning the log data;
the data storage unit is used for storing the cleaned log data into a database corresponding to the data type of the log data;
the system comprises a fault data acquisition unit, a fault analysis unit and a fault analysis unit, wherein the fault data acquisition unit is used for inputting log data stored in each database into a preset network fault analysis model to obtain network fault data, and the network fault analysis model is obtained by taking log training data of various types as training samples and taking a label of whether the log training data is the network fault data as a sample label for training;
and the fault network node determining unit is used for determining the network node with the network fault based on the network fault data.
Preferably, the apparatus further comprises:
the data editing unit is used for dividing, compressing and backing up the log data stored in each database;
and the updating unit is used for updating the log data stored in each database at regular time based on preset updating conditions.
Preferably, the apparatus further comprises:
a scheme determining unit, configured to determine a solution of the network fault corresponding to the network fault data in a preset network fault solution table, so as to provide a solution of the network fault for a user; wherein the network failure solution table includes network failure data and corresponding network failure solutions.
A network troubleshooting device comprising a memory and a processor;
the memory is used for storing programs;
the processor is configured to execute the program to implement the steps of the network troubleshooting method as described in any one of the foregoing introductions.
A readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the network troubleshooting method as described in any of the preceding introductions.
According to the technical scheme, the log training data of various types can be used as training samples in advance, and the label of whether the log training data is the network fault data is used as a sample label for training to obtain a network fault analysis model so as to analyze whether the log data of each network node is the network fault data. After the network fault analysis model is trained, log data collected by probes arranged at each network node can be obtained, wherein the probe of each network node is matched with the equipment type of the network node; after the log data is obtained, the log data can be cleaned so as to eliminate repeated, incomplete and error log data in the log data. Then, the cleaned log data can be stored in a database corresponding to the data type of the log data; after the log data are stored, the log data stored in each database can be input into a preset network fault analysis model to obtain network fault data; after confirming that network fault data exists in the log data, a network node with a network fault can be determined based on the network fault data. According to the method and the system, the log data of each network node is reported to the data platform in a unified mode for processing, all network nodes are effectively formed into a full link for data analysis, and further, the data platform is adopted for cleaning, dimension reduction and analysis of the data, so that the difficulty of data analysis is effectively reduced, the accuracy of real-time data analysis is improved, and the efficiency of network fault identification is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for implementing network troubleshooting according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a network obstacle deflector according to an example of the present application;
fig. 3 is a block diagram of a hardware structure of a network troubleshooting device disclosed in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In order to develop a network troubleshooting scheme, research is conducted around design targets of low consumption, transparent application level, good expansibility and the like.
Through research, the applicant develops a large-scale distributed system monitoring tool written by Java and supporting cluster and single machine deployment, and the large-scale distributed system monitoring tool is a complete set of services from probe, collector and storage to Web interface. For the collection of the log data of each network node, in order to better run the scheme, the applicant tries to collect the log data of each network node by using a Java probe, but finds that the log data of each network node is collected by using the Java probe after practice, the language support range is small, and the requirements of a device end, a gateway layer and other ends of video monitoring application are difficult to meet; in addition, because the monitoring tool adopts a mode based on bytecode injection to realize rapid network troubleshooting, but the mode is found to be obscure and unintelligible for other research personnel after practice, other research personnel are difficult to modify according to the user requirements based on the scheme researched and developed by the application, and the method has high customization difficulty and high research and development cost, and is not beneficial to market popularization.
Therefore, after research, the applicant develops another monitoring device capable of completing one-stop data acquisition, storage, query and display, provides a Finagle framework interface, supports mainstream languages such as Scala, Node, Go and the like, and mainly uses modified classes and own containers to provide tracking and acquisition of system service link data. However, experiments show that the monitoring granularity of the monitoring scheme is not fine enough, and the display layer can only see the interface level and cannot see more detailed call link relation information. Secondly, the monitoring device is complex in configuration, is based on Brave, only provides a simple API call interface, and needs to customize development codes or add corresponding configuration information if an integrated framework is needed. In addition, this monitoring device needs the bottom drive to support just can realize data acquisition, and its research and development cost is also very high, is unfavorable for marketing equally.
Therefore, it is necessary how to build a set of information system full-link devices conforming to video monitoring application to deeply perform deep fault diagnosis on a monitoring system device side, a gateway layer, a resource pool, a server side and the like so as to solve the problem of difficulty in troubleshooting of the current video monitoring application.
In order to solve the problem, the application provides a network troubleshooting method, which can be used for quickly positioning a network node with a network fault when a system fails and quickly providing a scheme for solving the fault problem.
The method can be applied to any equipment capable of realizing network fault elimination, and optionally, the equipment capable of realizing network fault elimination can be a terminal with data processing capability, such as a tablet computer, a mobile phone, a digital television and the like.
With reference to fig. 1, a flow of a network troubleshooting method according to an embodiment of the present application is described below, where the flow may include the following steps:
step S101, obtaining log data collected by probes arranged on each network node, wherein the probe of each network node is matched with the equipment type of the network node.
Specifically, the log data of each network node records the procedural event record data generated by each network node. Each piece of log data contains time, object, place, event, and the like. By looking at the log data of a network node, it is possible to know which user is specifically, at what time, on which device or in what application system, what specific operation is done. The log data is mainly sourced from servers, storage, network equipment, security equipment, operating systems, middleware, databases, business systems and the like. The log data can be divided into two categories, an IT hardware device state log and an application system log. The hardware device status log includes the CPU or memory usage status of the server, the health status such as storage device temperature or disk capacity, the status of network device traffic or behavior analysis, and the like. The application system log comprises log data of Windows, Linux and Unix operating systems, database log data of Oracle, DB2, SQL Server, Mysql and the like, middleware log data of Apache, Weblogic, Tomcat and the like, and business system log data of bank, finance and the like. The log data is the process data of all IT system operations, so for IT system operation and maintenance personnel, if the log data can be stored and managed in a centralized manner through the log management platform, the problem of fault location and troubleshooting can be solved. For risk management departments, a log management platform can also be used for performing series correlation analysis on log data of different systems, so as to control or reduce enterprise operation risks.
Therefore, whether each network node has a network fault can be analyzed through the log data of each network node. When a network node fails, the log data of each network node is analyzed, and the network node which fails can be quickly found. Therefore, the log data of each network node can be acquired so as to be used for analyzing whether each network node has a fault or not, and maintenance personnel can maintain the network node with the fault quickly.
The log data of each network node can be collected by a probe arranged on each network node. The probe of each network node is matched with the equipment type of the network node, different probes can be set in advance according to the equipment types of the network nodes so as to be used for collecting log data of the network nodes, different probes are adopted for collecting the log data aiming at different types of network nodes, independent installation and independent plug deployment can be realized aiming at the network nodes, a single network node is failed and reassembled, the deployment of other network nodes of the whole link cannot be influenced, and the method is suitable for a service architecture scene related to multi-terminal multi-language at the same time.
For example, for Android and iOS applications, a mobile terminal SDK probe is used, omnibearing monitoring of performances such as APP network requests, stuck, abnormal collapse, static resource loading and the like is achieved by means of an AOP technology, and meanwhile, a visual embedded point technology is used to complete real-time issuing requirements of an online environment embedded point;
aiming at a WEB/WAP terminal, using JS SDK to finish the collection of browser basic information, page performance data, JS error data, AJAX performance data and page resource loading detail data;
for a PC end and a TV end, the end-side data reporting is realized by using an API mode;
aiming at the camera equipment, the log data acquisition is realized through a unified SDK probe and API combination mode.
And step S102, cleaning the log data.
Specifically, the IT system continuously generates a large amount of log data in the process of supporting the operation of the support service. Core transaction systems such as commercial banks, internet banking or ATMs in the financial industry need to provide real-time services without interruption, and log data at TB level and even PB level may be generated every day. The log data are generated at every moment, so that the data volume is large, and the log data are generally dispersed in each storage device, so that IT is difficult for IT operation and maintenance personnel to find the relevance among the log data in a manual mode; moreover, there are various types of log data, and the format and elements of log data records of each type are different, for example:
text type log data: log data is recorded in TXT text, which can open up the view directly.
System type log data: log data for all operations is recorded in the system, but it can also be saved in support of export in TXT text.
SNMP-like log data: because the network equipment, namely the SNMP agent, sends the equipment state to the network management software, namely SNMP management, a log analysis manufacturer can directly obtain SNMP log data through the network management software;
database type log data: taking a relatively common Oracle database as an example, the Oracle database is composed of a database file, a control file and a log file, the log file is divided into a redo log file and an archived log file in the Oracle database, the redo log file is an indispensable file for normal operation of the Oracle database, the operation process of the database is recorded and used for backup and restoration, and the log data of the relational database can be read through an ODBC open database interconnection API.
Repeated, wrong and incomplete log data records may exist among various types of log data, in order to improve the data processing efficiency, after the log data of each network node is obtained, the log data can be cleaned, data noise can be removed by cleaning the log data, repeated, incomplete and wrong log data in each network node are effectively removed, and the data analysis efficiency is improved.
Step S103, saving the cleaned log data to a database corresponding to the data type of the log data.
Specifically, in order to better store the cleaned log data, the cleaned log data can be stored in a database corresponding to the data type of the log data, and the log data can be stored in a classified manner, so that the storage efficiency and the storage space utilization rate of the log data can be improved, and the storage cost can be reduced.
For example, after the log data is cleaned, the log data is dumped into Redis, Hive, MySQL, Hbase and ElasticSearch. Particularly, when the daily data volume of the video monitoring service is 5 times, the HBase table storage is optimized, and after large table compression and data expiration management optimization are increased, the storage is reduced by 73.8%, so that the storage cost is not increased on the premise of effectively guaranteeing the service.
And step S104, inputting the log data stored in each database into a preset network fault analysis model to obtain network fault data.
Specifically, after the cleaned log data is stored in each database according to the type of the log data, the log data stored in each database may be input into a preset network fault analysis model to obtain network fault data.
The network fault analysis model is obtained by taking various types of log training data as training samples and taking a label of whether the log training data is the network fault data as a sample label for training. Whether the log data is network fault data can be determined through the network fault analysis model. In addition, the network fault analysis model also adopts a dimension reduction technology in the process of analyzing the log data, mainly utilizes a PCA dimension reduction technology to process the log data, and reduces the dimension of the features of the log data from n features to k, wherein n > k. And reducing the dimension of the data to be analyzed by using a dimension reduction technology so as to reduce the difficulty of data analysis. The log data can be more easily processed and used in a low dimension; the relevant characteristics of the log data can be more easily highlighted, and particularly, important characteristics can be more clearly displayed in the data.
Step S105, based on the network fault data, determining the network node with the network fault.
Specifically, it is possible to determine which network node each piece of log data was generated by analyzing the log data, and therefore, after it is determined that the log data is network fault data, it is possible to determine which network node has a network fault from the log data corresponding to the fault data.
The network fault data may include topology data of a monitoring system, and a network node with a network fault may be quickly located through the topology data of the monitoring system.
According to the technical scheme, the log data can be cleaned, so that repeated, incomplete and error log data in the log data can be removed. The cleaned log data can be stored in a database corresponding to the data type of the log data and stored in a classified mode, the storage efficiency and the storage space utilization rate of the log data can be improved, and the storage cost is not increased on the premise that the service is effectively guaranteed. According to the method and the system, the log data of each network node is reported to the data platform in a unified mode for processing, all network nodes are effectively formed into a full link for data analysis, and further, the data platform is adopted for cleaning, dimension reduction and analysis of the data, so that the difficulty of data analysis is effectively reduced, the accuracy of real-time data analysis is improved, and the efficiency of network fault identification is improved.
In view of the fact that in practical applications, the volume of the log data is large, and sometimes the log data needs to be backed up and saved, therefore, in step S103, after saving the cleaned log data to the database corresponding to the data type of the log data, some embodiments of the present application may add a process of dividing, compressing, backing up, and updating the log data, and then introduce the process, where the process may include the following steps:
in step S1, the log data stored in each database is divided, compressed, and backed up.
Specifically, after the cleaned log data is saved in the database corresponding to the data type of the log data, the log data saved in each database may be divided, the log data is divided into a plurality of relatively small-sized data, and then the divided log data is compressed, so as to reduce the storage space of the log data and improve the storage utilization rate of the database.
The log data is divided and compressed, so that the storage cost can be saved, and the log data can be conveniently and visually displayed; in addition, the algorithm overhead is reduced, and the data analysis efficiency is improved.
In step S2, the log data stored in each database is updated at regular time intervals based on the preset update condition.
Specifically, after the cleaned log data is saved in the database corresponding to the data type of the log data, an update condition may be set to update the log data saved in each database at regular time.
For example, the log data saved in the data may be set to be updated once a day.
According to the technical scheme, after the cleaned log data are stored in the database corresponding to the data type of the log data, the log data can be divided and compressed, so that the storage space of the log data is reduced, the storage utilization rate of the database is improved, and the storage cost is not increased on the premise of effectively guaranteeing the service; in addition, the log data can be read for backup and periodic update.
In view of the fact that in practical applications, besides the network node where the fault occurs, a feasible solution needs to be provided for the network fault, and therefore, after the network node where the network fault data is located is confirmed, the embodiment of the present application may add a process of providing a solution for the network fault data, where the process is specifically as follows:
determining a solution of the network fault corresponding to the network fault data in a preset network fault solution table so as to provide a solution of the network fault for a user; wherein the network failure solution table includes network failure data and corresponding network failure solutions.
Specifically, a solution of a network fault corresponding to some common network fault types may be set in advance according to the network fault types. Therefore, when the operation and maintenance personnel do network troubleshooting, an optional network fault solution can be provided for the operation and maintenance personnel. After the log data is determined to be network fault data, a network fault solution corresponding to the network fault data can be inquired, and a network fault solution is provided for a user.
According to the technical scheme, the embodiment of the application can provide a solution of the network fault for the user aiming at the network fault data, so that the network fault can be solved, and the communication can be recovered in time.
According to the embodiment of the application, a visual interface can be added besides the fact that whether the log data are the network fault data or not can be confirmed, so that a user can inquire the network fault data in real time.
Generally speaking, a monitoring system can support the real-time query capability of billions of data, a visual interface is increased, and different functional emphasis points can be realized for different roles according to different role requirements of video monitoring application.
For example, customer service personnel can intelligently remove obstacles on the user feedback problems through the visual interface; technicians can quickly position abnormal codes and optimize system performance in time through the visual interface; the operation and maintenance personnel can check information such as service abnormity alarm, alarm threshold configuration, system resource bottleneck early warning and the like through the visual interface.
In the following, the network troubleshooting device provided by the embodiment of the present application is introduced, and the network troubleshooting device described below and the network troubleshooting method described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a network obstacle deflector disclosed in the embodiment of the present application.
As shown in fig. 2, the network troubleshooting device may include:
a log data acquisition unit 101 configured to acquire log data of each network node;
a data cleaning unit 102, configured to clean the log data;
a data saving unit 103, configured to save the cleaned log data to a database corresponding to a data type of the log data;
a fault data obtaining unit 104, configured to input log data stored in each database into a preset network fault analysis model to obtain network fault data, where the network fault analysis model is obtained by training using log training data of each type as a training sample and using a label indicating whether the log training data is the network fault data as a sample label;
a faulty network node determination unit 105, configured to determine, based on the network fault data, a network node where a network fault occurs.
The device of the embodiment of the application can acquire log data of each network node; after the log data is obtained, the log data can be cleaned so as to eliminate repeated, incomplete and error log data in the log data. Then, the cleaned log data can be stored in a database corresponding to the data type of the log data; after the log data are stored, the log data stored in each database can be input into a preset network fault analysis model to obtain network fault data; after confirming that network fault data exists in the log data, a network node with a network fault can be determined based on the network fault data. According to the method and the system, the log data of each network node is reported to the data platform in a unified mode for processing, all network nodes are effectively formed into a full link for data analysis, and further, the data platform is adopted for cleaning, dimension reduction and analysis of the data, so that the difficulty of data analysis is effectively reduced, the accuracy of real-time data analysis is improved, and the efficiency of network fault identification is improved.
Further optionally, the network obstacle deflector may further include:
the data editing unit is used for dividing, compressing and backing up the log data stored in each database;
and the updating unit is used for updating the log data stored in each database at regular time based on preset updating conditions.
Further optionally, the data cleansing unit is configured to perform deduplication and error removal processing on the log data, so as to remove invalid and duplicate log data.
Further optionally, the network obstacle deflector may further include:
a scheme determining unit, configured to determine a solution of the network fault corresponding to the network fault data in a preset network fault solution table, so as to provide a solution of the network fault for a user; wherein the network failure solution table includes network failure data and corresponding network failure solutions.
Further optionally, the network obstacle deflector may further include:
and the visual setting unit is used for setting a visual interface so that a user can inquire the network fault data in real time.
The detailed processing flow of each unit included in the network troubleshooting device may refer to the related description of the network troubleshooting method, and is not described herein again.
The network troubleshooting device provided by the embodiment of the application can be applied to network troubleshooting equipment, such as a terminal: mobile phones, computers, etc. Optionally, fig. 3 shows a block diagram of a hardware structure of the network fault removal device, and referring to fig. 3, the hardware structure of the network fault removal device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.
In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4.
The processor 1 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits or the like configured to implement embodiments of the present invention;
the memory 3 may include a high-speed RAM memory, and may further include a non-volatile memory (non-volatile memory) or the like, such as at least one disk memory;
wherein the memory stores a program and the processor can call the program stored in the memory, the program for: and realizing each processing flow in the terminal network troubleshooting scheme.
Embodiments of the present application further provide a readable storage medium, where the storage medium may store a program adapted to be executed by a processor, where the program is configured to: and realizing each processing flow of the terminal in the network troubleshooting scheme.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. The various embodiments may be combined with each other. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of network troubleshooting, comprising:
acquiring log data acquired by probes arranged on each network node, wherein the probe of each network node is matched with the equipment type of the network node;
cleaning the log data;
saving the cleaned log data to a database corresponding to the data type of the log data;
inputting the log data stored in each database into a preset network fault analysis model to obtain network fault data, wherein the network fault analysis model is obtained by taking log training data of various types as training samples and taking a label of whether the log training data is the network fault data as a sample label for training;
and determining the network node with the network fault based on the network fault data.
2. The network troubleshooting method of claim 1, wherein after the saving the cleaned log data to a database corresponding to a data type of the log data, the method further comprises:
dividing, compressing and backing up the log data stored in each database;
and updating the log data stored in each database at regular time based on preset updating conditions.
3. The network troubleshooting method of claim 1, wherein the data washing the log data comprises:
and carrying out duplicate removal and debugging processing on the log data so as to remove invalid and repeated log data.
4. A method of network troubleshooting as in claim 1 further comprising:
determining a network fault solution corresponding to the network fault data in a preset network fault solution table so as to provide the network fault solution for a user;
wherein the network failure solution table includes network failure data and corresponding network failure solutions.
5. A network troubleshooting method according to any one of claims 1-4 characterized in that the method further comprises:
and setting a visual interface for a user to inquire the network fault data in real time.
6. A network troubleshooting device, comprising:
the system comprises a log data acquisition unit, a log data acquisition unit and a log data processing unit, wherein the log data acquisition unit is used for acquiring log data acquired by probes arranged on each network node, and the probe of each network node is matched with the equipment type of the network node;
the data cleaning unit is used for cleaning the log data;
the data storage unit is used for storing the cleaned log data into a database corresponding to the data type of the log data;
the system comprises a fault data acquisition unit, a fault analysis unit and a fault analysis unit, wherein the fault data acquisition unit is used for inputting log data stored in each database into a preset network fault analysis model to obtain network fault data, and the network fault analysis model is obtained by taking log training data of various types as training samples and taking a label of whether the log training data is the network fault data as a sample label for training;
and the fault network node determining unit is used for determining the network node with the network fault based on the network fault data.
7. The network troubleshooting device of claim 6 further comprising:
the data editing unit is used for dividing, compressing and backing up the log data stored in each database;
and the updating unit is used for updating the log data stored in each database at regular time based on preset updating conditions.
8. The network troubleshooting device of claim 6 further comprising:
a scheme determining unit, configured to determine a solution of the network fault corresponding to the network fault data in a preset network fault solution table, so as to provide a solution of the network fault for a user; wherein the network failure solution table includes network failure data and corresponding network failure solutions.
9. A network troubleshooting device comprising a memory and a processor;
the memory is used for storing programs;
the processor, configured to execute the program, implementing the steps of the network troubleshooting method of any one of claims 1-5.
10. A readable storage medium, having stored thereon a computer program, characterized in that the computer program, when being executed by a processor, carries out the steps of the network troubleshooting method as claimed in any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111396187.8A CN114095333A (en) | 2021-11-23 | 2021-11-23 | Network troubleshooting method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111396187.8A CN114095333A (en) | 2021-11-23 | 2021-11-23 | Network troubleshooting method, device, equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114095333A true CN114095333A (en) | 2022-02-25 |
Family
ID=80303469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111396187.8A Pending CN114095333A (en) | 2021-11-23 | 2021-11-23 | Network troubleshooting method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114095333A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114978877A (en) * | 2022-05-13 | 2022-08-30 | 京东科技信息技术有限公司 | Exception handling method and device, electronic equipment and computer readable medium |
| CN115617953A (en) * | 2022-11-15 | 2023-01-17 | 成都九洲电子信息系统股份有限公司 | Intelligent diagnosis method and system for network service link fault |
| CN115914065A (en) * | 2022-12-06 | 2023-04-04 | 中国联合网络通信集团有限公司 | Terminal self-service diagnosis method, device, equipment and medium |
| CN117278383A (en) * | 2023-11-21 | 2023-12-22 | 航天科工广信智能技术有限公司 | Internet of things fault investigation scheme generation system and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110545195A (en) * | 2018-05-29 | 2019-12-06 | 华为技术有限公司 | network fault analysis method and device |
| CN110855502A (en) * | 2019-11-22 | 2020-02-28 | 叶晓斌 | Fault cause determination method and system based on time-space analysis log |
| CN111177095A (en) * | 2019-12-10 | 2020-05-19 | 中移（杭州）信息技术有限公司 | Log analysis method and device, computer equipment and storage medium |
| CN111585799A (en) * | 2020-04-29 | 2020-08-25 | 杭州迪普科技股份有限公司 | Network fault prediction model establishing method and device |
| CN111817891A (en) * | 2020-07-07 | 2020-10-23 | 中国联合网络通信集团有限公司 | Network fault processing method and device, storage medium and electronic equipment |
-
2021
- 2021-11-23 CN CN202111396187.8A patent/CN114095333A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110545195A (en) * | 2018-05-29 | 2019-12-06 | 华为技术有限公司 | network fault analysis method and device |
| CN110855502A (en) * | 2019-11-22 | 2020-02-28 | 叶晓斌 | Fault cause determination method and system based on time-space analysis log |
| CN111177095A (en) * | 2019-12-10 | 2020-05-19 | 中移（杭州）信息技术有限公司 | Log analysis method and device, computer equipment and storage medium |
| CN111585799A (en) * | 2020-04-29 | 2020-08-25 | 杭州迪普科技股份有限公司 | Network fault prediction model establishing method and device |
| CN111817891A (en) * | 2020-07-07 | 2020-10-23 | 中国联合网络通信集团有限公司 | Network fault processing method and device, storage medium and electronic equipment |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114978877A (en) * | 2022-05-13 | 2022-08-30 | 京东科技信息技术有限公司 | Exception handling method and device, electronic equipment and computer readable medium |
| CN114978877B (en) * | 2022-05-13 | 2024-04-05 | 京东科技信息技术有限公司 | Abnormality processing method, abnormality processing device, electronic equipment and computer readable medium |
| CN115617953A (en) * | 2022-11-15 | 2023-01-17 | 成都九洲电子信息系统股份有限公司 | Intelligent diagnosis method and system for network service link fault |
| CN115914065A (en) * | 2022-12-06 | 2023-04-04 | 中国联合网络通信集团有限公司 | Terminal self-service diagnosis method, device, equipment and medium |
| CN117278383A (en) * | 2023-11-21 | 2023-12-22 | 航天科工广信智能技术有限公司 | Internet of things fault investigation scheme generation system and method |
| CN117278383B (en) * | 2023-11-21 | 2024-02-20 | 航天科工广信智能技术有限公司 | Internet of things fault investigation scheme generation system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114095333A (en) | Network troubleshooting method, device, equipment and readable storage medium | |
| US11868373B2 (en) | Method and apparatus for monitoring an in-memory computer system | |
| CN106953740B (en) | Processing method, client, server and system for page access data in application | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN111752793A (en) | System abnormity monitoring method and device, computer equipment and storage medium | |
| US9116973B2 (en) | Method and apparatus for monitoring an in-memory computer system | |
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN103488793A (en) | User behavior monitoring method based on information retrieval | |
| US9235982B2 (en) | Determining alert criteria in a network environment | |
| CN112445583A (en) | Task management method, task management system, electronic device, and storage medium | |
| CN114780335A (en) | Correlation method and device of monitoring data, computer equipment and storage medium | |
| CN111865673A (en) | Automatic fault management method, device and system | |
| US20160139961A1 (en) | Event summary mode for tracing systems | |
| CN113032252A (en) | Method and device for collecting buried point data, client device and storage medium | |
| CN113791943A (en) | Website real-time monitoring method, system, equipment and storage medium | |
| CN117215867A (en) | Service monitoring method, device, computer equipment and storage medium | |
| CN112149975A (en) | APM monitoring system and method based on artificial intelligence | |
| CN115794545A (en) | Automatic processing method of operation and maintenance data and related equipment thereof | |
| CN110764985A (en) | Method, system, equipment and readable storage medium for recording command line | |
| CN113934595A (en) | Data analysis method and system, storage medium and electronic terminal | |
| CN116701525A (en) | Early warning method and system based on real-time data analysis and electronic equipment | |
| CN113342619A (en) | Log monitoring method and system, electronic device and readable medium | |
| US20200184428A1 (en) | Using Distributed Databases for Network Regression Analysis | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN115580528A (en) | Fault root cause positioning method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240321 Address after: Unit 1, Building 1, China Telecom Zhejiang Innovation Park, No. 8 Xiqin Street, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province, 311100 Applicant after: Tianyi Shilian Technology Co.,Ltd. Country or region after: Zhong Guo Address before: Room 1423, No. 1256 and 1258, Wanrong Road, Jing'an District, Shanghai 200040 Applicant before: Tianyi Digital Life Technology Co.,Ltd. Country or region before: Zhong Guo |