CN112202638B - Data processing method, device, equipment and computer storage medium - Google Patents
Data processing method, device, equipment and computer storage medium Download PDFInfo
- Publication number
- CN112202638B CN112202638B CN202011057380.4A CN202011057380A CN112202638B CN 112202638 B CN112202638 B CN 112202638B CN 202011057380 A CN202011057380 A CN 202011057380A CN 112202638 B CN112202638 B CN 112202638B
- Authority
- CN
- China
- Prior art keywords
- data
- dial
- testing
- dial testing
- dialing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0811—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking connectivity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application discloses a data processing method, a data processing device, data processing equipment and a computer storage medium, and relates to the technical field of cloud computing. The specific implementation scheme is as follows: acquiring dial testing data corresponding to a label combination, wherein the dial testing data corresponding to the label combination comprises all labels in the label combination; acquiring the concentration degree of normal and abnormal conditions corresponding to the label combination according to the normal and abnormal conditions of the acquired dial testing data; and selecting the data to be detected according to the concentration degrees. According to the method and the device, the data with non-concentrated normal or abnormal conditions can be filtered, and the data processing amount of whether the dial testing source or the dial testing target is abnormal or not according to the subsequent analysis of the detection data is reduced.
Description
Technical Field
The application relates to the technical field of computers, in particular to the technical field of cloud computing.
Background
Software modules running on the network can be referred to as services, which enable users to access data on the network or use APPs (applications) at different places through different terminal devices. In the process of using the service by the user, the user terminal needs to interact with the service backend, and some interaction also needs to be performed between the service backend. If there is connectivity problem with the network carrying these interactions, the user cannot use the service normally.
In order to determine whether there is a problem in network connectivity, the range of the problem, etc., a common way is to perform a dial test, i.e., send a packet and check whether there is a normal return. By collecting and analyzing the dial-up test data, a network connectivity detection result can be given. Such connectivity checks may be used in Network Performance Management (NPM) to send fault alarms and to assist operation and maintenance personnel in organizing fault notifications. At present, the fault range given by connectivity detection based on dial-up test data is inaccurate.
Disclosure of Invention
The application provides a method, a device, equipment and a computer storage medium for data processing.
According to an aspect of the present application, there is provided a data processing method including:
acquiring dial testing data corresponding to the label combination, wherein the dial testing data corresponding to the label combination comprises all labels in the label combination;
acquiring the concentration degree of normal and abnormal conditions corresponding to the label combination according to the normal and abnormal conditions of the acquired dial testing data;
and selecting the data to be detected according to the concentration degrees.
According to another aspect of the present application, there is provided a data processing apparatus comprising:
the dial testing data acquisition module is used for acquiring dial testing data corresponding to the label combination, and the dial testing data corresponding to the label combination comprises all labels in the label combination;
the concentration degree module is used for acquiring the concentration degree of the normal abnormal conditions corresponding to the label combination according to the normal abnormal conditions of the acquired dial testing data;
and the data module to be detected is used for selecting data to be detected according to each concentration degree.
According to another aspect of the present application, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform a method provided by any one of the embodiments of the present application.
According to another aspect of the present application, there is provided a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are configured to cause a computer to perform the method provided by any one of the embodiments of the present application.
According to another aspect of the application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method as described above.
According to the technical implementation mode, the concentration degree of normal or abnormal dial testing data in all the dial testing data corresponding to the label combination is determined according to the dial testing data corresponding to the label combination, then the dial testing data to be detected are selected according to the concentration degree, so that the dial testing data can be preliminarily screened, the dial testing data with low concentration degree are removed, the workload of determining the subsequent dial testing data conditions is reduced, and the detection efficiency is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a data processing method according to another embodiment of the present application;
FIG. 3 is a schematic diagram of a data processing method according to another embodiment of the present application;
FIG. 4 is a schematic diagram of a data processing method according to another embodiment of the present application;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a data processing apparatus according to another embodiment of the present application;
FIG. 7 is a schematic diagram of a data processing apparatus according to another embodiment of the present application;
FIG. 8 is a schematic diagram of a data processing apparatus according to another embodiment of the present application;
fig. 9 is a block diagram of an electronic device for implementing the data processing method according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
An embodiment of the present application provides a data processing method, as shown in fig. 1, including:
step S11: acquiring dial testing data corresponding to the label combination, wherein the dial testing data corresponding to the label combination comprises all labels in the label combination;
step S12: acquiring the concentration degree of normal and abnormal conditions corresponding to the label combination according to the normal and abnormal conditions of the acquired dial testing data;
step S13: and selecting the data to be detected according to the concentration degrees.
In this embodiment, the dial-up test data has a plurality of tags, and the tag combination includes at least one of the plurality of tags. The label is the dial testing source information of the dial testing data or the dial testing target information of the dial testing data and can correspond to the source of the abnormal occurrence in the dial testing process. For example, the label may be a province of the dial testing source, a province of the dial testing target, an operator of the dial testing source, an operator of the dial testing target, and the like, and the source of the corresponding abnormal condition may be a province of the dial testing source, a province of the dial testing target, an operator of the dial testing source, an operator of the dial testing target, and the like.
In a particular example, the tag combinations may be combinations of tags in which dial-up data exception conditions may be concentrated. For example, there are 2 anomalies in 10 pieces of dial testing data with the dial testing source being province A and the dial testing target being province B; if there are 3 abnormal pieces of 8 pieces of dial test data with the dial test source of province A and the dial test target of province C, the label "province A" can be determined as a label combination in which abnormal conditions may be concentrated.
For another example, there are 3 exceptions in 10 pieces of dial testing data with the dial testing source of province A, the dial testing source operator of corporation D, the dial testing target of province B and the dial testing target operator of corporation E; if there are 3 abnormal pieces of dial test data with the dial test source of province A, the dial test source operator of corporation D, the dial test target of province C and the dial test target operator of corporation F, the label "province A and D operators" can be determined as the label combination with the possible concentration of abnormal conditions.
In a specific operation, if the tag combinations are tag combinations in which abnormal conditions may be concentrated, the model may be used to perform preliminary screening, for example, the dial testing data is input into the model, the model analyzes the dial testing data, and the tag combinations are output. In order to save the amount of calculation, when determining the tag combination, prediction can be performed only by the model, and the concentration degree of the specific abnormal dial testing data in all the dial testing data corresponding to the tag combination is not calculated.
The label included in the label combination may be a label corresponding to the dial testing source, may also be a label corresponding to the dial testing target, and may also include both a label corresponding to the dial testing source and a label corresponding to the dial testing target.
The label combination can also select any label corresponding to the dial testing source and/or any label corresponding to the dial testing target.
In this embodiment, the normal abnormal condition may indicate that the dial test data is normal, or the dial test data is abnormal.
According to the normal and abnormal conditions of each piece of acquired dial testing data, the concentration degree of the normal and abnormal conditions corresponding to the label combination is acquired, which may be the concentration degree of the abnormal dial testing data in the dial testing data corresponding to the label combination. Or acquiring the concentration degree of the normal dial testing data in the dial testing data corresponding to the label combination, and then calculating the concentration degree of the abnormal dial testing data according to the concentration degree of the normal dial testing data.
In this embodiment, the data to be detected is selected according to each concentration degree, and the dial-up test data with the concentration degree higher than the set threshold may be selected as the data to be detected. Or eliminating the dial testing data of the label combination with low concentration degree in the dial testing data, and taking the rest data as the data to be detected.
The purpose of dial testing is to detect whether the dial testing source or the dial testing target is abnormal. These call tests typically include: the dial-up test is performed from each operator in each provincial city of the country to the service entrance, and the dial-up test is performed from the machine or container where each service is located to the machine or container depending on the service. And analyzing by collecting dial-up test data to give a network connectivity detection result. The dial testing source or the dial testing target has faults and the dial testing source or the dial testing target can show that the dial testing data is abnormal. If the dial testing source and the dial testing target are judged to be abnormal only according to whether the dial testing data are abnormal, the problem that only one of the dial testing source and the dial testing target is abnormal and the normal abnormal judgment of the other is influenced can occur. By adopting the data processing method of the embodiment, the dial testing data corresponding to the label combination with low degree of abnormal dial testing data concentration can be removed, and the residual data can be used as the data to be detected. Or selecting the dial testing data corresponding to the label combination with high degree of concentration of the abnormal dial testing data as the data to be detected.
After the data to be detected is determined, normal or abnormal conditions of the dial testing source and the dial testing target are determined by adopting an abnormal detection method of the dial testing data according to the data to be detected.
In this embodiment, according to the dial testing data corresponding to the tag combination, the concentration degree of the normal or abnormal dial testing data in all the dial testing data corresponding to the tag combination is determined, and then the dial testing data to be detected is selected according to the concentration degree, so that the dial testing data can be preliminarily screened, the dial testing data with low concentration degree are removed, the workload consumed by determination of the subsequent dial testing data condition is reduced, and the detection efficiency is improved.
For example, in general, if it is necessary to detect whether there is an abnormality in the dial test source or the dial test target, the tag combinations in the abnormality set are predicted first. And then according to the dial testing data corresponding to the label combination, determining the concentration degree of the abnormal dial testing data corresponding to the label combination in all the dial testing data corresponding to the label combination. And finally, screening abnormal dial testing data with low concentration degree to obtain residual abnormal dial testing data which are used as data to be detected, and further performing abnormal detection on the dial testing source or the dial testing target.
When the dial testing data is abnormal, the province of the dial testing source may have communication link abnormality, the operator of the dial testing source may have communication link abnormality, the address of the dial testing source may have communication link abnormality, or some aspects of the dial testing target may have abnormality. When the fault source corresponding to any label of the dial testing data is abnormal, the dial testing data is abnormal. Therefore, through screening, the situation that when a communication link is abnormal due to a possible abnormal source corresponding to a certain label, other possible abnormal sources of the dial testing source or the dial testing target are judged to be abnormal can be eliminated, and the accuracy of abnormality judgment is improved.
In one embodiment, the obtaining, according to each obtained dial testing data, a concentration degree of normal and abnormal conditions of the dial testing data corresponding to the tag combination includes:
calculating the divergence of abnormal dial testing data in the dial testing data corresponding to the label combination;
divergence is taken as the degree of concentration.
In the present embodiment, the divergence may be a numerical value used to measure the difference between two probabilities. Specifically, in this embodiment, the divergence may be a difference between a proportion of normal dial test data of the tag combination in the dial test data and a proportion of abnormal dial test data in the dial test data.
In this embodiment, the divergence may be calculated using a divergence function, such as a JS divergence function, a Relative Entropy divergence function, a Jensen Shannon (JS, Jensen Shannon) divergence function, an F divergence function, a Bregman divergence function, a Wasserstein distance function, and the like.
In this embodiment, the divergence is used as the concentration degree of the normal and abnormal conditions of the tag combination, so that the concentration degree of the normal or abnormal conditions in the dial test data of the tag combination can be accurately obtained.
In one embodiment, calculating the divergence of abnormal dial testing data in the dial testing data corresponding to the label combination comprises:
calculating a first quantity of normal dial testing data and a second quantity of abnormal dial testing data in the dial testing data corresponding to the label combination;
calculating a first proportion of the first quantity to the total quantity of the dial testing data corresponding to the label combination, and a second proportion of the second quantity to the total quantity of the dial testing data corresponding to the label combination;
and taking the Hailinge distance between the first proportion and the second proportion as the abnormal dialing test data divergence in the dialing test data corresponding to the label combination.
In the embodiment, by calculating the hailing distance, the abnormal data concentration degree of the label combination can be accurately obtained, and then the dial-up test data is screened, so that a more accurate screening result can be obtained.
In one embodiment, selecting the data to be detected according to the concentration degree comprises:
and selecting abnormal dial testing data corresponding to the label combination with the concentration degree higher than the set threshold value as to-be-detected data.
In this embodiment, the abnormal dial testing data corresponding to the label combination with the concentration degree higher than the set threshold may mean that the concentration degree of the abnormal data in the total number of the dial testing data corresponding to the label combination is higher than the set threshold.
And the dial testing data corresponding to the label combination with the abnormal concentration degree higher than the set threshold value is used as the data to be detected, so that the normal one of the dial testing source and the dial testing target can be excluded from the data to be detected, and the related dial testing data is influenced by the abnormal other one to present an abnormal condition. The calculation amount of subsequent abnormity judgment is reduced, and the abnormity judgment accuracy is improved.
In one embodiment, as shown in fig. 2, the data processing method further includes:
step S21: and determining whether the dial testing source and/or the dial testing target related to the label combination are abnormal or not according to the data to be detected.
In this embodiment, the dialing test source and/or the dialing test target related to the tag combination may be an abnormal source of the dialing test source and/or an abnormal source of the dialing test target corresponding to the tag combination. For example, if the domain name system of the dial testing source and/or the domain name system of the dial testing target exist, whether the domain name system of the dial testing source and/or the domain name system of the dial testing target have abnormality or failure can be determined according to the data to be detected.
In one example, determining whether a dial test source and/or a dial test target associated with a tag combination is abnormal according to data to be detected includes:
determining an abnormal threshold value according to a label combination corresponding to the data to be detected;
aiming at a third proportion of abnormal dial testing data in the dial testing data corresponding to the label combination of the data to be detected in the total dial testing data of the label combination of the data to be detected;
if the third proportion is higher than the abnormal threshold value, determining that the dial testing data corresponding to the label combination is abnormal;
and determining at least one of the dial testing source or the dial testing target as abnormal according to the information of the dial testing source and the dial testing target in the label combination.
In actual operation, corresponding abnormal threshold values are set corresponding to different dial testing sources or different dial testing targets. For example, for the D operator in province a, the stability of the service operation status is relatively low, a high anomaly threshold may be set for the D operator in province a, and when there is an anomaly in more dial test data, the D operator in province a is considered to be anomalous.
In other specific examples, whether an anomaly exists in the dial test source and/or the dial test target may also be determined in other ways.
Through further analysis of the data to be detected, specific reasons of occurrence of the abnormity, such as provincial communication link abnormity, communication link abnormity of operators and the like, can be found out, so that valuable reference information is provided for timely discovering the abnormity and solving the abnormity.
In one embodiment, the combination of tags includes at least one of the following tags:
the method comprises the steps of dialing an address of a testing source, dialing a province of the testing source, dialing an operator of the testing source, dialing an address of a testing target, dialing a province of the testing target, dialing an operator of the testing target, dialing a data center of the testing source, dialing a data center of the testing target, dialing a domain name system server of the testing source, dialing a domain name system server of the testing target, dialing a domain name system analysis result of the testing source and dialing a domain name system analysis result of the testing target.
In this embodiment, the label in the label combination may reflect the source or the reason of the failure, for example, the reason of the failure may be that at least one of the address of the dial test source, the province of the dial test source, the operator of the dial test source, the address of the dial test target, the province of the dial test target, the operator of the dial test target, the data center of the dial test source, the data center of the dial test target, the domain name system server of the dial test source, the domain name system server of the dial test target, the domain name system resolution result of the dial test source, and the domain name system resolution result of the dial test target has a failure. By the aid of the labels, the range of abnormal reasons can be narrowed, and more accurate basis is provided for finding the abnormality and repairing the fault.
In one embodiment, as shown in fig. 3, the data processing method further includes:
step S31: collecting dial testing data, wherein the dial testing data comprises data for dial testing between a dial testing source and a dial testing target;
step S32: and marking the condition information and the label of the dial testing data, wherein the condition information is normal or abnormal.
In this embodiment, when dialing, the dial test source may send dial test information to the dial test target, and the dial test information sending process generally includes: and (4) carrying out dial testing from each operator in each province city in the country to a service entrance, carrying out dial testing from a machine or container where each service is located to a machine or container depending on the service, and returning response information after a dial testing target receives the test information. The dial testing data collected may be data collected at the dial testing source and used for dial testing of the dial testing target. The dial testing data can also be actively reported by the dial testing source and collected according to the data reported by the dial testing source.
In this embodiment, marking the status information of the dial testing data may include marking that the detection result of each piece of dial testing data is normal or abnormal, for example, the dial testing target responds normally within a specified time, and the other situations are abnormal.
In this embodiment, the label for marking the dial-up test data may include: labeling at least one of the following as tag (label) on the dial-up test data: the source IP of the dial testing (IP of the dial testing source), the province of the dial testing source, the province of the dial testing target, the operator of the dial testing source, the operator of the dial testing target, the IP of the dial testing target, the dial testing, the data center of the source, the data center of the dial testing target, the DNS server of the dial testing source, the DNS server of the dial testing target, the DNS analysis result of the dial testing source, the DNS analysis result of the dial testing target and other information. In the examples given below, tag is in brackets:
for example, a dial test is performed from a dial test point [ source IP ═ x.x.x ] x ] of [ province ═ beijing ] [ operator ═ north ] to [ domain name ═ www.baidu.com ], a CNAME (alias) record resolved from [ DNS ═ 114.114.114.114] is [ CNAME ═ www.a.shifen.com ], an (Address) record is [ target IP ═ y.y.y.y ], an a record belongs to [ target area ═ north China data center ], and a dial test result is normal.
For another example, a dial test is performed from a dial test point of the [ source area ═ north China data center ] [ source IP ═ y.y.y.y ] to the [ target area ═ arry north China region ] [ target IP ═ z.z.z.z.z.z ] and the dial test result is abnormal.
In this embodiment, the dial testing data is collected and then marked, so that the label combination to be analyzed can be obtained according to the marked dial testing data.
In an example of the application, tag combinations in the abnormal dialing test set are selected, dialing test data can be input into a machine learning model or a deep learning model, and tag combinations influenced by other faults are preliminarily filtered by the model to obtain the tag combinations needing further calculation of the concentration degree of the abnormal data. tag combinations include at least one of the tagged tags. During preliminary filtering, the higher the proportion of the tag combination contained in abnormal dial testing data is, and the lower the proportion of the tag combination contained in normal dial testing data is, the more concentrated the dial testing abnormality is on the tag combination is considered. Specifically, the operations as shown in fig. 4 are performed:
step S41: and selecting a tag combination in the dial testing abnormal set from the dial testing data complete set.
Step S42: and if all the current result sets are normal or abnormal samples, finishing the selection.
Step S43: the candidate tag combination is set to null ({ }).
Step S44: selecting the tag combination value with the most concentrated abnormality, and calculating the divergence between the normal dial testing data and the abnormal dial testing data (namely the specific content of the tag) in the dial testing data corresponding to the tag combination. For a tag value (e.g., beijing, province), the current data set is divided into two parts according to normal and abnormal data, Divergence (Divergence) between the two parts is recorded as the degree of abnormal concentration, and the Divergence function includes, but is not limited to, hailing distance. Taking the hailing distance as an example:
recording the total number of dial tests as n, wherein the total number of dial tests is m;
note that a tag is includediThe number of dial measurements is niWherein the number of dial testing normality is mi；
The sample with normal dial test contains tagiIn a ratio of
The abnormal sample contains tagiIn a ratio of
Then tagiDegree of abnormal concentration (divergence)
And judging whether the divergence between the normal dial testing data and the abnormal dial testing data in the dial testing data corresponding to the tag combination is higher than a threshold value or not according to the divergence calculated in the step S44.
Step S45: and if the divergence is higher than the threshold value, adding the selected tag combination value into the candidate tag combination ({ province ═ Beijing }), starting from the data containing the candidate tag combination in the data corpus, and returning to the step S44.
Step S46: if the divergence is lower than the threshold value, the current candidate tag combination is output, the data containing the tag combination is excluded from the dial test result, and the step S44 is returned.
And repeating the steps S44-S45 until the dial testing data corresponding to all the tag combinations are over concentrated, and the corresponding tag combinations are added into the candidate tag combinations or the dial testing data corresponding to the tag combinations are removed.
Step S47: and combining the dial testing data corresponding to all the candidate tags into the dial testing data to be detected.
And further detecting the subsequent dial testing data to be detected, and judging the normal abnormal condition of the dial testing source or the dial testing target. When detecting that the dial testing source or the dial testing target is possibly abnormal, sending a fault alarm and assisting operation and maintenance personnel to arrange fault notification through Network Performance Management (NPM).
An embodiment of the present application further provides a data processing apparatus, as shown in fig. 5, the apparatus may include:
a dial testing data obtaining module 51, configured to obtain dial testing data corresponding to the tag combination, where the dial testing data corresponding to the tag combination includes all tags in the tag combination;
the concentration degree module 52 is configured to obtain a concentration degree of a normal abnormal condition corresponding to the tag combination according to a normal abnormal condition of each obtained dial test data;
and the data to be detected module 53 is configured to select data to be detected according to each concentration degree.
In one embodiment, as shown in fig. 6, the concentration level module 52 further includes:
a divergence calculation unit 61 for calculating divergence of abnormal dial-up test data in the dial-up test data corresponding to the label combination;
and a divergence processing unit 62 for regarding the divergence as a concentration degree.
In one embodiment, the divergence calculation unit is further configured to:
calculating a first quantity of normal dial testing data and a second quantity of abnormal dial testing data in the dial testing data corresponding to the label combination;
calculating a first proportion of the first quantity to the total quantity of the dial testing data corresponding to the label combination, and a second proportion of the second quantity to the total quantity of the dial testing data corresponding to the label combination;
and taking the Hailinge distance between the first proportion and the second proportion as the abnormal dialing test data divergence in the dialing test data corresponding to the label combination.
In one embodiment, the data module to be detected is further configured to:
and selecting abnormal dial testing data corresponding to the label combination with the concentration degree higher than the set threshold value as to-be-detected data.
In one embodiment, as shown in fig. 7, the data processing apparatus further includes:
and the data to be detected detection module 71 is configured to determine whether the dial test source and/or the dial test target related to the tag combination are/is abnormal according to the data to be detected.
In one embodiment, the combination of tags includes at least one of the following tags:
the method comprises the steps of dialing an address of a testing source, dialing a province of the testing source, dialing an operator of the testing source, dialing an address of a testing target, dialing a province of the testing target, dialing an operator of the testing target, dialing a data center of the testing source, dialing a data center of the testing target, dialing a domain name system server of the testing source, dialing a domain name system server of the testing target, dialing a domain name system analysis result of the testing source and dialing a domain name system analysis result of the testing target.
In one embodiment, as shown in fig. 8, the data processing apparatus further includes:
a collection module 81 for collecting dial testing data, wherein the dial testing data includes data for dial testing between a dial testing source and a dial testing target;
and the marking module 82 is used for marking the condition information and the label of the dial testing data, wherein the condition information is normal or abnormal.
The functions of each module/unit in each apparatus in the embodiment of the present application may refer to the corresponding description in the above method, and are not described herein again.
There is also provided, in accordance with an embodiment of the present application, an electronic device, a readable storage medium, and a computer program product.
As shown in fig. 9, it is a block diagram of an electronic device according to the data processing method of the embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 9, the electronic apparatus includes: one or more processors 901, memory 902, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 9 illustrates an example of a processor 901.
The memory 902, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the data processing method in the embodiment of the present application (for example, the dial-up test data acquisition module 51, the concentration level module 52, and the data to be detected module 53 shown in fig. 5). The processor 901 executes various functional applications of the server and data processing by executing non-transitory software programs, instructions, and modules stored in the memory 902, that is, implements the data processing method in the above method embodiment.
The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the data processing electronic device, and the like. Further, the memory 902 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 902 may optionally include memory located remotely from the processor 901, which may be connected to data processing electronics over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the data processing method may further include: an input device 903 and an output device 904. The processor 901, the memory 902, the input device 903 and the output device 904 may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example.
The input device 903 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the data processing electronics, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick, or other input device. The output devices 904 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and Virtual Private Server (VPS) service. The server may also be a server of a distributed system, or a server incorporating a blockchain.
According to the technical scheme of the embodiment of the application, the concentration degree of normal or abnormal dial testing data in all the dial testing data corresponding to the label combination is determined according to the dial testing data corresponding to the label combination, then the dial testing data to be detected are selected according to the concentration degree, so that the dial testing data can be preliminarily screened, the dial testing data with low concentration degree are removed, the workload for determining the subsequent dial testing data condition is reduced, and the detection efficiency is improved.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present application can be achieved, and the present invention is not limited herein.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (17)
1. A method of data processing, comprising:
acquiring dial testing data corresponding to a label combination, wherein the dial testing data corresponding to the label combination comprises all labels in the label combination;
acquiring the concentration degree of normal and abnormal conditions corresponding to the label combination according to the normal and abnormal conditions of the acquired dial testing data; the concentration degree is the difference between the proportion of normal dial testing data corresponding to the label combination and the proportion of abnormal dial testing data;
and selecting the data to be detected according to the concentration degrees.
2. The method according to claim 1, wherein the obtaining a concentration degree of normal abnormal conditions corresponding to the tag combination according to the normal abnormal conditions of the obtained dial-up test data comprises:
calculating the divergence of abnormal dial testing data in the dial testing data corresponding to the label combination;
and taking the divergence as the concentration degree.
3. The method of claim 2, wherein the calculating divergence of abnormal dial-up test data in the dial-up test data corresponding to the tag combination comprises:
calculating a first quantity of normal dial testing data and a second quantity of abnormal dial testing data in the dial testing data corresponding to the label combination;
calculating a first proportion of the first quantity to the total quantity of the dial testing data corresponding to the label combination, and a second proportion of the second quantity to the total quantity of the dial testing data corresponding to the label combination;
and taking the Hailinge distance between the first proportion and the second proportion as the abnormal dialing test data divergence in the dialing test data corresponding to the label combination.
4. The method according to any one of claims 1 to 3, wherein said selecting data to be detected according to each said concentration degree comprises:
and selecting abnormal dial testing data corresponding to the label combination with the concentration degree higher than the set threshold value as the data to be tested.
5. The method of any of claims 1 to 3, further comprising:
and determining whether the dial testing source and/or the dial testing target related to the label combination are abnormal or not according to the data to be detected.
6. The method of any one of claims 1 to 3, wherein the tag combination comprises at least one of:
the method comprises the steps of dialing an address of a testing source, dialing a province of the testing source, dialing an operator of the testing source, dialing an address of a testing target, dialing a province of the testing target, dialing an operator of the testing target, dialing a data center of the testing source, dialing a data center of the testing target, dialing a domain name system server of the testing source, dialing a domain name system server of the testing target, dialing a domain name system analysis result of the testing source and dialing a domain name system analysis result of the testing target.
7. The method of any of claims 1 to 3, further comprising:
collecting dial testing data, wherein the dial testing data comprises data for dial testing between a dial testing source and a dial testing target;
and marking the status information and the label of the dial testing data, wherein the status information is normal or abnormal.
8. A data processing apparatus comprising:
the system comprises a dial testing data acquisition module, a dial testing data acquisition module and a label combination testing module, wherein the dial testing data acquisition module is used for acquiring dial testing data corresponding to a label combination, and the dial testing data corresponding to the label combination comprises all labels in the label combination;
the concentration degree module is used for acquiring the concentration degree of the normal abnormal conditions corresponding to the label combination according to the normal abnormal conditions of the acquired dial testing data; the concentration degree is the difference between the proportion of normal dial testing data corresponding to the label combination and the proportion of abnormal dial testing data;
and the data module to be detected is used for selecting data to be detected according to the concentration degrees.
9. The apparatus of claim 8, wherein the concentration module further comprises:
the divergence calculation unit is used for calculating the divergence of abnormal dial testing data in the dial testing data corresponding to the label combination;
a divergence processing unit for taking the divergence as the concentration degree.
10. The apparatus of claim 9, wherein the divergence calculation unit is further to:
calculating a first quantity of normal dial testing data and a second quantity of abnormal dial testing data in the dial testing data corresponding to the label combination;
calculating a first proportion of the first quantity to the total quantity of the dial testing data corresponding to the label combination, and a second proportion of the second quantity to the total quantity of the dial testing data corresponding to the label combination;
and taking the Hailinge distance between the first proportion and the second proportion as the abnormal dialing test data divergence in the dialing test data corresponding to the label combination.
11. The apparatus according to any one of claims 8 to 10, wherein the data module to be detected is further configured to:
and selecting abnormal dial testing data corresponding to the label combination with the concentration degree higher than the set threshold value as the data to be tested.
12. The apparatus of any of claims 8 to 10, further comprising:
and the data detection module to be detected is used for determining whether the dial testing source and/or the dial testing target related to the label combination are abnormal or not according to the data to be detected.
13. The apparatus of any one of claims 8 to 10, wherein the tag combination comprises at least one of:
the method comprises the steps of dialing an address of a testing source, dialing a province of the testing source, dialing an operator of the testing source, dialing an address of a testing target, dialing a province of the testing target, dialing an operator of the testing target, dialing a data center of the testing source, dialing a data center of the testing target, dialing a domain name system server of the testing source, dialing a domain name system server of the testing target, dialing a domain name system analysis result of the testing source and dialing a domain name system analysis result of the testing target.
14. The apparatus of any of claims 8 to 10, further comprising:
the system comprises a collection module, a test acquisition module and a test acquisition module, wherein the collection module is used for collecting dial test data which comprises data for dial test between a dial test source and a dial test target;
and the marking module is used for marking the status information and the label of the dial testing data, wherein the status information is normal or abnormal.
15. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.
17. A computer program product comprising a computer program comprising computer instructions that instruct a computing device to perform the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057380.4A CN112202638B (en) | 2020-09-29 | 2020-09-29 | Data processing method, device, equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057380.4A CN112202638B (en) | 2020-09-29 | 2020-09-29 | Data processing method, device, equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112202638A CN112202638A (en) | 2021-01-08 |
| CN112202638B true CN112202638B (en) | 2022-03-01 |
Family
ID=74007149
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011057380.4A Active CN112202638B (en) | 2020-09-29 | 2020-09-29 | Data processing method, device, equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112202638B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860509A (en) * | 2021-01-14 | 2021-05-28 | 金茂数字科技有限公司 | Dial testing alarm method and device |
| CN115348615B (en) * | 2022-10-20 | 2022-12-20 | 南京智轩诚网络科技有限公司 | Low-delay 5G communication method based on big data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017114152A1 (en) * | 2015-12-29 | 2017-07-06 | 华为技术有限公司 | Service dial testing method, apparatus and system |
| WO2018063701A1 (en) * | 2016-10-01 | 2018-04-05 | Intel Corporation | Unsupervised machine learning ensemble for anomaly detection |
| CN108028778A (en) * | 2015-07-22 | 2018-05-11 | 动态网络服务股份有限公司 | Generate the mthods, systems and devices of information transmission performance warning |
| CN108270778A (en) * | 2017-12-29 | 2018-07-10 | 中国互联网络信息中心 | A kind of DNS domain name abnormal access detection method and device |
| CN110175114A (en) * | 2019-04-19 | 2019-08-27 | 深圳壹账通智能科技有限公司 | Test terminal, APP and camera compatibility test method and storage medium |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103220190A (en) * | 2013-04-22 | 2013-07-24 | 华为技术有限公司 | Method and device of fault detection |
| US9923910B2 (en) * | 2015-10-05 | 2018-03-20 | Cisco Technology, Inc. | Dynamic installation of behavioral white labels |
| CN106533762B (en) * | 2016-11-17 | 2020-09-08 | 贵州白山云科技股份有限公司 | Equipment switching method and equipment switching device |
| US11062230B2 (en) * | 2017-02-28 | 2021-07-13 | International Business Machines Corporation | Detecting data anomalies |
| CN108737205A (en) * | 2017-04-18 | 2018-11-02 | 中兴通讯股份有限公司 | Dial testing method, apparatus and system |
| US20190102718A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Techniques for automated signal and anomaly detection |
| US20190354850A1 (en) * | 2018-05-17 | 2019-11-21 | International Business Machines Corporation | Identifying transfer models for machine learning tasks |
| CN109391618B (en) * | 2018-10-18 | 2021-09-03 | 平安科技（深圳）有限公司 | Method and system for establishing communication link |
| CN109525736B (en) * | 2018-12-05 | 2022-01-14 | 华为技术服务有限公司 | Voice dial testing loopback method and device |
| CN110740065B (en) * | 2019-10-29 | 2022-04-15 | 中国联合网络通信集团有限公司 | Method, device and system for identifying degradation fault point |
-
2020
- 2020-09-29 CN CN202011057380.4A patent/CN112202638B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108028778A (en) * | 2015-07-22 | 2018-05-11 | 动态网络服务股份有限公司 | Generate the mthods, systems and devices of information transmission performance warning |
| WO2017114152A1 (en) * | 2015-12-29 | 2017-07-06 | 华为技术有限公司 | Service dial testing method, apparatus and system |
| WO2018063701A1 (en) * | 2016-10-01 | 2018-04-05 | Intel Corporation | Unsupervised machine learning ensemble for anomaly detection |
| CN108270778A (en) * | 2017-12-29 | 2018-07-10 | 中国互联网络信息中心 | A kind of DNS domain name abnormal access detection method and device |
| CN110175114A (en) * | 2019-04-19 | 2019-08-27 | 深圳壹账通智能科技有限公司 | Test terminal, APP and camera compatibility test method and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| An Evaluation of Service Composition Technologies Applied to Network Management;Ricardo Lemos Vianna等;《2007 10th IFIP/IEEE International Symposium on Integrated Network Management》;20070625;全文 * |
| 云计算中网络通信服务目标数据高效检测仿真;胡东明等;《计算机仿真》;20180215(第02期);全文 * |
| 基于信息熵技术的网络流量异常监测研究;邓小清;《信息与电脑(理论版)》;20171208(第23期);全文 * |
| 基于深度信念网络的服务可靠性预测方法研究;邱志国;《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》;20190515;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112202638A (en) | 2021-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112202638B (en) | Data processing method, device, equipment and computer storage medium | |
| US9262295B2 (en) | Multiple configuration item status value analysis | |
| CN111224843B (en) | Resource link monitoring method, device, equipment and storage medium | |
| CN111614483A (en) | Link monitoring method and device, storage medium and computer equipment | |
| CN112270399A (en) | Operator registration processing method and device based on deep learning and electronic equipment | |
| Mart et al. | Observability in kubernetes cluster: Automatic anomalies detection using prometheus | |
| CN112559086A (en) | Applet page rendering method and device, electronic equipment and readable storage medium | |
| CN111275011A (en) | Mobile traffic light detection method and device, electronic equipment and storage medium | |
| CN113409284A (en) | Circuit board fault detection method, device, equipment and storage medium | |
| US8073946B1 (en) | Monitoring of metrics to identify abnormalities in a large scale distributed computing environment | |
| CN105574150A (en) | Data processing method and device | |
| CN105515909B (en) | A kind of data acquisition test method and apparatus | |
| JP5208324B1 (en) | Information system management apparatus, information system management method, and program | |
| CN110032505A (en) | Software quality determining device, software quality determine that method and software quality determine program | |
| CN110874311A (en) | Database detection method and device, computer equipment and storage medium | |
| US20120110500A1 (en) | System and method for message grouping | |
| CN112015995A (en) | Data analysis method, device, equipment and storage medium | |
| CN112287922A (en) | Meter reading method, device, equipment and storage medium | |
| US11487746B2 (en) | Business impact analysis | |
| CN110737560B (en) | Service state detection method and device, electronic equipment and medium | |
| CN117424743A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110943887B (en) | Probe scheduling method, device, equipment and storage medium | |
| CN110825951A (en) | Webpage processing method and device and electronic equipment | |
| CN111538656B (en) | Monitoring method, device and equipment for gradient inspection and storage medium | |
| JP2017167578A (en) | Incident management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |