CN112637009B - Data link detection system, method and device - Google Patents
Data link detection system, method and device Download PDFInfo
- Publication number
- CN112637009B CN112637009B CN202011476016.1A CN202011476016A CN112637009B CN 112637009 B CN112637009 B CN 112637009B CN 202011476016 A CN202011476016 A CN 202011476016A CN 112637009 B CN112637009 B CN 112637009B
- Authority
- CN
- China
- Prior art keywords
- data
- consumption
- network node
- information
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Abstract
The embodiment of the specification provides a data link detection system, a method and a device, wherein the data link detection system comprises: a first network node and a second network node; the first network node is configured to collect service source data of a target service, execute a data processing task corresponding to the target service to process the service source data, generate a task execution log of the data processing task and a data processing result, and transmit the task execution log and the data processing result to the second network node; the second network node is configured to perform data consumption on the received data processing result, generate corresponding consumption information, compare the consumption information with log information in the task execution log, and determine a detection result of a data link between the first network node and the second network node according to a comparison result.
Description
Technical Field
The embodiment of the specification relates to the technical field of computers, in particular to a data link detection system. One or more embodiments of the present specification also relate to a data link detection method, a data link detection apparatus, a computing device, and a computer-readable storage medium.
Background
With the development of science and technology, the development of networks is more rapid, the life, modern industrial manufacturing and aspects of users can not be separated from the network environment, and the current network environment is mainly divided into: internet networks and private intranet networks. In short, a private intranet network is generally a network that is not connected to an internet network or is connected to an internet network only in a small part, for example: video networking networks, military system networks, and the like.
Therefore, to implement data transmission across networks, it is usually necessary to use an Object Storage Service (OSS) as an intermediate medium to implement data transmission between a first network and a second network, specifically, source data is processed and calculated in the first network, the calculated data is encrypted and written to the OSS, a data transmission scheduling task acquires a data file stored in the OSS and transmits the data file to the second network through a data transmission channel, and the second network receives the data file and then distributes and consumes data in the data file. However, such a data transmission method across networks cannot monitor the data link for abnormalities, and therefore, an effective method is needed to solve such problems.
Disclosure of Invention
In view of this, the present specification provides a data link detection system. One or more embodiments of the present disclosure also relate to a data link detection method, a data link detection apparatus, a computing device, and a computer-readable storage medium, so as to solve the technical deficiencies in the prior art.
According to a first aspect of embodiments herein, there is provided a data link detection system, comprising:
a first network node and a second network node;
the first network node is configured to collect service source data of a target service, execute a data processing task corresponding to the target service to process the service source data, generate a task execution log of the data processing task and a data processing result, and transmit the task execution log and the data processing result to the second network node;
the second network node is configured to perform data consumption on the received data processing result, generate corresponding consumption information, compare the consumption information with log information in the task execution log, and determine a detection result of a data link between the first network node and the second network node according to a comparison result.
Optionally, the first network node is further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, generating a task execution log of the data processing task, and storing the task execution log into a log list;
determining log update information in the log list and transmitting the log update information to the second network node.
Optionally, the first network node is further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, and generating a task execution log of the data processing task;
and calling a custom function to store the task execution log into a log list.
Optionally, the first network node is further configured to:
and calling a bridging module to pull the log updating information in the log list and transmitting the log updating information to the second network node.
Optionally, the first network node is further configured to:
and calling a bridging module to pull the log update information in the log list, calling an information transmission interface of the second network node, and transmitting the log update information to a detection module of the second network node through the information transmission interface.
Optionally, the second network node includes: the system comprises a detection module and a data consumption module;
the data consumption module configured to: and receiving the data processing result, performing data consumption on the data processing result, and sending the generated consumption information to the detection module.
Optionally, the detection module is configured to:
receiving the task execution log and the consumption information;
comparing the consumption information with the log information in the task execution log, and determining a detection result of a data link between the first network node and the second network node according to a comparison result.
Optionally, the detection module is configured to:
receiving the task execution log and the consumption information;
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
comparing the log identification with the consumption result identification, and determining a detection result of a data link between the first network node and the second network node according to the comparison result.
Optionally, the second network node further includes an alarm module;
the detection module further configured to: sending an abnormal prompt message to the alarm module when the data link is detected to be abnormal;
the alarm module is configured to alarm based on the abnormal prompt information.
Optionally, the detection module is further configured to:
sending data consumption notification information to the data consumption module, and receiving consumption information returned by the data consumption module;
calculating the data consumption duration of the data consumption module according to the consumption information;
and determining a detection result of the data link between the first network node and the second network node according to the size relation between the data consumption duration and a preset data consumption duration threshold value.
Optionally, the detection module is further configured to:
polling and detecting consumption information returned by the data consumption module according to a first preset time period;
and sending abnormal prompt information to the alarm module under the condition that the consumption information returned by the data consumption module is not detected.
According to a second aspect of embodiments herein, there is provided a data link detection method, including:
receiving a task execution log and a data processing result transmitted by a first network node, wherein the task execution log and the data processing result are generated by executing a data processing task corresponding to a target service to process service source data of the target service;
performing data consumption on the data processing result to generate corresponding consumption information;
comparing the consumption information with the log information in the task execution log, and determining a detection result of a data link between the first network node and the second network node according to the comparison result.
Optionally, the comparing the consumption information with the log information in the task execution log includes:
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
and comparing the log identification with the consumption result identification.
Optionally, the data link detection method further includes:
calculating data consumption duration according to the consumption information;
and determining a detection result of the data link between the first network node and the second network node according to the size relation between the data consumption duration and a preset data consumption duration threshold value.
Optionally, the data link detection method further includes:
and under the condition that the data link is detected to be abnormal, generating abnormal prompt information and giving an alarm based on the abnormal prompt information.
Optionally, the data link detection method further includes:
and under the condition that the consumption information is not detected, generating abnormal prompt information and giving an alarm based on the abnormal prompt information.
According to a third aspect of embodiments herein, there is provided a data link detection apparatus, including:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is configured to receive a task execution log and a data processing result transmitted by a first network node, and the task execution log and the data processing result are generated by executing a data processing task corresponding to a target service to process service source data of the target service;
the data consumption module is configured to perform data consumption on the data processing result and generate corresponding consumption information;
and the comparison module is configured to compare the consumption information with log information in the task execution log, and determine a detection result of a data link between the first network node and the second network node according to a comparison result.
According to a fourth aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions that when executed by the processor implement the steps of the data link detection method.
According to a fifth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data link detection method.
In one embodiment of the present description, a first network node executes a data processing task to process service source data, and then generates a task execution log, and synchronizes the generated task execution log to a second network node, which is beneficial to improving efficiency of synchronizing data execution information in a data link across networks.
Drawings
FIG. 1 is a schematic diagram of a data link detection system provided in one embodiment of the present description;
fig. 2 is a process flow diagram of a data link detection method according to an embodiment of the present disclosure;
fig. 3 is a flowchart illustrating a processing procedure of a data link detection method according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of a data link detection apparatus provided in an embodiment of the present specification;
fig. 5 is a block diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.
The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present specification relate are explained.
And OSS: the Object Storage Service (OSS) provides a network-based data access Service for users.
In the present specification, a data link detection system is provided, and the present specification relates to a data link detection method, a data link detection apparatus, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.
Fig. 1 is a schematic diagram illustrating a data link detection system provided according to an embodiment of the present specification, including:
a first network node 102 and a second network node 104;
the first network node 102 is configured to collect service source data of a target service, execute a data processing task corresponding to the target service to process the service source data, generate a task execution log of the data processing task and a data processing result, and transmit the task execution log and the data processing result to the second network node 104;
the second network node 104 is configured to perform data consumption on the received data processing result, generate corresponding consumption information, compare the consumption information with log information in the task execution log, and determine a detection result of a data link between the first network node 102 and the second network node 104 according to a comparison result.
Specifically, the data link detection system provided in the embodiments of the present specification is configured to perform anomaly detection on a data link between the first network node 102 and the second network node 104, so as to implement anomaly detection on the data link across networks.
In practical applications, the target service may be any service that needs to store data across a network, such as a transaction service, a claim settlement service, or a public service; since the service source data, that is, the initial service data of the target service, which has not been subjected to data processing or data statistics, needs to be stored in the first network node 102 to the second network node 104, in order to ensure the data transmission efficiency and the reliability of the data transmission result, the stability of the data link between the first network node 102 and the second network node 104 needs to be detected during the data transmission process.
Taking the target service as a transaction service as an example, the service source data may be transaction times, transaction results, and the like of the commodity in the target time period, after the first network node 102 collects the commodity transaction times and the transaction results of the transaction service in the target time period, the first network node may perform statistics and processing on the commodity transaction times and the transaction results by executing at least one preset data processing task (such as data task 1, data task 2, and data task 3 shown in fig. 1), generate a commodity transaction total number and various transaction result total numbers (commodity transaction success total number and commodity transaction failure total number in the target time period), and generate a task execution log of the data processing task; and then transmits the task execution log and the data processing results (the total number of commodity transactions and the total number of various transaction results in the target time period) to the second network node 104.
After receiving the task execution log and the data processing result, the second network node 104 may first perform data consumption on the data processing result (for example, write the data processing result into a database, or perform secondary processing on the data processing result, etc.), generate consumption information, and compare the consumption information with the task execution log, so as to determine an abnormal detection result of the data link between the first network node 102 and the second network node 104 according to the comparison result.
In particular implementation, the first network node 102 is further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, generating a task execution log of the data processing task, and storing the task execution log into a log list;
determining log update information in the log list and transmitting the log update information to the second network node 104.
Further, the first network node 102 is further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, and generating a task execution log of the data processing task;
and calling a custom function to store the task execution log into a log list.
Specifically, the first network node 102 performs statistics and processing on the service source data of the target service by executing at least one data processing task corresponding to the target service, generates a task execution log of the at least one data processing task, and stores the task execution log in a log list.
In practical application, the user-defined function can be called, the task execution log is processed according to a preset log information storage format, and a processing result is stored in a log list.
For example, the format of the log information stored in the log list is: and if the data processing task name, the data volume, the batch number and the creation time are not the same, the user-defined function can be called to process the task execution log according to the format, and the processing result is stored in the log list as log information.
In addition, after storing the task execution log in the log list, log update information in the log list may be determined, and the log update information may be transmitted to the second network node 104.
The task execution log is processed through the custom function, and the processing result is stored in the log list, so that the writing speed of the log information in the log list is favorably improved, and the network operation performance is favorably improved.
In a specific implementation, the first network node 102 includes a bridging module, configured to transmit the log information in the log list to the second network node 104, that is, the first network node 102 is further configured to: invoking a bridge module to pull the log update information in the log list, and transmitting the log update information to the second network node 104.
Specifically, the bridge module in the first network node 102 may transmit the task execution log generated by executing the data processing task to the second network node 104, so that the first network node 102 may invoke the bridge module to transmit the task execution log to the second network node 104; alternatively, the first network node 102 may call a bridge module to pull the log update information in the log list, and the bridge module transmits the log update information to the second network node 104.
Further, the second network node 104 includes a detection module, configured to implement anomaly detection on a data link between the first network node 102 and the second network node 104, specifically by comparing log information and consumption information, so that the first network node 102 transmits a task execution log to the second network node 104, specifically, the first network node 102 calls the bridge module to transmit the task execution log to the detection module of the second network node 104 through an information transmission interface of the second network node 104; or, the first network node 102 calls a bridge module to pull the log update information in the log list, and transmits the log update information to the detection module of the second network node 104 through the information transmission interface of the second network node 104.
Specifically, the bridging module (bridging) in the first network node transmits the task execution log or the log update information in the log list to the detection module (quality-platform) of the second network node through the public network port of the second network node.
Additionally, the second network node 104 further comprises a data consumption module configured to: and receiving the data processing result, performing data consumption on the data processing result, and sending the generated consumption information to the detection module.
Specifically, the first network node 102 processes or counts the service source data by executing a data processing task, generates a data processing result, and then transmits the data processing result to a data consumption module (Consumer) of the second network node 104, so that the data consumption module performs data consumption on the data processing result (writes the data processing result into a database, or performs secondary processing on the data processing result, and the data consumption module sends the generated consumption information to a detection module of the second network node 104, that is, the data consumption module sends an update message of the consumption state to the detection module (quality-platform) in real time.
Wherein the consumption information includes, but is not limited to: the size of the received data volume, the time of starting consumption, the time of completion of consumption, the batch number, the task name, and the like.
Further, the detection module is configured to:
receiving the task execution log and the consumption information;
comparing the consumption information with the log information in the task execution log, and determining a detection result of the data link between the first network node 102 and the second network node 104 according to the comparison result.
Specifically, the detection module (quality-platform) is responsible for receiving task update messages of the data producing end (log list) of the first network node 102 and the data consuming end (data consuming module) of the second network node 104, and performing judgments such as data volume consistency and task execution exception.
In practical applications, after receiving a task execution log or a log update message transmitted by a bridge module of the first network node 102 and receiving consumption information sent by a data consumption module of the second network node 104, the detection module may compare the consumption information with log information in the task execution log, or compare the consumption information with the log update information, so as to determine an abnormal detection result of the data link according to a comparison result.
As mentioned above, the format of the log information stored in the log list is: the data processing task name + data volume + batch number + creation time, and the consumption information generated by data consumption on the data processing result includes but is not limited to: the size of the received data volume, the time of starting consumption, the time of completion of consumption, the batch number, the task name, and the like. Therefore, after receiving the task execution log and the consumption information, the detection module can directly compare the consumption information with the log information in the task execution log so as to determine the abnormal detection result of the data link according to the comparison result.
Further, the detection module is configured to:
receiving the task execution log and the consumption information;
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
comparing the log identifier with the consumption result identifier, and determining a detection result of the data link between the first network node 102 and the second network node 104 according to the comparison result.
Specifically, after receiving the task execution log and the consumption information, the detection module may further process the task execution log and the consumption information according to the same rule, generate a log identifier corresponding to the task execution log and a consumption result identifier corresponding to the consumption information, and compare the log identifier and the consumption result identifier to determine an abnormal detection result of the data link according to a comparison result.
The rule may be a hash algorithm or a message digest algorithm (MD5 algorithm), and may be determined according to actual requirements, which is not limited herein.
The second network node processes the generated consumption information and the task execution log according to the same rule to generate a corresponding task identifier, and the detection module matches the data link tasks with the same task identifier to compare the data volume or the task state, and determines an abnormal detection result of the data link according to the comparison result, so that the monitoring of the data link across the network is realized.
In this embodiment of the present specification, the second network node 104 further includes an alarm module;
the detection module further configured to: sending an abnormal prompt message to the alarm module when the data link is detected to be abnormal;
the alarm module is configured to alarm based on the abnormal prompt information.
Specifically, the detection module (quality-platform) determines that an alarm is triggered when the data link is abnormal by comparing log information with consumption information or comparing a log identifier with a consumption result identifier, specifically, the detection module (quality-platform) triggers by sending an abnormal prompt message to the alarm module (Alert), and the alarm module receives the alarm information of the detection module and alarms according to a configured mode.
And when the data link is detected to be abnormal, alarming is carried out to provide effective abnormal information for the user, so that the user can be ensured to accurately position the abnormal reason, and the speed of removing the abnormal condition can be increased.
In this embodiment of the present specification, the detection module may further perform anomaly detection on the data link by:
sending data consumption notification information to the data consumption module, and receiving consumption information returned by the data consumption module;
calculating the data consumption duration of the data consumption module according to the consumption information;
determining a detection result of the data link between the first network node 102 and the second network node 104 according to a size relationship between the data consumption duration and a preset data consumption duration threshold.
Specifically, the detection module (quality-platform) determines whether the consumption timeout exists in the data consumption module, and if so, determines that the data link is abnormal.
And determining that the consumption of the data consumption module is overtime under the condition that the data consumption time of the data consumption module is greater than a preset consumption time threshold.
Because the consumption information includes the consumption starting time and the consumption completing time, in practical application, the data consumption duration is calculated according to the consumption information, namely the data consumption duration is calculated according to the consumption starting time and the consumption completing time included in the consumption information, and the data link is determined to have abnormity under the condition that the data consumption duration is determined to be greater than the preset data consumption duration threshold value.
In addition, the detection module may further perform anomaly detection on the data link by:
polling and detecting consumption information returned by the data consumption module according to a first preset time period;
and sending abnormal prompt information to the alarm module under the condition that the consumption information returned by the data consumption module is not detected.
Specifically, the detection module (quality-platform) determines whether the data link has an abnormality by determining whether consumption information returned by the data consumption module is received.
Specifically, after receiving the task execution log transmitted by the first network node 102, the detection module may determine whether the data link is abnormal by determining whether the data consumption module returns consumption information; if the detection module receives the task execution log of the first network node 102 but does not receive the consumption information returned by the data consumption module in the second network node 104, it is determined that the data link is abnormal, and an abnormal prompt message may be sent to the alarm module.
In addition, after receiving the task execution log transmitted by the first network node 102, the detection module includes: the consumption information also includes the size of the received data volume, so the detection module can compare the data volume in the task execution log with the data volume in the consumption information to determine whether the data volume sent by the first network node 102 is consistent with the data volume received by the second network node 104, thereby determining whether the data link is abnormal.
Further, if the data volume of the data received by the detection module from the first network node 102 is inconsistent with the data volume of the data received by the data consumption module in the second network node 104, it is determined that the data link is abnormal, and an abnormal prompt message may be sent to the alarm module.
In one embodiment of the present description, after the first network node executes the data processing task to process the service source data, the generated task execution log is written into the log list, and regularly pull the log update information in the log list and synchronize to the second network node, which is beneficial to improving the efficiency of synchronizing the data execution information in the data link across the network, in addition, the second network node consumes the received data processing result, processes the generated consumption information and the task execution log according to the same rule, generates corresponding task identification, the detection module matches the data link task with the same task identification, compares the data quantity sent or received by the two network nodes or compares the task state of the two network nodes, and determining the abnormal detection result of the data link according to the comparison result, thereby realizing the monitoring of the data link across the network.
Fig. 2 shows a process flow diagram of a data link detection method provided in accordance with one embodiment of the present specification, which includes steps 202 to 206.
And step 204, performing data consumption on the data processing result to generate corresponding consumption information.
Specifically, the data link detection method described in the embodiment of this specification is applied to a second network node, where a first network node collects service source data of a target service, executes a data processing task corresponding to the target service to process the service source data, generates a task execution log of the data processing task and a data processing result, and transmits the task execution log and the data processing result to the second network node.
And performing data consumption on the received data processing result by a second network node to generate corresponding consumption information, comparing the consumption information with log information in the task execution log, and determining a detection result of a data link between the first network node and the second network node according to a comparison result.
In the embodiment of the present specification, comparing the consumption information with the log information in the task execution log may specifically be implemented in the following manner:
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
and comparing the log identification with the consumption result identification.
In specific implementation, the data consumption duration may be calculated according to the consumption information, and the detection result of the data link between the first network node and the second network node may be determined according to a size relationship between the data consumption duration and a preset data consumption duration threshold.
In addition, under the condition that the data link is detected to be abnormal, abnormal prompt information is generated, and an alarm is given based on the abnormal prompt information; alternatively, the first and second electrodes may be,
and under the condition that the consumption information is not detected, generating abnormal prompt information and giving an alarm based on the abnormal prompt information.
After the first network node executes the data processing task to process the service source data, the embodiment of the present specification writes the generated task execution log into a log list, and regularly pulls the log update information in the log list and synchronizes to the second network node, which is beneficial to improving the efficiency of synchronizing the data execution information in the data link of the cross-network, in addition, the second network node consumes the received data processing result, processes the generated consumption information and the task execution log according to the same rule, generates corresponding task identification, the detection module matches the data link task with the same task identification, compares the data quantity sent or received by the two network nodes or compares the task state of the two network nodes, and determining the abnormal detection result of the data link according to the comparison result, thereby realizing the monitoring of the data link across the network.
The above is an exemplary scheme of the data link detection method of this embodiment. It should be noted that the technical solution of the data link detection method and the technical solution of the second network node side in the data link detection system belong to the same concept, and details that are not described in detail in the technical solution of the data link detection method can be referred to the description of the technical solution of the data link detection system.
The following description further describes the data link detection method by taking the application of the data link detection method provided in this specification to a transaction service as an example, with reference to fig. 3. Fig. 3 shows an interaction diagram of a data link detection method provided in an embodiment of the present specification, and specific steps include steps 302 to 318.
Step 302, the first network node collects service source data of the target transaction service from the terminal.
Step 304, the first network node executes a data processing task corresponding to the target transaction service to process the service source data, and generates a task execution log of the data processing task and a data processing result.
Step 306, the first network node calls a custom function to store the task execution log into a log list.
Step 308, the first network node pulls the log update information in the log list and transmits the log update information to the second network node.
Specifically, the first network node calls a bridge module in a node to pull the log update information in the log list, calls an information transmission interface of the second network node, and transmits the log update information to the second network node through the information transmission interface.
In practical application, the second network node includes a detection module, the first network node calls a bridge module in a node to pull the log update information in the log list, calls an information transmission interface of the second network node, and transmits the log update information to the detection module of the second network node through the information transmission interface.
In step 310, the first network node transmits the data processing result to the second network node.
And step 312, the second network node performs data consumption on the data processing result to generate corresponding consumption information.
Specifically, the second network node comprises a detection module, a data consumption module and an alarm module; and the data consumption module receives the data processing result, performs data consumption on the data processing result, and sends the generated consumption information to the detection module.
And step 314, processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier.
Step 316, comparing the log identifier and the consumption result identifier, and determining a detection result of the data link between the first network node and the second network node according to the comparison result.
And step 318, generating abnormal prompt information under the condition that the data link is detected to be abnormal, and alarming to the terminal based on the abnormal prompt information.
Specifically, the detection module sends an abnormality prompt message to the alarm module when detecting that the data link is abnormal; and the alarm module gives an alarm based on the abnormal prompt information.
After the first network node executes the data processing task to process the service source data, the embodiment of the present specification writes the generated task execution log into a log list, and regularly pulls the log update information in the log list and synchronizes to the second network node, which is beneficial to improving the efficiency of synchronizing the data execution information in the data link of the cross-network, in addition, the second network node consumes the received data processing result, processes the generated consumption information and the task execution log according to the same rule, generates corresponding task identification, the detection module matches the data link task with the same task identification, compares the data quantity sent or received by the two network nodes or compares the task state of the two network nodes, and determining the abnormal detection result of the data link according to the comparison result, thereby realizing the monitoring of the data link across the network.
Corresponding to the above method embodiment, this specification further provides an embodiment of a data link detection apparatus, and fig. 4 shows a schematic diagram of a data link detection apparatus provided in an embodiment of this specification. As shown in fig. 4, the apparatus includes:
a receiving module 402, configured to receive a task execution log and a data processing result transmitted by a first network node, where the task execution log and the data processing result are generated by executing a data processing task corresponding to a target service to process service source data of the target service;
a data consumption module 404 configured to perform data consumption on the data processing result to generate corresponding consumption information;
a comparison module 406, configured to compare the consumption information with log information in the task execution log, and determine a detection result of a data link between the first network node and the second network node according to a comparison result.
Optionally, the alignment module 406 includes:
the processing submodule is configured to process the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
and the comparison submodule is configured to compare the log identifier with the consumption result identifier.
Optionally, the data link detection apparatus further includes:
a calculation module configured to calculate a data consumption duration according to the consumption information;
a detection result determining module configured to determine a detection result of the data link between the first network node and the second network node according to a size relationship between the data consumption duration and a preset data consumption duration threshold.
Optionally, the data link detection apparatus further includes:
the first alarm module is configured to generate abnormal prompt information and alarm based on the abnormal prompt information under the condition that the data link is detected to be abnormal.
Optionally, the data link detection apparatus further includes:
and the second alarm module is configured to generate abnormal prompt information and alarm based on the abnormal prompt information under the condition that the consumption information is not detected.
The above is a schematic scheme of the data link detection apparatus of this embodiment. It should be noted that the technical solution of the data link detection apparatus and the technical solution of the data link detection method belong to the same concept, and details that are not described in detail in the technical solution of the data link detection apparatus can be referred to the description of the technical solution of the data link detection method.
FIG. 5 illustrates a block diagram of a computing device 500 provided in accordance with one embodiment of the present description. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is coupled to memory 510 via bus 530, and database 550 is used to store data.
Computing device 500 also includes access device 540, access device 540 enabling computing device 500 to communicate via one or more networks 560. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 540 may include one or more of any type of network interface, e.g., a Network Interface Card (NIC), wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 500, as well as other components not shown in FIG. 5, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 5 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.
Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 500 may also be a mobile or stationary server.
Wherein the memory 510 is used for storing computer executable instructions, and the processor 520 is used for implementing the steps of the data link detection method when executing the computer executable instructions.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data link detection method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data link detection method.
An embodiment of the present specification also provides a computer readable storage medium storing computer instructions which, when executed by a processor, are used for implementing the steps of the data link detection method.
The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data link detection method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data link detection method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, and software distribution medium, etc. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.
Claims (19)
1. A data link detection system across a network, comprising:
a first network node and a second network node;
the first network node is configured to collect service source data of a target service, execute a data processing task corresponding to the target service to process the service source data, generate a task execution log of the data processing task and a data processing result, and transmit the task execution log and the data processing result to the second network node;
the second network node is configured to perform data consumption on the received data processing result, generate corresponding consumption information, compare the consumption information with log information in the task execution log, determine a detection result of a data link between the first network node and the second network node according to the comparison result, and generate abnormal prompt information and alarm based on the abnormal prompt information under the condition that the consumption information is not detected.
2. The system for data link detection across a network of claim 1, the first network node further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, generating a task execution log of the data processing task, and storing the task execution log into a log list;
determining log update information in the log list and transmitting the log update information to the second network node.
3. The system for data link detection across a network according to claim 1 or 2, the first network node further configured to:
executing at least one data processing task corresponding to the target service to process the service source data, and generating a task execution log of the data processing task;
and calling a custom function to store the task execution log into a log list.
4. The system of data link detection across a network of claim 2, the first network node further configured to:
and calling a bridging module to pull the log updating information in the log list and transmitting the log updating information to the second network node.
5. The system for data link detection across a network of claim 4, the first network node further configured to:
and calling a bridging module to pull the log update information in the log list, calling an information transmission interface of the second network node, and transmitting the log update information to a detection module of the second network node through the information transmission interface.
6. The system for data link detection across a network of claim 1, the second network node, comprising: the system comprises a detection module and a data consumption module;
the data consumption module configured to: and receiving the data processing result, performing data consumption on the data processing result, and sending the generated consumption information to the detection module.
7. The system of claim 6, the detection module configured to:
receiving the task execution log and the consumption information;
comparing the consumption information with the log information in the task execution log, and determining a detection result of a data link between the first network node and the second network node according to the comparison result.
8. The system of claim 6, the detection module configured to:
receiving the task execution log and the consumption information;
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
comparing the log identification with the consumption result identification, and determining a detection result of a data link between the first network node and the second network node according to the comparison result.
9. The system for data link detection across a network of claim 8, the second network node, further comprising an alarm module;
the detection module further configured to: sending an abnormal prompt message to the alarm module when the data link is detected to be abnormal;
the alarm module is configured to alarm based on the abnormal prompt information.
10. The system for data link detection across a network of claim 6, the detection module further configured to:
sending data consumption notification information to the data consumption module, and receiving consumption information returned by the data consumption module;
calculating the data consumption duration of the data consumption module according to the consumption information;
and determining a detection result of the data link between the first network node and the second network node according to the size relation between the data consumption duration and a preset data consumption duration threshold value.
11. The system for data link detection across a network of claim 6, the detection module further configured to:
polling and detecting consumption information returned by the data consumption module according to a first preset time period;
and sending abnormal prompt information to an alarm module under the condition that the consumption information returned by the data consumption module is not detected.
12. A cross-network data link detection method is applied to a second network node and comprises the following steps:
receiving a task execution log and a data processing result transmitted by a first network node, wherein the task execution log and the data processing result are generated by executing a data processing task corresponding to a target service to process service source data of the target service;
performing data consumption on the data processing result to generate corresponding consumption information;
comparing the consumption information with log information in the task execution log, determining a detection result of a data link between the first network node and the second network node according to the comparison result, and generating abnormal prompt information and giving an alarm based on the abnormal prompt information under the condition that the consumption information is not detected.
13. The method of claim 12, wherein comparing the consumption information to log information in the task execution log comprises:
processing the log information and the consumption information in the task execution log according to a preset rule to generate a corresponding log identifier and a consumption result identifier;
and comparing the log identification with the consumption result identification.
14. The method of data link detection across a network of claim 12, further comprising:
calculating data consumption duration according to the consumption information;
and determining a detection result of the data link between the first network node and the second network node according to the size relation between the data consumption duration and a preset data consumption duration threshold value.
15. The method of data link detection across a network according to any one of claims 12 to 14, further comprising:
and under the condition that the data link is detected to be abnormal, generating abnormal prompt information and giving an alarm based on the abnormal prompt information.
16. The method of data link detection across a network of claim 12, further comprising:
and under the condition that the consumption information is not detected, generating abnormal prompt information and giving an alarm based on the abnormal prompt information.
17. An apparatus for data link detection across a network, comprising:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is configured to receive a task execution log and a data processing result transmitted by a first network node, and the task execution log and the data processing result are generated by executing a data processing task corresponding to a target service to process service source data of the target service;
the data consumption module is configured to perform data consumption on the data processing result and generate corresponding consumption information;
and the comparison module is configured to compare the consumption information with log information in the task execution log, determine a detection result of a data link between the first network node and the second network node according to the comparison result, and generate abnormal prompt information and give an alarm based on the abnormal prompt information under the condition that the consumption information is not detected.
18. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the steps of the method for detecting a data link across a network according to any one of claims 12 to 16.
19. A computer readable storage medium storing computer instructions which, when executed by a processor, carry out the steps of the method of data link detection across a network according to any one of claims 12 to 16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476016.1A CN112637009B (en) | 2020-12-15 | 2020-12-15 | Data link detection system, method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476016.1A CN112637009B (en) | 2020-12-15 | 2020-12-15 | Data link detection system, method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112637009A CN112637009A (en) | 2021-04-09 |
| CN112637009B true CN112637009B (en) | 2022-08-12 |
Family
ID=75313339
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011476016.1A Active CN112637009B (en) | 2020-12-15 | 2020-12-15 | Data link detection system, method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112637009B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114584490B (en) * | 2022-03-25 | 2024-04-09 | 阿里巴巴（中国）有限公司 | Data transmission detection method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103812705A (en) * | 2014-02-26 | 2014-05-21 | 国家电网公司 | Computer generating method for internetwork fault analysis-location mechanism |
| CN105706393A (en) * | 2013-11-05 | 2016-06-22 | 瑞典爱立信有限公司 | Method and system of supporting operator commands in link aggregation group |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8644137B2 (en) * | 2006-02-13 | 2014-02-04 | Cisco Technology, Inc. | Method and system for providing safe dynamic link redundancy in a data network |
| US9892240B2 (en) * | 2015-04-01 | 2018-02-13 | Telefonaktiebolaget Lm Ericsson (Publ) | System and method for providing a content consumption journal to users in a multi-device environment |
| CN110502426A (en) * | 2019-07-08 | 2019-11-26 | 中国工商银行股份有限公司 | The test method and device of distributed data processing system |
| CN111711544B (en) * | 2020-05-15 | 2021-11-09 | 北京奇艺世纪科技有限公司 | Link dial testing method and device, electronic equipment and storage medium |
-
2020
- 2020-12-15 CN CN202011476016.1A patent/CN112637009B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105706393A (en) * | 2013-11-05 | 2016-06-22 | 瑞典爱立信有限公司 | Method and system of supporting operator commands in link aggregation group |
| CN103812705A (en) * | 2014-02-26 | 2014-05-21 | 国家电网公司 | Computer generating method for internetwork fault analysis-location mechanism |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112637009A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109558748B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109861856B (en) | Method and device for notifying system fault information, storage medium and computer equipment | |
| CN110932892B (en) | Block chain based information early warning method and device, related nodes and storage medium | |
| CN111343009B (en) | Service alarm notification method and device, storage medium and electronic equipment | |
| CN111026749B (en) | Service alarm method and device | |
| CN112637009B (en) | Data link detection system, method and device | |
| CN110875841A (en) | Alarm information pushing method and device and readable storage medium | |
| CN114138500B (en) | Resource scheduling system and method | |
| CN111327466A (en) | Alarm analysis method, system, equipment and medium | |
| CN115001989A (en) | Equipment early warning method, device, equipment and readable storage medium | |
| CN109766198B (en) | Stream processing method, device, equipment and computer readable storage medium | |
| CN108667826B (en) | Scheduling device and scheduling method based on four-mode heterogeneous redundant processor | |
| CN111949418B (en) | Subscription message processing method and device | |
| CN116684256B (en) | Node fault monitoring method, device and system, electronic equipment and storage medium | |
| CN112783618A (en) | Task scheduling monitoring system, computer equipment and storage medium | |
| CN116483663A (en) | Abnormality warning method and device for platform | |
| CN113422739B (en) | Data transmission method, sending end and data transmission system | |
| CN115168139A (en) | Early warning method and system based on monitoring prediction | |
| CN113297039A (en) | Data monitoring method and device | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN113014587A (en) | API detection method and device, electronic equipment and storage medium | |
| CN111258845A (en) | Detection of event storms | |
| CN111382035A (en) | Global matching device and method for alarm triggering rules of operation and maintenance system | |
| CN116016265B (en) | Message all-link monitoring method, device, system, equipment and storage medium | |
| CN114676166B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |