CN114567536A

CN114567536A - Abnormal data processing method and device, electronic equipment and storage medium

Info

Publication number: CN114567536A
Application number: CN202210172222.6A
Authority: CN
Inventors: 张扬扬
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-02-24
Filing date: 2022-02-24
Publication date: 2022-05-31
Anticipated expiration: 2042-02-24
Also published as: CN114567536B

Abstract

The disclosure provides an abnormal data processing method and device, electronic equipment and a storage medium, and relates to the field of data processing, in particular to the field of the Internet of things. The specific implementation scheme is as follows: carrying out anomaly detection on a target node, and determining an abnormal node and an anomaly type of the abnormal node; determining a corresponding exception handling type according to the node type and the exception type of the exception node; and processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node. The embodiment of the disclosure can improve the exception handling efficiency and improve the forwarding accuracy.

Description

Abnormal data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing, and in particular, to the field of internet of things, and more particularly, to an abnormal data processing method and apparatus, an electronic device, and a storage medium.

Background

In recent years, Internet of things (IoT) technology that can access various devices or goods to a network has received much attention.

Under the scene of the internet of things, a user can use the rule engine service to process data uploaded to the internet of things platform by the internet of things equipment through configuration rules and further forward the data to the specified data destination service.

Disclosure of Invention

The disclosure provides an abnormal data processing method, an abnormal data processing device, an electronic device and a storage medium.

According to an aspect of the present disclosure, there is provided an exception data processing method, including:

carrying out anomaly detection on a target node, and determining an abnormal node and an anomaly type of the abnormal node;

determining a corresponding exception handling type according to the node type and the exception type of the exception node;

and processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node.

According to an aspect of the present disclosure, there is provided an exception data processing apparatus including:

the node abnormal type determining module is used for carrying out abnormal detection on the target node and determining an abnormal node and the abnormal type of the abnormal node;

the abnormal processing type determining module is used for determining a corresponding abnormal processing type according to the node type and the abnormal type of the abnormal node;

and the abnormal sending data processing module is used for processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of exception data handling according to any of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of processing abnormal data according to any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of exception data handling of any one of the embodiments of the present disclosure.

The embodiment of the disclosure can improve the exception handling efficiency and the forwarding accuracy.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of an exception data handling method disclosed in accordance with an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of another disclosed exception data handling method in accordance with an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of another disclosed exception data handling method in accordance with an embodiment of the present disclosure;

fig. 4 is a schematic diagram of an internet of things platform system disclosed in an embodiment of the present disclosure

FIG. 5 is a schematic diagram of an exception data handling apparatus according to an embodiment of the present disclosure;

FIG. 6 is a block diagram of an electronic device for implementing an exception data handling method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of embodiments of the present disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a flowchart of an abnormal data processing method disclosed in an embodiment of the present disclosure, and this embodiment may be applied to a case where, in a case where data is forwarded to a target node, whether an abnormality exists in the data forwarding target node is detected. The method of this embodiment may be executed by an abnormal data processing apparatus, which may be implemented in a software and/or hardware manner, and is specifically configured in an electronic device with a certain data operation capability, where the electronic device may be a server device.

S101, carrying out abnormity detection on the target node, and determining an abnormal node and an abnormal type of the abnormal node.

The electronic equipment provided by the embodiment of the disclosure is used for forwarding data to a target node. The target node is used for processing the data. In fact, the target node is a node corresponding to a destination of the electronic device to forward the data. The electronic equipment comprises a control module and a data module. The control module is used for configuring data sending rules by a user, and one data sending rule comprises a data source, a data processing logic statement, a data forwarding destination and the like. The data source refers to a source node of data. The source node may be a device accessing the internet of things, for example, an intelligent sound box, an intelligent kitchen appliance, or a camera. The data processing logic statement is used for processing data, wherein the data processing logic statement can be null, that is, direct forwarding is not processed. The data forwarding destination determines to which target node the data is to be sent. The data module is used for processing the data according to a data sending rule configured in advance in the control module, specifically receiving the data from a data source, executing a corresponding processing logic statement, and forwarding the corresponding processing logic statement to a corresponding destination node.

And carrying out anomaly detection on the target node, wherein the anomaly detection is used for detecting whether the data can be normally forwarded to the target node. An abnormal node may refer to a node that cannot normally send data to arrive. The exception type may refer to a type of an exception cause that data cannot be normally transmitted. Illustratively, the exception type may include a network type, specifically, a type in which data cannot be normally transmitted due to a problem occurring in a network between the electronic device and the target node.

For example, in an application scenario of the internet of things, the electronic device may be a platform system of the internet of things, and the target node may be a database server or the like. The internet of things equipment collects data, the data are sent to the electronic equipment through the access equipment (such as a switch), the electronic equipment processes or does not process (transparently transmit) the received data and forwards the received data to the target node, and the target node can be used for further processing or storing the forwarded data.

And S102, determining a corresponding exception handling type according to the node type and the exception type of the exception node.

The exception handling type may refer to a type of an exception handling manner, and is used to determine a handling manner of data of the abnormal node. And correspondingly configuring different exception handling modes aiming at different node types and different exception types. For example, the exception type is a network type, and the network may be generally waited to return to normal, so that the exception handling manner corresponding to the exception handling type may be to wait for a period of time to resend the data. For example, the exception type is a self exception of the target node, and it may be specifically determined whether rapid repair is possible in a short period according to the type of the node, and in the case that rapid repair is not possible, the exception handling manner corresponding to the exception handling type may be to notify a user who forwards data that the data cannot be normally sent. In addition, there are other cases where the configuration may be performed according to a specific application scenario.

S103, processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node.

The data to be sent corresponding to the abnormal node is the data for sending the abnormal node. In general, the electronic device cannot handle the exception of the target node, and can only handle the data of the transmission target node.

In the prior art, data sent to a target node is abnormal, which can cause the loss of the sent data; if the abnormal condition occurs for a long time, the electronic equipment initiates multiple unnecessary forwarding operations, and the service resources are wasted; the user can not perceive some exceptions needing to be processed by the user, so that the data destination forwarding exception can not be recovered all the time.

According to the technical scheme disclosed by the invention, the abnormal node and the abnormal type are determined by carrying out abnormality detection on the target node, the processing mode of the data to be sent of the abnormal node is determined according to the abnormal type and the node type, the data sent of the abnormal node is subjected to abnormal processing quickly and timely, and the corresponding abnormal processing mode can be carried out aiming at different abnormal scenes, so that the efficiency and the accuracy of data processing are improved, the data loss is reduced, the waste of data sending resources is reduced, and the stability of data sending is improved.

Fig. 2 is a flowchart of another abnormal data processing method disclosed in an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and may be combined with the above optional embodiments. The exception handling type includes an automatic handling type or a manual handling type.

S201, carrying out abnormity detection on the target node, and determining an abnormal node and an abnormal type of the abnormal node.

S202, determining a corresponding exception handling type according to the node type and the exception type of the exception node, wherein the handling type comprises an automatic handling type or a manual handling type.

The automatic processing type is that the electronic equipment directly processes data to be transmitted according to a processing mode corresponding to a preset automatic processing type without manual intervention, and meanwhile, a user does not sense the data. Illustratively, the processing mode corresponding to the automatic processing type may be data retransmission, data discarding, data saving, and the like. The manual processing type means that manual intervention is needed, the electronic device prompts the user about the abnormality and also prompts the reason of the abnormality so that the user can know the abnormality more and more carefully and correctly process the data to be sent, and the electronic device can process the data to be sent according to the operation input by the user aiming at the abnormality processing type.

And S203, processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node.

Optionally, the processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node includes: and under the condition that the exception processing type corresponding to the exception node comprises an automatic processing type, retransmitting the data to be transmitted corresponding to the exception node.

The automatic processing type may be an exception processing type for exceptions that may be quickly repaired. At this time, the data to be sent can be resent, so that the data to be sent is sent to the abnormal node in time after the sending abnormality is solved, and the data sending accuracy is improved.

Wherein the retransmission may be a periodic multiple transmission. In order to avoid wasting too much service resources, the number of retransmissions is limited, for example, 10. And if the retransmission fails, continuing to retransmit the next time until the retransmission times are more than or equal to the preset time threshold. When the number of times of retransmission is greater than or equal to the preset number threshold, the data to be transmitted may be discarded, or the data to be transmitted, information (such as a source node) associated with the data to be transmitted, and information about an abnormal cause and the like may be combined according to an error node configured in advance by a user and forwarded to the error node. By forwarding the wrong node, data and abnormal information can be reserved as much as possible, and a user can conveniently recover data, troubleshoot problems and the like.

By configuring the automatic processing type, the data to be sent corresponding to the abnormal node is retransmitted to the abnormal node, and the data to be sent can be sent to the abnormal node in time aiming at the abnormal node which is quickly repaired, so that the data sending efficiency and success rate are improved, and the stability and fault tolerance of the electronic equipment for sending the data are improved.

Optionally, the processing, according to the exception handling type corresponding to the exception node, the data to be sent corresponding to the exception node includes: under the condition that the exception processing type corresponding to the exception node comprises a manual processing type, modifying the state of the exception node into an exception state, notifying a user, and stopping sending data to be sent corresponding to the exception node; and under the condition that the state of the abnormal node is updated to be a normal state, retransmitting the data to be transmitted corresponding to the abnormal node.

The electronic device records states of each target node and each abnormal node, and the states may include a normal state or an abnormal state. The state of the abnormal node may include a normal state or an abnormal state. The state of the target node is usually a normal state, and the target node in the general abnormal state is an abnormal node. The abnormal state can be read and displayed so that a user can perceive that an abnormal condition exists when data is sent to the abnormal node. And the notification user is used for prompting the user to quickly know that the data to be sent cannot normally send the abnormal node. The manual processing type of exception condition is an exception that cannot be quickly repaired, and at this time, stopping sending data may mean not sending data to be sent, and caching the data to be sent locally, so as to reduce waste of service resources and wait for a user to complete exception handling. And the state of the abnormal node is updated to be a normal state, which indicates that the existing abnormality is processed and completed, and the data can be continuously sent to the abnormal node. At this time, the electronic device may resend the data to be sent, so as to implement stable and accurate sending of the data to be sent.

The updating of the state of the abnormal node to the normal state may be receiving a modification operation of a user for the state, or detecting a re-edit of the data sending rule, and updating the state of the abnormal node corresponding to the data sending rule to the normal state. Further, an event that another state is updated to the normal state may be detected, and the setting may be performed as necessary.

In one particular example, an electronic device includes a control module and a data module. In the control module, a data transmission rule further includes a rule status and a destination status. The rule state and the destination state are used for the user to perceive the behavior of the data transmission rule. Rule states include disabled, running, abnormal or blocked, etc.; the destination state includes normal or abnormal, and the abnormal state adds related abnormal information, such as abnormal reason. The multiple data sending rules correspond to the same target node. The destination state may refer to a state of a target node to which the data transmission rule corresponds. And when the state of one abnormal node is in an abnormal state, determining the rule state of the data sending rule corresponding to the abnormal node as the abnormal state. In addition, the rule state such as disable or block is a state determined according to other events, and can be modified manually.

The user can read the rule state and the destination state from the control module, determine that abnormal conditions exist when data are sent to the abnormal node, and intervene in processing in time. Specifically, the data to be sent may be sent to the abnormal node, and the rule state and the destination state of the corresponding data sending rule are set as abnormal states. Meanwhile, short message service is called to inform the user of corresponding abnormal information, the abnormal information is displayed in a page of the control module, and the data module stops data forwarding operation of corresponding abnormal nodes. After the user finishes processing the exception, the data sending rule is edited again, so that the rule state of the data sending rule is updated to be the running state, and the destination state is updated to be the normal state, and the data module can retransmit the data to the corresponding exception node.

The state of the abnormal node is updated by configuring the manual processing type, the user is informed to process the abnormality in time, the data to be sent is sent to the abnormal node again under the condition that the abnormal state is updated to the normal state, the abnormal node which cannot be repaired in time can be informed to intervene in time to process the abnormality, and the data to be sent is sent to the abnormal node under the condition that the abnormal processing is completed, so that unnecessary resource consumption is avoided, resource waste is reduced, the accuracy of data sending is improved, and the stability and fault tolerance of the electronic equipment for sending the data are improved.

Optionally, the determining a corresponding exception handling type according to the node type and the exception type of the exception node includes: and determining the exception handling type of the exception node according to the node type and the exception type of the exception node and the corresponding relation between the preset type and the exception handling type, wherein the node type comprises an authority type or an unauthorized type, and the exception type comprises a network type, a service type or a user behavior causing type.

The exception type and the node type are used together to determine an exception handling type of the exception node. The authority can be understood as whether the authority has the operation and maintenance performance. The target node with permission type can be a node with high exception recovery speed and good operation and maintenance performance. Specifically, the target node with the permission type generally has a corresponding technical team to carry out operation and maintenance, has more perfect operation and maintenance mechanisms such as monitoring and alarming, can generally recover in a short time after the service is abnormal, does not need a user to care about the operation and maintenance, and generally does not last too long for the corresponding abnormal data forwarding condition. And the target node without permission type can refer to a node with slow abnormal repair speed and poor operation and maintenance performance. Specifically, the monitoring operation and maintenance capability of the target node of the no-permission type is generally poor, and after the service is abnormal, the user needs to be reminded of recovering the corresponding service in time, and the recovery generally needs a long time.

The network type generally indicates that a network anomaly exists between the electronic device and the target node, for example, the response time is greater than or equal to a preset time threshold. The service type indicates that the target node itself has an exception, e.g., service unavailable. The user behavior causing type represents an abnormality of the target node due to a behavior that the user acts as or does not act, such as resource deletion, insufficient written quota and the like. The resource deletion may refer to deletion of a database or a topic (topic) to be written in the abnormal node by the data to be sent. In addition, part of data to be sent needs to be written into the abnormal node according to the writing amount, and the insufficient writing amount can mean that the remaining writing amount is insufficient to write the data to be sent, so that the data to be sent cannot be written into the abnormal node, and the data to be sent fails to be sent.

And the corresponding relation between the preset type and the exception handling type is used for determining the corresponding exception handling type according to the exception type and the node type. Illustratively, under the authorized type, the network type and the service type correspond to an automatic processing type, and the user behavior causing type corresponds to a manual processing type; under the no-permission type, the network type corresponds to an automatic processing type, and the service type and the user behavior causing type correspond to a manual processing type. It can be understood that the network type is a network exception, and generally, regardless of the operation and maintenance performance of the node, the network type can be quickly repaired by waiting for a period of time, and an automatic processing type exception handling mode can be adopted for data processing. The service type abnormity repairing time depends on the operation and maintenance performance of the node, so that the service type abnormity under the authorized type can be quickly repaired and corresponds to the automatic processing type; the service type abnormity under the no-permission type can not be quickly repaired, and the service type abnormity corresponds to a manual processing type. And the abnormity of the type caused by the user behavior can not be quickly repaired, and the abnormity is associated with the user behavior and can only be processed through manual intervention, so that the type caused by the user behavior corresponds to the manual processing type no matter the node type.

By configuring the contents of the node type and the exception type and combining the two types, the exception handling type is determined, different handling can be accurately carried out respectively aiming at different exceptions and the performances of different nodes, the accuracy of exception handling is improved, and therefore exceptions are rapidly solved.

According to the technical scheme, the exception handling type is divided into the automatic handling type and the manual handling type, the data to be sent can be handled, the handling accuracy of the data sent abnormally is improved, meanwhile, the user can be prompted in time, so that the user can handle the exception in time, the exception handling efficiency is improved, and the waste of processing resources for the exception which cannot be handled automatically is reduced.

Fig. 3 is a flowchart of another abnormal data processing method disclosed in an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and may be combined with the above optional embodiments. The anomaly detection of the target node is embodied as: determining an abnormal detection mode corresponding to the target node according to the connection type of the target node; and carrying out anomaly detection on the target node by adopting an anomaly detection mode corresponding to the target node.

S301, according to the connection type of the target node, determining an abnormal detection mode corresponding to the target node.

The connection type is used for determining a connection mode between the electronic equipment and the target node. The connection type may include a long connection or a short connection. Different connection modes correspond to different anomaly detection modes.

S302, performing anomaly detection on the target node by adopting an anomaly detection mode corresponding to the target node, and determining an anomalous node and an anomaly type of the anomalous node.

Under the condition that the electronic equipment establishes long connection with a target node, a multi-layer intermediate network exists in a network between the electronic equipment and the target node, if the electronic equipment only performs connection layer abnormity detection, the service availability of an application layer (or also called a service layer) is not ensured, and the abnormity detection of the application layer is required. Therefore, under the condition that the long connection is established between the electronic equipment and the target node, the application layer abnormity detection is periodically carried out, and the long connection abnormity detection result is obtained. The destination node establishing the long connection may comprise a Kafka system. The anomaly detection of the application layer may be to periodically send data and detect whether the function of the application layer is normal or correct. Periodic detection is also used to detect whether a long connection is available.

Under the condition that the electronic device establishes short connection with a target node, data to be sent is generally acquired, a data sending rule is inquired, and the target node corresponding to the data to be sent is determined. And sending the data to be sent to a target node, and determining the short connection abnormity detection result of the target node according to the information fed back by the target node or the counted response time. If the data sending rule needs to process the data to be sent, a processing step of the data to be sent is added in the previous step. The target node for establishing the short connection may include a Time Series Database (TSDB) or a function computing system, etc.

And determining whether the target node is an abnormal node according to the long connection abnormality detection result or the short connection abnormality detection result, and determining the abnormality type according to the content of the abnormality detection result.

S303, determining a corresponding exception handling type according to the node type and the exception type of the exception node.

S304, processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node.

Optionally, the data to be sent includes: data of the Internet of things; the method further comprises the following steps: receiving the data of the Internet of things, and acquiring forwarding information corresponding to the data of the Internet of things, wherein the forwarding information comprises a target node corresponding to the data of the Internet of things; and processing the data of the Internet of things according to the forwarding information, and forwarding the data of the Internet of things to a corresponding target node.

The data to be sent are data of the Internet of things. The embodiment of the disclosure is applied to the application scene of the Internet of things. The electronic device may be an internet of things platform system. And the Internet of things platform system receives the data acquired by the Internet of things equipment and forwards the data through the access equipment. Forwarding information corresponding to the data of the Internet of things is used for forwarding the data of the Internet of things. And determining forwarding information corresponding to the data of the Internet of things according to a data sending rule corresponding to the data of the Internet of things. The data sending rule comprises a source node (namely the Internet of things equipment) and a target node, and also comprises a data processing logic statement. And inquiring a data sending rule corresponding to the data of the Internet of things according to the source node of the data of the Internet of things, analyzing a target node in the corresponding data sending rule, and determining the target node as forwarding information corresponding to the data of the Internet of things. And if the data processing logic statement exists, determining the target node and the data processing logic statement as forwarding information corresponding to the data of the Internet of things. And acquiring an abnormal detection result of the target node, processing the data of the Internet of things according to the data processing logic statement in the forwarding information under the condition that the target node is a non-abnormal node, and forwarding the data of the Internet of things to the target node in the forwarding information.

And carrying out exception processing on the data of the Internet of things under the condition that the target node is an exception node. Illustratively, when the target node is an abnormal node and the exception handling type is an automatic handling type, the data of the internet of things is handled according to the data handling logic statement in the forwarding information and forwarded to the target node in the forwarding information, and when the data of the internet of things fails to be sent, the data of the internet of things is pre-stored for subsequent resending. Meanwhile, under the condition that the target node is an abnormal node and the abnormal processing type is an automatic processing type, the pre-stored data of the Internet of things is retransmitted for a limited time.

And under the condition that the target node is an abnormal node and the abnormal processing type is a manual processing type, stopping sending the Internet of things data, informing a user, modifying the state of the abnormal node into an abnormal state, caching the Internet of things data, and sending prestored Internet of things data to the abnormal node under the condition that the abnormal state is modified into a normal state. And re-determining the abnormal node as the target node, wherein the target node is equivalent to processing newly received data of the internet of things according to the data processing logic statement in the forwarding information under the condition that the target node is a non-abnormal node, and forwarding the newly received data of the internet of things to the target node in the forwarding information.

By applying the abnormal data processing method to the application scene of the internet of things data forwarding, the forwarding accuracy of the internet of things data can be improved, and the stability of the internet of things forwarding system can be improved.

According to the technical scheme, the abnormal detection mode of the target node is determined according to the connection type between the target node and the target node, different abnormal detection modes can be determined according to different connection modes, accurate abnormal detection is achieved, the accuracy rate of abnormal data processing is improved, abnormality is timely detected, and the instantaneity of abnormal detection is improved.

Fig. 4 is a scene diagram of an internet of things platform system disclosed according to an embodiment of the present disclosure. The abnormal data processing method can be applied to an Internet of things platform system, and the electronic equipment is the Internet of things platform system.

As shown in fig. 4, the internet of things platform system includes a control module 401 and a data module 402. The control module 401 is configured with data transmission rules in advance. Each data sending rule is provided with a data source, a data processing logic statement and a target node. Wherein the target node may be replaced with the error node. The data transmission rule is also configured with a rule state and the target node is also configured with a state, i.e. a destination state. The data source refers to a source of the internet of things device data to be received. The data processing logic statement is used for specifying data processing logic, namely how to process, for example, calculate, the data of the internet of things. The target node is used for appointing a forwarding destination of the processed data of the Internet of things. The target nodes may include target nodes of an authorized type or target nodes of an unauthorized type. The error node is used for designating a destination to which some internet of things data with abnormal forwarding can be sent, for example, the internet of things data which fails to be retransmitted for multiple times. The rule state may be determined based on the state of the target node. When the state of the target node is an abnormal state, the rule state is an abnormal state; and under the condition that the state of the target node is a normal state, the rule state is a running state. The rule state may also be determined based on other parameters, which are not particularly limited.

The internet of things device 403 sends the internet of things data to the message center 404, and the message center 404 sends the internet of things data to the data receiving module 405 in the data module 402. Where the message center 404 may be a message queue, such as a broker message queue. The data receiving module 405 in the data module 402 sends the received data of the internet of things to the data processing and forwarding module 406, and the data processing and forwarding module 406 correspondingly processes the data of the internet of things according to the data sending rule corresponding to the internet of things in the control module 401 and forwards the data of the internet of things to the target node 407 or the error node 408. The data processing and forwarding module 406 may perform exception detection on the target node 407 in each data sending rule, determine a node type and an exception type of the exception node, determine an exception handling type, and perform exception handling on the data of the internet of things.

The node type of the target node may be preconfigured in the control module 401. The node type includes a privileged type or an unauthorized type. Illustratively, the target nodes of the authoritative type may include Kafka systems, TSDBs, or function computing systems deployed in a specified public cloud. The target node of the no-permission type may include a Kafka system or an http system or the like deployed in a system other than the specified public cloud. In fact, the system of the public cloud deployment is appointed, the operation and maintenance are carried out by a corresponding technical team, a relatively perfect operation and maintenance mechanism such as monitoring and alarming is provided, after the service is abnormal, the system can be recovered in a short time, a user does not need to care about the operation and maintenance, and the corresponding abnormal data forwarding condition generally lasts for a short time. The monitoring operation and maintenance capability of a system deployed in a network system except a specified public cloud is generally poor, and after a service is abnormal, a user needs to be reminded of recovering the corresponding service in time, and the service generally needs a long time to be recovered. Therefore, the service exception of the target node without the authority type requires some processing modes of exception processing types different from the service exception of the target node with the authority type to process the data to be sent.

The connection type of the connection established between the platform system of the internet of things and the target node may include a short connection type or a long connection type. The connection type may be determined based on the target node. Illustratively, the connection type of the Kafka system is a long connection type; the connection type of the TSDB or function calculation system is a short connection type. And the Internet of things platform system receives the Internet of things data and determines a corresponding target node.

And the Internet of things platform system establishes long connection with the corresponding target node for the first time, or sends Internet of things data to the corresponding target node by adopting the pre-established long connection. The method comprises the steps that under the condition that long connection is established for the first time with a target node or long connection is maintained with the target node, the platform system of the Internet of things periodically carries out abnormity detection on a connection layer and an application layer, determines a long connection abnormity detection result of the target node, judges whether the target node is an abnormal node or not, and determines the abnormity type of the abnormal node according to the long connection abnormity detection result. The Internet of things platform system periodically detects the abnormality of the maintained long-connection target nodes. And under the condition that the abnormal processing type of the abnormal node is determined to be the manual processing type, disconnecting the long connection, and under the condition that the state of the abnormal node is updated to be the normal state and the data to be sent is sent to the abnormal node again, reestablishing the long connection and sending the data to be sent. The anomaly detection of the long-connection type target node is independent of the data sending operation of the Internet of things.

And under the condition that the connection type of the target node corresponding to the data of the Internet of things is determined to be the short connection type, processing the newly received data of the Internet of things according to the data processing logic statement in the forwarding information, and forwarding the data of the Internet of things to the target node in the forwarding information. And determining a short connection abnormity detection result of the target node according to the information fed back by the target node or the statistical response time, judging whether the target node is an abnormal node, and determining the abnormity type of the abnormal node according to the short connection abnormity detection result. It can be understood that the target node of the short connection type can perform anomaly detection under the condition of sending data of the internet of things each time.

The anomaly detection of the long-connection type target node specifically includes that the operation of acquiring the related meta-information in the target node is performed regularly by using the platform system of the internet of things and the operation is used for checking whether the business operation between the platform system of the internet of things and the target node is normal. The short connection type target node abnormality detection specifically includes acquiring information fed back by the target node and response time, and detecting whether an abnormality exists and an abnormality type of the existing abnormality.

The exception type is preconfigured. The exception type may specifically include a network type, a service type, or a user behavior causing type. Further differentiation of node types is also required. Exemplary exceptions may include: the network type, the service type or the user behavior under the authorized type causes the type abnormity; the type of exception is caused by the network type, the service type or the user behavior under the no-permission type. The exception of the network type under the authorized type may be: a network exception, such as a response timeout exception, specifying a system deployed in a public cloud. Exceptions to the type of service under the authoritative type may be: services of a system deployed in a given public cloud are unavailable, such as TSDB services deployed in a given public cloud. The user behavior causing type exceptions in the authorized type may be: system resources deployed in the specified public cloud are deleted or written with insufficient quota, for example, a database of the TSDB deployed in the specified public cloud is deleted or a topic of the Kafka system deployed in the specified public cloud is deleted, and the like, as well as TSDB deployed in the specified public cloud with insufficient writing quota.

The exception of the network type under the no-permission type may be: network anomalies for systems deployed in systems outside of the public cloud are specified. The exception to the service type under the no-permission type may be: services of a system deployed in a system outside the specified public cloud are not available, such as a Kafka system service itself deployed in a system outside the specified public cloud being abnormal or an http system service itself deployed in a system outside the specified public cloud being abnormal. The user behavior causing type exception in the no-permission type may be: system resources deployed in systems outside the specified public cloud are deleted or written to an insufficient amount, for example, topic of Kafka system deployed in systems outside the specified public cloud is deleted.

Meanwhile, under the authorized type, the network type and the service type are configured to correspond to the automatic processing type, and the user behavior causing type corresponds to the manual processing type; under the no-permission type, the network type corresponds to an automatic processing type, and the service type and the user behavior causing type correspond to a manual processing type.

The automatic processing type exception handling method is that the data of the internet of things is retransmitted for limited times, and under the condition that the retransmission times are more than or equal to the times threshold value, the data of the internet of things, the associated information, the exception reason and other information are combined and forwarded to the error node. The manual processing type exception handling mode is that the state of an exception node is modified into an exception state, the exception state is displayed in a page of the control module, a user is informed, and the data module stops sending the data of the Internet of things corresponding to the exception node; and under the condition that the user re-edits the data sending rule through the control module, updating the state of the abnormal node in the data sending rule into a normal state, and re-sending the data of the Internet of things corresponding to the abnormal node through the data module.

According to the technical scheme, different exception checking mechanisms are adopted on the basis of the node type of the target node and according to the connection modes of different target nodes, the data destination forwarding exception is detected, the corresponding exception handling type is determined according to the detected exception type and the node type, and different exception handling is performed aiming at different forwarding exceptions; on the other hand, for some exceptions, the data sending rule and the destination state are changed, and the corresponding target node is stopped from forwarding data, so that unnecessary resource consumption of the Internet of things receipt system can be reduced, and for exception notification of the user, the rule state of the data sending rule and the exception state of the destination state are displayed in the page of the control module, so that the user can be reminded to recover the service of the target node in time, and correct forwarding of the Internet of things data is ensured.

Fig. 5 is a structural diagram of an abnormal data processing apparatus in an embodiment of the present disclosure, and the embodiment of the present disclosure is applied to a case where whether an abnormality exists in a data forwarding target node in a case where data is forwarded to the target node. The device is realized by software and/or hardware and is specifically configured in electronic equipment with certain data operation capacity.

An exception data handling apparatus 500, as shown in figure 5, comprises: a node exception type determining module 501, an exception handling type determining module 502 and an exception sending data processing module 503; wherein the content of the first and second substances,

a node exception type determining module 501, configured to perform exception detection on a target node, and determine an exception node and an exception type of the exception node;

an exception handling type determining module 502, configured to determine a corresponding exception handling type according to the node type and the exception type of the exception node;

an exception sending data processing module 503, configured to process the to-be-sent data corresponding to the exception node according to the exception handling type corresponding to the exception node.

Further, the exception handling type includes an automatic handling type or a manual handling type.

Further, the exception sending data processing module 503 includes: and the abnormal data retransmission unit is used for retransmitting the data to be transmitted corresponding to the abnormal node under the condition that the abnormal processing type corresponding to the abnormal node comprises an automatic processing type.

Further, the exception sending data processing module 503 includes: the user prompting unit is used for modifying the state of the abnormal node into an abnormal state under the condition that the abnormal processing type corresponding to the abnormal node comprises a manual processing type, informing a user and stopping sending the data to be sent corresponding to the abnormal node; and the normal data retransmission unit is used for retransmitting the data to be transmitted corresponding to the abnormal node under the condition that the state of the abnormal node is updated to be a normal state.

Further, the exception handling type determining module 502 includes: and the processing type detection unit is used for determining the exception processing type of the exception node according to the node type and the exception type of the exception node and the corresponding relation between the preset type and the exception processing type, wherein the node type comprises an authority type or an unauthorized type, and the exception type comprises a network type, a service type or a user behavior causing type.

Further, the node anomaly type determining module 502 includes: an anomaly detection mode determining unit, configured to determine an anomaly detection mode corresponding to the target node according to a connection type with the target node; and the abnormality detection unit is used for carrying out abnormality detection on the target node by adopting an abnormality detection mode corresponding to the target node.

Further, the data to be transmitted includes: data of the internet of things; the abnormal data processing device further comprises: the internet of things data acquisition module is used for receiving the internet of things data and acquiring forwarding information corresponding to the internet of things data, wherein the forwarding information comprises a target node corresponding to the internet of things data; and the Internet of things data forwarding module is used for processing the Internet of things data according to the forwarding information and forwarding the Internet of things data to the corresponding target node.

The abnormal data processing device can execute the abnormal data processing method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of executing the abnormal data processing method.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order. It should be noted that the data collected and forwarded by the source node in the present disclosure is not information for a specific user, and cannot reflect personal information of a specific user.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 6 illustrates a schematic area diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 performs the respective methods and processes described above, such as the abnormal data processing method. For example, in some embodiments, the exception data handling method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM602 and/or the communication unit 609. When a computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the above described method of exception data handling may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the exception data handling method by any other suitable means (e.g. by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or area diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An exception data processing method, comprising:

carrying out anomaly detection on a target node, and determining an abnormal node and an abnormal type of the abnormal node;

2. The method of claim 1, wherein the exception handling type comprises an automatic handling type or a manual handling type.

3. The method according to claim 2, wherein the processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node comprises:

and under the condition that the exception processing type corresponding to the exception node comprises an automatic processing type, retransmitting the data to be transmitted corresponding to the exception node.

4. The method according to claim 2, wherein the processing the data to be sent corresponding to the abnormal node according to the abnormal processing type corresponding to the abnormal node comprises:

under the condition that the exception processing type corresponding to the exception node comprises a manual processing type, modifying the state of the exception node into an exception state, notifying a user, and stopping sending data to be sent corresponding to the exception node;

and under the condition that the state of the abnormal node is updated to be a normal state, retransmitting the data to be transmitted corresponding to the abnormal node.

5. The method of claim 2, wherein said determining a corresponding exception handling type based on the node type and the exception type of the exception node comprises:

and determining the exception handling type of the exception node according to the node type and the exception type of the exception node and the corresponding relation between the preset type and the exception handling type, wherein the node type comprises an authority type or an unauthorized type, and the exception type comprises a network type, a service type or a user behavior causing type.

6. The method of claim 1, wherein the detecting the anomaly of the target node comprises:

determining an abnormal detection mode corresponding to the target node according to the connection type of the target node;

and carrying out anomaly detection on the target node by adopting an anomaly detection mode corresponding to the target node.

7. The method of claim 1, wherein the data to be transmitted comprises: data of the internet of things;

the method further comprises the following steps:

receiving the data of the Internet of things, and acquiring forwarding information corresponding to the data of the Internet of things, wherein the forwarding information comprises a target node corresponding to the data of the Internet of things;

and processing the data of the Internet of things according to the forwarding information, and forwarding the data of the Internet of things to a corresponding target node.

8. An exception data handling apparatus comprising:

the node exception type determining module is used for carrying out exception detection on the target node and determining an exception node and an exception type of the exception node;

the exception handling type determining module is used for determining a corresponding exception handling type according to the node type and the exception type of the exception node;

9. The apparatus of claim 8, wherein the exception handling type comprises an automatic handling type or a manual handling type.

10. The apparatus of claim 9, wherein the exception-sending data processing module comprises:

and the abnormal data retransmission unit is used for retransmitting the data to be transmitted corresponding to the abnormal node under the condition that the abnormal processing type corresponding to the abnormal node comprises an automatic processing type.

11. The apparatus of claim 9, wherein the exception-sending data processing module comprises:

the user prompting unit is used for modifying the state of the abnormal node into an abnormal state under the condition that the abnormal processing type corresponding to the abnormal node comprises a manual processing type, informing a user and stopping sending the data to be sent corresponding to the abnormal node;

and the normal data retransmission unit is used for retransmitting the data to be transmitted corresponding to the abnormal node under the condition that the state of the abnormal node is updated to be a normal state.

12. The apparatus of claim 9, wherein the exception handling type determination module comprises:

and the processing type detection unit is used for determining the exception processing type of the exception node according to the node type and the exception type of the exception node and the corresponding relation between the preset type and the exception processing type, wherein the node type comprises an authority type or an unauthorized type, and the exception type comprises a network type, a service type or a user behavior causing type.

13. The apparatus of claim 8, wherein the node anomaly type determination module comprises:

an anomaly detection mode determining unit, configured to determine an anomaly detection mode corresponding to the target node according to a connection type with the target node;

and the abnormality detection unit is used for carrying out abnormality detection on the target node by adopting an abnormality detection mode corresponding to the target node.

14. The apparatus of claim 8, the data to be transmitted comprising: data of the internet of things;

further comprising:

the internet of things data acquisition module is used for receiving the internet of things data and acquiring forwarding information corresponding to the internet of things data, wherein the forwarding information comprises a target node corresponding to the internet of things data;

and the Internet of things data forwarding module is used for processing the Internet of things data according to the forwarding information and forwarding the Internet of things data to the corresponding target node.

15. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of exception data handling of any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the abnormal data processing method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of exception data handling according to any one of claims 1 to 7.