CN117873609A - Data processing method and device, storage medium and electronic equipment - Google Patents

Abstract

The disclosure provides a data processing method, a data processing device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: receiving a data acquisition task carrying an acquisition requirement; determining a plurality of processing rules corresponding to the data acquisition tasks and a rule calling sequence according to the acquisition requirements; wherein a first processing rule of the plurality of processing rules is an input parsing rule; calling a protocol executor to acquire data according to the input analysis rule to obtain original acquired data; and sequentially calling a plurality of processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task. The method can improve the efficiency of the data acquisition process and can carry out highly flexible configuration on the processing rules and the rule calling sequences related to the data acquisition process.

Description

Data processing method and device, storage medium and electronic equipment

Technical Field

The disclosure relates to the field of computer technology, and in particular, to a data processing method and device, a storage medium and electronic equipment.

Background

With the development of computer technology and network technology, data collection of devices via a network remotely has become possible. The data collection may be implemented using a micro-service architecture.

In the related art, equipment to be acquired (such as a 5GC network element) is generally distinguished according to the type of a network element, the type of the network element, the type of acquired data, the type of a protocol and the like, corresponding configuration, performance, alarm and log data acquisition services are respectively developed, a micro service code is required to be rewritten every time an acquisition service is developed, when the acquisition process is changed, the micro service code is required to be redeveloped, and then a package test and a reissue update are required, so that the maintenance difficulty of the micro service is high, and the data acquisition efficiency is reduced.

It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The disclosure aims to provide a data processing method, a device, an electronic device and a storage medium, so as to improve the efficiency of a data acquisition process and realize high-flexibility configuration of a data acquisition process.

Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.

According to one aspect of the present disclosure, there is provided a data processing method including: receiving a data acquisition task carrying an acquisition requirement; determining a plurality of processing rules corresponding to the data acquisition tasks and a rule calling sequence according to the acquisition requirements; wherein a first processing rule of the plurality of processing rules is an input parsing rule; calling a protocol executor to acquire data according to the input analysis rule to obtain original acquired data; and sequentially calling a plurality of processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task.

In one embodiment of the present disclosure, determining a plurality of processing rules and rule calling orders corresponding to data acquisition tasks according to acquisition requirements includes: accessing a rule database, wherein a plurality of preconfigured candidate rule chains are stored in the rule database; searching a target rule chain matched with the acquisition requirement in a plurality of candidate rule chains; wherein the acquisition requirements include at least one of: equipment information to be acquired, data information to be acquired and an acquisition protocol; and determining a plurality of processing rules and a rule calling sequence according to the target rule chain.

In one embodiment of the present disclosure, invoking a protocol executor to perform data acquisition according to an input parsing rule to obtain original acquired data includes: acquiring input analysis configuration information corresponding to the input analysis rule, wherein the input analysis configuration information comprises one or more data input types; and calling a protocol executor corresponding to each data input type to acquire data, so as to obtain the original acquired data of each data input type.

In one embodiment of the present disclosure, the input parsing configuration information includes a data parsing script corresponding to a data input type; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and comprises the following steps: and calling an input analysis rule, and analyzing the original acquired data of the corresponding data input type by utilizing each data analysis script to obtain analyzed data of each data input type.

In one embodiment of the present disclosure, the plurality of processing rules further includes a normalized merge rule, where the normalized merge rule exists after the input parsing rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a normalized merging rule to obtain a normalized compound script corresponding to the normalized compound rule; and carrying out normalization processing and merging processing on the analyzed data by utilizing a normalization combination script to obtain normalized data.

In one embodiment of the present disclosure, the plurality of processing rules further includes a storage rule; if the standard combination rule does not exist, the storage rule exists after the analysis rule is input; if the canonical combining rule exists, the storage rule exists after the canonical combining rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a storage rule to obtain storage configuration information corresponding to the storage rule; and storing the analyzed data or the normalized data by using the storage configuration information to obtain a storage result.

In one embodiment of the present disclosure, the plurality of processing rules further includes a notification rule, the notification rule existing after the storage rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a notification rule, and acquiring notification configuration information corresponding to the notification rule; and notifying the storage result by using the notification configuration information.

According to another aspect of the present disclosure, there is provided a data processing apparatus comprising: the receiving module is used for receiving a data acquisition task carrying an acquisition requirement; the rule determining module is used for determining a plurality of processing rules and rule calling sequences corresponding to the data acquisition tasks according to the acquisition requirements; wherein a first processing rule of the plurality of processing rules is an input parsing rule; the acquisition module is used for calling the protocol executor to acquire data according to the input analysis rule to obtain original acquired data; and the data processing module is used for sequentially calling a plurality of processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task.

According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described data processing method.

According to still another aspect of the present disclosure, there is provided an electronic apparatus including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method described above via execution of the executable instructions.

The data processing method provided by the embodiment of the disclosure can determine a plurality of corresponding processing rules and rule calling sequences according to the human acquisition requirements of the received data acquisition task, call a protocol executor according to the input analysis rules therein to acquire original acquisition data, and then call the plurality of processing rules in sequence according to the rule calling sequences to process the original acquisition data so as to complete the data acquisition task; wherein, a plurality of processing rules and rule calling sequences are capable of being updated and arranged on line. Therefore, the method can realize automatic execution of the data acquisition task based on a plurality of processing rules and the rule calling sequence, and improves the efficiency of the data acquisition process; the content of a plurality of processing rules for processing the data acquisition task and the rule calling sequence can be well arranged, expanded or modified, and the processing rules, scripts, configuration information and rule chains related to the data acquisition process can be highly flexibly configured, so that the maintainability and the expandability of the whole data processing process are high.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the data processing method of embodiments of the present disclosure may be applied;

FIG. 2 illustrates a flow chart of a data processing method of one embodiment of the present disclosure;

FIG. 3 illustrates a flow chart of obtaining raw acquisition data in a data processing method of one embodiment of the present disclosure;

FIG. 4 illustrates a flow chart of executing a rule chain in a data processing method of one embodiment of the present disclosure;

FIG. 5 illustrates a flow architecture diagram of a data processing method of one embodiment of the present disclosure;

FIG. 6 illustrates a block diagram of a data processing apparatus of one embodiment of the present disclosure; and

fig. 7 shows a block diagram of a data processing computer device in an embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present disclosure, the meaning of "a plurality" is at least two, such as two, three, etc., unless explicitly specified otherwise.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the data processing method of embodiments of the present disclosure may be applied.

As shown in fig. 1, the system architecture may include a server 101, a network 102, and a client 103. Network 102 is the medium used to provide communication links between clients 103 and server 101. Network 102 may include various connection types such as wired, wireless communication links, or fiber optic cables, among others.

In an exemplary embodiment, the client 103 in data transmission with the server 101 may include, but is not limited to, a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an AR (Augmented Reality) device, a VR (Virtual Reality) device, a smart wearable device, and the like. Alternatively, the operating system running on the electronic device may include, but is not limited to, an android system, an IOS system, a linux system, a windows system, and the like.

The server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like. In some practical applications, the server 101 may also be a server of a network platform, and the network platform may be, for example, a transaction platform, a live broadcast platform, a social platform, or a music platform, which is not limited in the embodiments of the present disclosure. The server may be one server or may be a cluster formed by a plurality of servers, and the specific architecture of the server is not limited in this disclosure.

In an exemplary embodiment, a data collection service capable of implementing the data processing method provided by the present disclosure may be deployed in the server 101, where the data collection service may receive a data collection task issued by the unified task scheduling center, and perform data collection processing in the client 103 by using the method provided by the present disclosure, so as to complete the data collection task. And a task processing result can be generated and notified to the unified task scheduling center.

In an exemplary embodiment, the procedure of the server 101 for implementing the data processing method may be: the server 101 receives a data acquisition task carrying an acquisition requirement; the server 101 determines a plurality of processing rules corresponding to the data acquisition task and a rule calling sequence according to the acquisition requirement; wherein a first processing rule of the plurality of processing rules is an input parsing rule; the server 101 calls a protocol executor to acquire data according to the input analysis rule, so as to obtain original acquired data; the server 101 sequentially calls a plurality of processing rules according to the rule call sequence to process the original collected data so as to complete the data collection task.

In addition, it should be noted that fig. 1 is only one application environment of the data processing method provided in the present disclosure. The number of servers 101, networks 102, and clients 103 in FIG. 1 is merely illustrative, and any number of clients, networks, and servers may be provided as desired.

In order for those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the steps of the data processing method in the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings and embodiments.

FIG. 2 illustrates a flow chart of a data processing method of one embodiment of the present disclosure. The method provided by the embodiments of the present disclosure may be performed by the server 101 or the client 103 as shown in fig. 1, but the present disclosure is not limited thereto.

In the following explanation, the server 101 is exemplified as an execution subject.

As shown in fig. 2, the data processing method provided by the embodiment of the present disclosure may include the following steps.

Step S201, receiving a data acquisition task carrying an acquisition requirement.

In this step, the data acquisition task may be issued from a unified task scheduling center, and the data acquisition task may instruct that the specified type of data be acquired in the specified network element/device. The configuration can be performed in the unified task scheduling center in advance and then the configuration can be issued at fixed time, or the data acquisition task can be temporarily established through the unified task scheduling center and then the configuration can be performed. The configuration may be to configure the content of the acquisition requirement, where the acquisition requirement may be regarded as scheduling task information for invoking a rule in a subsequent step, and the acquisition requirement may include information about a manufacturer, a model, a network element type, an acquired data type, a network type, an acquired protocol, an acquisition period, and the like of the network element/device to be acquired.

Step S203, determining a plurality of processing rules and rule calling sequences corresponding to the data acquisition tasks according to the acquisition requirements; wherein a first processing rule of the plurality of processing rules is an input parsing rule.

In this step, the plurality of processing rules may correspond to the acquisition requirement, for example, if the acquisition requirement indicates that the step to be executed includes acquiring data and storing data, the plurality of processing rules may be two rules responsible for executing the acquisition data and the storing data; for another example, if the acquisition requirement indicates that the data need to be cleaned and merged after the data is acquired and then stored, the plurality of processing rules may be four rules responsible for acquiring the data, cleaning the data, merging the data, and storing the data.

In some implementations, the plurality of processing rules may be rules in a pre-configured rule chain, and a rule calling order of the plurality of processing rules may be indicated in a structure of the rule chain. The rule chains are formed by a plurality of processing rules in advance, so that the processing rules can be conveniently called, and unified configuration and updating are convenient.

The input parsing rule may indicate the data type of the data to be collected, the parsing method for each data type, and the like.

In some embodiments, step S203 may include: accessing a rule database, wherein a plurality of preconfigured candidate rule chains are stored in the rule database; searching a target rule chain matched with the acquisition requirement in a plurality of candidate rule chains; wherein the acquisition requirements may include at least one of: acquiring a demand side, equipment information to be acquired, data information to be acquired and an acquisition protocol; and determining a plurality of processing rules and a rule calling sequence according to the target rule chain.

In this embodiment, some candidate rule chains with high universality may be preconfigured based on the historical data acquisition task, or rule chains matched with the data acquisition task may be configured before the data acquisition task is established, and then stored in a rule database, so that the matched rule chains are found to be invoked when the data acquisition task is received.

Because the task and the rule chain are not directly bound, the coupling degree is low and is determined through matching, in some practical applications, the rule chain can be configured or updated on line at the front end, for example, the online configuration of the rule, online java/groovy script programming, online rule chain arrangement, pulling of rule and rule chain data and the like can be realized by matching with a front end interface. In this way, the content of the rule chain can be well expanded or modified, so that the maintainability and the expandability of the whole data processing process are high.

Step S205, a protocol executor is called to collect data according to the input analysis rule, and original collected data is obtained.

In this step, the protocol executor is a component or module for executing different types of protocols, and may support a plurality of different data protocols. The protocol executor may be responsible for communication and interaction for different protocols, may communicate with remote devices/network elements or services using corresponding protocols according to configured rules, and perform corresponding operations.

Fig. 3 illustrates a flowchart of obtaining raw acquisition data in a data processing method of one embodiment of the present disclosure, as shown in fig. 3, in some embodiments, step S205 may include the following steps.

Step S301, obtaining input analysis configuration information corresponding to the input analysis rule, where the input analysis configuration information includes one or more data input types.

The input analysis configuration information may be preconfigured, and may specifically be configured according to a service requirement of a user. The input resolution configuration information may be stored in a rule base from which it may be pulled when the input resolution rule is invoked.

In some implementations, the data input types may be SshInput, telnetInput, sftpInput, sslInput, reftfulInput, snmpInput, etc., which represent different types of input data or input rules for configuring the input parsing rules in the system.

Wherein SshInput represents the input type of communication and execution command through SSH protocol. The target device information of the SSH connection and the SSH command to be executed may be configured.

Telnet input represents the type of input to communicate and execute commands via the Telnet protocol. The target device information of the Telnet connection and the Telnet command to be executed may be configured.

Sftplnput denotes the input type for file transfer via SFTP protocol. Target server information, file paths, and other related rules for the SFTP connection may be configured.

Sslnput denotes an input type for secure communication through the SSL protocol. The target server information and other related rules of the SSL connection may be configured.

RestfulInput represents the type of input for communication and data acquisition through RESTful API. Information such as URL of RESTful API, HTTP method, parameters, etc. may be configured.

The SNMP input indicates an input type for network device management and monitoring through SNMP protocol. The target device information of the SNMP connection and SNMP data to be acquired can be configured.

Step S303, calling a protocol executor corresponding to each data input type to acquire data, and obtaining original acquired data of each data input type.

In this embodiment, input information (such as a data type) required for the operation of the protocol executor may be indicated in the input parsing configuration information of the input parsing rule. The protocol executor may be invoked to collect in a target device, which may be specified in the collection requirements or may be preconfigured in the input parsing configuration information.

In some practical applications, the data collection needs to be performed for some devices, for example, some conditions need to be met, and the communication permission of the devices needs to be obtained, so that the data collection can be performed. Based on the account information, the input analysis configuration information can also contain account information required by connection login, and then connection and login can be performed according to the account information, and then a protocol executor is called for data acquisition.

Step S207, sequentially calling a plurality of processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task.

In this step, when the plurality of processing rules are all invoked, it can be regarded as the execution of the data acquisition task is completed. In practical applications, the task result of the data collection task is related to the last processing rule of the plurality of processing rules, for example, if the last processing rule is for storage, the task result may be a storage result (such as a storage path, a storage file name, etc.) obtained by the completion of storage, and if the last processing rule is for notification, the task result may be a final notification message (such as a message sent to a designated system describing the data collection content).

According to the data processing method provided by the disclosure, a plurality of corresponding processing rules and rule calling sequences can be determined according to the human acquisition requirements of the received data acquisition task, a protocol executor is called according to the input analysis rules to acquire the original acquisition data, and then the plurality of processing rules are called in sequence according to the rule calling sequences to process the original acquisition data, so that the data acquisition task is completed; wherein, a plurality of processing rules and rule calling sequences are capable of being updated and arranged on line. Therefore, the method can realize automatic execution of the data acquisition task based on a plurality of processing rules and the rule calling sequence, and improves the efficiency of the data acquisition process; the content of a plurality of processing rules for processing the data acquisition task and the rule calling sequence can be well arranged, expanded or modified, and the processing rules, scripts, configuration information and rule chains related to the data acquisition process can be configured with high flexibility, so that the maintainability and the expandability of the whole data processing process are high.

In some embodiments, the input parsing configuration information includes a data parsing script corresponding to a data input type; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and comprises the following steps: and calling an input analysis rule, and analyzing the original acquired data of the corresponding data input type by utilizing each data analysis script to obtain analyzed data of each data input type.

In this embodiment, the input parsing rule may further include a parsing rule java/groovy script (i.e., a data parsing script) for an execution result (i.e., original collected data), after the parsing rule script completes operation, return data (i.e., parsed data) may be obtained, and then the obtained return data may be used as input of a next rule. The next rule may be a different processing rule in a different rule chain, for example, a canonical compound rule or a stored rule, etc., depending on the arrangement in the rule chain.

In some embodiments, the plurality of processing rules further includes a normalized merge rule that exists after the input parsing rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a normalized merging rule to obtain a normalized compound script corresponding to the normalized compound rule; and carrying out normalization processing and merging processing on the analyzed data by utilizing a normalization combination script to obtain normalized data.

In this embodiment, the canonical combining rule may include normalized csv header information, field separators, and normalized rule script information; wherein csv header information, field separators may be used to determine the presence format of normalized data, and normalization rule script information may be used to perform normalized merge operations on the data.

The normalized merge rule may receive input (i.e., parsed data) from a plurality of input parsing rules, then perform a normalization/merge operation on the input data using the normalized rule script to parse and map the input data onto csv header information to obtain a return result (i.e., normalized data), and then may pass the return result to the next rule as input. The next rule may be a different processing rule in a different rule chain, for example, may be a storage rule, may be another normalized merging rule, may be a notification rule directly, or the like, and may specifically depend on the arrangement in the rule chain.

In some embodiments, the plurality of processing rules further comprises a storage rule; if the standard combination rule does not exist, the storage rule exists after the analysis rule is input; if the canonical combining rule exists, the storage rule exists after the canonical combining rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a storage rule to obtain storage configuration information corresponding to the storage rule; and storing the analyzed data or the normalized data by using the storage configuration information to obtain a storage result.

In this embodiment, the storage rule may include multiple storage modes (for example, sending to MQ, storing to a single file, sending to UDP Server, etc.), and information such as a path mode and a file name mode of storage, where the storage rule may receive an output transmitted from a previous rule as input data (for example, parsed data or normalized data) of the storage rule, then perform a storage operation on the input data according to the storage mode, and after the execution is completed, may return a storage result. The storage result may include information such as a path where the file is located, a file name, and the like. The stored results may be stored in a file system.

In some practical applications, the storage mode, the storage path and the storage name mode may be preconfigured, and may be specifically configured according to the service requirement of the user. The rule base is responsible for receiving these configuration information and storing it in the rule base, from which configuration information relating to the stored rule can be pulled when the stored rule is invoked.

In some embodiments, the plurality of processing rules further includes a notification rule that exists after the storage rule; the method comprises the steps of sequentially calling a plurality of processing rules according to a rule calling sequence to process the original acquired data, and further comprises the following steps: calling a notification rule, and acquiring notification configuration information corresponding to the notification rule; and notifying the storage result by using the notification configuration information.

In some implementations, the notification rules may include a subject configuration of the notification, a content dynamic configuration of the notification, and so on. The notifying of the storage result may be generating a notification message, notifying the storage result to an upper subsystem (e.g., an upper subsystem of the server 101) through the notification message, and notifying the upper subsystem to consume data under a designated file path and then use the consumed data, thereby completing a service closed loop of all data acquisition processes.

FIG. 4 illustrates a flow chart of executing a rule chain in a data processing method of one embodiment of the present disclosure, as illustrated in FIG. 4, the process of executing a rule chain may include the following steps.

Step S401, calling an input analysis rule, and acquiring input analysis configuration information corresponding to the input analysis rule, wherein the input analysis configuration information comprises a data analysis script corresponding to a data input type; and analyzing the original acquired data of the corresponding data input type by utilizing each data analysis script to obtain analyzed data of each data input type.

Step S403, calling a canonical combining rule; and obtaining a normalized merging script corresponding to the normalized merging rule, and performing normalized processing merging processing on the analyzed data by using the normalized merging script to obtain normalized data.

Step S405, calling a storage rule; and acquiring storage configuration information corresponding to the storage rule, and storing the analyzed data or the normalized data by utilizing the storage configuration information to obtain a storage result.

Step S407, calling a notification rule; and acquiring notification configuration information corresponding to the notification rule, and notifying the storage result by using the notification configuration information.

In some practical applications, the above processing rules may be flexibly configured, and the configuration content may be as follows:

when the input parsing rule is configured, if the input is of an instruction type, configuring an instruction; if the input is SFTP, configuring an SFTP path to be acquired, a file filter, whether to store a temporary file, a temporary file path, whether to decompress and the like; if the input is kafka, then it is necessary to configure kafka topic, consumption group groupId, etc.; if the input is RESTFUL, url, method, parameter list, header information, etc. need to be configured.

When the normalized merge rule is configured, the input content may be a groovy script.

When the storage rule is configured, a normalized title, a storage path and a file name rule can be configured

When the notification rule is configured, the type of notification, topic (topic) of the notification, the content of the notification, and the like may be configured.

Other details of the embodiment of fig. 4 may be found in the other embodiments described above.

FIG. 5 is a schematic flow chart of a data processing method according to one embodiment of the disclosure, as shown in FIG. 5, including a unified task scheduling center 501, a general data acquisition service 502, and a rule base 503; the general data collection service 502 includes a device to be collected 5031, a device to be collected 5032, a device to be collected 5033, a protocol executor 5041 corresponding to the device to be collected 5031, a protocol executor 5042 corresponding to the device to be collected 5032, a protocol executor 5043 corresponding to the device to be collected 5033, a rule chain executor 505, and a rule chain 506, wherein the rule chain 506 is composed of an input analysis rule 5061, a standard combination rule 5062, a storage rule 5063, and a notification rule 5064 in sequence.

Referring to fig. 5, a general data collection service 502 may receive a data collection task issued by a unified task scheduling center 501, and determine a rule chain 506 from a rule base 503 according to the data collection task.

Then, the data input types are acquired from the input analysis rules 5061 in the rule chain 506, and the protocol executors (including the protocol executors 5041-5043) corresponding to the data input types are called to respectively perform data acquisition on the equipment to be acquired (including the equipment to be acquired 5031-5033) so as to obtain the original acquired data of the data input types.

The rule chain executor 505 is then used to call the rule chain 506, specifically, the input parsing rule 5061, the canonical combining rule 5062, the storage rule 5063 and the notification rule 5064 in the rule chain 506 are sequentially called to process the original collected data, so as to complete the data collection task.

It is noted that the above-described figures are only schematic illustrations of processes involved in a method according to an exemplary embodiment of the invention, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.

FIG. 6 illustrates a block diagram of a data processing apparatus 600 of one embodiment of the present disclosure; as shown in fig. 6, includes: the receiving module 601 is configured to receive a data acquisition task carrying an acquisition requirement; the rule determining module 602 is configured to determine a plurality of processing rules and rule calling sequences corresponding to the data acquisition task according to the acquisition requirement; wherein a first processing rule of the plurality of processing rules is an input parsing rule; the acquisition module 603 is configured to invoke the protocol executor to perform data acquisition according to the input parsing rule, so as to obtain original acquired data; the data processing module 604 is configured to sequentially invoke a plurality of processing rules according to the rule invoking sequence to process the original collected data, so as to complete the data collection task.

According to the data processing device provided by the disclosure, a plurality of corresponding processing rules and rule calling sequences can be determined according to the human acquisition requirements of the received data acquisition task, a protocol executor is called according to the input analysis rules to acquire the original acquisition data, and then the plurality of processing rules are called in sequence according to the rule calling sequences to process the original acquisition data, so that the data acquisition task is completed; wherein, a plurality of processing rules and rule calling sequences are capable of being updated and arranged on line. Therefore, the method can realize automatic execution of the data acquisition task based on a plurality of processing rules and the rule calling sequence, and improves the efficiency of the data acquisition process; the content of a plurality of processing rules for processing the data acquisition task and the rule calling sequence can be well arranged, expanded or modified, and the processing rules, scripts, configuration information and rule chains related to the data acquisition process can be highly flexibly configured, so that the maintainability and the expandability of the whole data processing process are high.

In some embodiments, the rule determination module 602 determines a plurality of processing rules and rule calling orders corresponding to the data acquisition tasks according to the acquisition requirements, including: accessing a rule database, wherein a plurality of preconfigured candidate rule chains are stored in the rule database; searching a target rule chain matched with the acquisition requirement in a plurality of candidate rule chains; wherein the acquisition requirements include at least one of: the method comprises the steps of collecting a demand party, equipment to be collected, data information to be collected and a collection protocol; and determining a plurality of processing rules and a rule calling sequence according to the target rule chain.

In some embodiments, the collecting module 603 invokes the protocol executor to collect data according to the input parsing rule, so as to obtain raw collected data, including: acquiring input analysis configuration information corresponding to the input analysis rule, wherein the input analysis configuration information comprises one or more data input types; and calling a protocol executor corresponding to each data input type to acquire data, so as to obtain the original acquired data of each data input type.

In some embodiments, the input parsing configuration information includes a data parsing script corresponding to a data input type; the data processing module 604 sequentially invokes a plurality of processing rules according to the rule invoking sequence to process the original collected data, including: and calling an input analysis rule, and analyzing the original acquired data of the corresponding data input type by utilizing each data analysis script to obtain analyzed data of each data input type.

In some embodiments, the plurality of processing rules further includes a normalized merge rule that exists after the input parsing rule; wherein, the data processing module 604 sequentially invokes a plurality of processing rules according to the rule invoking sequence to process the original collected data, and further includes: calling a normalized merging rule to obtain a normalized compound script corresponding to the normalized compound rule; and carrying out normalization processing and merging processing on the analyzed data by utilizing a normalization combination script to obtain normalized data.

In some embodiments, the plurality of processing rules further comprises a storage rule; if the standard combination rule does not exist, the storage rule exists after the analysis rule is input; if the canonical combining rule exists, the storage rule exists after the canonical combining rule; wherein, the data processing module 604 sequentially invokes a plurality of processing rules according to the rule invoking sequence to process the original collected data, and further includes: calling a storage rule to obtain storage configuration information corresponding to the storage rule; and storing the analyzed data or the normalized data by using the storage configuration information to obtain a storage result.

In some embodiments, the plurality of processing rules further includes a notification rule that exists after the storage rule; wherein, the data processing module 604 sequentially invokes a plurality of processing rules according to the rule invoking sequence to process the original collected data, and further includes: calling a notification rule, and acquiring notification configuration information corresponding to the notification rule; and notifying the storage result by using the notification configuration information.

Other details of the embodiment of fig. 6 may be found in the other embodiments described above.

Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

Fig. 7 shows a block diagram of a data processing computer device in an embodiment of the present disclosure. It should be noted that the illustrated electronic device is only an example, and should not impose any limitation on the functions and application scope of the embodiments of the present invention.

An electronic device 700 according to this embodiment of the invention is described below with reference to fig. 7. The electronic device 700 shown in fig. 7 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.

As shown in fig. 7, the electronic device 700 is embodied in the form of a general purpose computing device. Components of electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one memory unit 720, and a bus 730 connecting the different system components, including the memory unit 720 and the processing unit 710.

Wherein the storage unit stores program code that is executable by the processing unit 710 such that the processing unit 710 performs steps according to various exemplary embodiments of the present invention described in the above-mentioned "exemplary methods" section of the present specification. For example, the processing unit 710 may perform the method as shown in fig. 2.

The memory unit 720 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 7201 and/or cache memory 7202, and may further include Read Only Memory (ROM) 7203.

The storage unit 720 may also include a program/utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 730 may be a bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and/or any device (e.g., router, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 750. Also, electronic device 700 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 over bus 730. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 700, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.

A program product for implementing the above-described method according to an embodiment of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

Furthermore, although the steps of the methods in the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in that particular order or that all illustrated steps be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

According to one aspect of the present disclosure, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the methods provided in the various alternative implementations of the above-described embodiments.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims (10)

1. A method of data processing, comprising:

receiving a data acquisition task carrying an acquisition requirement;

determining a plurality of processing rules and rule calling sequences corresponding to the data acquisition task according to the acquisition requirement; wherein a first processing rule of the plurality of processing rules is an input parsing rule;

Invoking a protocol executor to acquire data according to the input analysis rule to obtain original acquired data;

and sequentially calling the processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task.

2. The method of claim 1, wherein determining a plurality of processing rules and rule calling orders corresponding to the data acquisition tasks based on the acquisition requirements comprises:

accessing a rule database, wherein a plurality of preconfigured candidate rule chains are stored in the rule database;

searching a target rule chain matched with the acquisition requirement in the plurality of candidate rule chains; wherein the acquisition requirements include at least one of: equipment information to be acquired, data information to be acquired and an acquisition protocol;

and determining the processing rules and the rule calling sequence according to the target rule chain.

3. The method of claim 1, wherein invoking a protocol executor for data collection according to the input parsing rule results in raw collection data, comprising:

acquiring input analysis configuration information corresponding to the input analysis rule, wherein the input analysis configuration information comprises one or more data input types;

And calling a protocol executor corresponding to each data input type to acquire data, so as to obtain the original acquired data of each data input type.

4. A method according to claim 3, wherein the input parsing configuration information includes a data parsing script corresponding to the data input type;

the method comprises the steps of sequentially calling the plurality of processing rules according to the rule calling sequence to process the original acquired data, and comprises the following steps:

and calling the input analysis rule, and analyzing the original acquired data of the corresponding data input type by utilizing each data analysis script to obtain analyzed data of each data input type.

5. The method of claim 4, wherein the plurality of processing rules further comprises a canonical merge rule, the canonical merge rule existing after the input parsing rule;

the method comprises the steps of calling the plurality of processing rules in sequence according to the rule calling sequence to process the original acquired data, and further comprises the steps of:

calling the normalized merging rule to obtain a normalized merging script corresponding to the normalized merging rule;

and carrying out normalization processing and merging processing on the analyzed data by utilizing the normalization combination script to obtain normalized data.

6. The method of claim 1 or 5, wherein the plurality of processing rules further comprises a storage rule; if the canonical union rule does not exist, the storage rule exists after the input analysis rule; if the normalized merging rule exists, the storage rule exists behind the normalized merging rule;

the method comprises the steps of calling the plurality of processing rules in sequence according to the rule calling sequence to process the original acquired data, and further comprises the steps of:

invoking the storage rule to obtain storage configuration information corresponding to the storage rule;

and storing the analyzed data or the normalized data by using the storage configuration information to obtain a storage result.

7. The method of claim 6, wherein the plurality of processing rules further comprises a notification rule, the notification rule residing after the storage rule;

the method comprises the steps of calling the plurality of processing rules in sequence according to the rule calling sequence to process the original acquired data, and further comprises the steps of:

invoking the notification rule to obtain notification configuration information corresponding to the notification rule;

And notifying the storage result by using the notification configuration information.

8. A data processing apparatus, comprising:

the receiving module is used for receiving a data acquisition task carrying an acquisition requirement;

the rule determining module is used for determining a plurality of processing rules and rule calling sequences corresponding to the data acquisition task according to the acquisition requirement; wherein a first processing rule of the plurality of processing rules is an input parsing rule;

the acquisition module is used for calling a protocol executor to acquire data according to the input analysis rule to obtain original acquired data;

and the data processing module is used for sequentially calling the plurality of processing rules according to the rule calling sequence to process the original acquired data so as to complete the data acquisition task.

9. A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the data processing method according to any of claims 1 to 7.

10. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which when executed by the one or more processors cause the one or more processors to implement the data processing method of any of claims 1 to 7.