CN111538491B - Data event processing method, device, equipment and storage medium - Google Patents
Data event processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111538491B CN111538491B CN202010331024.0A CN202010331024A CN111538491B CN 111538491 B CN111538491 B CN 111538491B CN 202010331024 A CN202010331024 A CN 202010331024A CN 111538491 B CN111538491 B CN 111538491B
- Authority
- CN
- China
- Prior art keywords
- data
- atomic
- processing
- module
- base class
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/31—Programming languages or programming paradigms
- G06F8/315—Object-oriented languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Computing Systems (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data event processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a data event to be processed; performing data decoupling processing on the data event to generate a plurality of atomic data modules; calling a preset base class corresponding to each atomic data module from a database; and processing the data event according to the preset base class and the data content of each atomic data module. The method and the device realize unified processing of the data events in different service fields, reduce repeated development work and save development resources.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data event processing method, apparatus, device, and storage medium.
Background
Each data interaction in a computer system can be considered an event, and a solution can provide a plurality of interaction events, each of which can be split into a plurality of different processes depending on what is being processed. Taking an ERP system as an example, a small ERP system has more than 100 external interaction methods on average. In processing a general event flow, the events are often processed linearly in steps. For example, in the event of log landing, the log landing events in one service scene are generally processed linearly in steps, and the log landing events in different service scenes are processed respectively.
With the development of computer information technology, more and more events need to be processed. If there are many events similar to the log floor, the events need to be processed separately, and the development cost is very high. There are many similar data event processing flows in external data warehousing operations such as an ERP (Enterprise Resource Planning ) system. In fact, similar process flows exist between different systems.
In the above environment, most of the work of the programmer is the process flow in the new and complete business fields with different sizes, and each process flow is different from the business point of view. However, from the process structure analysis, several important links are different in size. Therefore, the side reflection programmers do a lot of repeated work unknowingly, and the utilization rate of development resources is low.
Disclosure of Invention
An objective of the embodiments of the present application is to provide a method, an apparatus, a device, and a storage medium for processing data events, which are used for abstracting a common processing mode of the data events by decoupling the data events, so as to process the data events in different service areas by adopting the common processing mode.
An embodiment of the present application provides a data event processing method, including: acquiring a data event to be processed; performing data decoupling processing on the data event to generate a plurality of atomic data modules; calling a preset base class corresponding to each atomic data module from a database; and processing the data event according to the preset base class and the data content of each atomic data module.
In an embodiment, the performing data decoupling processing on the data event to generate a plurality of atomic data modules includes: decomposing the processing flow of the data event into a plurality of sub-flows; splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as one atomic data module.
In an embodiment, the calling, from the database, a preset base class corresponding to each atomic data module includes: classifying a plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information; and calling the preset base class corresponding to each class in the database according to the classification result information.
In one embodiment, the processing the data event according to the preset base class and the data content of each atomic data module includes: judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module; if the preset base class does not contain all the logic for processing the atomic data modules, establishing a derived class for each atomic data module according to the preset base class and the data content; and executing the derived class, and generating processing result information of the data event.
In one embodiment, the method further comprises: and if the preset base class comprises all the logic for processing the atomic data modules, executing the preset base class corresponding to each atomic data module, and generating processing result information of the data event.
In an embodiment, the processing the data event according to the preset base class and the data content of each atomic data module further includes: judging whether an atom set belongs to the same preset base class or not, wherein the atom set comprises more than a preset number of atom data modules; if the atomic set exists, distributing a thread for each atomic data module in the atomic set; and executing threads corresponding to each atomic data module respectively, and generating processing result information of the atomic set.
In one embodiment, the method further comprises: judging whether the atomic data module which is not successfully processed exists in the atomic set; and if the atomic data module which is not successfully processed exists in the atomic set, generating rollback data according to the processing result information of the atomic data module which is successfully processed.
In one embodiment, the method further comprises: and if all the atomic data modules in the atomic set are successfully processed, adjusting the processing parameters of each atomic data module according to the result information of successful processing.
A second aspect of an embodiment of the present application provides a data event processing apparatus, including: the acquisition module is used for acquiring the data event to be processed; the decoupling module is used for carrying out data decoupling processing on the data event to generate a plurality of atomic data modules; the calling module is used for calling the preset base class corresponding to each atomic data module from the database; and the processing module is used for processing the data event according to the preset base class and the data content of each atomic data module.
In an embodiment, the decoupling module is configured to: decomposing the processing flow of the data event into a plurality of sub-flows; splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as one atomic data module.
In one embodiment, the calling module is configured to: classifying a plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information; and calling the preset base class corresponding to each class in the database according to the classification result information.
In one embodiment, the processing module is configured to: judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module; if the preset base class does not contain all the logic for processing the atomic data modules, establishing a derived class for each atomic data module according to the preset base class and the data content; and executing the derived class, and generating processing result information of the data event.
In one embodiment, the method further comprises: and the execution module is used for executing the preset base class corresponding to each atomic data module and generating the processing result information of the data event if the preset base class contains all the logic for processing the atomic data module.
In an embodiment, the processing module is further configured to: judging whether an atom set belongs to the same preset base class or not, wherein the atom set comprises more than a preset number of atom data modules; if the atomic set exists, distributing a thread for each atomic data module in the atomic set; and executing threads corresponding to each atomic data module respectively, and generating processing result information of the atomic set.
In one embodiment, the method further comprises: the judging module is used for judging whether the atomic data module which is not successfully processed exists in the atomic set; and the generation module is used for generating rollback data according to the processing result information of the atomic data module which is successfully processed if the atomic data module which is not successfully processed exists in the atomic set.
In one embodiment, the method further comprises: and the adjustment module is used for adjusting the processing parameters of each atomic data module according to the result information of successful processing if each atomic data module in the atomic set is processed successfully.
A third aspect of the embodiments of the present application provides an electronic device, including: a memory for storing a computer program; a processor configured to perform the method of the first aspect of the embodiments of the present application and any of the embodiments thereof, to process data events.
A fourth aspect of the present application provides a non-transitory electronic device readable storage medium, comprising: a program which, when run by an electronic device, causes the electronic device to perform the method of the first aspect of the embodiments of the present application and any of the embodiments thereof.
According to the data event processing method, device, equipment and storage medium, the data event is decoupled into the plurality of atomic data modules, and the atomic data modules are processed by adopting the preset base class, so that the whole data event is processed. The preset base class is used as a set of public processing mode, so that unified processing of data events in different service fields is realized, repeated development work is reduced, and development resources are saved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating a method for processing data events according to an embodiment of the present application;
FIG. 3A is a flow chart illustrating a method for processing data events according to an embodiment of the present application;
FIG. 3B is a code schematic diagram of a virtual method according to an embodiment of the present application;
FIG. 4A is a flow chart illustrating a method for processing data events according to an embodiment of the present application;
FIG. 4B is a code schematic diagram of an assembly strategy according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a data event processing device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the present application, the terms "first," "second," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.
As shown in fig. 1, the present embodiment provides an electronic apparatus 1 including: at least one processor 11 and a memory 12, one processor being exemplified in fig. 1. The processor 11 and the memory 12 are connected through the bus 10, and the memory 12 stores instructions executable by the processor 11, so that the electronic device 1 can execute all or part of the flow of the method in the embodiment described below, to abstract a common processing mode of the data event by decoupling the data event, and to process the data event in different service areas using the common processing mode.
In an embodiment, the electronic device 1 may be a mobile phone, a notebook computer, a desktop computer, or the like.
Please refer to fig. 2, which is a data event processing method according to an embodiment of the present application, the method may be executed by the electronic device 1 shown in fig. 1, and may be applied to a processing scenario of a log information floor event, so as to implement decoupling of the data event, abstract a common processing mode of the data event, and process the data event in different service areas by adopting the common processing mode. The method comprises the following steps:
step 201: and acquiring the data event to be processed.
In this step, the data event may be a computer-based data-in operation task, and the data event may originate from a different data system, such as an external data-in operation task, a log-landing task, etc., and may receive, in real time, the operation task to be processed, so as to obtain the data event to be processed.
Step 202: and carrying out data decoupling processing on the data event to generate a plurality of atomic data modules.
In this step, taking a log-to-floor event as an example, firstly, data decoupling is performed among a plurality of service modules of the log-to-floor event, and the service data of the log-to-floor event is divided into a plurality of minimum modules, namely, a plurality of atomic data modules are generated.
Step 203: and calling the preset base class corresponding to each atomic data module from the database.
In this step, the preset base class is a common base class of multiple data events, and the processing flow of similar data events can be abstracted based on historical data to form a common processing mode, and the preset base class is generated and stored in a database. When the method is used, the corresponding preset base class can be directly called from the database according to the characteristics of the atomic data module, the base class does not need to be reconfigured, and development resources are saved.
Step 204: and processing the data event according to the preset base class and the data content of each atomic data module.
In this step, after the preset base class is called, each atomic data module can be directly processed by adopting the preset base class, so as to complete the processing of the whole data event.
According to the data event processing method, the data event is decoupled into the plurality of atomic data modules, and the atomic data modules are processed by adopting the preset base class, so that the whole data event is processed. By using the preset base class as a set of common processing mode, unified processing of data events in different service fields is realized, repeated development work is reduced, and development resources are saved.
Please refer to fig. 3A, which is a data event processing method according to an embodiment of the present application, the method may be executed by the electronic device 1 shown in fig. 1, and may be applied to a processing scenario of a log information floor event, so as to implement decoupling of the data event, abstract a common processing mode of the data event, and process the data event in different service areas by adopting the common processing mode. The method comprises the following steps:
step 301: and acquiring the data event to be processed. See the description of step 201 in the above embodiments for details.
Step 302: the processing flow of the data event is broken down into a plurality of sub-flows.
In this step, the whole processing procedure of the data event is firstly decoupled, and the public data is extracted, and all the decoupled sub-flow modules are connected through the public data. Taking an event of log system floor log information as an example, firstly decomposing the whole flow of the log floor event, and mutually decoupling each service module; such as breaking up the entire flow of log floor events into: data verification, data assembly, data submission, post-submission processing and asynchronous processing are divided into five sub-flows.
Step 303: splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as an atomic data module.
In this step, the minimum atomization can be achieved according to the time sequence of each sub-flow, the dimension of the service module, and the like. And determining whether to continue splitting the sub-flow according to the data condition in the actual scene. For example, the data submitting sub-flow can be split into a plurality of table operation tasks, and each table operation task can be regarded as an atomic data module.
Step 304: and classifying the plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information.
In this step, each data event has its own time information and service attribute information, and the time information may be execution time of the process flow, and the service attribute information may be a service field of the data event. And extracting and structuring trunk data of the data event, wherein the trunk data becomes a tie between each atomic data module. For example, entity classes of each table operation task can be used as MasterLogDo attributes, and all data assembly is realized by assigning values to the MasterLogDo attributes. The real parameters used for the later table preservation operation are MasterLogDo. And then sorting the atomic data modules, and classifying the atomic data modules from the fields of processing sequence, service attribute and the like to generate classification result information. For example, the classification result information may include: atomic data modules contained in the sub-process data verification are classified into one type, atomic data modules contained in the data assembly are classified into one type, atomic data modules contained in each table operation (namely data submission) are classified into one type, atomic data modules contained in post-submission processing are classified into one type, and atomic data modules contained in asynchronous processing are classified into one type. Then, a plurality of atomic data modules can be arranged on a time axis according to the processing sequence, and then the atomic data modules in the same service field are arranged together in parallel according to the service field.
In one embodiment, the classification result information may be arranged in a time-axis manner as follows:
summarizing the following nodes according to the processing sequence, and classifying atoms into each node:
a: initialization of
b: verification (Validata) -data verification
c: pre-execution (BeforeAction) -assembling data
d: in execution (Action) -save table data
e: post-execution (AfterAction) -post-commit processing including failure processing and success processing
f: asynchronous processing (AsyncAction) -asynchronous processing
The steps can realize centralized management of all modules, improve code readability and reduce code maintenance cost.
Step 305: and calling a preset base class corresponding to each class in the database according to the classification result information.
In this step, the preset base class may be generated according to the historical data, for example, based on the whole flow of the historical data event, different time points are divided according to the sequence of processing—this time point is referred to as a time node. Extracting and abstracting the time nodes to form a set of common processing modes, and further generating corresponding preset base classes.
In one embodiment, as shown in FIG. 3B, the time nodes may be summarized based on historical data events: the method comprises the steps of executing, post-executing and asynchronous processing, and abstracting the five time nodes into class ProcessorBase to form corresponding virtual methods respectively: check Validata (), pre-execution BeforeAction (result), in-execution Action (result), post-execution AfterAction (result), asynchronous process AsyncProcess (result). And further respectively generating preset base classes, and calling corresponding virtual methods Validata (), beforeAction (result), action (result), afterAction (result) and AsyncProcess (result) to encapsulate a processing flow (Process).
After the packaging processing Process, the interface is extracted from the preset base class, so that the preset base class realizes the Process method of the interface. When the method is used, the preset base class corresponding to each category is identified in the classification result information, and the preset base class corresponding to each category is called from the database. For example, in actual use, an interface is declared, and a Process procedure is called through a Process method of the interface.
Step 306: and judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module. If yes, go to step 309, otherwise go to step 307.
In this step, since the newly added data event is often different from the historical data event, the preset base class does not necessarily include all logic of all newly added data events. Therefore, according to the data content of each atomic data module, whether all logic for processing the atomic data module is contained in the preset base class can be judged, and further, according to different judging results, different processing modes are executed.
Step 307: and establishing a derived class for each atomic data module according to the preset base class and the data content.
In this step, if all the logic for processing the atomic data module is not included in the preset base class, it is indicated that the preset base class cannot complete the complete processing of the data event, a derivative class may be established for the corresponding atomic data module according to the actual situation of the data event, and the derivative class inherits the corresponding preset base class. The derived classes can rewrite the methods of pre-execution, in-execution, post-execution and asynchronous processing according to actual needs.
Step 308: and executing the derived class, and generating the processing result information of the data event.
In this step, each atomic data module may be processed by executing the derived class, and processing result information of the data event may be generated according to the execution result.
Step 309: and executing the preset base class corresponding to each atomic data module to generate the processing result information of the data event.
In this step, if all the logic of the atomic data processing module is included in the preset base class, it means that the processing of the data event can be completed without creating a derivative class and directly executing the corresponding preset base class, and the processing result information can be generated.
In one embodiment, a monitoring mechanism may be established for the entire data event handling process, such as may be implemented by AOP (Aspect Oriented Programming, tangent plane oriented programming) monitoring. The RequestMonitor and Running methods are implemented by delegation (Func and Action). The RequestMonitor is the time taken to record the pre-Invoke request argument and post-Invoke () return content, and calculate and record Invoke () and run the time taken before and after Invoke (). AOP monitoring can cover the whole Process flow, and can wrap the actions and the afteractions in the Process () of the base class by using a RequestMonitor, and the following codes are RequestMonitor wrapping Action () examples:
PerformanceMonitoring.RequestMonitor(_＝>Action(result),result,"ProcessorFun Process Action",Tags)；
please refer to fig. 4, which is a data event processing method according to an embodiment of the present application, the method may be executed by the electronic device 1 shown in fig. 1, and may be applied to a processing scenario of a log information floor event, so as to implement decoupling of the data event, abstract a common processing mode of the data event, and process the data event in different service areas by adopting the common processing mode. The method comprises the following steps:
step 401: and acquiring the data event to be processed. See the description of step 201 in the above embodiments for details.
Step 402: the processing flow of the data event is broken down into a plurality of sub-flows. See the description of step 302 in the above embodiments for details.
Step 403: splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as an atomic data module. See for details the description of step 303 in the above embodiments.
Step 404: and classifying the plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information. See the description of step 304 in the above embodiments for details.
Step 405: and calling a preset base class corresponding to each class in the database according to the classification result information. See the description of step 305 in the above embodiments for details.
Step 406: and judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module. If yes, go to step 408, otherwise go to step 407. See the description of step 306 in the above embodiments for details.
Step 407: if all logic for processing the atomic data modules is not contained in the preset base class, a derivative class is established for each atomic data module according to the preset base class and the data content. And proceeds to step 408. See for details the description of step 307 in the above embodiments.
Step 408: judging whether an atom set belongs to the same preset base class and/or derivative class, wherein the atom set comprises more than a preset number of atom data modules. If yes, go to step 409, otherwise go to step 414.
In this step, if all the logic for processing the atomic data modules is included in the preset base class or all the logic for processing the atomic data modules is included in the newly created derivative class, it is further determined whether the number of the atomic data modules included in the same preset base class or derivative class exceeds the preset number (i.e., whether the atomic set exists), and if the preset base class and the derivative class exist at the same time, it is determined whether the atomic set exists. If so, step 409 is entered. Otherwise, the atomic data modules contained in the same preset base class or derivative class are not more than the preset number, and multithreading is not needed. For example, the volume of data transmitted by the log floor processing flow is not large, the number of assembled data tables is only 4, and the number of the assembled data tables is lower than the preset number. If the multi-thread concurrent operation is adopted, the time can be saved, but the effect is not obvious, the multi-thread is not easy to control, and the risk is high, so that the multi-thread processing is not needed. Whether or not a multithreaded execution policy is used in the flow may depend on the actual traffic scenario.
In an embodiment, the preset number may be determined according to historical statistics, for example, when the number of atomic data modules in the atomic collection is greater than a certain number, and multithreading concurrency is required, the number may be set to be the preset number.
Step 409: each atomic data module in the atomic set is assigned a thread.
In this step, if there is an atom set, a thread is assigned to each atom data module in the atom set. Considering that the atomic data modules in the atomic set have the same time node (such as Validata, beforeAction, action), the coupling degree of each atomic data module is low, the level is the same, the atomic number is variable, the atomic structure is similar, and policy management can be performed. Taking Action nodes as examples, the following takes the actual service as a storage of each table data for detailed explanation:
1) Policy definition
Each atomic data module is classified-one class processes one module, and the class provides service to the outside through a single instance. For example, in a log floor event, each table operation corresponds to a class.
Definition: the Process () for processing the main traffic (forward flow) and Cancel the Cancel () method are performed, and the Cancel () for traffic rollback when the main flow fails. The Process () method inserts and modifies table data for the log master table, and cancer () is data inserted before rollback. Common arguments cts, masterLogDO and result are provided for Process () and Cancel (). The extraction interface IStrategy defines the Process () and cancer () methods.
2) Assembly strategy
Data is prepared, as shown in fig. 4B, real parameters are prepared for each policy Process () method, and the data must be structured (typed). In the floor log processing flow, this operation can be performed in a BeforeAction, which belongs to the assembly data. All data of the tables to be saved are then assigned to masterlog do.
Preparation policies, as shown in FIG. 4B, are the same as preparing data, also in BeforeAction, except that instead of assigning a value to MasterLogDo, all table policies that hold data are placed in a same level policy array, list < IStratey >.
Step 410: and executing threads corresponding to each atomic data module respectively to generate processing result information of the atomic set.
In this step, the execution of the policies may bind the threads to the policies, create threads by taskfactor and bind the policies, each thread corresponding to a policy. When executed, the execution time, here the policy enforcement portal, is recorded using the performancemonitor.
Step 411: and judging whether an atomic data module which is not successfully processed exists in the atomic set. If yes, go to step 412, otherwise go to step 413.
In this step, before each policy execution, it is determined whether to cancel (IsCancelationRequested) or not, and if so, the process returns directly. The exception or failure of the execution process belongs to unsuccessful processing, and task is performed, and the successful execution process saves a successful strategy into a successful strategy set (success list).
Step 412: and generating rollback data according to the processing result information of the atomic data module which is successfully processed.
In this step, if there is an atomic data module that has not been successfully processed in the atomic set, this indicates that the policy was canceled (IsCancelationRequested) before execution, or an exception occurred. The policy execution is unsuccessful, a cancer () method provided by the policy is called to generate rollback data, the rollback data can be realized by a successful policy set (success list) in the policy execution, the same principle in Action is utilized, the policy in the multi-thread binding success list set is utilized, and then the cancer () of the policy is called to realize the data rollback.
Step 413: and adjusting the processing parameters of each atomic data module according to the result information of successful processing. And proceeds to step 414.
In this step, the policy, whether successful or unsuccessful in execution, will enter the subsequent processing where the AfterAction of the abstract main flow process is engaged. In the whole Process of log landing, if each atomic data module in the atomic set is processed successfully, the method indicates that all the policies are executed successfully, the AfterAction supplements the main key of the main table (main table) of the log to other tables as external keys, and the engineering is the same as the execution of the policies in the Action, and whether the operations are added or modified is judged in the Process () method of the table policies according to parameters.
Step 414: and executing a preset base class and/or a derivative class corresponding to each atomic data module, and generating processing result information of the data event.
In this step, for the preset base class and/or the derivative class corresponding to the atomic data module that does not need multithreading concurrent processing, the corresponding preset base class and/or derivative class is directly executed. The data processing results of each atomic set and the execution results of other preset base classes and derivative classes of the log floor event which do not need multithreading are combined to form the processing result information of the whole data event. Details of the relevant matters are described in the above embodiments with respect to steps 308 and 309.
According to the data event processing method, the whole event processing process is decoupled, common data are extracted, all decoupled atomic data modules are connected through the common data, then the atomic data modules are arranged on a time axis according to the processing sequence, and then the atomic modules in the same field are arranged in parallel together in a concentrated mode according to the service field. The whole flow is divided into different time points according to the sequence of processing, and the time points are called as time nodes.
The time nodes are extracted and abstracted to form a set of common processing modes, and all the processing procedures are unified to the common processing modes. For the atomic modules arranged in parallel, a public access method can be extracted for the atomic modules, a failure rollback method is formulated, a strategy mode is evolved, and an atomic strategy is collected by a set. The collected atomic strategies are processed by using a multithreading technology, a public access method can be extracted for the atomic modules arranged in parallel, a failure rollback method is formulated, a strategy mode is evolved, and the atomic strategies are collected by a set. The collected atomic strategies are processed concurrently by using the multithreading technology.
Please refer to fig. 5, which illustrates a data event processing apparatus 500 according to an embodiment of the present application, which is applicable to the electronic device 1 illustrated in fig. 1 and can be applied to a processing scenario of a log information floor event, so as to decouple the data event, abstract a common processing mode of the data event, and process the data event in different service areas by adopting the common processing mode. The device comprises: the principle relationship of the acquisition module 501, the decoupling module 502, the calling module 503 and the processing module 504 is as follows:
an acquisition module 501, configured to acquire a data event to be processed. See the description of step 201 in the above embodiments for details.
The decoupling module 502 is configured to perform data decoupling processing on the data event, and generate a plurality of atomic data modules. See the description of step 202 in the above embodiments for details.
And the calling module 503 is configured to call the preset base class corresponding to each atomic data module from the database. See for details the description of step 203 in the above embodiments.
And the processing module 504 is configured to process the data event according to the preset base class and the data content of each atomic data module. See the description of step 204 in the above embodiments for details.
In one embodiment, the decoupling module 502 is configured to: the processing flow of the data event is broken down into a plurality of sub-flows. Splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as an atomic data module. See the description of steps 302 through 303 in the above embodiments for details.
In one embodiment, the calling module 503 is configured to: and classifying the plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information. And calling a preset base class corresponding to each class in the database according to the classification result information. See the description of steps 304-305 in the above embodiments for details.
In one embodiment, the processing module 504 is configured to: and judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module. If all logic for processing the atomic data modules is not contained in the preset base class, a derivative class is established for each atomic data module according to the preset base class and the data content. And executing the derived class, and generating the processing result information of the data event. See the description of steps 306-308 in the above embodiments for details.
In one embodiment, the method further comprises: the execution module 505 is configured to execute the preset base class corresponding to each atomic data module if the preset base class includes all the logic for processing the atomic data modules, and generate processing result information of the data event. See the description of step 309 in the above embodiments for details.
In one embodiment, the processing module 504 is further configured to: judging whether an atom set belongs to the same preset base class or not, wherein the atom set comprises more than a preset number of atom data modules. If an atom set exists, a thread is allocated to each atom data module in the atom set. And executing threads corresponding to each atomic data module respectively to generate processing result information of the atomic set. See the description of steps 408-410 in the above embodiments for details.
In one embodiment, the method further comprises: a judging module 506, configured to judge whether an atomic data module that is not successfully processed exists in the atomic set. And the generating module 507 is configured to generate rollback data according to the processing result information of the atomic data module that has been successfully processed if there is an atomic data module that has not been successfully processed in the atomic set. See the description of steps 411 to 412 in the above embodiments for details.
In one embodiment, the method further comprises: and the adjusting module 508 is configured to adjust a processing parameter of each atomic data module according to the result information of the successful processing if each atomic data module in the atomic set has been processed successfully. See for details the description of step 413 in the above embodiments.
For a detailed description of the data event processing apparatus 500, please refer to the description of the related method steps in the above embodiment.
The embodiment of the invention also provides a non-transitory electronic device readable storage medium, which comprises: a program which, when run on an electronic device, causes the electronic device to perform all or part of the flow of the method in the above-described embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD), etc. The storage medium may also comprise a combination of memories of the kind described above.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.
Claims (16)
1. A method of processing a data event, comprising:
acquiring a data event to be processed;
performing data decoupling processing on the data event to generate a plurality of atomic data modules;
calling a preset base class corresponding to each atomic data module from a database; the preset base class is a public base class of various data events;
processing the data event according to the preset base class and the data content of each atomic data module;
wherein, the processing the data event according to the preset base class and the data content of each atomic data module includes:
judging whether an atom set belongs to the same preset base class or not, wherein the atom set comprises more than a preset number of atom data modules;
if the atomic set exists, distributing a thread for each atomic data module in the atomic set;
and executing threads corresponding to each atomic data module respectively, and generating processing result information of the atomic set.
2. The method of claim 1, wherein said subjecting the data event to a data decoupling process generates a plurality of atomic data modules, comprising:
decomposing the processing flow of the data event into a plurality of sub-flows;
splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as one atomic data module.
3. The method of claim 1, wherein the calling, from the database, the preset base class corresponding to each of the atomic data modules includes:
classifying a plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information;
and calling the preset base class corresponding to each class in the database according to the classification result information.
4. The method of claim 1, wherein said processing said data event based on said preset base class and data content of each of said atomic data modules comprises:
judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module;
if the preset base class does not contain all the logic for processing the atomic data modules, establishing a derived class for each atomic data module according to the preset base class and the data content;
and executing the derived class, and generating processing result information of the data event.
5. The method as recited in claim 4, further comprising:
and if the preset base class comprises all the logic for processing the atomic data modules, executing the preset base class corresponding to each atomic data module, and generating processing result information of the data event.
6. The method as recited in claim 1, further comprising:
judging whether the atomic data module which is not successfully processed exists in the atomic set;
and if the atomic data module which is not successfully processed exists in the atomic set, generating rollback data according to the processing result information of the atomic data module which is successfully processed.
7. The method as recited in claim 6, further comprising:
and if all the atomic data modules in the atomic set are successfully processed, adjusting the processing parameters of each atomic data module according to the result information of successful processing.
8. A data event processing apparatus, comprising:
the acquisition module is used for acquiring the data event to be processed;
the decoupling module is used for carrying out data decoupling processing on the data event to generate a plurality of atomic data modules;
the calling module is used for calling the preset base class corresponding to each atomic data module from the database; the preset base class is a public base class of various data events;
the processing module is used for processing the data event according to the preset base class and the data content of each atomic data module;
wherein, the processing the data event according to the preset base class and the data content of each atomic data module includes:
judging whether an atom set belongs to the same preset base class or not, wherein the atom set comprises more than a preset number of atom data modules;
if the atomic set exists, distributing a thread for each atomic data module in the atomic set;
and executing threads corresponding to each atomic data module respectively, and generating processing result information of the atomic set.
9. The apparatus of claim 8, wherein the decoupling module is to:
decomposing the processing flow of the data event into a plurality of sub-flows;
splitting each sub-flow into a plurality of table operation tasks, wherein each table operation task serves as one atomic data module.
10. The apparatus of claim 8, wherein the invoking module is configured to:
classifying a plurality of atomic data modules according to the time information and the service attribute information of the data event to generate classification result information;
and calling the preset base class corresponding to each class in the database according to the classification result information.
11. The apparatus of claim 8, wherein the processing module is to:
judging whether all logic for processing the atomic data modules is contained in the preset base class according to the data content of each atomic data module;
if the preset base class does not contain all the logic for processing the atomic data modules, establishing a derived class for each atomic data module according to the preset base class and the data content;
and executing the derived class, and generating processing result information of the data event.
12. The apparatus as recited in claim 11, further comprising:
and the execution module is used for executing the preset base class corresponding to each atomic data module and generating the processing result information of the data event if the preset base class contains all the logic for processing the atomic data module.
13. The apparatus as recited in claim 8, further comprising:
the judging module is used for judging whether the atomic data module which is not successfully processed exists in the atomic set;
and the generation module is used for generating rollback data according to the processing result information of the atomic data module which is successfully processed if the atomic data module which is not successfully processed exists in the atomic set.
14. The apparatus as recited in claim 13, further comprising:
and the adjustment module is used for adjusting the processing parameters of each atomic data module according to the result information of successful processing if each atomic data module in the atomic set is processed successfully.
15. An electronic device, comprising:
a memory for storing a computer program;
a processor configured to perform the method of any one of claims 1 to 7 to process data events.
16. A non-transitory electronic device-readable storage medium, comprising: program which, when run by an electronic device, causes the electronic device to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331024.0A CN111538491B (en) | 2020-04-24 | 2020-04-24 | Data event processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331024.0A CN111538491B (en) | 2020-04-24 | 2020-04-24 | Data event processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111538491A CN111538491A (en) | 2020-08-14 |
| CN111538491B true CN111538491B (en) | 2023-07-14 |
Family
ID=71967569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010331024.0A Active CN111538491B (en) | 2020-04-24 | 2020-04-24 | Data event processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111538491B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113885863A (en) * | 2021-10-21 | 2022-01-04 | 上海波顿诺华智能科技有限公司 | Method, device, electronic equipment and storage medium for realizing SOA (service oriented architecture) service |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB736774A (en) * | 1950-10-06 | 1955-09-14 | Atomic Energy Authority Uk | Improvements in or relating to target structures for electromagnetic separators |
| CN106325980A (en) * | 2015-06-30 | 2017-01-11 | 中国石油化工股份有限公司 | Multi-thread concurrent system |
| CN107545030A (en) * | 2017-07-17 | 2018-01-05 | 阿里巴巴集团控股有限公司 | Processing method, device and the equipment of data genetic connection |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4873557B2 (en) * | 2007-01-12 | 2012-02-08 | ルネサスエレクトロニクス株式会社 | Data processing apparatus and semiconductor integrated circuit |
| US8065293B2 (en) * | 2007-10-24 | 2011-11-22 | Microsoft Corporation | Self-compacting pattern indexer: storing, indexing and accessing information in a graph-like data structure |
| CN103336694B (en) * | 2013-07-08 | 2016-06-22 | 北京航空航天大学 | The assemble method of a kind of entity behavior modeling and system |
| CN106569786A (en) * | 2015-10-10 | 2017-04-19 | 阿里巴巴集团控股有限公司 | Development method of service application and system |
| CN107918556A (en) * | 2016-10-11 | 2018-04-17 | 北京京东尚科信息技术有限公司 | A kind of timed task performs method and apparatus in the parallel of multiple servers |
| CN108256716B (en) * | 2016-12-29 | 2020-10-16 | 同方威视技术股份有限公司 | Method and apparatus for configuring/executing processes and/or atomic processes based on tables |
-
2020
- 2020-04-24 CN CN202010331024.0A patent/CN111538491B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB736774A (en) * | 1950-10-06 | 1955-09-14 | Atomic Energy Authority Uk | Improvements in or relating to target structures for electromagnetic separators |
| CN106325980A (en) * | 2015-06-30 | 2017-01-11 | 中国石油化工股份有限公司 | Multi-thread concurrent system |
| CN107545030A (en) * | 2017-07-17 | 2018-01-05 | 阿里巴巴集团控股有限公司 | Processing method, device and the equipment of data genetic connection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111538491A (en) | 2020-08-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7023718B2 (en) | Selecting a query to execute against a real-time data stream | |
| CN103559118B (en) | A kind of method for auditing safely based on AOP and annotating information system | |
| US20110219383A1 (en) | Processing model-based commands for distributed applications | |
| US20160188385A1 (en) | Optimized system for analytics (graphs and sparse matrices) operations | |
| CN102375731A (en) | Coding-free integrated application platform system | |
| US11366713B2 (en) | System and method for automatically identifying and resolving computing errors | |
| US20170206119A1 (en) | System and method for achieving specific behaviors by intercepting file access calls in a mainframe rehosting platform | |
| Schmutz et al. | Service-oriented architecture: an integration blueprint: a real-world SOA strategy for the integration of heterogeneous enterprise systems: successfully implement your own enterprise integration architecture using the trivadis integration architecture blueprint | |
| US9542656B2 (en) | Supporting ETL processing in BPEL-based processes | |
| US10108473B2 (en) | System and method for automatic error classification in integration systems | |
| CN111538491B (en) | Data event processing method, device, equipment and storage medium | |
| US20140372488A1 (en) | Generating database processes from process models | |
| US8930960B2 (en) | Methods and systems for object interpretation within a shared object space | |
| CN113031964B (en) | Big data application management method, device, equipment and storage medium | |
| WO2021036987A1 (en) | Method and device for achieving operation and maintenance monitoring | |
| CN114116509A (en) | Program analysis method, program analysis device, electronic device, and storage medium | |
| US10657476B2 (en) | Just in time compilation (JIT) for business process execution | |
| US11023839B2 (en) | Workflow integration | |
| EP2601627B1 (en) | Transaction processing system and method | |
| US20070299849A1 (en) | Logging of application messages | |
| CN111782363A (en) | Method and flow system for supporting multi-service scene calling | |
| Wang et al. | Mapcheckreduce: an improved mapreduce computing model for imprecise applications | |
| AU2021287457B2 (en) | "Log Data Compliance" | |
| KR101824084B1 (en) | Data processing method applicable to various analysis models for business analysis service system based on cloud computing | |
| Pradhan | Role of Scripting Language on Unix Operating System for Risk Assessment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |