CN115827754A

CN115827754A - Data processing method in isomorphic environment based on abstract strategy model

Info

Publication number: CN115827754A
Application number: CN202211491652.0A
Authority: CN
Inventors: 姚宏宇; 朱朝强; 田越; 孙彦辉
Original assignee: BEIJING YOYO TIANYU SYSTEM TECHNOLOGY CO LTD
Current assignee: BEIJING YOYO TIANYU SYSTEM TECHNOLOGY CO LTD
Priority date: 2022-11-25
Filing date: 2022-11-25
Publication date: 2023-03-21

Abstract

Embodiments of the application provide a data processing method, device, equipment and computer readable storage medium in a homogeneous environment based on an abstract policy model. The method comprises the steps of matching a corresponding strategy model for each data packet; sending the data packet matched with the strategy model to a corresponding data processing node, processing the data packet based on the strategy model, and outputting the processed data to a target library; wherein the policy execution environment of each data processing node is homogeneous. In this way, high reliability of data processing tasks is achieved, installation, deployment, expansion and contraction are simpler based on the homogeneous environment of the data processing nodes, any data packet in the transmission process can be executed on any data processing node in a strategy mode, and data processing drifting is achieved.

Description

Data processing method in isomorphic environment based on abstract strategy model

Technical Field

Embodiments of the present application relate to the field of data processing, and in particular, to a data processing method, apparatus, device, and computer-readable storage medium in a homogeneous environment based on an abstract policy model.

Background

In enterprise-level applications, multi-source heterogeneous data sources with different specifications often exist behind application systems of different business lines, customized development needs to be performed according to the heterogeneous data sources no matter an ETL tool or a stream processing middleware is used, and if the standards of the data sources change, related programs need to be changed. The processing logic of the data is complex and scattered and is hidden in the code logic, and the data is lack of uniform management configuration and difficult to multiplex;

different data processing execution environments are complex and various, a uniform operation environment is lacked, rapid deployment is difficult, traditional data processing architectures are based on a Master/Slave architecture (with a center), complete isomorphism of the operation environment cannot be achieved, and if a problem occurs in a data processing task, the task needs to be executed again. If the node executing the task has a problem, the data is difficult to distribute to other non-isomorphic nodes for continuous operation, and the whole process of data processing is difficult to track due to the interruption of the task, thereby bringing great difficulty to data management.

Disclosure of Invention

According to the embodiment of the application, a data processing scheme based on an abstract strategy model in a homogeneous environment is provided.

In a first aspect of the present application, a data processing method in a homogeneous environment based on an abstract policy model is provided. The method comprises the following steps:

matching a corresponding strategy model for each data packet;

sending the data packet matched with the strategy model to a corresponding data processing node, processing the data packet based on the strategy model, and outputting the processed data to a target library;

wherein the policy execution environment of each data processing node is homogeneous.

Further, the strategy model is obtained by the following method:

abstracting operation logic for reading, processing and/or writing data into a strategy model; the policy model comprises a policy unique identifier, a policy definition, an executive body of the policy and/or a sending target.

Further, the policy definition includes data source, data type, ordering rules, processing priority, and/or data dependencies.

Further, the matching of the corresponding policy model for each data packet includes:

one or more policy models are matched for each data.

Further, the data processing node includes a transport layer, a policy execution environment, a policy repository, job management, scheduling, exchange management, and/or logging functionality.

Further, the sending the data packet matched with the policy model to the corresponding data processing node, and processing the data based on the policy model includes:

and if the processing node can not complete the processing of the data packet, sending the data packet matched with the strategy model to a new exchange node for processing according to the strategy model.

Further, the method also comprises the following steps:

and if the processing node can not complete the processing of the data packet, scheduling a plurality of nodes to be combined according to the strategy model to complete the execution of all strategies.

In a second aspect of the present application, a data processing apparatus in a homogeneous environment based on an abstract policy model is provided. The device comprises:

the matching module is used for matching the corresponding strategy model for each data packet;

the processing module is used for sending the data packet matched with the strategy model to a corresponding data processing node, processing the data packet based on the strategy model and outputting the processed data to a target library;

In a third aspect of the present application, an electronic device is provided. The electronic device includes: a memory having a computer program stored thereon and a processor implementing the method as described above when executing the program.

In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the method as according to the first aspect of the present application.

According to the data processing method in the isomorphic environment based on the abstract strategy model, the corresponding strategy model is matched for each data packet; sending the data packet matched with the strategy model to a corresponding data processing node, processing the data packet based on the strategy model, and outputting the processed data to a target library; the strategy execution environment of each data processing node is isomorphic, and high reliability of all data processing tasks is achieved.

It should be understood that what is described in this summary section is not intended to limit key or critical features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will become apparent from the following description.

Drawings

The above and other features, advantages and aspects of various embodiments of the present application will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:

FIG. 1 is a flow diagram of a data processing method in a homogeneous environment based on an abstract policy model according to an embodiment of the application;

FIG. 2 is a schematic diagram of a data packet according to an embodiment of the present application;

FIG. 3 is a schematic diagram of policy enforcement according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a homogeneous execution environment according to an embodiment of the present application;

FIG. 5 is a block diagram of a data processing apparatus in a homogeneous environment based on an abstract policy model according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a terminal device or a server suitable for implementing the embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter associated objects are in an "or" relationship.

FIG. 1 shows a flow diagram of a data processing method in a homogeneous environment based on an abstract policy model according to an embodiment of the present disclosure. The method comprises the following steps:

and S110, matching a corresponding strategy model for each data packet.

Wherein the policy model may be constructed by:

abstracting any operation of data processing into a strategy model; including read, process, and/or write operations.

Further, the policy model comprises a policy unique identifier, a policy definition, an executor of a policy and/or a sending target.

For example, a sorting operation is performed on data, a policy may set a unique identifier, a policy definition may include a data source, a data type, a sorting rule, a processing priority, and the like, an executor of the policy may be a piece of sorting algorithm, and a sending target may specify a message topic of a next processing unit.

In some embodiments, the policy model is registered in a rule policy repository, each processing policy having a unique identifier; the rule policy repository is a collection of all policies. When data is to be converted, a processing policy unique identification is attached to each piece of data. And the data strategy executor finds a processing strategy model according to the strategy unique identifier and uses a strategy engine to process data.

The rule strategy library is defined by a code mode and is combined with a configuration mode to carry out system built-in;

further, the policy models in the rule model policy library are all policy models that pass verification.

In some embodiments, the policy model (policy definition) comprises:

1. the metadata describes:

input data field: field name, field type, field length, field description, etc.;

output data field: field name, field type, field length, field description, etc.;

description of data: data type, data format, data description, etc.;

2. class name and parameters of the data processing executor (taking the string processing executor as an example):

an actuator: processing a character string;

parameters are as follows: converting upper case to lower case, and counting;

3. run-time variables:

log level, LAST _ VALUE, JVM memory parameters, etc.;

4. upstream information:

upstream packet information, such as unique identification, processing results, etc.;

5. data dependency relationship:

when the data packet is processed, the data of other data packets is depended on, such as multi-table combination and the like;

6. executing the state data:

real-time information of the current actuator for processing the data packet;

7. data verification information:

such as packet size, number of records, number of upstream records, current processing progress, etc.

Further, after matching the corresponding policy model for each data packet, the data packet (data matched with the policy model) as shown in fig. 2 is obtained.

Further, when the data includes multiple operations, a corresponding plurality of policy models may be matched.

S120, sending the packet data matched with the strategy model to a corresponding data processing node, processing the data packet based on the strategy model, and outputting the processed data to a target library

In some embodiments, the policy execution flow is shown in fig. 3, and includes:

strategy design: setting a specific strategy model (strategy definition) through a graphical interface of a management node;

and (3) policy verification: trial running the strategy model and verifying the execution result of the strategy model;

and (3) policy issuing: issuing a strategy model qualified in trial operation (only the issued strategy model can be operated), namely registering the strategy model qualified in trial operation to a rule strategy library;

strategy operation: after the switching node receives the data packet, the strategy executor analyzes the strategy in the data packet, then starts a corresponding strategy processing thread to process the data, and reports the processing progress of the data to the monitor.

In some embodiments, the policy execution environment of each data processing node is the same, as shown in FIG. 4, and each node includes transport layers, policy execution environments, policy libraries, and job management, scheduling, exchange management, and/or logging functions. After the data processing node receives the data packet matched with the strategy model, the strategy executor analyzes the strategy in the data packet, the role played by the processing node is determined according to the strategy, and the data processing node only starts different functions according to the different played roles to process data, so that the premise and guarantee are provided for the high availability and the high reliability of the node. I.e., the execution of the same or similar policies, may be performed by one module. The module is a Java (or other language) program package running on the switching node, loads and calls different program packages, and completes access to different data processing strategies and execution of specific processing strategies.

In some embodiments, if the processing node cannot complete the processing of the data, the data matched with the policy model is sent to a new switching node for processing according to the policy model; or scheduling a plurality of nodes to combine according to the strategy model to complete the execution of all strategies.

In some embodiments, in a distributed system, before a processing node processes a data packet, the data packet may be split, for example, by splitting a row and splitting a column, and the split data packet may be sent to a corresponding processing node for processing.

According to the embodiment of the disclosure, the following technical effects are achieved:

the data and the processing logic are combined through the strategy model, so that the unified configuration management of the data processing logic is realized, and convenience is provided for data management based on data blood relationship analysis; the data processing nodes are completely isomorphic, so that the installation, deployment and expansion are simpler, any data packet in the transmission process can be processed on the data nodes, and the drift of data processing is realized.

Meanwhile, because the self (data packet) is provided with the processing logic, the processing logic does not need to be added on the replaced node even if the processing node is replaced or the multi-processing node is needed, so that the whole system has high expansibility, and the working efficiency is improved.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are exemplary embodiments and that the acts and modules referred to are not necessarily required in this application.

The above is a description of method embodiments, and the embodiments of the present application are further described below by way of apparatus embodiments.

FIG. 5 shows a block diagram of a data processing apparatus 500 in a homogeneous environment based on an abstract policy model according to an embodiment of the application As shown in FIG. 5, the apparatus 500 comprising:

a matching module 510, configured to match a corresponding policy model for each data packet;

the processing module 520 is configured to send the data packet matched with the policy model to a corresponding data processing node, process the data packet based on the policy model, and output the processed data to the target library;

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the described module may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

Fig. 6 shows a schematic structural diagram of a terminal device or a server suitable for implementing the embodiments of the present application.

As shown in fig. 6, the terminal device or the server includes a Central Processing Unit (CPU) 601 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the terminal device or the server are also stored. The CPU601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.

In particular, the above method flow steps may be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present application may be implemented by software or hardware. The described units or modules may also be provided in a processor. Wherein the designation of a unit or module does not in some way constitute a limitation of the unit or module itself.

As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may be separate and not incorporated into the electronic device. The computer readable storage medium stores one or more programs that, when executed by one or more processors, perform the methods described herein.

The foregoing description is only exemplary of the preferred embodiments of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the application referred to in the present application is not limited to the embodiments with a particular combination of the above-mentioned features, but also encompasses other embodiments with any combination of the above-mentioned features or their equivalents without departing from the spirit of the application. For example, the above features may be replaced with (but not limited to) features having similar functions as those described in this application.

Claims

1. A data processing method in a isomorphic environment based on an abstract strategy model is characterized by comprising the following steps:

matching a corresponding strategy model for each data packet;

2. The method of claim 1, wherein the policy model is obtained by:

abstracting operation logic for reading, processing and/or writing data into a strategy model; the strategy model comprises a strategy unique identifier, a strategy definition, an executive body of the strategy and/or a sending target.

3. The method of claim 2, wherein the policy definition comprises a data source, a data type, a sorting rule, a processing priority, and/or a data dependency.

4. The method of claim 3, wherein matching the corresponding policy model for each packet comprises:

one or more policy models are matched for each packet.

5. The method of claim 4, wherein the data processing node comprises a transport layer, a policy execution environment, a policy repository, job management, scheduling, exchange management, and/or logging functionality.

6. The method of claim 5, wherein the sending of the data matched with the policy model to the corresponding data processing node, the processing of the data based on the policy model comprises:

and if the processing node can not complete the processing of the data, sending the data packet matched with the strategy model to a new exchange node for processing according to the strategy model.

7. The method of claim 6, further comprising:

8. A data processing apparatus in a homogeneous environment based on an abstract policy model, comprising:

9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.