CN115562654A - Operation method and device based on general data operation framework and electronic equipment - Google Patents

Operation method and device based on general data operation framework and electronic equipment Download PDF

Info

Publication number
CN115562654A
CN115562654A CN202211128998.4A CN202211128998A CN115562654A CN 115562654 A CN115562654 A CN 115562654A CN 202211128998 A CN202211128998 A CN 202211128998A CN 115562654 A CN115562654 A CN 115562654A
Authority
CN
China
Prior art keywords
data
node
computing
framework
read
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211128998.4A
Other languages
Chinese (zh)
Inventor
任金昊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CLP Cloud Digital Intelligence Technology Co Ltd
Original Assignee
CLP Cloud Digital Intelligence Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CLP Cloud Digital Intelligence Technology Co Ltd filed Critical CLP Cloud Digital Intelligence Technology Co Ltd
Priority to CN202211128998.4A priority Critical patent/CN115562654A/en
Publication of CN115562654A publication Critical patent/CN115562654A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/36—Software reuse
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Stored Programmes (AREA)

Abstract

The invention provides an operation method and device based on a general data operation framework and electronic equipment, wherein the method comprises the following steps: reading data of different data sources, wherein the data are read into the input nodes through the same interface; performing operation on data by using a pre-configured operation function; and outputting the calculated data to a specified position. The operation method based on the universal data operation framework exposes the corresponding interfaces for data input, operation and output, different data sources and the operation framework can be accessed into the framework through the realization interfaces, and a standard operation component is defined, so that repeated multiplexing can be realized at one time.

Description

Operation method and device based on general data operation framework and electronic equipment
Technical Field
The invention relates to the technical field of information technology application, in particular to an operation method and device based on a general data operation framework and electronic equipment.
Background
In IT (information technology) application scenarios, there are a large number of data operation requirements, and the technical differences of these requirements are mainly reflected in the differences of data sources and the differences of operation tools, these differences make different projects need to be developed separately for specific requirements, and similar logics in different requirements cannot be reused due to no standard atom definition, resulting in many repeated jobs. Users of various technologies and middleware are mostly technical personnel, and a use threshold exists for users who are not technically present.
Disclosure of Invention
The technical problem to be solved by the invention is that a large amount of data operation needs exist in an IT application scene, the technical differences of the needs are mainly reflected in the difference of data sources and the difference of operation tools, different projects need to be separately developed according to specific needs, and similar logics in different needs cannot be reused due to the fact that no standard atomic definition exists, so that a large amount of repeated work is caused.
The technical scheme adopted by the invention is that in the operation method based on the general data operation framework, the operation framework comprises the following steps: an input node, at least one compute node, and an output node, the method comprising:
reading data of different data sources, wherein the data are read into the input node through the same interface;
computing the data by using a pre-configured computing function;
and outputting the calculated data to a specified position.
In one embodiment, the performing the operation on the data by using the preconfigured operation function includes:
and connecting the operation functions pre-configured by a plurality of the computing nodes to the same computing node in series for realization.
In one embodiment, the reading data of different data sources, wherein the data is read through the same interface, includes:
and reading the read data into a designated computing node in at least one computing node.
In one embodiment, the outputting the calculated data to a specified location includes:
and directly outputting the data calculated by one of the calculation nodes to a designated position.
In one embodiment, said performing, by using a preconfigured operation function, an operation on said data includes:
and processing the read data by using a pre-configured sub-process data processing algorithm, and outputting the processed data to the next computing node or the output node.
In one embodiment, the performing the operation on the data by using the preconfigured operation function includes:
and directly outputting the acquired current data to the next computing node or the output node.
Another aspect of the present invention provides an arithmetic device based on a general data arithmetic framework, including:
an input node configured to read data of different data sources, wherein the data is read into the input node through the same interface;
at least one compute node configured to compute the data using a preconfigured compute function;
an output node configured to output the data after the operation to a specified location.
Another aspect of the present invention provides an electronic device, including: memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the method as claimed in any one of the above.
Another aspect of the invention provides a computer storage medium having a computer program stored thereon, which when executed by a processor implements the steps of the method as recited in any one of the above.
By adopting the technical scheme, the invention at least has the following advantages:
the invention provides an operation method based on a general data operation frame, which exposes corresponding interfaces for data input, operation and output by designing a set of general abstract frame processes and a calculation method thereof, different data sources and operation frames can be accessed into the frame through the realization interfaces, standard operation components are defined, and multiple times of multiplexing can be realized at one time.
Drawings
FIG. 1 is a flowchart of a general data computation framework based computation method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an architecture of an application instance computing framework according to an embodiment of the present invention;
FIG. 3 is a block diagram of a computing device based on a generic data computation framework according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
To further explain the technical means and effects of the present invention adopted to achieve the intended purpose, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments.
The description of the method flow in the present specification and the steps of the flow chart in the drawings of the present specification are not necessarily strictly performed by the step numbers, and the execution order of the method steps may be changed. Moreover, certain steps may be omitted, multiple steps may be combined into one step execution, and/or a step may be broken down into multiple step executions.
A first embodiment of the present invention provides an operation method based on a general data operation framework, as shown in fig. 1, including the following specific steps:
step S1, reading data of different data sources, wherein the data are read into the input node through the same interface.
And S2, operating the data by using a pre-configured operation function.
And S3, outputting the calculated data to a specified position.
Referring to fig. 1, the method provided in this embodiment will be described in detail in steps.
Step S1, reading data of different data sources, wherein the data are read into the input node through the same interface.
In this embodiment, the method is implemented based on a pre-configured computing framework. The computing framework at least comprises an input node, a computing node and an output node.
In this embodiment, by selecting and configuring the interface, different types of data sources can be read, and data is read to the input node in the computing framework through the unified interface.
In this embodiment, the data source may be a database, a target system, or any data that may be obtained from a data source, which is not limited herein.
In this embodiment, there may be one or more reading nodes, or a corresponding reading logic unit may be configured in the computing node, and the read data is read to a designated computing node in at least one computing node.
And step S2, performing operation on the data by using a pre-configured operation function.
Illustratively, the operation function may be implemented by two operation components, i.e., mysql data source, mysql and memory, and the defined operation component types include: intersection, union, complement, difference, deduplication, filtering, adding constants, custom sql, statistics (maximum, minimum, average, total, sum, etc.), arithmetic operations (add, subtract, multiply, divide, power, square, etc.), string operations (join, intercept, length, etc.), ranking.
In this embodiment, the calculation function is implemented by pre-configuring the calculation nodes in the calculation framework. Specifically, there may be multiple computing nodes with different computing functions, and the computing functions pre-configured for the multiple computing nodes may be implemented by connecting the computing functions in series to the same computing node. It may also be implemented by setting logic units therebetween to achieve any possible combination of computing flows, which will not be limited herein.
In this embodiment, a logic unit configured to output current data may be configured in a compute node, and configured to directly obtain current data after computation processing by the compute node.
In this embodiment, a computing node may be configured as a processing node with a sub-process, and it is understood that the input of the processing node is the output of the previous node (input node or previous computing node), and the output of the processing node is the input of the next node (next computing node or output node). The specific configuration in the sub-process can be reasonably adjusted according to actual conditions, which will not be limited herein.
In this embodiment, the computing node may also not configure any computation logic implementation algorithm, that is, a computing node may directly output the current data obtained from the previous node to the next node in the computation flow without any processing.
And S3, outputting the calculated data to a specified position.
In this embodiment, the output may be completed by an output unit in the computing framework, or may be obtained directly from the computing unit.
In this embodiment, the designated location of the data output may be, for example, a database, a target system, etc., which will not be limited herein.
In the general data operation framework-based operation method provided by this embodiment, a set of general abstract framework processes and a calculation method thereof are designed, so that corresponding interfaces are exposed for data input, operation and output, different data sources and operation frameworks can be accessed into the framework through the implementation interfaces, and standard operation components are defined, so that multiple multiplexing can be realized at one time.
The second embodiment of the present invention, which is an application example of the present invention, is described on the basis of the above embodiments.
In this embodiment, the operation process is abstracted into one flow, and the flow includes an input node, an operation node, and an output node. The input node is responsible for reading data of a specified data source, the operation node comprises arithmetic operation, logic operation and other operation functions, and the output node is responsible for outputting a final operation result to a specified position (such as a database, a target system and the like).
That is to say, this embodiment specifically provides a computing framework, which corresponds to the flow of the computing method, and the framework may include: an input node, an operation node and an output node.
Referring to fig. 2, in the present embodiment, for convenience of subsequent description, an overall flow is defined as a Model object, each node (input node, compute node, output node) in the flow is defined as an Operation object, and one Model holds a plurality of start nodes. The Operation comprises a plurality of Input objects, an Operator object and a plurality of Output objects. The Input object is responsible for specific data reading operation, the Operator object is responsible for operation, the Output is responsible for rendering operation of operation result,
firstly, obtaining an operation data set List < RowsData > by the Input, then transmitting the data set to the Operator for logic operation, and finally transmitting an operation result RowsData returned by the Operator to the Output for rendering. The data Object is encapsulated as a RowsData, which is essentially a List < Map < String, object > structure, and the data streamed in the flow is the Object.
The RowsData may be the data itself or may be a data identifier (e.g., a table name). For three types of objects, namely Input/Operator/Output, the framework has already realized the implementation of two sets of operation components, namely a mysql data source, a mysql and a memory, and the defined operation component types comprise: intersection, union, complement, difference, deduplication, filtering, adding constants, custom sql, statistics (maximum, minimum, mean, total, sum, etc.), arithmetic operations (add, subtract, multiply, divide, power, square, etc.), string operations (join, intercept, length, etc.), ranking.
In addition to the implementation for the corresponding data sources and middleware, the framework itself provides several special purpose implementations to assist in the framework functions, including:
OperationOutput: holding an Operation object of a next node of a current node, packaging an Operation result of the current node into DirectInput, adding the DirectInput into the next node, and calling an Operation method of the next node. Through the implementation, a plurality of operations can be connected in series to form one flow.
DirectInput: the method is mainly used for OperationOutput, and is an Input based on an internal memory, namely directly holding RowsData and returning the RowsData when calling a data loading method.
DirectOutput, like DirectInput, is also implemented based on memory. The purpose of acquiring the Operation data of the node can be achieved by additionally adding one DirectOutput to a certain Operation.
The ModelOperator internally holds a Model object (sub-process) for realizing the function of the sub-process. The operation logic is to set the input parameters of the node to the sub-processes according to the mapping relation, trigger the sub-processes, and output the result of the appointed node of the sub-processes as the result of the node.
BlankOperator-an Operator without logic, used in the input and output nodes, only for passing data. Since the primary role in the Input nodes is the Input objects and the primary role in the Output nodes is the Output objects.
In addition, in the present embodiment, the following functional modules may be configured in the framework.
1. A ModelContext module;
the ModelContext module is a context for process operation, and is configured to be responsible for maintaining global variables, node operation states, process operation caches, process listeners, and the like.
2. A ModelExecutCAche module;
the ModelExecutCAche module is a flow operation cache. If two identical data source nodes exist in one calculation process or two identical submodels exist, one of the two data source nodes is executed and the result data is cached, and the other data source node waits for the completion of the previous execution and acquires the loaded and executed result from the cache so as to optimize the operation performance of the process.
3. An operationListener module;
the Operation listener module is a listener triggered by each key node in the Operation execution process, and is used for realizing the extended function by a user. The monitor is divided into three levels, the priority levels from high to low are respectively a default monitor (built-in framework), a global monitor (defined by users, applicable to all processes), and a process monitor (defined by users, applicable to a single process instance), and the monitors of all levels can also define the priority level to specify the internal calling sequence.
4. A parser module;
in order to conveniently develop a visualization page to create conditions for non-technical users, the process support is defined in a json form, and the parser module is responsible for converting json into the objects such as Model/Operation/Input/Operator/Output.
5. A ModelRepository module;
the framework does not limit the storage mode of the process definition, so the interface is developed to be used as a process warehouse, and the business system can be realized according to actual design. In the implementation of the process warehouse, a json defining mode needs to be obtained in a self-defined mode, and then a parser module is used for converting the json into a Model object.
6. A Configuration module;
the Configuration module is a Configuration class of the framework, and parameters including model warehouse implementation, global listener, and the like can be configured here.
In a third embodiment of the present invention, an arithmetic device based on a generic data arithmetic framework, as shown in fig. 3, includes:
the input node is configured to read data of different data sources, wherein the data are read into the input node through the same interface;
at least one compute node configured to operate on data using a preconfigured operation function;
an output node configured to output the operated data to a specified location.
A fourth embodiment of the present invention, an electronic device, can be understood as a physical device with reference to fig. 4, and includes a processor and a memory storing processor-executable instructions, and when the instructions are executed by the processor, the following operations are performed:
s1, reading data of different data sources, wherein the data are read into an input node through the same interface;
s2, operating the data by utilizing a pre-configured operation function;
and S3, outputting the calculated data to a specified position.
In the fifth embodiment of the present invention, the flow of the general data operation framework based operation method in this embodiment is the same as that in the first and second embodiments, but the difference is that in terms of engineering implementation, this embodiment can be implemented by software plus a necessary general hardware platform, and certainly, the present embodiment can also be implemented by hardware, but the former is a better implementation mode in many cases. With this understanding in mind, the method of the present invention may be embodied in the form of a computer software product stored on a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and including instructions for causing a device (e.g., a network device such as a base station) to perform the method of the present invention.
In summary, compared to the prior art, the operation method based on the universal data operation framework provided by the present invention at least has the following advantages:
1) The invention discloses a method for realizing multiplexing of data sources and operation frames, which comprises the steps of designing a set of universal abstract frames, exposing corresponding interfaces for data input, operation and output, accessing different data sources and operation frames into the frames through the realization interfaces, defining standard operation components and realizing multiplexing for multiple times at one time.
2) The invention can additionally develop the process configuration page and reduce the use threshold of non-technical personnel.
3) One operational flow definition in the invention can switch different operational tools at the bottom layer to support the operation of data with different magnitudes.
4) The invention supports the monitoring of the running state of the process; the node executes a key position setting monitor mechanism, and more functions are conveniently expanded.
While the invention has been described in connection with specific embodiments thereof, it is to be understood that it is intended by the appended drawings and description that the invention may be embodied in other specific forms without departing from the spirit or scope of the invention.

Claims (9)

1. An operation method based on a general data operation framework, wherein the operation framework comprises: an input node, at least one compute node, and an output node, the method comprising:
reading data of different data sources, wherein the data are read into the input node through the same interface;
operating the data by utilizing a pre-configured operation function;
and outputting the calculated data to a specified position.
2. The method of claim 1, wherein the performing operations on the data using the pre-configured computing functionality comprises:
and connecting the operation functions pre-configured by a plurality of the computing nodes to the same computing node in series for realization.
3. The computing method based on general data operation framework as claimed in claim 1, wherein the reading data of different data sources, wherein the data are read through the same interface, comprises:
and reading the read data into a designated computing node in at least one computing node.
4. The computing method based on the universal data computing framework as claimed in claim 1, wherein the outputting the computed data to a designated location comprises:
and directly outputting the data calculated by one of the calculation nodes to a designated position.
5. The method of claim 1, wherein the performing operations on the data using the pre-configured computing functionality comprises:
and processing the read data by using a pre-configured sub-process data processing algorithm, and outputting the processed data to the next computing node or the output node.
6. The method of claim 1, wherein the performing operations on the data using the pre-configured computing functionality comprises:
and directly outputting the acquired current data to the next computing node or the output node.
7. An arithmetic device based on a general data arithmetic framework, the arithmetic framework comprising:
an input node configured to read data of different data sources, wherein the data is read into the input node through the same interface;
at least one compute node configured to compute the data using a preconfigured compute function;
an output node configured to output the data after the operation to a specified location.
8. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the method according to any one of claims 1 to 6.
9. A computer storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
CN202211128998.4A 2022-09-16 2022-09-16 Operation method and device based on general data operation framework and electronic equipment Pending CN115562654A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211128998.4A CN115562654A (en) 2022-09-16 2022-09-16 Operation method and device based on general data operation framework and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211128998.4A CN115562654A (en) 2022-09-16 2022-09-16 Operation method and device based on general data operation framework and electronic equipment

Publications (1)

Publication Number Publication Date
CN115562654A true CN115562654A (en) 2023-01-03

Family

ID=84740127

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211128998.4A Pending CN115562654A (en) 2022-09-16 2022-09-16 Operation method and device based on general data operation framework and electronic equipment

Country Status (1)

Country Link
CN (1) CN115562654A (en)

Similar Documents

Publication Publication Date Title
CN109684319B (en) Data cleaning system, method, device and storage medium
CN107665228A (en) A kind of related information querying method, terminal and equipment
US20180150530A1 (en) Method, Apparatus, Computing Device and Storage Medium for Analyzing and Processing Data
CN111818175B (en) Enterprise service bus configuration file generation method, device, equipment and storage medium
CN113094125B (en) Business process processing method, device, server and storage medium
CN112860730A (en) SQL statement processing method and device, electronic equipment and readable storage medium
CN110888672B (en) Expression engine implementation method and system based on metadata architecture
CN115422866A (en) Method for simulating logic system design on simulator and related equipment
CN115794213A (en) Configurable object management method, device and equipment based on embedded system
CN113190576B (en) Data processing method, apparatus, computer device and readable storage medium
CN110019207B (en) Data processing method and device and script display method and device
JP2023553220A (en) Process mining for multi-instance processes
CN116301735B (en) Method, device and storage medium for organizing software elements into software data links
CN113296760A (en) Method for generating model code, computer device and readable storage medium
CN115562654A (en) Operation method and device based on general data operation framework and electronic equipment
Lallchandani et al. Slicing UML architectural models
CN112506943B (en) Heterogeneous data service providing method, device, equipment and medium
CN114254029A (en) Index processing method and device, storage medium and equipment
CN113377368A (en) Project development method, device, server and storage medium
CN111160403A (en) Method and device for multiplexing and discovering API (application program interface)
CN112506944B (en) Data standard conversion access method, device, equipment and medium between service systems
CN116301758B (en) Rule editing method, device, equipment and medium based on event time points
CN111679807B (en) Demand management method and device
CN113504912B (en) Real-time task processing method and device, storage medium and electronic device
CN113689173B (en) Modeling device and modeling method of business logic representation model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination