CN113778545A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113778545A CN113778545A CN202011241197.XA CN202011241197A CN113778545A CN 113778545 A CN113778545 A CN 113778545A CN 202011241197 A CN202011241197 A CN 202011241197A CN 113778545 A CN113778545 A CN 113778545A
- Authority
- CN
- China
- Prior art keywords
- target
- data processing
- data
- node
- link
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000003860 storage Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 287
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000006870 function Effects 0.000 claims description 42
- 230000008569 process Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 5
- 238000004806 packaging method and process Methods 0.000 claims description 5
- 238000009941 weaving Methods 0.000 claims description 4
- 238000011161 development Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 239000003795 chemical substances by application Substances 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012856 packing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000006317 isomerization reaction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
- G06F9/45508—Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
- G06F9/45512—Command shells
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring target data to be processed and a target data processing identifier corresponding to the target data; determining target link configuration information corresponding to a target link for processing target data based on link configuration information corresponding to each pre-configured data processing identifier and a target data processing identifier, wherein the target link configuration information comprises: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node; loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node; and processing the target data based on the target execution sequence and each target instance object. By the technical scheme of the embodiment of the invention, the dynamic configuration generation of the data processing link can be realized, thereby improving the development efficiency and the flexibility of data processing.
Description
Technical Field
The present invention relates to internet technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
With the rapid development of internet technology, software servitization has received more and more attention as an innovative software application mode. Generally, a business process often needs to interact with many data interfaces to realize corresponding business functions. For a data interface of a collaboration service or a streaming data interface of business data, processing operations such as splitting, packaging, isomerism, and reassembly are often performed on received data based on business requirements, and data obtained after processing is returned.
At present, a developer can directly write a code corresponding to processing logic inside a data interface according to a service requirement, so that data processing is performed in a mode of directly executing the code. A configuration file described by JSON (JSON Object Notation), XML (Extensible Markup Language), or the like may also be defined, so that the data interfaces that are mutually called perform data processing operations according to the configuration file.
However, in the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
when the service requirement changes, the code needs to be rewritten, or the configuration file needs to be reconfigured, and the rewritten code or the configuration file needs to be reissued online to complete the change of the service requirement, thereby greatly reducing the development efficiency. Moreover, once the two existing data processing methods are implemented, the data processing link inside the data interface cannot be dynamically changed based on the service requirements, that is, the execution sequence of each node in the data processing link cannot be dynamically adjusted, or the nodes are recombined to generate a new data processing link, so that the flexibility of data processing is reduced, and the personalized requirements of users cannot be met.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, apparatus, device, and storage medium to implement dynamic configuration generation of a data processing link, and do not need to release online again when a service requirement changes, thereby improving development efficiency and flexibility of data processing, and further satisfying a user personalized requirement.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
acquiring target data to be processed and a target data processing identifier corresponding to the target data;
determining target link configuration information corresponding to a target link for processing the target data based on link configuration information corresponding to each pre-configured data processing identifier and the target data processing identifier, wherein the target link configuration information includes: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node;
and processing the target data based on the target execution sequence and each target instance object.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, including:
the target data acquisition module is used for acquiring target data to be processed and a target data processing identifier corresponding to the target data;
a target link configuration information determining module, configured to determine, based on link configuration information corresponding to each preconfigured data processing identifier and the target data processing identifier, target link configuration information corresponding to a target link for processing the target data, where the target link configuration information includes: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
a target instance object generation module, configured to load and compile the data processing script corresponding to each target node, and generate a target instance object corresponding to each target node;
and the target data processing module is used for processing the target data based on the target execution sequence and each target instance object.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data processing method as provided by any of the embodiments of the invention.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data processing method provided in any embodiment of the present invention.
The embodiment of the invention has the following advantages or beneficial effects:
by pre-configuring the link configuration information corresponding to each data processing identifier, namely dynamically configuring the data processing script corresponding to each node in the link for processing each service data and the execution sequence corresponding to each node, when processing the target data corresponding to the target data processing identifier, the data processing script corresponding to each target node in the target link configuration information can be loaded and compiled to generate the target instance object corresponding to each target node, and the target data is processed based on the target execution sequence of each target node and each target instance object, so that the target link for processing the target data can be dynamically generated based on the target link configuration information, thereby realizing the dynamic configuration generation of the data processing link, greatly improving the flexibility of data processing, and further meeting the personalized requirements of users, and each target node in the target link carries out data processing based on the data processing script, so that when the service requirement changes, the data processing script in the configuration information can be adjusted to take effect without rewriting logic codes and re-releasing the logic codes, and the development efficiency is improved.
Drawings
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing framework according to an embodiment of the present invention;
FIG. 3 is an example of a data processing flow according to an embodiment of the present invention;
fig. 4 is a flowchart of a data processing method according to a second embodiment of the present invention;
fig. 5 is an example of a target node creation process according to the second embodiment of the present invention;
fig. 6 is an example of a target link creation process according to a second embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, which is applicable to a case of performing data processing on data acquired by a data interface, and especially applicable to a scenario in which processing operations such as splitting, packing, isomerism, and reassembly of data are required. The method may be performed by a data processing apparatus, which may be implemented by means of software and/or hardware, integrated in an electronic device having data processing capabilities. As shown in fig. 1, the method specifically includes the following steps:
and S110, acquiring target data to be processed and a target data processing identifier corresponding to the target data.
The target data may be service data that is acquired by calling a data interface and needs to be subjected to data processing operation, such as service data that needs to be subjected to processing operations such as splitting, packing, isomerism, and/or reassembly. The data processing identifier may be used to identify processing requirements of the service data. When the data processing requirements are different, namely the data processing logics are different, the corresponding data processing identifiers are also different, so that different data processing requirements can be distinguished by using different data processing identifiers, and different service requirements can be met. For example, the service type identifier corresponding to the data to be processed may be used as the data processing identifier. The target data processing identification may refer to a data processing identification for processing the target data.
Specifically, by calling a data interface of the collaboration service or a streaming data interface of the business data, the data interface of the collaboration service or the streaming data interface of the business data can acquire target data to be processed and a target data processing identifier corresponding to the target data, so that the target data is processed based on a data processing logic corresponding to the target data processing identifier.
S120, determining target link configuration information corresponding to a target link for processing target data based on link configuration information corresponding to each pre-configured data processing identifier and a target data processing identifier, wherein the target link configuration information comprises: and the data processing script corresponding to each target node in the target link and the target execution sequence corresponding to each target node.
The link configuration information corresponding to the data processing identifier may be data processing logic information corresponding to a link for processing the to-be-processed data corresponding to the data processing identifier. Each link configuration information includes: and the data processing script corresponding to each node in the link and the target execution sequence corresponding to each node. Each link may correspond to a data processing logic. Each node in a link may refer to each step in the data processing logic. The execution sequence corresponding to each node may refer to the sequential execution sequence of each step. Each data processing can pass through one or more steps, namely each link can pass through one or more nodes, and the specific passing through nodes and the execution sequence of each node can be dynamically determined based on the link configuration information, so that the link can be dynamically and flexibly configured, the data processing logic can be modified and adjusted at any time in a self-defined manner, and the calling system does not need to be modified in an invasive manner, thereby improving the development efficiency.
Specifically, the link configuration information corresponding to each data processing identifier may be preconfigured based on the service requirement, and each configured link configuration information is stored, so that when processing the target data, the target link configuration information corresponding to the target data processing identifier may be determined based on the target data processing identifier. By analyzing the target link configuration information, a data processing script corresponding to each target node in the target link for processing the target data and a target execution sequence corresponding to each target node can be obtained.
S130, loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node.
Wherein the target instance object may be constructed based on the data processing script for executing an instantiation object of the script processing logic. Each target node corresponds to one target instance object, so that the data processing script corresponding to the corresponding target node is executed through the target instance object, and the corresponding data processing logic function is realized.
Specifically, for each target node, reading a data processing script corresponding to the target node, loading and compiling the data processing script, compiling the data processing script in the character string format into an executable binary script, thereby constructing a script processing class, instantiating the script processing class, and obtaining a target instance object corresponding to each target node. In this embodiment, the data processing script corresponding to each target node may be packaged in a componentized manner by using Java, so that the nodes in the actual link can be dynamically and flexibly assembled, thereby meeting different data processing requirements.
And S140, processing the target data based on the target execution sequence and each target instance object.
Specifically, after the target instance object corresponding to each target node in the target link is generated, the calling sequence of each target instance object may be obtained based on the target execution sequence corresponding to each target node, so as to dynamically generate the target link, implement dynamic configuration generation of the data processing link, and process the target data by using the dynamically generated target link. In this embodiment, the data processing link inside the data interface may be dynamically adjusted by adjusting the link configuration information, for example, the execution sequence of each node in the data processing link is dynamically adjusted, or a new data processing link is dynamically generated by recombining the nodes, so that the flexibility of data processing is greatly improved, and the personalized requirements of the user are further satisfied. Moreover, each target node in the target link performs data processing based on the data processing script, so that when the data processing logic of each node needs to be adjusted, the data processing script in the configuration information can be adjusted to take effect through the visual view, and the logic code does not need to be rewritten and the logic code does not need to be reissued, so that the development efficiency is improved, and the expandability of data processing is further improved.
Exemplarily, S140 may include: and sequentially calling the data processing functions in the target instance objects corresponding to each target node based on the target execution sequence, and sequentially processing the target data.
The data processing function may refer to a function inside the target instance object for executing the corresponding data processing logic. Specifically, the data processing function in the target instance object corresponding to each target node may be sequentially called based on the target execution sequence in the target link, so that the target data sequentially performs data processing at each target node in sequence, and thus a final data processing result may be obtained at the last target node, and the final data processing result may be returned, thereby completing the entire data processing process.
Illustratively, FIG. 2 presents a schematic diagram of a data processing framework. As shown in fig. 2, the data processing framework may include: an input interface, a node assembler component, a link generator component, and a data processing output. The input interface may be an interface for receiving data to be processed in a data interface of a collaboration service or a streaming data interface of business data. The node assembler component may include a dynamic script processor. The dynamic scenario processor may be configured to process the data processing scenario to assemble each node in the production link for processing data. The link generator component may include: the system comprises a configuration management module, a configuration analysis module and a link generation module. The configuration management module may be configured to query management link configuration information. The configuration parsing module may be a module for parsing the link configuration information. The link generation module may be a module for dynamically generating an executable link. Fig. 3 gives an example of a data processing flow. As shown in fig. 3, in this embodiment, the target data to be processed and the target data processing identifier corresponding to the target data may be acquired through the input interface. The link configuration information corresponding to each pre-stored data processing identifier in the database can be stored in the cache unit, so that the configuration management module can query and determine the target link configuration information in the cache unit based on the target data processing identifier, thereby improving the query efficiency and further improving the data processing efficiency. The dynamic script processor module can perform processing operations such as loading and compiling on the data processing script corresponding to each target node in the target link configuration information, and generate a target instance object corresponding to each target node, namely, an assembler component corresponding to each target node. The configuration analysis module can analyze the link configuration information to obtain a target execution sequence corresponding to each target node. The link generation module may generate a target link for processing target data based on the assembler component and the target execution sequence corresponding to each target node, for example, the target link in fig. 3 includes three target nodes, and the execution sequence is target node 1, target node 2, and target node 3, so that the target data may sequentially flow through target node 1, target node 2, and target node 3 for data processing, and an obtained final data processing result is returned to a calling party, thereby completing the entire data processing flow.
In the technical solution of this embodiment, by configuring link configuration information corresponding to each data processing identifier in advance, that is, dynamically configuring a data processing script corresponding to each node in a link for processing each service data and an execution sequence corresponding to each node, when processing target data corresponding to a target data processing identifier, the data processing script corresponding to each target node in the target link configuration information can be loaded and compiled to generate a target instance object corresponding to each target node, and the target data is processed based on the target execution sequence of each target node and each target instance object, so that a target link for processing the target data can be dynamically generated based on the target link configuration information, thereby implementing dynamic configuration generation of the data processing link and greatly improving flexibility of data processing, and then the individual requirements of the user are met, and each target node in the target link carries out data processing based on the data processing script, so that when the service requirements change, the data processing script in the configuration information can be effective only by adjusting, logic codes do not need to be rewritten, the logic codes do not need to be released again, and the development efficiency is improved.
On the basis of the above technical solution, S110 may include: and acquiring target data to be processed and a target data processing identifier corresponding to the target data, which are sent by a calling party through a calling input end interface. Illustratively, as shown in FIG. 2, the caller may include, but is not limited to: an external WEB network interface, a remote service interface or an internal calling interface.
Specifically, the data processing process in this embodiment may be applicable to complex multidimensional processing of data in scenes such as between WEB systems, between service systems, or between modules inside a system, and the like, thereby greatly improving the application range of the scenes. Moreover, the conversion of the data format can be completed based on the data processing script, so that the operations of rewriting logic codes and repeatedly releasing online after the interface data structure is changed are avoided, the data processing script can be adjusted to take effect, the labor cost is reduced, and the development efficiency is improved.
On the basis of the technical scheme, the method further comprises the following steps: and storing the target instance object corresponding to each target node into the memory, and canceling the release operation of the memory when the target data processing is finished.
Specifically, after the target instance object corresponding to each target node is generated, each target instance object may be stored in the memory, and the memory is not released, so as to cache the generated target instance objects, so that the stored target instance objects corresponding to the same node may be directly utilized in the next data processing, and the instance objects do not need to be generated in real time each time the same node is assembled, thereby further improving the data processing efficiency.
Illustratively, S130 may include: detecting whether a generated target instance object corresponding to each target node in a target link exists or not according to a pre-stored instance object corresponding to each node; if yes, directly acquiring a generated target instance object corresponding to the target node; if not, loading and compiling the data processing script corresponding to the target node to generate a target instance object corresponding to the target node.
Specifically, for each target node in the target link, matching may be performed in pre-stored instance objects corresponding to each node, and whether an instance object corresponding to the same node is stored is detected, and if so, it is indicated that the data processing script corresponding to the target node has been loaded and compiled to generate a corresponding instance object, and at this time, the instance object may be used as the target instance object corresponding to the target node, so that when the same node is included in different links, the generated target instance object corresponding to the node may be directly used to perform the data processing logic of the node, and the target instance object does not need to be reconstructed, thereby further improving the data processing efficiency.
Example two
Fig. 4 is a flowchart of a data processing method according to a second embodiment of the present invention, and in this embodiment, further optimization is performed on "processing target data based on a target execution sequence and each target processing object", where explanations of terms that are the same as or corresponding to the above embodiments are not repeated herein.
Referring to fig. 4, the data processing method provided in this embodiment specifically includes the following steps:
s410, target data to be processed and a target data processing identifier corresponding to the target data are obtained.
S420, determining target link configuration information corresponding to a target link for processing target data based on the link configuration information and the target data processing identifier corresponding to each preconfigured data processing identifier, where the target link configuration information includes: and the data processing script corresponding to each target node in the target link and the target execution sequence corresponding to each target node.
S430, loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node.
S440, packaging each target instance object, and generating an assemblable target proxy object corresponding to each target instance object.
The target proxy object may be a proxy provided for the target instance object, so as to control access to the target instance object, facilitate extension of a logic function implemented by the target instance object, and further improve extensibility. Each target instance object corresponds to a target proxy object.
In particular, each target instance object may be encapsulated as assemblable using dynamic proxy techniques and cut-oriented programming, and functions may be cut into target proxy objects. The section-oriented programming can be a technology for realizing dynamic and unified addition of functions to functions without modifying source codes through a precompilation mode and a running-time dynamic proxy mode.
Illustratively, the assembler component in the data processing architecture of fig. 2 may include, in addition to the dynamic script processor, a dynamic assembly agent constructor, such that each target instance object may be encapsulated by the dynamic assembly agent constructor to generate a corresponding target agent object. For example, FIG. 5 provides an example of a target node creation process. As shown in fig. 5, when each target node needs to be constructed, the assembler component first calls the dynamic assembly proxy constructor, the dynamic assembly proxy constructor then calls the dynamic scenario processor, the dynamic scenario processor calls the configuration management module based on the target data processing identifier, the configuration management module returns the target link configuration information corresponding to the target data processing identifier, so that the dynamic scenario processor generates a corresponding target instance object based on the data processing scenario corresponding to the target node, and returns the target instance object to the dynamic assembly proxy constructor, and the dynamic assembly proxy constructor encapsulates the target instance object to generate a corresponding target proxy object, thereby completing the assembly construction process of one target node.
Exemplarily, after S440, the method may further include: acquiring a logic code for realizing a non-data processing logic function, and determining a first proxy object to be expanded from each target proxy object; and weaving the logic code into the first proxy object so as to realize the data processing logic function and the non-data processing logic function corresponding to the first proxy object by calling the expanded first proxy object.
Where a non-data processing logic function may refer to operations other than data processing operations, for example, operations other than data splitting, packing, isomerisation, and reassembly. For example, the non-data processing logic function may be a function for implementing monitoring and reporting, and the like.
Specifically, when a third party needs to switch in a data processing process to realize a non-data processing logic function, a logic code for realizing the non-data processing logic function and a first proxy object for realizing the non-data processing logic function in each target proxy object can be obtained based on the third party requirement, so that the logic code can be directly woven into the first proxy object without modifying a script, the function expansion of the first proxy object is realized, the data processing logic function and the non-data processing logic function corresponding to the first proxy object can be simultaneously realized when the expanded first proxy object is called subsequently, the function expansion efficiency is improved, and the personalized requirements of a user are further met.
And S450, sequentially calling the data processing functions in the target proxy objects corresponding to the target nodes based on the target execution sequence, and sequentially processing the target data.
The data processing function may refer to a function inside the target instance object corresponding to the target proxy object, and used for executing the corresponding data processing logic. Specifically, the data processing function in the target proxy object corresponding to each target node may be sequentially called based on the target execution sequence in the target link, so that the target data sequentially performs data processing at each target node in sequence, and thus a final data processing result may be obtained at the last target node, and the final data processing result may be returned, thereby completing the entire data processing process.
Illustratively, FIG. 6 gives an example of a target link creation process. As shown in fig. 6, in the construction phase, the link generation module may invoke the configuration analysis module, and the configuration analysis module may invoke the configuration management module, so that the configuration management module returns the target link configuration information corresponding to the acquired target data processing identifier, and the configuration analysis module may analyze the target link configuration to obtain the target execution sequence of each target node, and return the target execution sequence to the link generation module, so that the link generation module may construct a specific data processing logic of the target link based on the target instance object and the target execution sequence corresponding to each target node, thereby implementing dynamic generation of the data processing link. In the data processing stage, target data flows through each target node in the target link, and after the target nodes are processed layer by layer, a final data processing result can be obtained.
According to the technical scheme of the embodiment, each target instance object is packaged to generate the target proxy object which corresponds to each target instance object and can be assembled, and the target data is processed by using each target proxy object, so that the access to the target instance object can be controlled more conveniently through the target proxy object, the logic function realized by the target instance object is conveniently expanded, the expandability is further improved, and the personalized requirements of users are further met.
On the basis of the above technical solution, the method may further include: and storing the target proxy object corresponding to each target node so as to directly process the target data by using the stored target proxy object in the following process.
Specifically, after the target proxy object corresponding to each target node is generated, each target proxy object may be stored in the memory, and when the target data processing is completed, the release operation of the memory is cancelled, so that each generated target proxy object may be cached, and further, the stored target proxy objects corresponding to the same node may be directly utilized in the next data processing, and it is not necessary to generate the instance object and the proxy object in real time each time the same node is assembled, thereby further improving the data processing efficiency.
Exemplarily, after S420, the method may further include: detecting whether a generated target proxy object corresponding to each target node in a target link exists or not according to a pre-stored proxy object corresponding to each node; if yes, directly acquiring a generated target proxy object corresponding to the target node; if not, loading and compiling the data processing script corresponding to the target node to generate a target processing object corresponding to the target node, packaging the target processing object, and generating an assemblable target proxy object corresponding to the target processing object.
Specifically, for each target node in the target link, matching may be performed in a pre-stored proxy object corresponding to each node, and whether a proxy object corresponding to the same node is stored is detected, and if so, it indicates that the data processing script corresponding to the target node has been loaded, compiled, and packaged, and a corresponding proxy object is generated, and at this time, the proxy object may be used as the target proxy object corresponding to the target node, so that when different links include the same node, the generated target proxy object corresponding to the node may be directly used to perform the data processing logic of the node, and the target proxy object does not need to be reconstructed, thereby further improving the data processing efficiency.
The following is an embodiment of a data processing apparatus according to an embodiment of the present invention, which belongs to the same inventive concept as the data processing methods of the above embodiments, and reference may be made to the above embodiments of the data processing method for details that are not described in detail in the embodiments of the data processing apparatus.
EXAMPLE III
Fig. 7 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention, which is applicable to a case of performing data processing on data acquired by a data interface. The device may specifically include: a target data acquisition module 710, a target link configuration information determination module 720, a target instance object generation module 730, and a target data processing module 740.
A target data obtaining module 710, configured to obtain target data to be processed and a target data processing identifier corresponding to the target data; a target link configuration information determining module 720, configured to determine, based on the link configuration information and the target data processing identifier corresponding to each preconfigured data processing identifier, target link configuration information corresponding to a target link for processing target data, where the target link configuration information includes: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node; a target instance object generation module 730, configured to load and compile the data processing script corresponding to each target node, and generate a target instance object corresponding to each target node; and the target data processing module 740 is configured to process the target data based on the target execution order and each target instance object.
Optionally, the target data processing module 740 is specifically configured to: and sequentially calling the data processing functions in the target instance objects corresponding to each target node based on the target execution sequence, and sequentially processing the target data.
Optionally, the apparatus further comprises:
and the instance object storage module is used for storing the target instance object corresponding to each target node into the memory and canceling the release operation of the memory when the target data processing is finished.
Optionally, the target instance object generating module 730 is specifically configured to:
detecting whether a generated target instance object corresponding to each target node in a target link exists or not according to a pre-stored instance object corresponding to each node; if yes, directly acquiring a generated target instance object corresponding to the target node; if not, loading and compiling the data processing script corresponding to the target node to generate a target instance object corresponding to the target node.
Optionally, the target data processing module 740 is further specifically configured to:
packaging each target instance object to generate an assemblable target proxy object corresponding to each target instance object; and sequentially calling the data processing functions in the target proxy objects corresponding to each target node based on the target execution sequence, and sequentially processing the target data.
Optionally, the apparatus further comprises:
the logic code acquisition module is used for acquiring logic codes for realizing non-data processing logic functions after generating assemblable target proxy objects corresponding to each target instance object, and determining a first proxy object to be expanded from each target proxy object;
and the logic code weaving module is used for weaving the logic code into the first proxy object so as to realize the data processing logic function and the non-data processing logic function corresponding to the first proxy object by calling the expanded first proxy object.
Optionally, the apparatus further comprises:
and the proxy object storage module is used for storing the target proxy object corresponding to each target node so as to directly process the target data by using the stored target proxy object in the following process.
Optionally, the target data obtaining module 710 is specifically configured to:
acquiring target data to be processed and a target data processing identifier corresponding to the target data, which are sent by a calling party through a calling input end interface; wherein, the caller includes: an external network interface, a remote service interface, or an internal invocation interface.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional module and the beneficial effect of executing the data processing method.
It should be noted that, in the embodiment of the data processing apparatus, the included units and modules are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
Example four
Fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. FIG. 8 illustrates a block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention. The electronic device 12 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in FIG. 8, electronic device 12 is embodied in the form of a general purpose computing device. The components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 8, and commonly referred to as a "hard drive"). Although not shown in FIG. 8, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data processing method provided by the embodiment of the present invention, the method including:
acquiring target data to be processed and a target data processing identifier corresponding to the target data;
determining target link configuration information corresponding to a target link for processing target data based on link configuration information corresponding to each pre-configured data processing identifier and a target data processing identifier, wherein the target link configuration information comprises: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node;
and processing the target data based on the target execution sequence and each target instance object.
Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.
EXAMPLE five
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a data processing method as provided in any of the embodiments of the invention, the method comprising:
acquiring target data to be processed and a target data processing identifier corresponding to the target data;
determining target link configuration information corresponding to a target link for processing target data based on link configuration information corresponding to each pre-configured data processing identifier and a target data processing identifier, wherein the target link configuration information comprises: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node;
and processing the target data based on the target execution sequence and each target instance object.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It will be understood by those skilled in the art that the modules or steps of the invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and optionally they may be implemented by program code executable by a computing device, such that it may be stored in a memory device and executed by a computing device, or it may be separately fabricated into various integrated circuit modules, or it may be fabricated by fabricating a plurality of modules or steps thereof into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (11)
1. A data processing method, comprising:
acquiring target data to be processed and a target data processing identifier corresponding to the target data;
determining target link configuration information corresponding to a target link for processing the target data based on link configuration information corresponding to each pre-configured data processing identifier and the target data processing identifier, wherein the target link configuration information includes: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node;
and processing the target data based on the target execution sequence and each target instance object.
2. The method of claim 1, wherein processing the target data based on the target execution order and the respective target instance objects comprises:
and sequentially calling a data processing function in the target instance object corresponding to each target node based on the target execution sequence, and sequentially processing the target data.
3. The method of claim 1, further comprising:
and storing the target instance object corresponding to each target node into a memory, and canceling the release operation of the memory when the target data processing is finished.
4. The method according to claim 1, wherein loading and compiling the data processing script corresponding to each target node to generate a target instance object corresponding to each target node comprises:
detecting whether a generated target instance object corresponding to each target node in the target link exists or not according to a pre-stored instance object corresponding to each node;
if yes, directly acquiring a generated target instance object corresponding to the target node;
if not, loading and compiling the data processing script corresponding to the target node to generate a target instance object corresponding to the target node.
5. The method of claim 1, wherein processing the target data based on the target execution order and the respective target processing objects comprises:
packaging each target instance object, and generating an assemblable target proxy object corresponding to each target instance object;
and sequentially calling a data processing function in the target proxy object corresponding to each target node based on the target execution sequence, and sequentially processing the target data.
6. The method of claim 5, after generating an assemblable target proxy object corresponding to each of the target instance objects, further comprising:
acquiring a logic code for realizing a non-data processing logic function, and determining a first proxy object to be expanded from each target proxy object;
and weaving the logic code into the first proxy object so as to realize the data processing logic function and the non-data processing logic function corresponding to the first proxy object by calling the expanded first proxy object.
7. The method of claim 5, further comprising:
and storing the target proxy object corresponding to each target node so as to directly process the target data by using the stored target proxy object in the following process.
8. The method according to claim 5, wherein obtaining target data to be processed and a target data processing identifier corresponding to the target data comprises:
acquiring target data to be processed and a target data processing identifier corresponding to the target data, which are sent by a calling party through a calling input end interface;
wherein the caller comprises: an external network interface, a remote service interface, or an internal invocation interface.
9. A data processing apparatus, comprising:
the target data acquisition module is used for acquiring target data to be processed and a target data processing identifier corresponding to the target data;
a target link configuration information determining module, configured to determine, based on link configuration information corresponding to each preconfigured data processing identifier and the target data processing identifier, target link configuration information corresponding to a target link for processing the target data, where the target link configuration information includes: a data processing script corresponding to each target node in the target link and a target execution sequence corresponding to each target node;
a target instance object generation module, configured to load and compile the data processing script corresponding to each target node, and generate a target instance object corresponding to each target node;
and the target data processing module is used for processing the target data based on the target execution sequence and each target instance object.
10. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-8.
11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011241197.XA CN113778545A (en) | 2020-11-09 | 2020-11-09 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011241197.XA CN113778545A (en) | 2020-11-09 | 2020-11-09 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113778545A true CN113778545A (en) | 2021-12-10 |
Family
ID=78835140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011241197.XA Pending CN113778545A (en) | 2020-11-09 | 2020-11-09 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113778545A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114615027A (en) * | 2022-02-24 | 2022-06-10 | 奇安信科技集团股份有限公司 | Behavior data processing method, behavior data processing device, behavior data processing equipment and storage medium |
| CN115134228A (en) * | 2022-06-28 | 2022-09-30 | 中国工商银行股份有限公司 | Environment link supply and detection method, device, equipment, medium and program product |
| CN115658078A (en) * | 2022-12-27 | 2023-01-31 | 金篆信科有限责任公司 | Database pre-compiling processing method, device, equipment and medium |
| CN115826980A (en) * | 2022-10-10 | 2023-03-21 | 中科三清科技有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN118055039A (en) * | 2024-04-11 | 2024-05-17 | 明阳产业技术研究院(沈阳)有限公司 | Network equipment management method and system based on telemetry data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385527A (en) * | 2010-09-01 | 2012-03-21 | 中兴通讯股份有限公司 | Device and method for improving execution efficiency of service scripts |
| CN106021543A (en) * | 2016-05-26 | 2016-10-12 | 中国农业银行股份有限公司 | Data pre-processing method and device |
| US20160364216A1 (en) * | 2015-06-15 | 2016-12-15 | Qualcomm Incorporated | Generating object code from intermediate code that includes hierarchical sub-routine information |
| CN109597661A (en) * | 2018-10-26 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of service function configuration method and device |
| CN110764839A (en) * | 2019-09-24 | 2020-02-07 | 苏宁云计算有限公司 | Business processing flow configuration method, business request processing method and device |
| CN110941488A (en) * | 2019-12-11 | 2020-03-31 | 中国建设银行股份有限公司 | Task processing method, device, equipment and storage medium |
-
2020
- 2020-11-09 CN CN202011241197.XA patent/CN113778545A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385527A (en) * | 2010-09-01 | 2012-03-21 | 中兴通讯股份有限公司 | Device and method for improving execution efficiency of service scripts |
| US20160364216A1 (en) * | 2015-06-15 | 2016-12-15 | Qualcomm Incorporated | Generating object code from intermediate code that includes hierarchical sub-routine information |
| CN106021543A (en) * | 2016-05-26 | 2016-10-12 | 中国农业银行股份有限公司 | Data pre-processing method and device |
| CN109597661A (en) * | 2018-10-26 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of service function configuration method and device |
| CN110764839A (en) * | 2019-09-24 | 2020-02-07 | 苏宁云计算有限公司 | Business processing flow configuration method, business request processing method and device |
| CN110941488A (en) * | 2019-12-11 | 2020-03-31 | 中国建设银行股份有限公司 | Task processing method, device, equipment and storage medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114615027A (en) * | 2022-02-24 | 2022-06-10 | 奇安信科技集团股份有限公司 | Behavior data processing method, behavior data processing device, behavior data processing equipment and storage medium |
| CN115134228A (en) * | 2022-06-28 | 2022-09-30 | 中国工商银行股份有限公司 | Environment link supply and detection method, device, equipment, medium and program product |
| CN115826980A (en) * | 2022-10-10 | 2023-03-21 | 中科三清科技有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN115826980B (en) * | 2022-10-10 | 2023-07-21 | 中科三清科技有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN115658078A (en) * | 2022-12-27 | 2023-01-31 | 金篆信科有限责任公司 | Database pre-compiling processing method, device, equipment and medium |
| CN115658078B (en) * | 2022-12-27 | 2023-03-17 | 金篆信科有限责任公司 | Database pre-compiling processing method, device, equipment and medium |
| CN118055039A (en) * | 2024-04-11 | 2024-05-17 | 明阳产业技术研究院(沈阳)有限公司 | Network equipment management method and system based on telemetry data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113778545A (en) | Data processing method, device, equipment and storage medium | |
| CN109032706B (en) | Intelligent contract execution method, device, equipment and storage medium | |
| US8191042B2 (en) | Continuation based declarative definition and composition | |
| US8560602B2 (en) | Data sharing in a stream processing system | |
| US9015685B2 (en) | Code analysis for simulation efficiency improvement | |
| CN112860260B (en) | Web-based cross-platform application construction tool and method in college scene | |
| CN111897541A (en) | Software interaction platform and method for automatically deploying resources in cloud environment | |
| CN110716748A (en) | Service processing method and device, computer readable medium and electronic equipment | |
| US20120291045A1 (en) | Registration and execution of highly concurrent processing tasks | |
| WO2013018204A1 (en) | Image processing software development method, image processing software development apparatus, and image processing software development program | |
| CN113467972A (en) | Communication interface construction method, communication interface construction device, computer equipment and storage medium | |
| CN114217887A (en) | Micro-service arranging method, micro-service executing method, micro-service arranging device and micro-service executing server | |
| CN110457132B (en) | Method and device for creating functional object and terminal equipment | |
| CN113448678A (en) | Application information generation method, deployment method, device, system and storage medium | |
| CN115390846A (en) | Compiling construction method and device, electronic equipment and storage medium | |
| CN116360735A (en) | Form generation method, device, equipment and medium | |
| CN118502992A (en) | Service arrangement method and related equipment | |
| US20110246967A1 (en) | Methods and systems for automation framework extensibility | |
| Banerjee et al. | Programming paradigm and the internet of things | |
| Fortier et al. | Dyninka: a FaaS framework for distributed dataflow applications | |
| CN109976713A (en) | Script processing method and device for Service Assembly | |
| CN113485686B (en) | Information system program generation method and device, electronic equipment and storage medium | |
| US11797277B2 (en) | Neural network model conversion method server, and storage medium | |
| CN114237686A (en) | Installation package generation method and device, electronic equipment and storage medium | |
| CN113835904A (en) | Remote procedure call control method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |