CN115328627A

CN115328627A - Data processing method and device

Info

Publication number: CN115328627A
Application number: CN202210915062.XA
Authority: CN
Inventors: 唐东帆; 范旭; 张威力; 邓金星; 陈旭东
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2022-08-01
Filing date: 2022-08-01
Publication date: 2022-11-11

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises: receiving a data acquisition request, wherein the data acquisition request comprises a page identifier; obtaining a DSL text corresponding to the page identifier; analyzing each task and the dependency relationship among the tasks from the DSL text, wherein the tasks correspond to data sources; and executing each task according to the dependency relationship to acquire result data corresponding to the data acquisition request from a data source corresponding to the task. According to the implementation method, the execution sequence of each task is determined according to the dependency relationship of each task analyzed from the DSL text, the data acquisition from a multi-data source is supported, the DSL text is generated according to the flow chart constructed by the visual interface, the purposes of changing, taking effect in real time and quickly getting online can be achieved, the development efficiency can be improved, and the method is suitable for fast-paced and multi-iteration interactive marketing scenes.

Description

Data processing method and device

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for data processing.

Background

The service party calls the data source to acquire data to perform service processing, and the authority needs to be applied to the data source provider before calling, so that the data acquisition efficiency is reduced. The service arranging system can provide the calling of the data source for a business party, but the existing service arranging system is difficult to adapt to a private data source, cannot acquire data from various data sources, and is difficult to meet business requirements.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data processing method and apparatus, which can support access to multiple data sources, have high development efficiency, can improve data acquisition efficiency, and are applicable to a fast-paced and multi-iteration interactive marketing scenario.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data processing method including:

receiving a data acquisition request, wherein the data acquisition request comprises a page identifier;

obtaining a DSL text corresponding to the page identifier;

analyzing each task and the dependency relationship among the tasks from the DSL text, wherein the tasks correspond to data sources;

and executing each task according to the dependency relationship to acquire result data corresponding to the data acquisition request from a data source corresponding to the task.

Optionally, executing each task according to the dependency relationship includes:

and constructing a directed acyclic graph according to the dependency relationship to determine the execution sequence of each task, and executing each task according to the execution sequence.

Optionally, executing each task according to the execution sequence includes:

acquiring tasks corresponding to all leaf nodes in the directed acyclic graph, and executing the tasks corresponding to all the leaf nodes in parallel;

and after the tasks corresponding to all the leaf nodes are executed, executing the tasks corresponding to the upper-level nodes of all the leaf nodes in parallel until all the tasks are executed.

Optionally, before acquiring the DSL text corresponding to the page identifier, the method includes:

and generating the DSL text according to a created flow chart, wherein the created flow chart comprises each task and the dependency relationship among the tasks.

Optionally, acquiring result data corresponding to the data acquisition request from a data source corresponding to the task includes:

acquiring original data corresponding to the data acquisition request from a data source corresponding to the task;

calling an adapter to perform adaptive conversion on the original data to determine the result data;

the adapter comprises one or more of a Java method, a built-in function, a SpEL expression and a FaaS platform custom Groovy script.

Optionally, the task includes one or more of a unified configuration center task, an RPC generalized remote call task, an HTTP call task, an MQ message sending task, an MYSQL database operation task, and a Redis operation task.

Optionally, obtaining the DSL text corresponding to the page identifier includes:

obtaining a DSL text corresponding to the page identification from a local cache;

obtaining the DSL text from a distributed cache if the DSL text is not present in the local cache.

According to still another aspect of the embodiments of the present invention, there is provided an apparatus for data processing, including:

the receiving module receives a data acquisition request, wherein the data acquisition request comprises a page identifier;

the obtaining module is used for obtaining the DSL text corresponding to the page identifier;

the analysis module is used for analyzing each task and the dependency relationship among the tasks from the DSL text, wherein the tasks correspond to data sources;

and the execution module executes each task according to the dependency relationship so as to acquire result data corresponding to the data acquisition request from a data source corresponding to the task.

According to another aspect of an embodiment of the present invention, there is provided an electronic apparatus including:

one or more processors;

a storage device to store one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data processing method provided by the present invention.

According to a further aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the method of data processing provided by the present invention.

One embodiment of the above invention has the following advantages or benefits: the corresponding DSL text is obtained through the page identification in the data obtaining request, each task and the dependency relationship among the tasks can be analyzed from the DSL text, and the execution sequence of each task is further determined.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic illustration of the main flow of a method of data processing according to an embodiment of the invention;

FIG. 2 is a schematic diagram of a main flow of another method of data processing according to an embodiment of the invention;

FIG. 3 is a flow chart illustrating a method of data processing according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention;

FIG. 5 is a flow chart of data processing using the data processing apparatus according to the embodiment of the present invention;

FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of a main flow of a method of data processing according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:

step S101: receiving a data acquisition request, wherein the data acquisition request comprises a page identifier;

step S102: obtaining a DSL text corresponding to the page identifier;

step S103: analyzing each task and the dependency relationship among the tasks from the DSL text, wherein the tasks correspond to the data source;

step S104: and executing each task according to the dependency relationship to acquire result data corresponding to the data acquisition request from the data source corresponding to the task.

In the embodiment of the invention, the data processing method can be used for the process that a business side acquires data from a data source connected with an application. For example, in the e-commerce field, the basic information can be obtained from the first page of the interactive marketing campaign; and basic information in a promotion activity page in a bank application system can be acquired.

In the embodiment of the present invention, the data obtaining request may be a request for obtaining data to be displayed in a current page, and the data obtaining request includes a page identifier. In the embodiment of the present invention, the data acquisition request includes trigger types such as an RPC (Remote Procedure Protocol) request, a timing schedule, and an MQ (Message Queue) Message. The RPC request may be a call request of an application, for example, a home page of an interactive playing method completes the pull of home page information through a call. The timing call request is to execute the acquisition of a DSL (Domain-Specific Language) text and execute a corresponding task at a timing, thereby achieving the purpose of executing the service logic at a timing. The MQ message is a subject subscribed upstream through the MQ message middleware, and whether the corresponding DSL text is acquired or not is judged according to the existence of the message content or the specific message content, and corresponding service logic is executed. The triggers of each data acquisition request have no dependency relationship and are completely independent of each other, and each trigger records corresponding trigger log information including information such as trigger type and trigger time. Optionally, the RPC request and the MQ message-triggered data acquisition request may be executed immediately, delayed, at a preset time interval, or the like.

Optionally, before obtaining the DSL text corresponding to the page identifier, the method includes: and generating DSL texts according to the created flow chart, wherein the created flow chart comprises each task and the dependency relationship among the tasks.

In the embodiment of the invention, a user edits and creates a flow chart in an H5 page through a visual operation interface, the flow chart comprises each task and the dependency relationship among the tasks, a DSL text can be generated according to the created flow chart, and the generated DSL text is stored in a database and is synchronized into a local cache and a distributed cache. The flow chart can be created by a mode of dragging nodes, and part of BPMN2.0 specifications are supported. The nodes correspond to the tasks, the tasks correspond to the data sources one by one, and logic inside the execution nodes is executed, namely the tasks are executed, so that data are acquired from the data sources corresponding to the tasks. The method for creating and generating the DSL text can aggregate related upstream services according to business requirements, can quickly develop interface services meeting the business requirements, avoids the steps of newly building applications, issuing applications, applying interfaces and the like, and can obviously improve the efficiency. Meanwhile, one-key release and real-time effect can be realized by modifying the flow chart, DSL text information or quoted Groovy (an agile development language based on a Java virtual machine) script content, the compiling and constructing processes are omitted, the purpose of quick online is achieved, and the defect that the online Java application service logic cannot be effective in real time because the online Java application service logic is modified by adopting the steps of code modification, compiling and packaging, mirror image construction and application release in the prior art is overcome.

In the embodiment of the present invention, when a large number of highly concurrent scenes are processed and a data acquisition request is received, if a database needs to be read each time to acquire a DSL text corresponding to a page identifier, the efficiency of acquiring the DSL text is reduced, so that the execution efficiency of a task cannot be ensured; and when the request amount reaches a certain degree, database blockage may occur, and then avalanche effect is caused to cause system breakdown. Optionally, obtaining the DSL text corresponding to the page identifier includes: obtaining a DSL text corresponding to the page identifier from a local cache; and under the condition that the DSL text does not exist in the local cache, obtaining the DSL text from the distributed cache. Obtaining DSL text from a local cache or a distributed cache may improve the efficiency of obtaining DSL text. The local cache may be a local Caffeine (high-performance cache library based on Java 1.8 Version) cache, and the distributed cache may be a Redis cache. Further, in the case that the DSL text corresponding to the page identifier does not exist in the distributed cache, the DSL text is obtained from the database, and is stored in the local cache and the distributed cache, so that the DSL text can be directly obtained from the local cache or the distributed cache in the subsequent process.

Different service logics are required to be executed according to data acquisition requests of different pages, and corresponding DSL texts can be acquired according to page identifiers in the data acquisition requests. The DSL text comprises the page identifier, so the corresponding DSL text can be obtained through the matching of the page identifier. The tasks and the dependency relationship among the tasks can be analyzed from the DSL text, and the scene code, the user type, the tasks and the like can also be analyzed. Wherein, the scene code is a scene identifier of the interactive activity; the user type can be a classification of the user according to a preset rule; the task may be a task that obtains data from a data source. Different DSL texts can implement scenario, thousands of people and thousands of faces of data aggregation and processing.

In the embodiment of the invention, the data acquisition request further comprises a version information field, and a DSL analyzer of a corresponding version can be selected to analyze the DSL text according to the version information field, so that the service abnormity caused by version upgrading is avoided.

In the embodiment of the invention, the tasks in the DSL text comprise one or more of a unified configuration center task, an RPC generalized remote call task, an HTTP call task, an MQ information sending task, an MYSQL database operation task and a Redis operation task. Accordingly, the data source includes one or more of a unified configuration center interface, an RPC service interface, an HTTP interface, MQ messages, a MYSQL database, a Redis cache. While performing the task, a connection is established with the data source to obtain the data. The method applies for the calling authority of each data source for the service party in advance so that the service party can call conveniently, the service party can apply for increasing the calling authority of the data source according to the service requirement of the service party, and subsequently other service parties do not need to apply for the authority again when accessing again.

In the embodiment of the present invention, the dependency relationship between tasks can be analyzed from the DSL text, and each task corresponds to each data node in the flowchart, and performs strong and weak dependency configuration on the data node, for example, the data node can be configured as a strong dependency node or a weak dependency node according to the dependency relationship, and if the task corresponding to the data node configured as the strong dependency node fails to be executed, the whole process fails to be executed, which further causes the whole application program to make mistakes and stop; if the task corresponding to the data node configured as the weak dependency node fails to be executed, the task can be continuously executed according to node failure logic preset by a user.

In the embodiment of the present invention, for each task, if all the tasks are executed in series, performance is low, and if all the tasks are executed in parallel, a dependency relationship between the tasks cannot be guaranteed.

In the embodiment of the present invention, as shown in fig. 2, each task is executed according to the dependency relationship, which includes:

step S201: constructing a directed acyclic graph according to the dependency relationship to determine the execution sequence of each task;

step S202: the respective tasks are executed according to the execution order.

According to the dependency relationship of each task, each data node corresponding to each task is constructed into a Directed Acyclic Graph (DAG), and a DAG shortest path algorithm is utilized to determine a better execution sequence of each task, so that the dependency relationship is guaranteed, and the execution efficiency is maximized. If the directed acyclic graph cannot be constructed, if data nodes with execution sequences being mutually dependent exist, failure is directly returned.

In the embodiment of the invention, each task is executed according to the dependency relationship, and when a scene of the tasks executed in parallel does not exist, all the tasks can be directly executed in series.

In the embodiment of the present invention, the executing of each task according to the execution sequence includes: acquiring tasks corresponding to all leaf nodes in the directed acyclic graph, and executing the tasks corresponding to all the leaf nodes in parallel; and after the tasks corresponding to all the leaf nodes are executed, executing the tasks corresponding to the nodes at the upper level of each leaf node in parallel until all the tasks are executed.

When executing tasks according to the directed acyclic graph, firstly acquiring tasks corresponding to all leaf nodes, putting the tasks corresponding to all the leaf nodes into a thread pool for parallel execution, inquiring the strong and weak dependency configuration of data nodes corresponding to each task for parallel execution, judging whether the task fails to be executed or not, if the task fails to be executed and the data nodes corresponding to the task are strong dependency nodes, jumping out of execution logic and returning a failure result, if the task fails to be executed, acquiring the tasks corresponding to the upper-level nodes of all the leaf nodes after the tasks corresponding to all the leaf nodes are executed, and executing in parallel until all the tasks are executed.

The different tasks differ in their functionality. The RPC generalized remote call task needs to serialize the input parameters into Json character strings, mainly comprises the input parameter values and the input parameter types represented by class full paths, and also needs to transmit the RPC interface names, the packet names, the method names and the timeout time. The embodiment of the invention adopts a generalization call mode to establish the RPC connection, can achieve the aim of calling the RPC service without an interface SDK (software tool development kit), and stores the result in a data context after the RPC bloom remote call task is executed; for the RPC generalized remote call task which has run once, the genericService (generalized call method) of the RPC generalized remote call task is put into a local Caffeine cache, so that the subsequent call is directly multiplexed with the previous connection, and the repeated initialization is avoided. The HTTP call task implements Post and Get requests through HttpClient and stores the request results in the data context. The MQ messaging task may support MQ message event listening and MQ message pushing. The MYSQL database operation task can be used for carrying out addition, deletion, modification and check on the MYSQL database. The Redis operation task supports Redis read-write operation.

The Check checking task can be parsed from the DSL text, and the Check checking task can use a Spring Expression Language (Spring Expression Language) Expression or refer to a Groovy script for data checking, and is used for terminating a flow in advance and returning a corresponding result when data is abnormal.

In the embodiment of the invention, the data adaptation task and the data output task can be analyzed from the DSL text. The data adaptation task corresponds to an adapter node in the flow chart, and the data output task corresponds to a data output node in the flow chart.

In an optional embodiment, when the task is executed, acquiring result data corresponding to the data acquisition request from a data source corresponding to the task includes: and acquiring original data corresponding to the data acquisition request from a data source corresponding to the task, extracting data of a preset field from the original data to obtain result data, or performing data conversion on the original data to obtain the result data.

Optionally, acquiring result data corresponding to the data acquisition request from a data source corresponding to the task includes: acquiring original data corresponding to the data acquisition request from a data source corresponding to the task; and calling an adapter to perform adaptive conversion on the original data so as to determine result data.

In the embodiment of the invention, when each task is executed, the connection with the data source is established, so that the original data corresponding to the data acquisition request can be acquired from the data source corresponding to each task, the original data is coarse-grained and unprocessed original data, and the service requirement is difficult to meet, therefore, the calling adapter is adopted to perform adaptive conversion and other processing on the original data. Aiming at service logics with different complexity degrees, the adapter can select one or more of a Java method, a built-in function, a SpEL expression and a FaaS platform custom Groovy script. Different adapters may be distinguished by a variable prefix. For example, the @ sign beginning indicates calling a Java method provided by the orchestration system, the $ sign beginning indicates calling a built-in function, the # sign beginning indicates that this is a piece of the SpEl expression, and the &' sign beginning indicates calling a Groovy script of the FaaS platform. The Java method is a common data source result conversion method, and is a method for settling after summarizing and inducing data mapping requirements of a large number of service parties on the data source for subsequent calling. The built-in function is mainly based on tool classes, such as currency unit conversion, time format conversion, character string encryption and decryption and the like. The SpEL expression supports reading object attributes, calling object methods and simple comparison and judgment, and is the most main and most common data adaptation tool. The Groovy script is suitable for a field conversion scene of complex logic, the Groovy script is firstly compiled on a FaaS platform (a public Service platform providing script compiling, script management, script publishing and script pulling, compiling and executing in the executing process), then a script name is introduced into a corresponding position of a DSL text as a variable, a data source result, namely original data, can be processed according to the Groovy script logic, and finally field data required by a Service is obtained. The four adapters support combined use and nested use, and can meet the data adaptation requirements of most service scenes.

For example, when a built-in function needs to be called for time format conversion, time data in original data acquired from a data source can be converted according to a time format to obtain an adapter mapping result, so as to determine result data.

The data output node is used for defining fields and formats returned to the client side, simultaneously supports the four adapters, can output the fields and the formats in a user-defined mode, and meets the requirements of the customized fields. After the adapter mapping result is obtained, the data output node can select a corresponding field for the adapter mapping result according to the self-defined field and format, or further perform data conversion to determine the result data. The data output node can also directly refer to the original data, select the corresponding field or add data conversion to determine the result data. And returning the result data to the client so that the client can display the result data.

Fig. 3 is a flowchart illustrating a data processing method according to an embodiment of the present invention. As shown in fig. 3, after receiving a data acquisition request triggered by any one of an external RPC request, an MQ message, and a timing task, where the data acquisition request includes a page identifier, acquiring a DSL text corresponding to the page identifier from a database, analyzing the DSL text to obtain each task and a dependency relationship between the tasks, constructing a directed acyclic graph according to the dependency relationship, determining an execution sequence of each task, executing each task according to the execution sequence, acquiring original data from a data source corresponding to the task, and performing adaptive conversion on the original data by using an adapter to obtain an adapter mapping result; and performing result assembly on the mapping result of the adapter to obtain result data, and returning the result data to the client.

The data processing method provided by the embodiment of the invention supports the access of multiple data sources, and the data is acquired by establishing connection between the DSL text and the data sources, so that the problem of difficult matching with private data sources in the prior art is solved; the method can realize high-performance loading of the data source, namely, efficiently acquire data in the data source, optimize the task execution sequence based on a DAG algorithm, construct a directed acyclic graph by taking tasks as nodes according to the dependency relationship of each task, obtain the efficient task execution sequence, execute the tasks in batches and in parallel, and improve the task execution efficiency; the method provides various adapters to carry out adaptation conversion on data, supports simple and complex logic scenes, can select the adapters according to the complexity of business logic, can call the adapters at data nodes and adaptation output nodes, can meet the requirements of customized fields, and can achieve the purposes of adaptation and validity check of data source results if a custom Groovy script is achieved through a Faas platform, enriches the use scenes, has good flexibility and meets the requirements of complex scenes; the method can realize real-time effect and quick online by modifying the flow chart, the DSL text or the quoted Groovy script content; when a flow chart is constructed, common BPMN2.0 specifications are supported, a simple and easy-to-use visual operation interface is provided, and a DSL text can be executed on other platforms supporting the BPMN2.0 specifications and has certain universality; when the flow chart is constructed and the DSL text is generated, the business application is established in a Low-Code mode, so that the method is friendly to non-technical personnel, the learning cost can be reduced, and the development efficiency is improved.

As shown in fig. 4, according to still another aspect of the embodiment of the present invention, there is provided an apparatus 400 for data processing, including:

a receiving module 401, configured to receive a data obtaining request, where the data obtaining request includes a page identifier;

an obtaining module 402, configured to obtain a DSL text corresponding to the page identifier;

the parsing module 403 is configured to parse the tasks and the dependency relationships among the tasks from the DSL text, where the tasks correspond to data sources;

and the execution module 404 executes each task according to the dependency relationship to acquire result data corresponding to the data acquisition request from a data source corresponding to the task.

In this embodiment of the present invention, the execution module 404 is further configured to: and constructing a directed acyclic graph according to the dependency relationship to determine the execution sequence of each task, and executing each task according to the execution sequence.

In this embodiment of the present invention, the execution module 404 is further configured to: the method comprises the following steps: acquiring tasks corresponding to all leaf nodes in the directed acyclic graph, and executing the tasks corresponding to all the leaf nodes in parallel; and after the tasks corresponding to all the leaf nodes are executed, executing the tasks corresponding to the nodes at the upper level of each leaf node in parallel until all the tasks are executed.

In this embodiment of the present invention, the apparatus 400 further includes a generating module, configured to: and before the DSL text corresponding to the page identifier is acquired, generating the DSL text according to the created flow chart, wherein the created flow chart comprises each task and the dependency relationship among the tasks.

In this embodiment of the present invention, the execution module 404 is further configured to: acquiring original data corresponding to the data acquisition request from a data source corresponding to the task; calling an adapter to perform adaptive conversion on the original data to determine result data; the adapter comprises one or more of a Java method, a built-in function, a SpEL expression and a FaaS platform custom Groovy script.

Optionally, the task includes one or more of a unified configuration center task, an RPC generalized remote call task, an HTTP call task, an MQ sending task, an MYSQL database operation task, and a Redis operation task.

In this embodiment of the present invention, the obtaining module 402 further includes: obtaining a DSL text corresponding to the page identification from a local cache; and under the condition that the DSL text does not exist in the local cache, obtaining the DSL text from the distributed cache.

The data processing device of the embodiment of the invention can comprise a foreground Web service and a background service arrangement system. Fig. 5 is a flowchart illustrating a method for processing data by using the data processing apparatus according to an embodiment of the present invention. A user operates an H5 visual interface, edits data nodes, data adaptation nodes and data output nodes in foreground Web service, and constructs a flow chart, wherein the data nodes correspond to tasks, and the tasks correspond to data sources; and the foreground Web service generates a DSL text according to the established flow chart, writes the generated DSL text into a database, and can also synchronize the generated DSL text into a local cache and a distributed cache. The service arranging system receives a data acquisition request, wherein the data acquisition request can be sent by a user through a client direct request interface, such as an RPC request, or can be sent by a foreground Web service request interface, such as timing scheduling; the service arranging system acquires a DSL text corresponding to the page identifier from the database according to the page identifier in the data acquisition request, and analyzes each task and the dependency relationship of each task in the DSL text; then determining the execution sequence of each task according to the dependency relationship so as to pull the original data from the data source corresponding to the task, and then calling an adapter by a data adaptation node to perform adaptation conversion on the original data to obtain an adapter mapping result; and the output node selects a self-defined field from the adapter mapping result to obtain result data, and returns the result data to be displayed at the client.

According to another aspect of an embodiment of the present invention, there is provided an electronic apparatus including: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by one or more processors, the one or more processors realize the data processing method provided by the invention.

Fig. 6 shows an exemplary system architecture 600 of a data processing apparatus or a method of data processing to which embodiments of the invention may be applied.

As shown in fig. 6, the system architecture 600 may include

terminal devices

601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the

terminal devices

601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

601, 602, 603 to interact with a server 605, via a network 604, to receive or send messages or the like. The

terminal devices

601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

601, 602, 603. The backend management server may analyze and perform other processing on the received data such as the information acquisition request, and feed back a processing result (for example, result data — only an example) to the terminal device.

It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data processing apparatus is generally disposed in the server 605.

It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present invention, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, an obtaining module, a parsing module, and an executing module. The names of these modules do not in some cases constitute a limitation on the module itself, for example, the acquisition module may also be described as a "module that acquires DSL text corresponding to a page identification".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a data acquisition request, wherein the data acquisition request comprises a page identifier; obtaining a DSL text corresponding to the page identifier; analyzing each task and the dependency relationship among the tasks from the DSL text, wherein the tasks correspond to the data source; and executing each task according to the dependency relationship to acquire result data corresponding to the data acquisition request from the data source corresponding to the task.

According to the technical scheme of the embodiment of the invention, the access of multiple data sources is supported, and the data is acquired by establishing connection between the DSL text and the data sources, so that the problem of difficult matching with private data sources in the prior art is solved; the method can realize high-performance loading of the data source, namely, efficiently acquire data in the data source, optimize the task execution sequence based on a DAG algorithm, construct a directed acyclic graph by taking tasks as nodes according to the dependency relationship of each task, obtain the efficient task execution sequence, execute the tasks in batches and in parallel, and improve the task execution efficiency; the method provides various adapters to perform adaptation conversion on data, supports simple and complex logic scenes, can select the adapters according to the complexity of business logic, can call the adapters at data nodes and adaptation output nodes, can meet the requirements of customized fields, and can achieve the purposes of adaptation and validity check of data source results if a user-defined Groovy script is achieved through a Faas platform, thereby enriching the use scenes, having good flexibility and meeting the requirements of complex scenes; the method can realize real-time effect and quick online by modifying the flow chart, the DSL text or the quoted Groovy script content; when a flow chart is constructed, common BPMN2.0 specifications are supported, a simple and easy-to-use visual operation interface is provided, and a DSL text can be executed on other platforms supporting the BPMN2.0 specifications and has certain universality; when the flow chart is constructed and the DSL text is generated, the business application is built in a Low-Code mode, the method is friendly to non-technical personnel, the learning cost can be reduced, and the development efficiency is improved.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data processing, comprising:

obtaining a DSL text corresponding to the page identifier;

and executing each task according to the dependency relationship so as to acquire result data corresponding to the data acquisition request from a data source corresponding to the task.

2. The method of claim 1, wherein performing each of the tasks according to the dependencies comprises:

3. The method of claim 2, wherein performing each of the tasks according to the execution order comprises:

and after the tasks corresponding to all the leaf nodes are executed, executing the tasks corresponding to the nodes at the upper level of each leaf node in parallel until all the tasks are executed.

4. The method of claim 1, wherein prior to obtaining the DSL text corresponding to the page identifier, comprising:

5. The method of claim 2, wherein retrieving result data corresponding to the data retrieval request from a data source corresponding to the task comprises:

6. The method of claim 1, wherein the tasks include one or more of a unified configuration center task, an RPC generalized remote call task, an HTTP call task, an MQ messaging task, a MYSQL database manipulation task, and a Redis manipulation task.

7. The method of claim 1, wherein obtaining the DSL text corresponding to the page identifier comprises:

obtaining a DSL text corresponding to the page identifier from a local cache;

8. An apparatus for data processing, comprising:

9. An electronic device, comprising:

one or more processors;

a storage device to store one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-7.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.