CN116303672A

CN116303672A - Data processing method and device, equipment and computer readable storage medium

Info

Publication number: CN116303672A
Application number: CN202310156414.2A
Authority: CN
Inventors: 蔡满天; 张茜; 凌海挺; 杜均; 王忠瑞
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2023-02-14
Filing date: 2023-02-14
Publication date: 2023-06-23

Abstract

The application relates to the field of artificial intelligence and discloses a data processing method, a device, equipment and a computer readable storage medium. The method can be used for processing financial business data by generating a directed acyclic graph according to a plurality of jobs; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; thereby rapidly determining the repeated node according to the directed acyclic graph; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged operation, if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node for data processing of the packaged operation, intuitively displaying the running condition of the node by introducing the visualized node view, and directly acquiring the calculation result of the multiplexing node to reduce the time consumption of the data processing process.

Description

Data processing method and device, equipment and computer readable storage medium

Technical Field

The present application relates to the field of artificial intelligence, and in particular, to a data processing method, apparatus, device, and computer readable storage medium.

Background

At present, in the field of artificial intelligence, a great amount of data is required for training an intelligent system, and an analyst needs to know not only business rules but also used databases and technical tools when performing data analysis, and certain databases and data analysis background knowledge are required for using the technical tools. For example, financial transactions involve multiple account data including, but not limited to, funds, stocks, insurance, etc., and because of the different types of related financial transaction data and different business rules, analysts need to use different technical tools to analyze different financial transaction data when analyzing related financial transaction data.

Along with the development of the artificial intelligence field, the data processing speed is higher, and particularly in the data explosion age, massive data needs to be processed, and a plurality of jobs are often required to be subjected to distributed data processing, so that the existing data processing method traverses each job to perform data processing, and the processed data is more, so that the processing time is long.

Disclosure of Invention

In order to solve the technical problems, embodiments of the present application provide a data processing method, a device, an apparatus, and a computer readable storage medium, respectively, so as to reduce a calculation process of multiplexing nodes in a data processing process, reduce time consumption of the data processing process, and thereby meet a requirement of a data processing speed in an artificial intelligence field.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned in part by the practice of the application.

According to an aspect of the embodiments of the present application, there is provided a data processing method, including: generating a directed acyclic graph according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; wherein the multiplexing node is a node repeatedly appearing in the directed acyclic graph; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging the plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged operation, and if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node to be used for carrying out data processing on the packaged operation.

In another embodiment, the generating a directed acyclic graph from a plurality of jobs includes:

acquiring metadata from the plurality of jobs, and taking each node as a node; wherein the metadata includes apertures of the plurality of jobs and a sequential relationship between the plurality of jobs;

and determining the connection sequence among the nodes according to the metadata, and connecting each node according to the connection sequence to obtain the directed acyclic graph.

In another embodiment, the method further comprises:

traversing each node, and taking the traversed node as a target node;

detecting whether a dependency relationship exists between the target node and a node above the target node;

if the fact that the target node and the previous node of the target node have the dependency relationship is detected, the view corresponding to the previous node of the target node is obtained, and the view corresponding to the target node is generated according to the view corresponding to the previous node of the target node.

In another embodiment, the method further comprises:

detecting whether an operation fault occurs in the operation process of the packaged operation;

if the occurrence of the operation fault is detected, acquiring a fault keyword corresponding to the operation fault, and matching the fault keyword with a preset fault keyword;

And if the matching is successful, processing the operation fault according to a preset processing strategy corresponding to the preset fault key.

In another embodiment, the detecting whether an operation fault occurs in the operation process of the packaged job includes:

acquiring an operation log generated in the operation process of the packaged operation;

and detecting whether the fault moment is recorded in the operation log or not so as to determine whether the operation fault occurs in the operation process of the packaged operation.

In another embodiment, the matching the fault keyword with a preset fault keyword includes:

if the matching fails, marking the view corresponding to the node running to the running fault, and recording the fault keyword for updating the preset fault keyword.

In another embodiment, before said running said packaged job, said method further comprises:

performing parameter analysis on the packaged operation to obtain operation resources required by operation of the packaged operation;

and distributing target operation resources which are more than the operation resources for the packaged job so as to enable the packaged job to operate according to the target operation resources.

According to an aspect of an embodiment of the present application, there is provided a data processing apparatus including:

a generation module configured to generate a directed acyclic graph from a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph;

the marking module is configured to mark multiplexing nodes in the directed acyclic graph and store calculation results of the multiplexing nodes; wherein the multiplexing node is a node repeatedly appearing in the directed acyclic graph;

the encapsulation module is configured to generate corresponding views for each node, mark the views corresponding to the multiplexing nodes, encapsulate the plurality of jobs and the views corresponding to all the nodes, and obtain encapsulated jobs;

and the operation module is configured to operate the packaged operation, and if the view corresponding to the multiplexing node is detected, the calculation result of the multiplexing node is obtained so as to be used for carrying out data processing on the packaged operation.

In another embodiment, the generating module includes:

an acquisition unit configured to acquire metadata from the plurality of jobs and to take each node as a node; wherein the metadata includes apertures of the plurality of jobs and a sequential relationship between the plurality of jobs;

And the generating unit is configured to determine the connection sequence among the nodes according to the metadata and connect each node according to the connection sequence so as to obtain the directed acyclic graph.

In another embodiment, the data processing apparatus further includes:

the node traversing unit is configured to traverse each node and take the traversed node as a target node;

a relationship detection unit configured to detect whether a dependency relationship exists between the target node and a node preceding the target node;

and the view generation unit is configured to acquire a view corresponding to the previous node of the target node and generate a view corresponding to the target node according to the view corresponding to the previous node of the target node if the dependency relationship between the target node and the previous node of the target node is detected.

In another embodiment, the data processing apparatus further includes:

an operation failure detection unit configured to detect whether an operation failure occurs in an operation process of the packaged job;

the fault matching unit is configured to acquire a fault keyword corresponding to the operation fault if the operation fault is detected, and match the fault keyword with a preset fault keyword;

And the fault processing unit is configured to process the operation fault according to a preset processing strategy corresponding to the preset fault key word if the matching is successful.

In another embodiment, the operation failure detection unit includes:

the operation log obtaining plate is configured to obtain an operation log generated in the operation process of the packaged operation;

and the operation log detection plate is configured to detect whether the operation log records fault time or not so as to determine whether the operation fault occurs in the operation process of the packaged operation.

In another embodiment, the fault matching unit includes:

and the fault updating plate is configured to mark the view corresponding to the node running to the running fault if the matching fails, and record the fault keyword for updating the preset fault keyword.

In another embodiment, the data processing apparatus further includes:

the analysis module is configured to perform parameter analysis on the packaged operation to obtain operation resources required by operation of the packaged operation;

and the resource allocation module is configured to allocate more target operation resources than the operation resources for the packaged job so as to enable the packaged job to operate according to the target operation resources.

According to an aspect of an embodiment of the present application, there is provided an electronic device including: a controller; and a memory for storing one or more programs which, when executed by the controller, perform the data processing method described above.

According to an aspect of the embodiments of the present application, there is also provided a computer-readable storage medium having stored thereon computer-readable instructions, which when executed by a processor of a computer, cause the computer to perform the above-described data processing method.

According to an aspect of embodiments of the present application, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the data processing method described above.

In the technical scheme provided by the embodiment of the application, a directed acyclic graph is generated according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph; therefore, the repeated node is rapidly determined according to the directed acyclic graph, only one calculation is needed for the multiplexing node, and the calculation result is stored, so that the subsequent direct calling is convenient, the calculation process of the multiplexing node in the data processing process is reduced, and the time consumption of the data processing process is reduced; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged operation, if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node for carrying out data processing on the packaged operation, introducing a visualized node view, intuitively displaying the running condition of each node, directly acquiring the calculation result of the multiplexing node by detecting the view corresponding to the multiplexing node, and saving the time of the data processing process without repeated calculation.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application. It is apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art. In the drawings:

FIG. 1 is a schematic illustration of one implementation environment to which the present application relates;

FIG. 2 is a flow chart of a data processing method according to an exemplary embodiment of the present application;

FIG. 3 is an exemplary directed acyclic graph;

FIG. 4 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 2;

FIG. 5 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 2;

FIG. 6 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 2;

FIG. 7 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 6;

FIG. 8 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 6;

FIG. 9 is a flow chart of another data processing method proposed based on the embodiment shown in FIG. 2;

FIG. 10 is a schematic diagram of a data processing apparatus according to an exemplary embodiment of the present application;

fig. 11 is a schematic diagram of a computer system of an electronic device according to an exemplary embodiment of the present application.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

Reference to "a plurality" in this application means two or more than two. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., a and/or B may represent: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

The present application relates to a data processing process in the field of artificial intelligence, and more particularly, to an improvement of the data processing process, so as to improve the data processing efficiency, so as to meet the data processing speed in the field of artificial intelligence, and referring to fig. 1, fig. 1 is a schematic diagram of an implementation environment related to the present application. The implementation environment includes a job input 100 and a server 200, and communication is performed between the job input 100 and the server 200 through a wired or wireless network.

The job input terminal 100 is used for transmitting a job input by a user to the server 200; server 200 generates a directed acyclic graph from a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged job, and if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node to be used for carrying out data processing on the packaged job.

The user operation input terminal 100 includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc., for example, any electronic device capable of receiving a user input operation, such as a smart phone, a tablet, a notebook, a computer, etc., and transmitting the operation to the server 200, which is not limited herein. The server 200 may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, where a plurality of servers may form a blockchain, and the servers are nodes on the blockchain, and the server 200 may also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network ), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited in this respect.

In the data explosion era, massive data needs to be processed, a plurality of jobs are often required to be processed in a distributed mode, the existing data processing method traverses each job to process data, the processed data is more, the processing time is long, the requirement of data processing speed in the artificial intelligence field cannot be met, and particularly in the financial field with larger data processing requirement, a plurality of financial service data needs to be processed rapidly.

For this reason, an exemplary embodiment of the present application shows a data processing method to reduce the computation process of multiplexing nodes in the data processing process, so as to reduce the time consumption of the data processing process, thereby meeting the requirement of the data processing speed in the artificial intelligence field, and referring specifically to fig. 2, fig. 2 is a flowchart of a data processing method shown in an exemplary embodiment of the present application, and the method may be specifically executed by the server 200 in the implementation environment shown in fig. 1. Of course, the method may also be applied to other implementation environments and executed by a server device in other implementation environments, which is not limited by the present embodiment. As shown in fig. 2, the method at least includes S210 to S240, which are described in detail as follows:

s210: generating a directed acyclic graph according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph.

Each job is used as a node, and the nodes are connected in sequence according to the relation, execution logic, execution sequence and the like among the jobs, so that a directed acyclic graph is generated.

The directed acyclic graph (DAG, directed acyclic graph) is a loop-free directed graph. As shown in fig. 3, fig. 3 is an exemplary directed acyclic graph. Wherein 1 to 6 represent nodes. The directed acyclic graph starts from a vertex and returns to the point through a plurality of edges, because the directed acyclic graph does not necessarily form a ring when one point in the directed graph reaches the other point through two routes, and therefore the directed acyclic graph does not necessarily convert to a tree, but any directed tree is a directed acyclic graph.

S220: marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph.

In the construction of the directed acyclic graph, there may be a node that repeatedly appears at least once, the node is used as a multiplexing node in the directed acyclic graph, and the multiplexing node is calculated once to obtain a calculation result, and the calculation result is stored, for example, in a database, so that subsequent call is facilitated, and repeated calculation is not required.

S230: and generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs.

When DAG analysis is performed on a plurality of jobs, each step correspondingly generates a view, that is, each node in this embodiment generates a corresponding view, marks the view corresponding to the multiplexing node, and encapsulates each job and the view corresponding to each node to obtain an encapsulated job.

S240: and running the packaged job, and if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node to be used for carrying out data processing on the packaged job.

In the operation process, whether the view corresponding to the multiplexing node is operated is detected in real time, and if the view corresponding to the multiplexing node is operated, the calculation result of the multiplexing node is directly obtained, so that the flow of data processing is quickened.

The embodiment generates a directed acyclic graph according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph; therefore, the repeated node is rapidly determined according to the directed acyclic graph, only one calculation is needed for the multiplexing node, and the calculation result is stored, so that the subsequent direct calling is convenient, the calculation process of the multiplexing node in the data processing process is reduced, and the time consumption of the data processing process is reduced; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged operation, if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node for carrying out data processing on the packaged operation, introducing a visualized node view, intuitively displaying the running condition of each node, directly acquiring the calculation result of the multiplexing node by detecting the view corresponding to the multiplexing node, and saving the time of a data processing process without repeated calculation so as to meet the requirement of the data processing speed in the artificial intelligence field.

The data processing method in this embodiment may be applied to a scenario of processing financial service data, for example, analyzing and processing service data such as funds, insurance, stocks, etc. for multiple accounts, and dividing the service data into different jobs according to different types of the fund service data. Then generating a directed acyclic graph according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph; marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph; generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging a plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs; and running the packaged operation, if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node for carrying out data processing on the packaged operation, thereby realizing the processing of different financial service data such as funds, insurance, stocks and the like of a plurality of accounts.

In the data explosion age, massive data needs to be processed, and in a big data environment, a distributed computing system is often needed. Distributed computation is not simple to clearly understand business logic SQL (Structured Query Language ) and can run, and tuning, error analysis and the like are often needed, and after analysis caliber exploration is finished, the caliber is often needed to be solidified.

For this reason, the data processing method according to another exemplary embodiment of the present application can be applied to a distributed computing system for data processing, and referring specifically to fig. 4, fig. 4 is a flowchart of another data processing method according to the embodiment shown in fig. 2. The method further includes at least S410 to S420 in S210 shown in fig. 2, and is described in detail below:

s410: acquiring metadata from a plurality of jobs, and taking each node as a node; wherein the metadata includes a caliber of the plurality of jobs and a sequential relationship between the plurality of jobs.

Some project with a plurality of jobs is collected, wherein Metadata (Metadata), also called intermediate data and relay data, including caliber, sequence and the like of the plurality of jobs is used as data (data about data) describing data, mainly describing data attribute (property) information, and is used for supporting functions such as indicating storage positions, historical data, resource searching, file recording and the like.

S420: and determining the connection sequence among the nodes according to the metadata, and connecting each node according to the connection sequence to obtain the directed acyclic graph.

Since the metadata is data describing data, it can describe a relationship, a connection order, etc. between the respective jobs, so that the connection order between the respective nodes can be determined, and the respective nodes are connected in accordance with the connection order, thereby constructing the directed acyclic graph.

The embodiment further illustrates that the relation among the jobs is determined according to the metadata in the jobs, so that the connection sequence among the nodes representing each job is determined, and each node is connected according to the connection sequence, so that the directed acyclic graph is formed.

In the analysis process of the directed acyclic graph, a certain relation may exist between nodes, and how to determine the relation between the nodes and how to strengthen the relation between the nodes is a technical problem to be solved.

For this reason, in another embodiment of the present application, a node having a dependency relationship is determined, and a view construction of the target node is performed according to an existing view of the node having the dependency relationship, specifically referring to fig. 5, fig. 5 is a flowchart of another data processing method proposed based on the embodiment shown in fig. 2. The method further includes at least S510 to S530 based on the data processing method shown in fig. 2, and the following details are described below:

S510: and traversing each node, and taking the traversed node as a target node.

Each node in the directed acyclic graph is traversed one by one to determine whether dependencies exist between related nodes.

S520: and detecting whether a dependency relationship exists between the target node and a node above the target node.

The dependency relationship refers to whether the job of the target node pointer has a dependable relationship with the job characterized by the previous node, namely, whether data dependency exists, wherein the data dependency is a constraint relationship between an attribute and an attribute in the relationship (whether the values among the attributes are equal or not).

S530: if the fact that the dependency relationship exists between the target node and the previous node of the target node is detected, the view corresponding to the previous node of the target node is obtained, and the view corresponding to the target node is generated according to the view corresponding to the previous node of the target node.

Illustratively, a view corresponding to a previous node of the target node is obtained, relevant data in the view is extracted, and processing, such as fusion, partial calling, replacement and the like, is performed according to the relevant data and the data provided by the target node, so as to generate the view corresponding to the target node.

The embodiment further illustrates how to generate a view corresponding to the target node when the target node has a dependency relationship with the previous node; if the dependency relationship exists, a view corresponding to the target node is generated according to a view corresponding to the previous node of the target node, so that the connection of the target node and the previous node on the respective views is enhanced.

In another embodiment of the present application, the packaged job is subjected to fault detection, and is optimized to ensure its normal operation, and referring specifically to fig. 6, fig. 6 is a flowchart of another data processing method according to the embodiment shown in fig. 2. The method further includes at least S610 to S630 based on the data processing method shown in fig. 2, and the following details are described below:

s610: and detecting whether an operation fault occurs in the operation process of the packaged operation.

S620: if the occurrence of the operation fault is detected, acquiring a fault keyword corresponding to the operation fault, and matching the fault keyword with a preset fault keyword.

S630: if the matching is successful, processing the operation fault according to a preset processing strategy corresponding to the preset fault key word.

The operation fault detection may be performed in real time during the operation of the packaged job, or may be performed after the whole operation is finished. If the operation fault is detected, a fault keyword, such as A, B, C, 1, 2, 3 and other letters and/or numerals, is generated, wherein the fault keyword corresponds to a preset processing strategy, such as a fault keyword is A, and corresponds to an A processing strategy, the fault keyword is B, and corresponds to a B processing strategy; in addition, it may happen that different keywords correspond to the same processing policy, e.g. the fault keywords C and D both correspond to the C processing policy.

As another example, the operation fault processing may also be performed according to a preset processing policy shown in table 1, where table 1 is a table corresponding to a preset fault keyword and a preset processing policy, specifically as follows:

TABLE 1

It should be noted that, in table 1, there is a preset fault keyword of "Unknown", in fact, not a preset fault keyword, and when an Unknown operation fault occurs, the "Unknown" is used to indicate that a preset processing policy is matched, that is, the operation fault cannot be processed according to the preset processing policy, and at this time, an alarm needs to be given to prompt a person to operate.

The embodiment further illustrates operation analysis of the packaged operation, and the corresponding fault processing strategy is matched according to the fault keywords so as to accurately process the operation faults, realize accurate fault removal and ensure normal operation of the packaged operation.

In another embodiment of the present application, it is described how to detect whether an operation failure occurs during the operation of a packaged job, and referring specifically to fig. 7, fig. 7 is a flowchart of another data processing method according to the embodiment shown in fig. 6. The method further includes at least S710 to S720 in S610 shown in fig. 6, and is described in detail below:

S710: and acquiring an operation log generated in the operation process of the packaged operation.

The embodiment further illustrates that after the packaged job is finished, whether the operation fault occurs in the operation process is judged according to the operation log generated in the operation process.

The operation log is a log for recording the fault time, fault keywords, and the like corresponding to the occurrence of faults in the operation process of the job. Wherein the fault keywords comprise preset fault keywords and non-preset fault keywords, the non-preset fault keywords have no corresponding preset processing strategies,

s720: and detecting whether a fault moment is recorded in the operation log or not so as to determine whether the packaged operation has an operation fault in the operation process.

In other embodiments, whether the packaged job has an operation fault in the operation process may be further determined by detecting whether a fault keyword is recorded in the operation log. Illustratively, fault keywords in the sparkEventLog are automatically captured, and the fault keywords are matched to corresponding processing strategies according to preset fault keywords.

In addition, the operation log in the embodiment also records the position of the operation fault, namely the corresponding view when the operation fault occurs, namely the position of the fault is automatically positioned without manual operation searching when the operation fault occurs, and the position is quickly and intuitively displayed and is more clear.

According to the embodiment, whether the operation faults occur in the operation process or not is determined by detecting whether the fault moment is recorded in the operation log, and whether the operation faults occur in the operation process or not can be accurately judged by detecting whether the fault moment is recorded or not.

In the process of detecting an operation fault, a situation may occur in which the fault keyword does not match with the preset fault keyword, for example, the fault keyword "unkenown" in table 1 above occurs.

For this reason, another embodiment of the present application will explain this situation in detail, and referring specifically to fig. 8, fig. 8 is a flowchart of another data processing method according to the embodiment shown in fig. 6. The method further includes S810 at least in S620 shown in fig. 6, and is described in detail below:

s810: if the matching fails, marking the view corresponding to the node which runs to the running fault, and recording the fault keywords for updating the preset fault keywords.

If the matching of the fault key word and the preset fault key word fails, the preset processing strategy indicating that the fault key word is not matched is adopted, and the new operation fault is indicated. Recording the fault keywords, adding the fault keywords into a database where the preset fault keywords are located, obtaining the preset fault keywords as heart, and matching the corresponding processing strategies for the fault keywords.

The embodiment describes the matching process of the fault keywords and the preset fault keywords, further describes the situation that the fault keywords are required to be recorded when the matching of the fault keywords and the preset fault keywords fails, and updates the preset fault keywords according to the fault keywords, namely, the fault keywords are used as new preset fault keywords, and a corresponding fault processing strategy is added, so that the normal operation of the operation is ensured.

In the distributed data processing process, how to optimize the operation resources for performing distributed processing, how to allocate reasonable operation resources for the packaged job is a technical problem to be solved, so another embodiment of the present application further describes how to allocate reasonable operation resources for the packaged job, and specifically please refer to fig. 9, and fig. 9 is a flowchart of another data processing method according to the embodiment shown in fig. 2. The method further includes at least S910 to S920 before S240 shown in fig. 2, and is described in detail below:

s910: and carrying out parameter analysis on the packaged operation to obtain operation resources required by operation of the packaged operation.

The parameter analysis in this embodiment is job operation resource analysis, including analysis of operation resources such as the memory required by the packaged job during operation, and parameters of the CPU (central processing unit ), and the like, and exemplary parameter analysis is performed on the packaged job: and reversely deducing the concurrency of the parallel computation of the big data according to the complexity of the computation operator fed back by the physical plan and the data quantity required to be computed, and configuring parameters such as a required memory, a CPU and the like.

The operation resource required by the operation of the packaged job in this embodiment is the minimum operation resource required by the operation of the job, that is, the minimum operation resource required by the normal operation of the packaged job.

S920: and distributing target operation resources which are more than the operation resources aiming at the packaged operation so as to enable the packaged operation to operate according to the target operation resources.

For example, the running resources required for running the packaged job are 100 units, and for the packaged job, 200 units of running resources are allocated to the packaged job so as to ensure that the packaged job runs normally. If the operation resource is divided into 80 units, the operation resource is obviously lower than the operation resource of 100 units required by the normal operation of the operation resource, and the packaged operation can not normally operate.

The embodiment illustrates that before running the packaged job, parameter analysis is performed on the packaged job, and running resources which are more than the minimum running resources of the packaged job are allocated to the packaged job, so that normal running of the packaged job in the running process is ensured, and the situation that the packaged job cannot normally run due to lack of the running resources is avoided.

Another aspect of the present application further provides a data processing apparatus, which corresponds to the data processing method provided in the foregoing embodiment, and they belong to the same concept, and particularly as shown in fig. 10, fig. 10 is a schematic structural diagram of the data processing apparatus according to an exemplary embodiment of the present application. Wherein the data processing device comprises:

A generation module 1010 configured to generate a directed acyclic graph from a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph.

A marking module 1030 configured to mark the multiplexing nodes in the directed acyclic graph and store the calculation results of the multiplexing nodes; the multiplexing nodes are repeatedly appeared nodes in the directed acyclic graph.

The encapsulation module 1050 is configured to generate a corresponding view for each node, mark the view corresponding to the multiplexing node, and encapsulate the multiple jobs and the views corresponding to all the nodes to obtain an encapsulated job.

And the operation module 1070 is configured to operate the packaged job, and if the view corresponding to the multiplexing node is detected, the calculation result of the multiplexing node is obtained, so as to be used for performing data processing on the packaged job.

In another embodiment, the generating module 1010 includes:

an acquisition unit configured to acquire metadata from a plurality of jobs and to take each node as a node; wherein the metadata includes a caliber of the plurality of jobs and a sequential relationship between the plurality of jobs.

In another embodiment, the data processing apparatus further comprises:

and the relation detection unit is configured to detect whether a dependency relation exists between the target node and a node above the target node.

And the view generation unit is configured to acquire a view corresponding to the previous node of the target node and generate a view corresponding to the target node according to the view corresponding to the previous node of the target node if the fact that the target node and the previous node of the target node have a dependency relationship is detected.

In another embodiment, the data processing apparatus further comprises:

and an operation fault detection unit configured to detect whether an operation fault occurs in the operation process of the packaged job.

The fault matching unit is configured to acquire a fault keyword corresponding to the operation fault if the operation fault is detected, and match the fault keyword with a preset fault keyword.

In another embodiment, an operation failure detection unit includes:

The operation log obtaining plate is configured to obtain an operation log generated in the operation process of the packaged operation.

The operation log detection plate is configured to detect whether a fault moment is recorded in the operation log or not so as to determine whether the packaged operation has an operation fault in the operation process or not.

In another embodiment, the fault matching unit includes:

In another embodiment, the data processing apparatus further comprises:

and the analysis module is configured to perform parameter analysis on the packaged operation to obtain operation resources required by operation of the packaged operation.

The resource allocation module is configured to allocate more target operation resources than operation resources for the packaged job so as to enable the packaged job to operate according to the target operation resources.

It should be noted that, the data processing apparatus provided in the foregoing embodiments and the data processing method provided in the foregoing embodiments belong to the same concept, and a specific manner in which each module and unit perform an operation has been described in detail in the method embodiment, which is not described herein again.

Another aspect of the present application also provides an electronic device, including: a controller; and a memory for storing one or more programs which, when executed by the controller, perform the data processing method described above.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a computer system of an electronic device according to an exemplary embodiment of the present application, which illustrates a schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiments of the present application.

It should be noted that, the computer system 1100 of the electronic device shown in fig. 11 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present application.

As shown in fig. 11, the computer system 1100 includes a central processing unit (Central Processing Unit, CPU) 1101 that can perform various appropriate actions and processes, such as performing the methods in the above-described embodiments, according to a program stored in a Read-Only Memory (ROM) 1102 or a program loaded from a storage section 1108 into a random access Memory (Random Access Memory, RAM) 1103. In the RAM 1103, various programs and data required for system operation are also stored. The CPU 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. An Input/Output (I/O) interface 1105 is also connected to bus 1104.

The following components are connected to the I/O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output portion 1107 including a Cathode Ray Tube (CRT), a liquid crystal display (Liquid Crystal Display, LCD), and a speaker; a storage section 1108 including a hard disk or the like; and a communication section 1109 including a network interface card such as a LAN (Local Area Network ) card, a modem, or the like. The communication section 1109 performs communication processing via a network such as the internet. The drive 1110 is also connected to the I/O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in drive 1110, so that a computer program read therefrom is installed as needed in storage section 1108.

In particular, according to embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1109, and/or installed from the removable media 1111. When executed by a Central Processing Unit (CPU) 1101, performs the various functions defined in the system of the present application.

It should be noted that, the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with a computer-readable computer program embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Where each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present application may be implemented by means of software, or may be implemented by means of hardware, and the described units may also be provided in a processor. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.

Another aspect of the present application also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method as before. The computer-readable storage medium may be included in the electronic device described in the above embodiment or may exist alone without being incorporated in the electronic device.

Another aspect of the present application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the data processing method provided in the above-described respective embodiments.

According to an aspect of the embodiments of the present application, there is also provided a computer system including a central processing unit (Central Processing Unit, CPU) which can perform various appropriate actions and processes, such as performing the method in the above embodiments, according to a program stored in a Read-Only Memory (ROM) or a program loaded from a storage section into a random access Memory (Random Access Memory, RAM). In the RAM, various programs and data required for the system operation are also stored. The CPU, ROM and RAM are connected to each other by a bus. An Input/Output (I/O) interface is also connected to the bus.

The following components are connected to the I/O interface: an input section including a keyboard, a mouse, etc.; an output section including a Cathode Ray Tube (CRT), a liquid crystal display (Liquid Crystal Display, LCD), and the like, and a speaker, and the like; a storage section including a hard disk or the like; and a communication section including a network interface card such as a LAN (Local Area Network ) card, a modem, or the like. The communication section performs communication processing via a network such as the internet. The drives are also connected to the I/O interfaces as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, and the like are mounted on the drive as needed so that a computer program read therefrom is mounted into the storage section as needed.

The foregoing is merely a preferred exemplary embodiment of the present application and is not intended to limit the embodiments of the present application, and those skilled in the art may make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data processing, comprising:

Generating a directed acyclic graph according to a plurality of jobs; wherein each job corresponds to a node in the directed acyclic graph;

marking multiplexing nodes in the directed acyclic graph, and storing calculation results of the multiplexing nodes; wherein the multiplexing node is a node repeatedly appearing in the directed acyclic graph;

generating corresponding views for each node, marking the views corresponding to the multiplexing nodes, and packaging the plurality of jobs and the views corresponding to all the nodes to obtain packaged jobs;

and running the packaged operation, and if detecting the view corresponding to the multiplexing node, acquiring a calculation result of the multiplexing node to be used for carrying out data processing on the packaged operation.

2. The method of claim 1, wherein the generating a directed acyclic graph from a plurality of jobs comprises:

3. The method according to claim 1, wherein the method further comprises:

traversing each node, and taking the traversed node as a target node;

4. The method according to claim 1, wherein the method further comprises:

5. The method of claim 4, wherein detecting whether an operational failure occurred during operation of the packaged job comprises:

6. The method of claim 4, wherein the matching the fault key with a preset fault key comprises:

7. The method of claim 1, wherein prior to said running said packaged job, said method further comprises:

8. A data processing apparatus, comprising:

9. An electronic device, comprising:

a controller;

a memory for storing one or more programs that, when executed by the controller, cause the controller to implement the data processing method of any of claims 1-7.

10. A computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor of a computer, cause the computer to perform the data processing method of any of claims 1 to 7.