CN117075877A - Data processing method, computing device and storage medium - Google Patents

Abstract

The invention relates to the field of big data, in particular to a data processing method, computing equipment and a storage medium, wherein a data platform is arranged in the computing equipment, and the method comprises the following steps: setting a plurality of data processing tasks in an editing area according to a data platform; connecting a plurality of data processing tasks through directed lines, and setting a directed acyclic graph comprising the plurality of data processing tasks; and creating a workflow instance of a workflow defined by the directed acyclic graph, and calling a job unit to execute the workflow instance to obtain a data processing result. According to the invention, the input data source is output after being processed according to the service requirement, and the DAG graphical development processing process is used, so that the development of the data processing task can be directly carried out in a dragging mode without writing codes.

Description

Data processing method, computing device and storage medium

Technical Field

The present invention relates to the field of big data, and in particular, to a data processing method, a computing device, and a storage medium.

Background

Any company, after a period of operation, generates a large amount of data that is directly related to business objectives. And business function settings of various departments are different, most departments can grasp corresponding data items in a database according to business demands, write SQL codes, generate various report views, thoroughly read data by combining business backgrounds, and output data analysis results and business suggestions with explicit guiding significance.

In the prior art, the data functions are scattered on each service line, and serious repeated pulling of the data is carried out, and the same data has different roles and specific meanings in different departments because of the difference of service definitions, so that the data requirements are more changeable in the departments, the code research and development period is long, the maintenance is difficult, the efficiency is low, and the departments cannot rapidly present the data processing results according to the data processing requirements of the departments.

For this reason, a new data processing method is required.

Disclosure of Invention

To this end, the present invention provides a data processing method in an effort to solve or at least alleviate the above-identified problems.

According to one aspect of the present invention there is provided a data processing method adapted to provide a data platform in a computing device, the method comprising: setting a plurality of data processing tasks in an editing area according to a data platform; connecting a plurality of data processing tasks through directed lines, and setting a directed acyclic graph comprising the plurality of data processing tasks; and creating a workflow instance of a workflow defined by the directed acyclic graph, and calling a job unit to execute the workflow instance to obtain a data processing result.

Optionally, in the method according to the present invention, the data platform is provided with a development component area, the development component area is provided with a plurality of component tags, and the setting a plurality of data processing tasks in the editing area according to the data platform includes: and adding data processing tasks according to the component tags, wherein each component tag is suitable for setting one data processing task.

Optionally, in the method according to the invention, the data processing task comprises a data input task, the method further comprising: and setting a first field name, a field type and service information corresponding to the field of the data table which needs to be subjected to data processing for the added data input task.

Optionally, in the method according to the present invention, the data processing task further comprises a data processing task, the method further comprising: setting screening conditions for data cleaning for the added data cleaning task; and/or the first field name of the setting data table is modified to the second field name.

Optionally, in the method according to the present invention, the data processing task further comprises a data output task, the method further comprising: setting data format conversion of one or more field names of data for the added data output task; and/or data filtering to set one or more field names.

Optionally, in the method according to the present invention, the data processing task further comprises a database processing task, the method further comprising: and setting database processing sentences for the added database processing.

Optionally, in the method according to the present invention, the data processing task further comprises a data output task, the method further comprising: the data table or the data drawing information is set for the added data output task so as to draw the data table or the data drawing from the output data.

Optionally, in the method according to the present invention, invoking the job unit to execute the workflow instance includes: when executing the workflow instance, creating a corresponding task instance for each task in the workflow to obtain a plurality of task instances; each task instance is executed according to the order of task connections in the directed acyclic graph.

According to another aspect of the present invention, there is provided a computing device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the data processing method according to the present invention.

According to yet another aspect of the present invention, there is provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a data processing method according to the present invention.

The data processing method in the invention is suitable for being executed in a computing device, wherein a data platform is arranged in the computing device, and the method comprises the following steps: setting a plurality of data processing tasks in an editing area according to a data platform; connecting a plurality of data processing tasks through lines, and setting a directed acyclic graph comprising the plurality of data processing tasks; and creating a workflow instance of the directed acyclic graph stroke workflow, and calling a job unit to execute the workflow instance to obtain a data processing result. According to the invention, the input data source is output after being processed according to the service requirement, and the DAG graphical development processing process is used, so that the development of the data processing task can be directly carried out in a dragging mode without writing codes.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which set forth the various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to fall within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. Like reference numerals generally refer to like parts or elements throughout the present disclosure.

FIG. 1 shows a schematic diagram of a data processing method 100 according to an exemplary embodiment of the invention;

FIG. 2 illustrates a block diagram of a computing device 200 according to an exemplary embodiment of the invention.

FIG. 3 illustrates a schematic diagram of setting up a plurality of data processing tasks according to an exemplary embodiment of the invention;

FIG. 4 illustrates a schematic diagram of a data entry task according to an exemplary embodiment of the present invention;

fig. 5 shows a schematic diagram of setting screening conditions according to an exemplary embodiment of the present invention;

FIG. 6 illustrates a schematic diagram of modifying field names according to an exemplary embodiment of the present invention;

FIG. 7 shows a schematic diagram of a setup database processing statement according to an exemplary embodiment of the invention;

FIG. 8 illustrates a schematic diagram of a setup data output task according to an exemplary embodiment of the present invention;

FIG. 9 illustrates a schematic diagram of a PowerJob task scheduling engine in accordance with an exemplary embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like reference numerals generally refer to like parts or elements.

Fig. 1 shows a schematic diagram of a data processing method 100 according to an exemplary embodiment of the invention. The data processing method 100 of the present invention is suitable for execution in a computing device.

FIG. 2 illustrates a block diagram of a computing device according to an exemplary embodiment of the invention. In a basic configuration, computing device 200 includes at least one processing unit 220 and system memory 210. According to one aspect, depending on the configuration and type of computing device, system memory 210 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 210 includes an operating system 211.

According to one aspect, operating system 211 is suitable, for example, for controlling the operation of computing device 200. Further, examples are practiced in connection with a graphics library, other operating systems, or any other application program and are not limited to any particular application or system. This basic configuration is illustrated in fig. 2 by those components within dashed line 215. According to one aspect, computing device 200 has additional features or functionality. For example, according to one aspect, computing device 200 includes additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape.

As set forth hereinabove, according to one aspect, program modules 212 are stored in system memory 210. According to one aspect, program modules 212 may include one or more application programs, the invention is not limited to the type of application program, e.g., applications further include: email and contacts applications, word processing applications, spreadsheet applications, database applications, slide show applications, drawing or computer-aided application, web browser applications, etc.

According to one aspect, the examples may be practiced in a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic components or a microprocessor. For example, examples may be practiced via a system on a chip (SOC) in which each or many of the components shown in fig. 2 may be integrated on a single integrated circuit. According to one aspect, such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operating via an SOC, the functionality described herein may be operated via dedicated logic integrated with other components of computing device 200 on a single integrated circuit (chip). Embodiments of the invention may also be practiced using other techniques capable of performing logical operations (e.g., AND, OR, AND NOT), including but NOT limited to mechanical, optical, fluidic, AND quantum techniques. In addition, embodiments of the invention may be practiced within a general purpose computer or in any other circuit or system.

According to one aspect, computing device 200 may also have one or more input devices 231, such as a keyboard, mouse, pen, voice input device, touch input device, or the like. Output devices 232 such as a display, speakers, printer, etc. may also be included. The foregoing devices are examples and other devices may also be used. Computing device 200 may include one or more communication connections 233 that allow communication with other computing devices 240. Examples of suitable communication connections 233 include, but are not limited to: RF transmitter, receiver and/or transceiver circuitry; universal Serial Bus (USB), parallel and/or serial ports. Computing device 200 may be communicatively connected to other computing devices 240 via communication connection 233.

Embodiments of the present invention also provide a non-transitory readable storage medium storing instructions for causing the computing device to perform a method according to embodiments of the present invention. The readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be any method or technology for information storage. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of readable storage media include, but are not limited to: phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transitory readable storage medium.

According to one aspect, communication media is embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., carrier wave or other transport mechanism) and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio Frequency (RF), infrared, and other wireless media.

It should be noted that although the above-described computing device only shows processing unit 220, system memory 210, input device 231, output device 232, and communication connection 233, the device may include other components necessary to achieve proper operation in a particular implementation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.

According to one embodiment of the invention, a computing device is provided with a data platform. The data platform is adapted to process data.

Returning to fig. 1, as shown in fig. 1, the data processing method of the present invention first performs step 110: and setting a plurality of data processing tasks in the editing area according to the data platform.

Fig. 3 shows a schematic diagram of setting up a plurality of data processing tasks according to an exemplary embodiment of the invention. As shown in fig. 3, the data platform is provided with a development component area and an editing area, the editing area is provided with one or more component tags, the development component area is provided with a plurality of component tags, the development component comprises a data input component, a data processing component, a data output component and a database processing component, and the number of specific components included in the development component area is not limited. Setting a plurality of data processing tasks in an editing area according to a data platform comprises: the data processing tasks are added according to the component tags, and each component tag is suitable for setting one data processing task. The development component also includes a node component for interfacing with a plurality of data processing tasks.

According to one embodiment of the invention, the data processing task comprises a data input task, the method further comprising: and setting a first field name, a field type and service information corresponding to the field of the data table which needs to be subjected to data processing for the added data input task. FIG. 4 illustrates a schematic diagram of a data entry task according to an exemplary embodiment of the present invention. As shown in fig. 4, specifically, a data source type, a source database, a reading mode, a read field table name, a reading condition and the like of the data input task may also be set, and the read field may also be set with a first field name, a field type and service information corresponding to the field. The first field name is an unmodified field name, the field type comprises an int type, a varchar type and the like, and the invention does not limit the specific types included in the field type. The service information corresponding to the field is the actual parameter in the specific service corresponding to the data of the field.

According to one embodiment of the invention, the data processing tasks further comprise data processing tasks, the method further comprising: setting screening conditions for data cleaning for the added data cleaning task; and/or the first field name of the setting data table is modified to the second field name. Fig. 5 shows a schematic diagram of setting a screening condition according to an exemplary embodiment of the present invention. As shown in fig. 5, a filtering condition may be set to filter out unnecessary fields.

Fig. 6 shows a schematic diagram of modifying field names according to an exemplary embodiment of the present invention. As shown in fig. 6, after the first field name is split according to the manner of symbols, words, keywords, etc., the required symbols, words or keywords are reselected and combined to obtain the modified second field name. When the field names are modified, the modified field names can be customized.

According to one embodiment of the invention, the data processing tasks further comprise data output tasks, the method further comprising: setting data format conversion of one or more field names of data for the added data output task; and/or data filtering to set one or more field names. Data format conversion refers to converting a first data type of data of a field to a second data type. The data filtering means that data filtering conditions are set, data which do not accord with a preset range in the data under the field are deleted, and the data which accord with the preset range are extracted.

According to one embodiment of the invention, the data processing tasks further comprise database processing tasks, the method further comprising: and setting database processing sentences for the added database processing. Fig. 7 shows a schematic diagram of a setup database processing statement according to an exemplary embodiment of the invention. As shown in fig. 7, the database gram is specifically implemented as an SQL database, and SQL statements can be set to process data, so as to implement a target function.

According to one embodiment of the invention, the data processing tasks further comprise data output tasks, the method further comprising: the data table or the data drawing information is set for the added data output task so as to draw the data table or the data drawing from the output data. Fig. 8 shows a schematic diagram of setting data output tasks according to an exemplary embodiment of the invention. As shown in fig. 8, one or more tasks may be set: and drawing a data table or a data graph according to the processed data.

Subsequently, step 120 is performed to connect the plurality of data processing tasks through the directed line, and a directed acyclic graph is set that includes the plurality of data processing tasks. According to one embodiment of the invention, the directed acyclic graph includes a plurality of sequentially connected data processing tasks, each data processing task being connected by a directed arrow, and the task processing flow is formed without a task processing loop.

Finally, step 130 is executed to create a workflow instance of the workflow defined by the directed acyclic graph, and call the job unit to execute the workflow instance to obtain a data processing result. According to one embodiment of the invention, when a workflow instance is executed, a corresponding task instance is created for each task in the workflow to obtain a plurality of task instances;

when each task instance is executed according to the order of task connections in the directed acyclic graph, task (Job) refers to task information describing what needs to be scheduled by PowerJob, including task name, scheduling time, processor information, etc. When the task is executed, the task instance is obtained after the task is instantiated, and the task instance records the runtime information of the task. When executing each task instance, the job unit is called, the job unit is an execution unit of the task instance, and at least one job unit is executed when executing one task instance. The directed acyclic graph includes a plurality of data processing tasks that can result in a Workflow (Workflow) for performing the data processing task orchestration. A workflow instance (workflow instance) is generated after the workflow is scheduled to be executed, and information of the workflow runtime is recorded.

According to one embodiment of the invention, the tasks can be processed by a task scheduling engine, and the task scheduling engine can be specifically realized as a secondary development of the PowerJob task scheduling engine so as to synchronize task execution state logs to a data development platform through the task scheduling engine. FIG. 9 illustrates a schematic diagram of a PowerJob task scheduling engine in accordance with an exemplary embodiment of the present invention. As shown in fig. 9, when the PowerJob task schedule is executed, the job_info table is queried to determine tasks needing to be executed recently, an instance_info table written into a database by an execution record is generated when the tasks are executed, and then the tasks are pushed into a time wheel to trigger the schedule at corresponding time points. Finally, calculating the next scheduling time and synchronizing to an instance_info table of the database.

The invention can process various data, has rich and various data sources, the data platform supports the function of registering the data sources, and can synchronize the data by collocating various input and output data sources. When the data is processed, the input data source is subjected to data filtering, character string replacement and data desensitization processing according to the service requirement, and then output. The full-quantity synchronization data is realized and then the increment synchronization data is continuously realized. When the workflow is constructed, the DAG is used for graphical development, codes are not required to be written, and the task development can be directly carried out in a dragging mode. The invention also provides a fault-tolerant processing mechanism, which processes possible errors in a plurality of modes in the process of data synchronization, retries the errors and classifies the errors so as to ensure the consistency of the client and the background data.

According to the invention, through project componentization data processing, data processing is combined with actual business application, and the processing of data extraction, processing, cleaning, output and the like are rapidly realized; the capability and the efficiency of data processing are improved, and the data processing capability and the efficiency are integrated with a statement module of an accounting system, so that the data analysis can be carried out in a combined way; the data requirements of different posts are met; the data development manages the tasks, and the component generates a DAG execution flow chart through the solution and the project; the data source is rich and various, the data platform supports the function of registering the data source, and various input and output data sources can be matched and formed to perform data synchronization. And supporting data processing according to service requirements, and outputting the input data source after data filtering, character string replacement and data desensitization processing. The full-quantity synchronization data is realized and then the increment synchronization data is continuously realized. The invention uses DAG to develop in a graphical way: the development of the task can be directly carried out in a dragging mode without writing codes. The invention also provides a fault-tolerant processing mechanism, which processes possible errors in a plurality of modes in the process of data synchronization, retries the errors and classifies the errors so as to ensure the consistency of the client and the background data.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects.

Those skilled in the art will appreciate that the modules or units or groups of devices in the examples disclosed herein may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into a plurality of sub-modules.

Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or groups of embodiments may be combined into one module or unit or group, and furthermore they may be divided into a plurality of sub-modules or sub-units or groups. Any combination of all features disclosed in this specification, as well as all processes or units of any method or apparatus so disclosed, may be employed, except that at least some of such features and or processes or units are mutually exclusive. Each feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments.

Furthermore, some of the embodiments are described herein as methods or combinations of method elements that may be implemented by a processor of a computer system or by other means of performing the functions. Thus, a processor with the necessary instructions for implementing the described method or method element forms a means for implementing the method or method element. Furthermore, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is for carrying out the functions performed by the elements for carrying out the objects of the invention.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the data processing method of the invention in accordance with instructions in said program code stored in the memory.

By way of example, and not limitation, computer readable media comprise computer storage media and communication media. Computer-readable media include computer storage media and communication media. Computer storage media stores information such as computer readable instructions, data structures, program modules, or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.

As used herein, unless otherwise specified the use of the ordinal terms "first," "second," "third," etc., to describe a general object merely denote different instances of like objects, and are not intended to imply that the objects so described must have a given order, either temporally, spatially, in ranking, or in any other manner.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments are contemplated within the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. The disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention.

Claims (10)

1. A data processing method adapted to run in a computing device having a data platform disposed therein, the method comprising:

setting a plurality of data processing tasks in an editing area according to the data platform;

connecting a plurality of data processing tasks through directed lines, and setting a directed acyclic graph comprising the plurality of data processing tasks;

and creating a workflow instance of the workflow defined by the directed acyclic graph, and calling a job unit to execute the workflow instance to obtain a data processing result.

2. The method of claim 1, wherein the data platform is provided with a development component area provided with a plurality of component tags, the setting a plurality of data processing tasks in an editing area according to the data platform comprising:

and adding data processing tasks according to the component tags, wherein each component tag is suitable for setting one data processing task.

3. The method of claim 2, wherein the data processing task comprises a data input task, the method further comprising:

and setting a first field name, a field type and service information corresponding to the field of the data table which needs to be subjected to data processing for the added data input task.

4. The method of claim 2, wherein the data processing tasks further comprise data processing tasks, the method further comprising:

setting screening conditions for data cleaning for the added data cleaning task;

and/or the first field name of the setting data table is modified to the second field name.

5. The method of claim 2, wherein the data processing tasks further comprise data output tasks, the method further comprising:

setting data format conversion of one or more field names of data for the added data output task;

and/or data filtering to set one or more field names.

6. The method of claim 2, wherein the data processing tasks further comprise database processing tasks, the method further comprising:

and setting database processing sentences for the added database processing.

7. The method of claim 2, wherein the data processing tasks further comprise data output tasks, the method further comprising:

the data table or the data drawing information is set for the added data output task so as to draw the data table or the data drawing from the output data.

8. The method of claim 1, wherein the invoking the job unit to execute the workflow instance comprises:

when executing the workflow instance, creating a corresponding task instance for each task in the workflow to obtain a plurality of task instances;

each task instance is executed according to the order of task connections in the directed acyclic graph.

9. A computing device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1-8.

10. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of claims 1-8.