CN114691112A - Data processing method and device and data processing server - Google Patents
Data processing method and device and data processing server Download PDFInfo
- Publication number
- CN114691112A CN114691112A CN202011586477.4A CN202011586477A CN114691112A CN 114691112 A CN114691112 A CN 114691112A CN 202011586477 A CN202011586477 A CN 202011586477A CN 114691112 A CN114691112 A CN 114691112A
- Authority
- CN
- China
- Prior art keywords
- data processing
- plug
- task
- configuration information
- ins
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Stored Programmes (AREA)
Abstract
The application discloses a data processing method, a data processing device and a data processing server, wherein the method comprises the following steps: receiving task configuration information of a data processing task, wherein the task configuration information comprises plug-in configuration information of one or more data processing plug-ins; and loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in. The technical scheme has the advantages that the data development process can be simplified in a plug-in mode, even a user who does not know the programming technology can simply edit task configuration information, so that the required data processing can be completed by using the existing plug-in, brand-new data development operation is not required to be performed according to each data processing requirement of a business side, the data development time is saved, the data development cost is reduced, and the data processing efficiency is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, and a data processing server.
Background
In the internet era, data is an important production material of enterprises, and many enterprises choose to use a data processing system (data platform) to manage, view, process and the like the data. For example, many data are derived from the service log, and different service lines may require different information in the service log, thereby placing different data processing requirements on the service log.
One solution is that a data engineer develops data according to the general requirements of a business party, writes a corresponding data processing program, and re-develops the data under the condition that the individual requirements of the business party cannot be met, so that the data development is continued endlessly if the requirements of the business party are continuously increased. Moreover, newly developed data processing programs are not necessarily compatible with existing data processing systems, and customized development cannot be reused by multiple business parties, thereby reducing resource utilization.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device and a data processing server, so that a user can conveniently, quickly and self-define data development, and data processing requirements are met.
The embodiment of the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides a data processing method, executed by a data processing server, including: receiving task configuration information of a data processing task, wherein the task configuration information comprises plug-in configuration information of one or more data processing plug-ins; and loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in.
In some embodiments, receiving task configuration information for a data processing task comprises: and providing a data processing task editing interface, and receiving task configuration information of the data processing task through the data processing task editing interface.
In some embodiments, the data processing task editing interface includes a list of plug-ins and a first editing region, and receiving task configuration information for the data processing task through the data processing task editing interface includes: responding to a selection request of one or more data processing plug-ins in the plug-in list, and displaying visual objects of the corresponding data processing plug-ins in a first editing area; and receiving the plug-in connection relation input through the first editing area, and generating corresponding plug-in configuration information according to the plug-in connection relation.
In some embodiments, the selection request of the data processing plug-in is triggered according to a selection operation of the data processing plug-in the plug-in list, and/or the selection request of the data processing plug-in is triggered according to an operation of dragging the data processing plug-in the plug-in list to the first editing area.
In some embodiments, the data processing task editing interface further comprises a second editing region, and receiving task configuration information for the data processing task through the data processing task editing interface further comprises: responding to a selection request of a visual object of a target data processing plug-in the first editing area, and displaying a configurable item of the target data processing plug-in the second editing area; plug-in configuration information for the target data processing plug-in is received via the configurable item.
In some embodiments, the method further comprises: receiving and storing a program package of the data processing plug-in, wherein the program package comprises a front end code and a back end code; and realizing the plug-in list and the first editing area according to the front end code, and realizing the configurable item of the data processing plug-in according to the back end code.
In some embodiments, the task configuration information further includes task base configuration information, the data processing task editing interface includes a third editing region, and receiving the task configuration information of the data processing task through the data processing task editing interface includes: and receiving the task basic configuration information input through the third editing area.
In some embodiments, the method further comprises: storing the task configuration information into a task configuration file in a markup language format, wherein each plug-in configuration information corresponds to a markup language node; loading the corresponding data processing plug-in according to the plug-in configuration information comprises: analyzing the task configuration file to obtain an analysis result and a plug-in dependency relationship corresponding to each data processing plug-in; and loading the corresponding data processing plug-in according to the analysis result and the plug-in dependency relationship.
In some embodiments, loading the corresponding data processing plug-in according to the parsing result and the plug-in dependency relationship includes: loading the analysis result of the target data processing plug-in a distributed container to obtain a plurality of examples of the target data processing plug-in; executing the data processing task according to the loaded data processing plug-in includes: data processing tasks are performed distributively according to the plug-in dependencies using respective instances of respective data processing plug-ins.
In a second aspect, an embodiment of the present application further provides a data processing apparatus, configured to implement the data processing method described above.
In a third aspect, an embodiment of the present application further provides a data processing server, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform a data processing method as described in any one of the above.
In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium storing one or more programs that, when executed by a database server including a plurality of application programs, cause the database server to perform the data processing method as described in any one of the above.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects: the data development process is simplified in a plug-in mode, even a user who does not know the programming technology can simply edit task configuration information, so that the required data processing is completed by using the existing plug-in, brand-new data development operation is not required to be carried out according to each data processing requirement of a business party, the data development time is saved, the data development cost is reduced, and the data processing efficiency is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 shows a schematic flow diagram of a data processing method according to an embodiment of the present application;
FIG. 2 depicts a data processing system architecture diagram according to an embodiment of the present application;
FIG. 3 shows a schematic diagram of a data processing task editing interface, according to one embodiment of the present application;
FIG. 4 shows a schematic diagram of data processing principles according to one embodiment of the present application;
FIG. 5 shows a schematic block diagram of a data processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a data processing server in an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The technical idea of the application is that data processing requirements which can be usually provided by a service party are split in advance, a plurality of steps in the data processing process are determined, and then an execution unit, namely a data processing plug-in corresponding to each step, is programmed and developed. After the business side puts forward the actual data processing requirement, a plurality of data processing plug-ins capable of executing the data processing requirement can be freely selected, and the data processing plug-ins are correspondingly configured, so that the configuration of a data processing task is completed, and then the data processing plug-ins can be correspondingly loaded according to the configuration of the data processing plug-ins only by executing the data processing task, so that the data processing is completed.
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 shows a flow diagram of a data processing method according to an embodiment of the present application, which may be specifically executed by a data processing server. As shown in fig. 1, the method includes:
step S110, receiving task configuration information of the data processing task, where the task configuration information includes plug-in configuration information of one or more data processing plug-ins.
The task configuration information may further include a task name, an execution time, and the like, and the core content of the task configuration information is to include the plug-in configuration information. This has the advantage that the user does not need to be concerned with the data processing logic at the code level, but only with the business logic that he needs to implement.
For example, a user needs to extract concerned business data from a business log, and during specific processing, a plurality of steps such as opening a log file, closing the log file, querying, extracting, checking and the like need to be involved, but how to implement each step is not the content actually concerned by the user, and the user only needs to determine the execution sequence of the steps.
Then, returning to the technical solution of the present application, the execution of each step may be implemented by using a data processing plug-in, what the user needs to configure is the connection relationship between these plug-ins to characterize the execution sequence and the data flow direction, and in addition, if such steps are extracted, the user may also configure which fields are specifically extracted.
And step S120, loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in.
It can be seen that the method shown in fig. 1 simplifies the data development process in a plug-in manner, and even a user who does not know the programming technology can simply edit task configuration information, so that the required data processing is completed by using the existing plug-in, and a brand-new data development operation is not required to be performed for each data processing requirement of a business party, thereby saving data development time, reducing data development cost, and improving data processing efficiency.
In some embodiments, receiving task configuration information for a data processing task comprises: and providing a data processing task editing interface, and receiving task configuration information of the data processing task through the data processing task editing interface.
The technical solution of the present application may be specifically implemented in a data processing system, for example, fig. 2 shows a data processing system architecture diagram according to an embodiment of the present application. The data processing system includes a management control center, a DB (Database, which may be understood as a broad Database) storage, a task core scheduling, an API Server, and a Yarn (a resource manager, and in other embodiments, other types of resource managers may be used). The management control center is mainly related to several functions of task creating, manual task running, task operation and maintenance and task monitoring, the DB mainly stores a task record generated by the newly created task, a data version and a task instance generated by the manual task running and a task running record, and the task operation and maintenance can read the task running record.
When the task runs, the task management can be realized according to task core scheduling, and a data version and a task instance are automatically generated. The specific interactive interface can be realized by using an API Server, and can also acquire the task state, and if the alarm condition is triggered, the operation and maintenance personnel is warned by a mail or a short message.
Specifically, the task can implement resource management allocation through Yarn, such as resource allocation of MR (Map Reduce, Map-Reduce), resource allocation of ETL (Extract-Transform-Load), etc., and the operation can be implemented depending on Shell script, Python script, MR program. The data can be stored by using GFS, Hive, HBase, HDFS, Local File and other modes.
It should be noted that fig. 2 shows only one possible data processing system example, and in practical application, the adjustment may be performed according to requirements.
In particular, the data processing system may provide an accessible data processing task editing interface through which task configuration information for the data processing task is received.
In some embodiments, the data processing task editing interface includes a list of plug-ins and a first editing region, and receiving task configuration information for the data processing task through the data processing task editing interface includes: responding to a selection request of one or more data processing plug-ins in the plug-in list, and displaying visual objects of the corresponding data processing plug-ins in a first editing area; and receiving the plug-in connection relation input through the first editing area, and generating corresponding plug-in configuration information according to the plug-in connection relation.
FIG. 3 shows a schematic diagram of a data processing task editing interface, according to one embodiment of the present application. As shown in fig. 3, the leftmost side is a plug-in list, which shows a plurality of plug-in groups such as a checking plug-in group, and a specific data processing plug-in each plug-in group, and of course, the directory structure of the data processing plug-in may be set according to requirements, for example, divided into a plurality of plug-in groups by function.
After selecting a data processing plug-in, the visual object of the data processing plug-in may be displayed in the first editing area, for example, the first editing area in the middle of fig. 3 already shows a plurality of selected data processing plug-ins, and the same data processing plug-in may be selected a plurality of times.
In some embodiments, the selection request of the data processing plug-in is triggered according to a selection operation of the data processing plug-in the plug-in list, and/or the selection request of the data processing plug-in is triggered according to an operation of dragging the data processing plug-in the plug-in list to the first editing area.
For example, after double-clicking one data processing plug-in the left plug-in list, the visual object is displayed in the first editing area, or the user may drag one data processing plug-in the plug-in list to the first editing area by a drag operation, and then display the visual object in the first editing area.
If the data processing plug-in the first editing area is not needed any more (for example, by wrong selection), deletion can also be performed, for example, by triggering a menu by a right key, selecting a deletion option, and the like. Fig. 3 also shows a functional area at the top, and the visual objects in the first editing area can be deleted, enlarged, abbreviated, etc.
The positions of the visual objects in the first editing area can be repositioned by dragging and the like, and the connection mode between the visual objects can be edited at will, which is not described herein any more.
In some embodiments, the data processing task editing interface further comprises a second editing region, and receiving task configuration information for the data processing task through the data processing task editing interface further comprises: responding to a selection request of a visual object of a target data processing plug-in the first editing area, and displaying a configurable item of the target data processing plug-in the second editing area; plug-in configuration information for the target data processing plug-in is received via the configurable item.
Fig. 3 shows a second editing area on the right side of the first editing area, and after a user selects a visual object in the first editing area, if the data processing plug-in corresponding to the visual object has configurable content, the configurable item of the target data processing plug-in can be presented in the second editing area, and fig. 3 shows four configurable parameters.
In some embodiments, the task configuration information further includes task base configuration information, the data processing task editing interface includes a third editing region, and receiving the task configuration information of the data processing task through the data processing task editing interface includes: and receiving the task basic configuration information input through the third editing area.
As shown in fig. 3, the rightmost side shows a third editing area where the user can enter basic configuration information for the task, such as authorization schema, task name, task specification, etc. It should be noted that, limited to the page space, the editing regions in fig. 3 can be folded, expanded or combined as required.
In some embodiments, the data processing method further comprises receiving and storing a package of data processing plug-ins, the package comprising front end code and back end code; and realizing the plug-in list and the first editing area according to the front end code, and realizing the configurable item of the data processing plug-in according to the back end code.
In particular, the data processing system may provide a data developer with a package development specification for the data processing system to enable the package to be compatible with the data processing system.
The front end code and the back end code contained in the program package are mainly used for realizing a data processing task editing interface, the front end code is used for realizing the interaction of a front end page, for example, after a data processing plug-in is dragged to a first editing area, the visualized object of the data processing plug-in is realized by depending on the front end code; the back-end code implementation correspondingly performs back-end response according to the interaction of the front-end page, for example, after a data processing plug-in the first editing area is selected, the configurable item displayed in the second editing area is implemented depending on the back-end code.
Of course, in order to ensure that the subsequent data processing plug-in can implement the corresponding function after being loaded, the program package should also include an execution end code.
In some embodiments, the data processing method further comprises: storing the task configuration information into a task configuration file in a markup language format, wherein each plug-in configuration information corresponds to a markup language node; loading the corresponding data processing plug-in according to the plug-in configuration information comprises: analyzing the task configuration file to obtain an analysis result and a plug-in dependency relationship corresponding to each data processing plug-in; and loading the corresponding data processing plug-in according to the analysis result and the plug-in dependency relationship.
Specifically, the task configuration information may be saved in a task configuration file in an xml format, and a corresponding data processing task may be formed in the data processing system, and the data processing task may be automatically executed according to the task configuration information or manually executed according to a request of a user.
The file in the markup language format such as Xml has a specific file structure, for example, comprises a plurality of nodes, so that the execution of subsequent data processing tasks is facilitated, and the parsing of the file can be conveniently realized by using a parser in the prior art. The execution sequence and the data flow of the data processing plug-ins can be determined according to the plug-in dependency relationship.
In some embodiments, loading the corresponding data processing plug-in according to the parsing result and the plug-in dependency relationship includes: loading the analysis result of the target data processing plug-in a distributed container to obtain a plurality of examples of the target data processing plug-in; executing the data processing task according to the loaded data processing plug-in includes: data processing tasks are performed distributively according to the plug-in dependencies using respective instances of respective data processing plug-ins.
For example, multiple instances of data processing plug-ins are implemented in the distributed container of Yarn. It can be seen that the configuration of the data processing task can be regarded as a data development process, i.e. a development state, and compared with the prior art, the configuration does not need a user to have a higher programming level, and convenient data development can be realized according to the existing plug-in; the execution of the data task can be regarded as a running state, and compared with the prior art, the convenient management and scheduling of resources can be realized by utilizing distributed containers. Fig. 4 shows a schematic diagram of a data processing principle according to an embodiment of the present application, and as shown in fig. 4, task configuration information input by a user corresponds to data development processing input in fig. 4, and includes multiple steps (steps), and specific task configuration information is written in a task configuration file in an xml format, and in a running state, according to the task configuration file, data processing plug-in instances (APPs) corresponding to the steps are formed in the Yarn contacts, and finally processed data is obtained.
It should be noted that all the execution subjects of the steps of the method provided above may be the same device, or different devices may be used as the execution subjects of the method. That is, the data processing server may be implemented by a single server device, or may be implemented by a cluster or the like.
The embodiment of the application also provides a data processing device, which is used for realizing the data processing method.
Specifically, fig. 5 shows a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. The apparatus may be embodied in a data processing server. As shown in fig. 5, the data processing apparatus 500 includes:
a configuration unit 510, configured to receive task configuration information of a data processing task, where the task configuration information includes plug-in configuration information of one or more data processing plug-ins.
And an executing unit 520, configured to load a corresponding data processing plug-in according to the task configuration information, and execute a data processing task according to the loaded data processing plug-in.
In some embodiments, the configuration unit 510 is configured to provide a data processing task editing interface, and receive task configuration information of the data processing task through the data processing task editing interface.
In some embodiments, the data processing task editing interface includes a plug-in list and a first editing area, and the configuration unit 510 is configured to, in response to a selection request for one or more data processing plug-ins in the plug-in list, display a visual object of the corresponding data processing plug-in the first editing area; and receiving the plug-in connection relation input through the first editing area, and generating corresponding plug-in configuration information according to the plug-in connection relation.
In some embodiments, the selection request of the data processing plug-in is triggered according to a selection operation of the data processing plug-in the plug-in list, and/or the selection request of the data processing plug-in is triggered according to an operation of dragging the data processing plug-in the plug-in list to the first editing area.
In some embodiments, the data processing task editing interface further includes a second editing area, and the configuration unit 510 is configured to, in response to a request for selecting a visual object of a target data processing plug-in the first editing area, present a configurable item of the target data processing plug-in the second editing area; plug-in configuration information for the target data processing plug-in is received via the configurable item.
In some embodiments, the task configuration information further includes task basic configuration information, the data processing task editing interface includes a third editing area, and the configuration unit 510 is configured to receive the task basic configuration information input through the third editing area.
In some embodiments, the apparatus further comprises a plug-in unit 530 for receiving and storing a package of data processing plug-ins, the package comprising front end code and back end code; and realizing the plug-in list and the first editing area according to the front end code, and realizing the configurable item of the data processing plug-in according to the back end code.
In some embodiments, the configuration unit 510 is configured to store the task configuration information into a task configuration file in a markup language format, where each piece of plug-in configuration information corresponds to a markup language node; the execution unit 520 is configured to analyze the task configuration file to obtain an analysis result and a plug-in dependency relationship corresponding to each data processing plug-in; and loading the corresponding data processing plug-in according to the analysis result and the plug-in dependency relationship.
In some embodiments, the execution unit 520 is configured to load the analysis result of the target data processing plug-in a distributed container to obtain multiple instances of the target data processing plug-in; executing the data processing task according to the loaded data processing plug-in includes: data processing tasks are performed distributively according to the plug-in dependencies using respective instances of respective data processing plug-ins.
It can be understood that, the data processing apparatus can implement the steps of the data processing method executed by the data processing server provided in the foregoing embodiment, and the related explanations about the data processing method are applicable to the data processing apparatus, and are not described herein again.
Fig. 6 is a schematic structural diagram of a data processing server according to an embodiment of the present application. Referring to fig. 6, at a hardware level, the data processing server includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the data processing server may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the non-volatile memory into the memory and then runs the computer program, thereby forming the data processing device on a logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
receiving task configuration information of a data processing task, wherein the task configuration information comprises plug-in configuration information of one or more data processing plug-ins; and loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in.
The method performed by the data processing apparatus according to the embodiment shown in fig. 1 of the present application may be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The data processing server may further execute the method executed by the data processing apparatus in fig. 1, and implement the functions of the data processing apparatus in the embodiment shown in fig. 5, which are not described herein again in this embodiment of the present application.
An embodiment of the present application further provides a computer-readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by a data processing server including a plurality of application programs, can cause the data processing server to execute the method performed by the data processing apparatus in the embodiment shown in fig. 1, and specifically is configured to perform:
receiving task configuration information of a data processing task, wherein the task configuration information comprises plug-in configuration information of one or more data processing plug-ins; and loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (12)
1. A data processing method performed by a data processing server, wherein the method comprises:
receiving task configuration information of a data processing task, wherein the task configuration information comprises plug-in configuration information of one or more data processing plug-ins;
and loading the corresponding data processing plug-in according to the task configuration information, and executing the data processing task according to the loaded data processing plug-in.
2. The method of claim 1, wherein the receiving task configuration information for a data processing task comprises:
and providing a data processing task editing interface, and receiving task configuration information of the data processing task through the data processing task editing interface.
3. The method of claim 2, wherein the data processing task editing interface comprises a list of plug-ins and a first editing area, and the receiving task configuration information for a data processing task via the data processing task editing interface comprises:
in response to a selection request of one or more data processing plug-ins in the plug-in list, displaying visual objects of the corresponding data processing plug-ins in the first editing area;
and receiving the plug-in connection relation input through the first editing area, and generating corresponding plug-in configuration information according to the plug-in connection relation.
4. The method according to claim 3, wherein the selection request of the data processing plug-in is triggered according to a selection operation of the data processing plug-ins in the plug-in list, and/or the selection request of the data processing plug-ins is triggered according to an operation of dragging the data processing plug-ins in the plug-in list to the first editing area.
5. The method of claim 3, wherein the data processing task editing interface further comprises a second editing area, and the receiving task configuration information for a data processing task via the data processing task editing interface further comprises:
responding to a selection request of a visual object of a target data processing plug-in the first editing area, and displaying a configurable item of the target data processing plug-in the second editing area;
plug-in configuration information for the target data processing plug-in is received via the configurable item.
6. The method of claim 5, wherein the method further comprises:
receiving and storing a program package of a data processing plug-in, wherein the program package comprises a front end code and a back end code;
and realizing the plug-in list and the first editing area according to the front end code, and realizing the configurable item of the data processing plug-in according to the back end code.
7. The method of claim 2, wherein the task configuration information further includes task base configuration information, the data processing task editing interface includes a third editing region, and receiving the task configuration information for the data processing task through the data processing task editing interface includes:
and receiving the task basic configuration information input through the third editing area.
8. The method of claim 1, wherein the method further comprises:
storing the task configuration information into a task configuration file in a markup language format, wherein each plug-in configuration information corresponds to a markup language node;
the loading of the corresponding data processing plug-in according to the plug-in configuration information includes:
analyzing the task configuration file to obtain an analysis result and a plug-in dependency relationship corresponding to each data processing plug-in;
and loading the corresponding data processing plug-in according to the analysis result and the plug-in dependency relationship.
9. The method of claim 8, wherein said loading a corresponding data processing plug-in according to the parsing result and the plug-in dependency comprises:
loading the analysis result of the target data processing plug-in a distributed container to obtain a plurality of examples of the target data processing plug-in;
the executing of the data processing task according to the loaded data processing plug-in includes: and executing the data processing task in a distributed manner according to the plug-in dependency relationship by utilizing each instance of each data processing plug-in.
10. A data processing apparatus applied to a data processing server, wherein the data processing apparatus comprises:
the configuration unit is used for receiving task configuration information of the data processing task, and the task configuration information comprises plug-in configuration information of one or more data processing plug-ins;
and the execution unit is used for loading the corresponding data processing plug-in according to the task configuration information and executing the data processing task according to the loaded data processing plug-in.
11. A data processing server, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the data processing method of any of claims 1 to 9.
12. A computer-readable storage medium storing one or more programs which, when executed by a data processing server comprising a plurality of application programs, cause the data processing server to perform the data processing method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011586477.4A CN114691112A (en) | 2020-12-29 | 2020-12-29 | Data processing method and device and data processing server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011586477.4A CN114691112A (en) | 2020-12-29 | 2020-12-29 | Data processing method and device and data processing server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114691112A true CN114691112A (en) | 2022-07-01 |
Family
ID=82129761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011586477.4A Pending CN114691112A (en) | 2020-12-29 | 2020-12-29 | Data processing method and device and data processing server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114691112A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116382813A (en) * | 2023-03-16 | 2023-07-04 | 成都考拉悠然科技有限公司 | Video real-time processing AI engine system for smart city management |
-
2020
- 2020-12-29 CN CN202011586477.4A patent/CN114691112A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116382813A (en) * | 2023-03-16 | 2023-07-04 | 成都考拉悠然科技有限公司 | Video real-time processing AI engine system for smart city management |
| CN116382813B (en) * | 2023-03-16 | 2024-04-19 | 成都考拉悠然科技有限公司 | Video real-time processing AI engine system for smart city management |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415832B (en) | Interface automation test method, device, equipment and storage medium | |
| CN109002362B (en) | Service method, device and system and electronic equipment | |
| CN107370786B (en) | General information management system based on micro-service architecture | |
| CN111124906B (en) | Tracking method, compiling method and device based on dynamic embedded points and electronic equipment | |
| CN110032599B (en) | Data structure reading and updating method and device, and electronic equipment | |
| CN110543483A (en) | Data auditing method and device and electronic equipment | |
| US10831565B2 (en) | Fault tolerant adapter system to consume database as a service | |
| CN110515795A (en) | A kind of monitoring method of big data component, device, electronic equipment | |
| CN110674105A (en) | Data backup method, system and server | |
| CN111553652B (en) | Service processing method and device | |
| CN115840687A (en) | Visual log link tracking method and system based on service level | |
| CN114691112A (en) | Data processing method and device and data processing server | |
| US20130127863A1 (en) | Determining an optimal sequence of status transitions for business objects | |
| CN112860412A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN110928941A (en) | Data fragment extraction method and device | |
| US9934019B1 (en) | Application function conversion to a service | |
| CN116483888A (en) | Program evaluation method and device, electronic equipment and computer readable storage medium | |
| CN112800030B (en) | Method and device for managing component data of flow and computer readable storage medium | |
| CN114677114A (en) | Approval process generation method and device based on graph dragging | |
| CN114253845A (en) | Automatic testing method and device for special-shaped architecture integration system | |
| CN106844056B (en) | Hadoop big data platform multi-tenant job management method and system | |
| CN114860566A (en) | Source code testing method and device, electronic equipment and storage medium | |
| CN111158873A (en) | System state machine implementation method and device, storage medium and processor | |
| CN112181407A (en) | Service implementation processing method, device, system, electronic equipment and storage medium | |
| CN112000366B (en) | System parameter configuration method and device based on code generator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |