CN113742040B - Method and device for quickly generating distributed batch processing task based on visual interface - Google Patents
Method and device for quickly generating distributed batch processing task based on visual interface Download PDFInfo
- Publication number
- CN113742040B CN113742040B CN202110906348.7A CN202110906348A CN113742040B CN 113742040 B CN113742040 B CN 113742040B CN 202110906348 A CN202110906348 A CN 202110906348A CN 113742040 B CN113742040 B CN 113742040B
- Authority
- CN
- China
- Prior art keywords
- configuration information
- task
- configuration
- visual interface
- batch processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 69
- 230000000007 visual effect Effects 0.000 title claims abstract description 59
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000009466 transformation Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 abstract description 9
- 238000007726 management method Methods 0.000 description 12
- 238000004590 computer program Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000008520 organization Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention relates to the technical field of data processing, and discloses a method and a device for quickly generating a distributed batch processing task based on a visual interface. The method comprises the following steps: receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information; organizing the configuration information into configuration files and storing the configuration files in a storage medium; receiving a task starting request of a user, calling a configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks. By implementing the embodiment of the invention, the configuration information can be filled in through the visual interface, the configuration file can be automatically generated, and the execution engine is used for analyzing the configuration file and scheduling the execution of batch processing operation so as to accelerate the construction and processing of batch processing tasks.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a device for quickly generating a distributed batch processing task based on a visual interface.
Background
The construction of the existing batch processing task of the data is completed by writing codes, for example, the construction can be realized through Waterdrop and the like, and because the configuration file of Waterdrop is a conf file and cannot be dynamically assigned, the configuration file needs to be rewritten when the configuration file is used each time, so that the configuration file is matched with the task to be batched. This approach requires a specially specialized software technician to accomplish, and the compiling process is relatively complex, requiring a lot of labor costs to achieve the requirements, and is inefficient.
Disclosure of Invention
Aiming at the defects, the embodiment of the invention discloses a method and a device for quickly generating a distributed batch processing task based on a visual interface, which fill configuration information through the visual interface, automatically generate a configuration file, analyze the configuration file through an execution engine and schedule execution batch processing operation, and accelerate the construction and processing of the batch processing task.
The first aspect of the embodiment of the invention discloses a method for rapidly generating a distributed batch processing task based on a visual interface, which comprises the following steps:
Receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information;
Organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
and receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks.
In a first aspect of the embodiment of the present invention, the visual interface is a web page.
As a preferred embodiment, in the first aspect of the embodiment of the present invention, the spark configuration information includes a name of a spark-app, a scheduling platform, and a resource; the Input configuration information comprises information of a data source and configuration information when reading data; the transformation configuration information comprises names of configuration SQL and temporary tables, wherein the SQL and the temporary tables are arranged into one or more groups; the Output configuration information comprises data falling information and configuration information of a falling library.
In a first aspect of the present invention, a task start request of a user is received, the configuration file is called, the configuration file is transferred into an execution engine in a background, the execution engine parses the configuration file information to generate a spark task, and a batch task is scheduled to be executed, including:
The management interface of the visual interface receives a task starting request generated by the operation of a user on the management interface;
Based on the task request, the configuration file is called through the management interface, and the called configuration file is transmitted to an execution engine of a background;
The execution engine analyzes the configuration information, generates a spark task, sends the spark task to the dispatching platform, dispatches and executes batch processing tasks through the dispatching platform based on the Input configuration information and the transform configuration information, and stores batch processing results in configuration information corresponding to Output.
In a first aspect of the embodiment of the present invention, the execution engine is java code, and is started in the form of a script java-jar.
The second aspect of the embodiment of the invention discloses a device for quickly generating a distributed batch processing task based on a visual interface, which comprises the following steps:
The receiving unit is used for receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information;
An organizing unit, configured to organize the configuration information into a configuration file, and store the configuration file in a storage medium;
the processing unit is used for receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks.
In a second aspect of the embodiment of the present invention, the visual interface is a web page.
In a second aspect of the embodiment of the present invention, the spark configuration information includes a name of a spark-app, a scheduling platform, and a resource; the Input configuration information comprises information of a data source and configuration information when reading data; the transformation configuration information comprises names of configuration SQL and temporary tables, wherein the SQL and the temporary tables are arranged into one or more groups; the Output configuration information comprises data falling information and configuration information of a falling library.
In a second aspect of the embodiment of the present invention, the processing unit includes:
the visual interface comprises a promoter unit, a visual interface and a display unit, wherein the promoter unit is used for receiving a task starting request generated by the operation of a user on the management interface through the management interface of the visual interface;
The calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to an execution engine of a background;
The scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in configuration information corresponding to Output.
In a second aspect of the embodiment of the present invention, the execution engine is java code, and is started in the form of a script java-jar.
A third aspect of an embodiment of the present invention discloses an electronic device, including: a memory storing executable program code; a processor coupled to the memory; the processor invokes the executable program code stored in the memory to perform the method for quickly generating the distributed batch task based on the visual interface disclosed in the first aspect of the embodiment of the present invention.
A fourth aspect of the embodiment of the present invention discloses a computer-readable storage medium storing a computer program, where the computer program causes a computer to execute a method for quickly generating a distributed batch task based on a visual interface disclosed in the first aspect of the embodiment of the present invention.
A fifth aspect of the embodiments of the present invention discloses a computer program product, which when run on a computer, causes the computer to execute a method for quickly generating a distributed batch task based on a visual interface disclosed in the first aspect of the embodiments of the present invention.
A sixth aspect of the embodiment of the present invention discloses an application publishing platform, where the application publishing platform is configured to publish a computer program product, where when the computer program product runs on a computer, the computer is caused to execute a method for quickly generating a distributed batch task based on a visual interface disclosed in the first aspect of the embodiment of the present invention.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
according to the embodiment of the invention, the configuration information is filled in through the visual interface, so that the method is simple and convenient, general personnel can also complete the method quickly under certain guidance, the configuration information can automatically generate the configuration file, and when the task is started, the execution engine analyzes the configuration file and dispatches to execute batch processing operation, so that the construction and processing of the batch processing task are accelerated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is a flow diagram of a method for quickly generating distributed batch processing tasks based on a visual interface according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an apparatus for quickly generating distributed batch processing tasks based on a visual interface according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by a person of ordinary skill in the art without making any inventive effort, are intended to be within the scope of the present invention, based on the embodiments of the present invention.
It should be noted that the terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present invention are used for distinguishing between different objects and not necessarily for describing a particular sequential or chronological order. The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a method and a device for quickly generating a distributed batch processing task based on a visual interface, which are simple and convenient to fill configuration information through the visual interface, can be quickly completed by general personnel under certain guidance, can automatically generate a configuration file through the configuration information, and can accelerate the construction and processing of the batch processing task by executing an engine to analyze the configuration file and schedule the execution of batch processing operation when the task is started.
Example 1
Referring to fig. 1, fig. 1 is a flow chart of a method for quickly generating a distributed batch task based on a visual interface according to an embodiment of the present invention. As shown in FIG. 1, the method for quickly generating the distributed batch processing task based on the visual interface comprises the following steps:
S110, receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information.
The user fills configuration information in a visual interface, such as a web page, and can fill the configuration information by himself after training of professionals, and the configuration information can be filled without professional ability of the user, so that the efficiency of batch task construction is improved, and rapid batch task construction is realized. Of course, in some other embodiments, the web page may also be provided with prompts for filling configuration information through floating windows, and the user may also complete filling configuration information based on the prompts, or may select corresponding configuration information in the web page through a drop-down option, where in the drop-down option, each configuration information to be selected may also be interpreted through the floating windows, and so on.
The configuration information includes spark configuration information, input configuration information, transform configuration information, and Output configuration information. The configuration information is consistent with the subsequent execution engine, so that the execution engine can analyze the configuration files formed by the configuration information organization to finish scheduling and executing batch processing tasks.
Specifically, spark configuration information may generally include a name of the spark-app, a scheduling platform, and resources for performing subsequent batch tasks; the Input configuration information comprises information of a data source and configuration information when reading data, and is used for calling data used by a batch processing task; the transformation configuration information comprises names of configuration SQL and temporary tables, the SQL and the temporary tables are arranged into one or more groups, processing instructions for determining each task, such as SQL can be a data increment instruction, a calculation instruction and the like, the SQL is associated with the temporary tables, the SQL and the temporary tables are generally in a plurality of groups because of batch processing tasks, and the result of SQL execution is mapped into one temporary table and gives a name to the temporary table so as to facilitate calling; the Output configuration information comprises data falling information and configuration information of a floor database, and is used for determining a database, a storage path and the like of data storage after batch processing tasks are completed.
S120, organizing the configuration information into a configuration file and storing the configuration file into a storage medium.
When the user completes the filling of the configuration information, the user can click the completion or next button, the configuration information is automatically organized into configuration files and stored in a corresponding storage medium, such as a database, wherein the storage positions can be in a user-specified catalog or a default catalog. The process may be insensitive to the user, or of course, after the organization is completed, a certain prompt may be given to the web page, for example, the configuration file is already organized and stored in the directory.
The configuration information is automatically organized into a configuration file, and the configuration file can be formed by adding a plug-in the web page, namely, a script is formed by writing codes and runs under the web page, and after the configuration information is filled in and clicked by a user, the script automatically organizes the configuration information to form the configuration file.
S130, receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks.
When the user (the user and the user who fills in the configuration information in step S110 may or may not be the same person) clicks the start or batch processing button in the management interface of the visual page, for example, the web page, a task start request is triggered, and after the management interface receives the task start request, the management interface retrieves the configuration file organized in step S120 according to the storage directory, and transmits the configuration file to the execution engine in the background.
The background execution engine can be written by adopting java codes and is started in the form of a script java-jar. The configuration file of the execution engine needs to be consistent with the relevant configuration information of the configuration file organized in step S120, so as to ensure that the spark task can be generated by the information analyzed by the execution engine.
The execution engine analyzes the configuration file, automatically generates a spark task, then sends the spark task to the dispatching platform, dispatches and executes batch processing tasks through the dispatching platform based on the Input configuration information and the transform configuration information, and stores batch processing results in a floor library corresponding to Output.
For a user for starting a task, the method realizes one-key batch processing operation and achieves the rapid completion of the batch processing task.
From the above, the configuration information is filled in through the visual interface, so that the method is simple and convenient, general personnel can also complete the method quickly under certain guidance, the configuration information can automatically generate the configuration file, and when the task is started, the execution engine analyzes the configuration file and dispatches to execute batch processing operation, so that the construction and processing of the batch processing task are accelerated.
Example two
Referring to fig. 2, fig. 2 is a schematic structural diagram of another apparatus for quickly generating a distributed batch task based on a visual interface according to an embodiment of the present invention. As shown in FIG. 2, the apparatus for rapidly generating distributed batch processing tasks based on a visual interface may include:
A receiving unit 210, configured to receive configuration information filled in the visual interface by a user, where the configuration information includes spark configuration information, input configuration information, transform configuration information, and Output configuration information;
an organizing unit 220, configured to organize the configuration information into a configuration file, and store the configuration file in a storage medium;
The processing unit 230 is configured to receive a task start request from a user, call the configuration file, transfer the configuration file into an execution engine in the background, parse the configuration file information by the execution engine, generate a spark task, and schedule and execute a batch task.
Preferably, the visual interface is a web page.
Preferably, the spark configuration information comprises a name of a spark-app, a scheduling platform and a resource; the Input configuration information comprises information of a data source and configuration information when reading data; the transformation configuration information comprises names of configuration SQL and temporary tables, wherein the SQL and the temporary tables are arranged into one or more groups; the Output configuration information comprises data falling information and configuration information of a falling library.
Preferably, the processing unit 230 includes:
the visual interface comprises a promoter unit, a visual interface and a display unit, wherein the promoter unit is used for receiving a task starting request generated by the operation of a user on the management interface through the management interface of the visual interface;
The calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to an execution engine of a background;
The scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in configuration information corresponding to Output.
Example III
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the invention. As shown in fig. 3, the electronic device may include:
a memory 310 in which executable program code is stored;
a processor 320 coupled to the memory 310;
Wherein the processor 320 invokes executable program code stored in the memory 310 to perform some or all of the steps in one method of quickly generating a distributed batch task based on a visual interface in accordance with the first embodiment.
The embodiment of the invention discloses a computer readable storage medium storing a computer program, wherein the computer program enables a computer to execute part or all of the steps in a method for quickly generating a distributed batch task based on a visual interface in the first embodiment.
The embodiment of the invention also discloses a computer program product, wherein when the computer program product runs on a computer, the computer is caused to execute part or all of the steps in the method for quickly generating the distributed batch processing task based on the visual interface.
The embodiment of the invention also discloses an application release platform, wherein the application release platform is used for releasing the computer program product, and when the computer program product runs on a computer, the computer is caused to execute part or all of the steps in the method for quickly generating the distributed batch processing task based on the visual interface in the first embodiment.
In various embodiments of the present invention, it should be understood that the size of the sequence numbers of the processes does not mean that the execution sequence of the processes is necessarily sequential, and the execution sequence of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, or a part contributing to the prior art or all or part of the technical solution, may be embodied in the form of a software product stored in a memory, comprising several requests for a computer device (which may be a personal computer, a server or a network device, etc., in particular may be a processor in a computer device) to execute some or all of the steps of the method according to the embodiments of the present invention.
In the embodiments provided herein, it should be understood that "B corresponding to a" means that B is associated with a, from which B can be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information.
Those of ordinary skill in the art will appreciate that some or all of the steps of the various methods of the described embodiments may be implemented by hardware associated with a program that may be stored in a computer-readable storage medium, including read-only memory (ROM), random-access memory (Random Access Memory, RAM), programmable read-only memory (Programmable Read-only memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-only memory, EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (Compact Disc Read-only memory, CD-ROM), or any other optical disk memory, magnetic disk memory, tape memory, or any other medium capable of being used to carry or store data that is readable by a computer.
The method and the device for quickly generating the distributed batch processing task based on the visual interface disclosed by the embodiment of the invention are described in detail, and specific examples are applied to the principle and the implementation mode of the invention, and the description of the above embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, the person skilled in the art will change the specific embodiments and application scope according to the idea of the present invention, and the present description should not be construed as limiting the present invention.
Claims (6)
1. A method for rapidly generating distributed batch processing tasks based on a visual interface, comprising:
Receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information; the spark configuration information comprises a name of a spark-app, a scheduling platform and resources; the Input configuration information comprises information of a data source and configuration information when reading data; the transformation configuration information comprises names of configuration SQL and temporary tables, wherein the SQL and the temporary tables are arranged into one or more groups; the Output configuration information comprises data falling information and configuration information of a falling library;
Organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
Receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine to generate a spark task, and scheduling and executing batch processing tasks, wherein the method specifically comprises the following steps of:
The management interface of the visual interface receives a task starting request generated by the operation of a user on the management interface;
Based on the task request, the configuration file is called through the management interface, and the called configuration file is transmitted to an execution engine of a background;
The execution engine analyzes the configuration information, generates a spark task, sends the spark task to the dispatching platform, dispatches and executes batch processing tasks through the dispatching platform based on the Input configuration information and the transform configuration information, and stores batch processing results in configuration information corresponding to Output.
2. The method for quickly generating distributed batch processing tasks based on a visual interface according to claim 1, wherein the visual interface is a web page.
3. The method for quickly generating a distributed batch task based on a visual interface according to any one of claims 1-2, wherein the execution engine is java code and is started in the form of a script java-jar.
4. An apparatus for rapidly generating a distributed batch processing task based on a visual interface, comprising:
The receiving unit is used for receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, input configuration information, transform configuration information and Output configuration information; the spark configuration information comprises a name of a spark-app, a scheduling platform and resources; the Input configuration information comprises information of a data source and configuration information when reading data; the transformation configuration information comprises names of configuration SQL and temporary tables, wherein the SQL and the temporary tables are arranged into one or more groups; the Output configuration information comprises data falling information and configuration information of a falling library;
An organizing unit, configured to organize the configuration information into a configuration file, and store the configuration file in a storage medium;
the processing unit is used for receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks, and specifically comprises the following steps:
the visual interface comprises a promoter unit, a visual interface and a display unit, wherein the promoter unit is used for receiving a task starting request generated by the operation of a user on the management interface through the management interface of the visual interface;
The calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to an execution engine of a background;
The scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in configuration information corresponding to Output.
5. The apparatus for rapidly generating distributed batch processing tasks based on a visual interface of claim 4, wherein the visual interface is a web page.
6. The apparatus for quickly generating a distributed batch task based on a visual interface according to any one of claims 4 to 5, wherein the execution engine is java code, and is started in the form of a script java-jar.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110906348.7A CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110906348.7A CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113742040A CN113742040A (en) | 2021-12-03 |
| CN113742040B true CN113742040B (en) | 2024-04-19 |
Family
ID=78730604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110906348.7A Active CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113742040B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608772A (en) * | 2017-08-23 | 2018-01-19 | 山东中创软件工程股份有限公司 | A kind of method and system of batch processing task configuration schedules |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| CN111475564A (en) * | 2020-04-13 | 2020-07-31 | 恒安嘉新（北京）科技股份公司 | Streaming data processing method, system, computer equipment and storage medium |
| CN111625269A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Web-based universal Spark task submission system and method |
| WO2021008570A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳前海微众银行股份有限公司 | Interface processing method and device based on compute engine |
| CN112732852A (en) * | 2020-12-31 | 2021-04-30 | 武汉大学 | Cross-platform space-time big data distributed processing method and software |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10027728B2 (en) * | 2015-03-06 | 2018-07-17 | Ebay Inc. | Systems and methods of streaming data |
-
2021
- 2021-08-09 CN CN202110906348.7A patent/CN113742040B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608772A (en) * | 2017-08-23 | 2018-01-19 | 山东中创软件工程股份有限公司 | A kind of method and system of batch processing task configuration schedules |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| WO2021008570A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳前海微众银行股份有限公司 | Interface processing method and device based on compute engine |
| CN111475564A (en) * | 2020-04-13 | 2020-07-31 | 恒安嘉新（北京）科技股份公司 | Streaming data processing method, system, computer equipment and storage medium |
| CN111625269A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Web-based universal Spark task submission system and method |
| CN112732852A (en) * | 2020-12-31 | 2021-04-30 | 武汉大学 | Cross-platform space-time big data distributed processing method and software |
Non-Patent Citations (1)
| Title |
|---|
| BDAP——一个基于Spark的数据挖掘工具平台;卜尧, 吴斌, 陈玉峰等;中国科学技术大学学报;20170430;第47卷(第4期);358-368 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113742040A (en) | 2021-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5574854A (en) | Method and system for simulating the execution of a computer program | |
| KR20070120095A (en) | Method and apparatus for providing process guidance | |
| CN108647141A (en) | Automatic test approach, device, computer-readable medium and electronic equipment | |
| JPH0612452A (en) | Group information access system | |
| CN106294094A (en) | The method of testing of game server, client, server and system | |
| CN110955409B (en) | Method and device for creating resources on cloud platform | |
| CN108959086B (en) | Program package testing deployment method, device, system, electronic equipment and storage medium | |
| CN110320378B (en) | Method, apparatus, device and computer readable storage medium for applying for a quality control test | |
| CN112365157A (en) | Intelligent dispatching method, device, equipment and storage medium | |
| CN113961332A (en) | Method and device for realizing workflow engine, electronic equipment and storage medium | |
| CN111736923A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN111367531A (en) | Code processing method and device | |
| CN110990001A (en) | IVR (Interactive Voice response) process execution method and device | |
| CN111078383A (en) | Timed task management method, device, equipment and storage medium | |
| CN113742040B (en) | Method and device for quickly generating distributed batch processing task based on visual interface | |
| EP3812884A1 (en) | Information presentation method and apparatus | |
| CN110851688A (en) | Page request method, device and equipment | |
| CN115016775A (en) | Interface simulation method and device, storage medium and electronic equipment | |
| US20180060787A1 (en) | Information processing method, information processing device and client terminal | |
| CN110674024A (en) | Electronic equipment integration test system and method thereof | |
| CN113468074A (en) | Application program version comparison monitoring method, device, medium and electronic equipment | |
| CN112181823A (en) | Test script execution method and device, server, electronic device and medium | |
| US20200301817A1 (en) | Automatic navigation from log statement to code | |
| CN110688430A (en) | Method and device for obtaining data bypass and electronic equipment | |
| US8321844B2 (en) | Providing registration of a communication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |