CN113742040A - Method and device for quickly generating distributed batch processing tasks based on visual interface - Google Patents
Method and device for quickly generating distributed batch processing tasks based on visual interface Download PDFInfo
- Publication number
- CN113742040A CN113742040A CN202110906348.7A CN202110906348A CN113742040A CN 113742040 A CN113742040 A CN 113742040A CN 202110906348 A CN202110906348 A CN 202110906348A CN 113742040 A CN113742040 A CN 113742040A
- Authority
- CN
- China
- Prior art keywords
- configuration information
- batch processing
- task
- configuration
- visual interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 40
- 230000008520 organization Effects 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract description 8
- 238000004590 computer program Methods 0.000 description 11
- 238000007726 management method Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention relates to the technical field of data processing, and discloses a method and a device for quickly generating distributed batch processing tasks based on a visual interface. The method comprises the following steps: receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information; organizing the configuration information into configuration files and storing the configuration files into a storage medium; receiving a task starting request of a user, calling a configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch processing task. By implementing the embodiment of the invention, the configuration information can be filled in through the visual interface, the configuration file is automatically generated, the configuration file is analyzed through the execution engine and the batch processing operation is scheduled and executed, and the construction and the processing of the batch processing task are accelerated.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a device for quickly generating distributed batch processing tasks based on a visual interface.
Background
The construction of the existing batch processing task of data needs to be completed by writing codes, for example, the construction can be realized by Waterdrop, and the like. This method requires a specially specialized software technician to complete the compilation, and the compilation process is relatively complex, requires a large amount of labor cost to meet the requirements, and is inefficient.
Disclosure of Invention
Aiming at the defects, the embodiment of the invention discloses a method and a device for quickly generating a distributed batch processing task based on a visual interface.
The first aspect of the embodiments of the present invention discloses a method for quickly generating a distributed batch processing task based on a visual interface, which includes:
receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information;
organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch processing task.
As a preferred embodiment, in the first aspect of the embodiment of the present invention, the visual interface is a web page.
As a preferred embodiment, in the first aspect of the embodiment of the present invention, the spark configuration information includes a name of a spark-app, a scheduling platform, and a resource; the Input configuration information comprises information of a data source and configuration information during data reading; the transform configuration information comprises names of configuration SQL and temporary tables, and the SQL and the temporary tables are arranged into one group or multiple groups; the Output configuration information comprises data drop information and configuration information of a drop base.
As a preferred embodiment, in the first aspect of the embodiments of the present invention, receiving a task start request from a user, retrieving the configuration file, and sending the configuration file to a background execution engine, where the configuration file information is analyzed by the execution engine to generate a spark task, and scheduling and executing a batch processing task includes:
a management interface of the visual interface receives a task starting request generated by a user in the operation of the management interface;
based on the task request, the configuration file is called through the management interface, and the called configuration file is transmitted to a background execution engine;
and the execution engine analyzes the configuration information, generates a spark task, sends the spark task to the scheduling platform, schedules and executes a batch processing task based on the Input configuration information and the transform configuration information through the scheduling platform, and stores a batch processing result in configuration information corresponding to Output.
As a preferred embodiment, in the first aspect of the embodiment of the present invention, the execution engine is java code, and is started in the form of java-jar.
The second aspect of the embodiments of the present invention discloses a device for quickly generating distributed batch processing tasks based on a visual interface, including:
the receiving unit is used for receiving configuration information filled in by a user on a visual interface, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information;
the organization unit is used for organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
and the processing unit is used for receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into a background execution engine, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch processing task.
As a preferred embodiment, in the second aspect of the embodiment of the present invention, the visual interface is a web page.
As a preferred embodiment, in the second aspect of the embodiment of the present invention, the spark configuration information includes a name of the spark-app, a scheduling platform, and a resource; the Input configuration information comprises information of a data source and configuration information during data reading; the transform configuration information comprises names of configuration SQL and temporary tables, and the SQL and the temporary tables are arranged into one group or multiple groups; the Output configuration information comprises data drop information and configuration information of a drop base.
As a preferred embodiment, in the second aspect of the embodiments of the present invention, the processing unit includes:
the starting sub-unit is used for receiving a task starting request generated by the operation of a user on a management interface through the management interface of the visual interface;
the calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to a background execution engine;
and the scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in the configuration information corresponding to the Output.
In a second aspect of an embodiment of the invention, the execution engine is java code, and is started in the form of a script java-jar.
A third aspect of an embodiment of the present invention discloses an electronic device, including: a memory storing executable program code; a processor coupled with the memory; the processor calls the executable program code stored in the memory to execute the method for quickly generating the distributed batch processing task based on the visual interface disclosed by the first aspect of the embodiment of the invention.
A fourth aspect of the embodiments of the present invention discloses a computer-readable storage medium, which stores a computer program, where the computer program enables a computer to execute the method for quickly generating a distributed batch processing task based on a visual interface disclosed in the first aspect of the embodiments of the present invention.
A fifth aspect of the embodiments of the present invention discloses a computer program product, which, when running on a computer, enables the computer to execute the method for quickly generating a distributed batch processing task based on a visual interface disclosed in the first aspect of the embodiments of the present invention.
A sixth aspect of the present invention discloses an application publishing platform, where the application publishing platform is configured to publish a computer program product, and when the computer program product runs on a computer, the computer is enabled to execute the method for quickly generating a distributed batch processing task based on a visual interface disclosed in the first aspect of the present invention.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
the embodiment of the invention fills in the configuration information through the visual interface, is simple and convenient, can be quickly completed by common personnel under certain guidance, can automatically generate the configuration file through the configuration information, and can accelerate the construction and the processing of batch processing tasks by analyzing the configuration file and scheduling and executing batch processing operations through the execution engine when the tasks are started.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and for a person of ordinary skill in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic flowchart illustrating a method for rapidly generating distributed batch processing tasks based on a visual interface according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of an apparatus for rapidly generating a distributed batch processing task based on a visual interface according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
It should be noted that the terms "first", "second", "third", "fourth", and the like in the description and the claims of the present invention are used for distinguishing different objects, and are not used for describing a specific order. The terms "comprises," "comprising," and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a method and a device for quickly generating a distributed batch processing task based on a visual interface, wherein configuration information is filled in through the visual interface, the method and the device are simple and convenient, general personnel can quickly complete the process under certain guidance, the configuration information can automatically generate a configuration file, when the task is started, the configuration file is analyzed and the batch processing operation is scheduled to be executed through an execution engine, the construction and the processing of the batch processing task are accelerated, and the detailed description is carried out in combination with the attached drawings.
Example one
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a method for quickly generating a distributed batch processing task based on a visual interface according to an embodiment of the present invention. As shown in fig. 1, the method for quickly generating a distributed batch processing task based on a visual interface includes the following steps:
s110, receiving configuration information filled in by a user on a visual interface, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information.
The user fills in the configuration information in a visual interface such as a web page, the configuration information can be filled in by himself after training of professionals, the configuration information can be filled in without the need that the user has professional ability, the efficiency of batch processing task construction is improved, and the rapid batch processing task construction is achieved. Of course, in some other embodiments, prompts for filling in configuration information may be set on the web page through a floating window or the like, the user may also complete filling in the configuration information based on the prompts, or may select corresponding configuration information in the web page through a pull-down option, or may perform corresponding explanation on each piece of configuration information to be selected through the floating window or the like in the pull-down option.
The configuration information includes spark configuration information, Input configuration information, transform configuration information, and Output configuration information. The configuration information is consistent with the subsequent execution engine, so that the execution engine can analyze the configuration file organized by the configuration information to complete the scheduling execution batch processing task.
Specifically, the spark configuration information may generally include the name of spark-app, a scheduling platform, and resources, which are used to perform subsequent batch processing tasks; the Input configuration information comprises information of a data source and configuration information during data reading and is used for calling data used by batch processing tasks; the transform configuration information includes names of configuration SQL and temporary tables, the SQL and temporary tables are arranged into one or more groups and used for determining processing instructions of each task, for example, the SQL can be data increment instructions, calculation instructions and the like, the SQL and the temporary tables are associated, because the SQL and the temporary tables are batch processing tasks, the SQL and the temporary tables are generally in multiple groups, and the result of SQL execution can be mapped into one temporary table and removes one name for the temporary table, so that the SQL and the temporary tables are convenient to call; the Output configuration information includes data drop information and configuration information of a floor library, and is used for determining a database, a storage path and the like of data storage after the batch processing task is completed.
And S120, organizing the configuration information into a configuration file and storing the configuration file into a storage medium.
When the user completes the configuration information, the user can click a complete or next button, and the configuration information is automatically organized into a configuration file and stored in a corresponding storage medium such as a database, wherein the storage position can be in a directory specified by the user or a default directory. The process may be non-sensory to the user, or of course, after the organization is completed, a certain prompt may be given to the web page, for example, the configuration file is organized and stored in the directory.
The configuration information is automatically organized into the configuration file, which can be completed by adding a plug-in a web page, namely, a script is formed by writing codes and runs under the web page, and after a user fills in the configuration information and clicks, the script automatically organizes the configuration information to form the configuration file.
S130, receiving a task starting request of a user, calling the configuration file, transmitting the configuration file to an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch task.
When a user (the user may be the same person as the user who fills the configuration information in step S110, or may not be the same person) clicks a start or batch button in a management interface of a visual page, such as a web page, a task start request is triggered to be formed, and after receiving the task start request, the management interface calls the configuration file organized and formed in step S120 according to the storage directory and transmits the configuration file to a background execution engine.
The background execution engine can be written by java codes and started in a java-jar script mode. The configuration file of the execution engine needs to be consistent with the relevant configuration information of the configuration file organized and formed in step S120, so as to ensure that the information parsed by the execution engine can generate a spark task.
The execution engine analyzes the configuration file, automatically generates a spark task, then sends the spark task to the scheduling platform, schedules and executes a batch processing task based on the Input configuration information and the transform configuration information through the scheduling platform, and stores a batch processing result in a floor base corresponding to the Output.
For a user starting the task, the one-key batch processing operation is realized, and the quick completion of the batch processing task is achieved.
The configuration information can be quickly completed under certain guidance, the configuration file can be automatically generated by the configuration information, and when the task is started, the configuration file is analyzed and the batch processing operation is scheduled to be executed by the execution engine, so that the construction and the processing of the batch processing task are accelerated.
Example two
Referring to fig. 2, fig. 2 is a schematic structural diagram of another apparatus for rapidly generating a distributed batch processing task based on a visual interface according to an embodiment of the present disclosure. As shown in fig. 2, the apparatus for rapidly generating a distributed batch processing task based on a visualization interface may include:
a receiving unit 210, configured to receive configuration information filled in by a user on a visual interface, where the configuration information includes spark configuration information, Input configuration information, transform configuration information, and Output configuration information;
an organizing unit 220, configured to organize the configuration information into a configuration file, and store the configuration file in a storage medium;
the processing unit 230 is configured to receive a task start request of a user, invoke the configuration file, transmit the configuration file to an execution engine in a background, analyze information of the configuration file by the execution engine, generate a spark task, and schedule and execute a batch task.
Preferably, the visual interface is a web page.
Preferably, the spark configuration information includes a name of spark-app, a scheduling platform, and a resource; the Input configuration information comprises information of a data source and configuration information during data reading; the transform configuration information comprises names of configuration SQL and temporary tables, and the SQL and the temporary tables are arranged into one group or multiple groups; the Output configuration information comprises data drop information and configuration information of a drop base.
Preferably, the processing unit 230 includes:
the starting sub-unit is used for receiving a task starting request generated by the operation of a user on a management interface through the management interface of the visual interface;
the calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to a background execution engine;
and the scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in the configuration information corresponding to the Output.
EXAMPLE III
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure. As shown in fig. 3, the electronic device may include:
a memory 310 storing executable program code;
a processor 320 coupled to the memory 310;
the processor 320 calls the executable program code stored in the memory 310 to perform part or all of the steps of the method for quickly generating the distributed batch processing task based on the visual interface in the first embodiment.
The embodiment of the invention discloses a computer-readable storage medium which stores a computer program, wherein the computer program enables a computer to execute part or all of the steps in the method for quickly generating the distributed batch processing task based on the visual interface in the first embodiment.
The embodiment of the invention also discloses a computer program product, wherein when the computer program product runs on a computer, the computer is enabled to execute part or all of the steps in the method for rapidly generating the distributed batch processing task based on the visual interface in the first embodiment.
The embodiment of the invention also discloses an application publishing platform, wherein the application publishing platform is used for publishing the computer program product, and when the computer program product runs on a computer, the computer is enabled to execute part or all of the steps in the method for rapidly generating the distributed batch processing task based on the visual interface in the first embodiment.
In various embodiments of the present invention, it should be understood that the sequence numbers of the processes do not mean the execution sequence necessarily in order, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated units, if implemented as software functional units and sold or used as a stand-alone product, may be stored in a computer accessible memory. Based on such understanding, the technical solution of the present invention, which is a part of or contributes to the prior art in essence, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for causing a computer device (which may be a personal computer, a server, a network device, or the like, and may specifically be a processor in the computer device) to execute part or all of the steps of the method according to the embodiments of the present invention.
In the embodiments provided herein, it should be understood that "B corresponding to a" means that B is associated with a from which B can be determined. It should also be understood, however, that determining B from a does not mean determining B from a alone, but may also be determined from a and/or other information.
It will be understood by those of ordinary skill in the art that some or all of the steps of the methods of the embodiments may be implemented by instructions associated with a program, which may be stored in a computer-readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc-Read-Only Memory (CD-ROM), or other Memory, a CD-ROM, or other disk, or a combination thereof, A tape memory, or any other medium readable by a computer that can be used to carry or store data.
The method and the device for rapidly generating the distributed batch processing task based on the visual interface disclosed by the embodiment of the invention are introduced in detail, a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as limiting the present invention.
Claims (10)
1. A method for rapidly generating distributed batch processing tasks based on a visual interface is characterized by comprising the following steps:
receiving configuration information filled in a visual interface by a user, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information;
organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into an execution engine of a background, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch processing task.
2. The method for rapidly generating distributed batch processing tasks based on a visualization interface as recited in claim 1, wherein the visualization interface is a web page.
3. The method for rapidly generating distributed batch processing tasks based on the visual interface of claim 1, wherein the spark configuration information comprises the name of spark-app, scheduling platform and resources; the Input configuration information comprises information of a data source and configuration information during data reading; the transform configuration information comprises names of configuration SQL and temporary tables, and the SQL and the temporary tables are arranged into one group or multiple groups; the Output configuration information comprises data drop information and configuration information of a drop base.
4. The method for rapidly generating distributed batch processing tasks based on the visual interface of claim 3, wherein the steps of receiving a task starting request of a user, calling the configuration file, transmitting the configuration file to a background execution engine, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing batch processing tasks comprise:
a management interface of the visual interface receives a task starting request generated by a user in the operation of the management interface;
based on the task request, the configuration file is called through the management interface, and the called configuration file is transmitted to a background execution engine;
and the execution engine analyzes the configuration information, generates a spark task, sends the spark task to the scheduling platform, schedules and executes a batch processing task based on the Input configuration information and the transform configuration information through the scheduling platform, and stores a batch processing result in configuration information corresponding to Output.
5. The method for rapidly generating the distributed batch processing task based on the visual interface as claimed in any one of claims 1 to 4, wherein the execution engine is java code and is started in a java-jar form.
6. An apparatus for rapidly generating distributed batch processing tasks based on a visual interface, comprising:
the receiving unit is used for receiving configuration information filled in by a user on a visual interface, wherein the configuration information comprises spark configuration information, Input configuration information, transform configuration information and Output configuration information;
the organization unit is used for organizing the configuration information into a configuration file and storing the configuration file into a storage medium;
and the processing unit is used for receiving a task starting request of a user, calling the configuration file, transmitting the configuration file into a background execution engine, analyzing the configuration file information by the execution engine, generating a spark task, and scheduling and executing a batch processing task.
7. The apparatus for rapidly generating distributed batch processing tasks based on visual interface of claim 6, wherein the visual interface is a web page.
8. The apparatus for rapidly generating distributed batch processing tasks based on the visual interface of claim 6, wherein the spark configuration information comprises the name of spark-app, scheduling platform and resources; the Input configuration information comprises information of a data source and configuration information during data reading; the transform configuration information comprises names of configuration SQL and temporary tables, and the SQL and the temporary tables are arranged into one group or multiple groups; the Output configuration information comprises data drop information and configuration information of a drop base.
9. The apparatus for rapidly generating distributed batch processing tasks based on the visual interface of claim 8, wherein the processing unit comprises:
the starting sub-unit is used for receiving a task starting request generated by the operation of a user on a management interface through the management interface of the visual interface;
the calling subunit is used for calling the configuration file through the management interface based on the task request and transmitting the called configuration file to a background execution engine;
and the scheduling subunit is used for analyzing the configuration information through an execution engine, generating a spark task, sending the spark task to the scheduling platform, scheduling and executing a batch processing task through the scheduling platform based on the Input configuration information and the transform configuration information, and storing a batch processing result in the configuration information corresponding to the Output.
10. The apparatus for rapidly generating distributed batch processing tasks based on the visual interface as claimed in any one of claims 6-9, wherein the execution engine is java code, and is started by a form of java-jar.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110906348.7A CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110906348.7A CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113742040A true CN113742040A (en) | 2021-12-03 |
| CN113742040B CN113742040B (en) | 2024-04-19 |
Family
ID=78730604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110906348.7A Active CN113742040B (en) | 2021-08-09 | 2021-08-09 | Method and device for quickly generating distributed batch processing task based on visual interface |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113742040B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160261727A1 (en) * | 2015-03-06 | 2016-09-08 | Connie W. Yang | Systems and methods of streaming data |
| CN107608772A (en) * | 2017-08-23 | 2018-01-19 | 山东中创软件工程股份有限公司 | A kind of method and system of batch processing task configuration schedules |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| CN111475564A (en) * | 2020-04-13 | 2020-07-31 | 恒安嘉新（北京）科技股份公司 | Streaming data processing method, system, computer equipment and storage medium |
| CN111625269A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Web-based universal Spark task submission system and method |
| WO2021008570A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳前海微众银行股份有限公司 | Interface processing method and device based on compute engine |
| CN112732852A (en) * | 2020-12-31 | 2021-04-30 | 武汉大学 | Cross-platform space-time big data distributed processing method and software |
-
2021
- 2021-08-09 CN CN202110906348.7A patent/CN113742040B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160261727A1 (en) * | 2015-03-06 | 2016-09-08 | Connie W. Yang | Systems and methods of streaming data |
| CN107608772A (en) * | 2017-08-23 | 2018-01-19 | 山东中创软件工程股份有限公司 | A kind of method and system of batch processing task configuration schedules |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| WO2021008570A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳前海微众银行股份有限公司 | Interface processing method and device based on compute engine |
| CN111475564A (en) * | 2020-04-13 | 2020-07-31 | 恒安嘉新（北京）科技股份公司 | Streaming data processing method, system, computer equipment and storage medium |
| CN111625269A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Web-based universal Spark task submission system and method |
| CN112732852A (en) * | 2020-12-31 | 2021-04-30 | 武汉大学 | Cross-platform space-time big data distributed processing method and software |
Non-Patent Citations (1)
| Title |
|---|
| 卜尧, 吴斌, 陈玉峰等: "BDAP——一个基于Spark的数据挖掘工具平台", 中国科学技术大学学报, vol. 47, no. 4, 30 April 2017 (2017-04-30), pages 358 - 368 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113742040B (en) | 2024-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20120330598A1 (en) | Method, program, and system for generating test cases | |
| CN110532084B (en) | Platform task scheduling method, device, equipment and storage medium | |
| CN110955409B (en) | Method and device for creating resources on cloud platform | |
| CN112581018B (en) | Method, system, device and storage medium for managing process tasks | |
| CN110320378B (en) | Method, apparatus, device and computer readable storage medium for applying for a quality control test | |
| CN110990001A (en) | IVR (Interactive Voice response) process execution method and device | |
| CN112365157A (en) | Intelligent dispatching method, device, equipment and storage medium | |
| CN108415998B (en) | Application dependency relationship updating method, terminal, device and storage medium | |
| CN113760763A (en) | Software testing method, device, server and system | |
| CN111078383A (en) | Timed task management method, device, equipment and storage medium | |
| CN111243580B (en) | Voice control method, device and computer readable storage medium | |
| CN111046640A (en) | Dynamic generation method and device of certification document | |
| CN112487163B (en) | Execution method of automatic flow and method and device for acquiring interface data of execution method | |
| CN113742040B (en) | Method and device for quickly generating distributed batch processing task based on visual interface | |
| CN110851688A (en) | Page request method, device and equipment | |
| CN106951236B (en) | Plug-in development method and device | |
| CN106294395A (en) | The method and device that a kind of task processes | |
| CN111045943B (en) | Mock data generation method, test system and readable storage medium | |
| CN112181823A (en) | Test script execution method and device, server, electronic device and medium | |
| CN109377177B (en) | Flow processing method, device, equipment and computer readable storage medium | |
| CN112181803B (en) | Data processing method and device | |
| CN116974653B (en) | Class loading method and system | |
| CN111782777B (en) | Method and device for generating information | |
| CN110598106B (en) | Method and device for pushing resource information, storage medium and electronic equipment | |
| CN109544097B (en) | Moon knot information processing method based on new product development, electronic device and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |