CN108280023A - Task executing method, device and server - Google Patents
Task executing method, device and server Download PDFInfo
- Publication number
- CN108280023A CN108280023A CN201710003855.3A CN201710003855A CN108280023A CN 108280023 A CN108280023 A CN 108280023A CN 201710003855 A CN201710003855 A CN 201710003855A CN 108280023 A CN108280023 A CN 108280023A
- Authority
- CN
- China
- Prior art keywords
- task
- database
- file
- script file
- database script
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/362—Software debugging
- G06F11/3628—Software debugging of optimised code
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/73—Program documentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a kind of task executing method, device and servers, this method includes the corresponding task description file of reading task, the path of database public packet of the record for executing task in task description file, the entity for indicating task database script file path and corresponding first parameter of task, the first parameter be used for the variable in the script file of replacement data library；By database script file push to calculate node, there is the public packet of database at calculate node；The public packet of database is called according to the path of the public packet of database, the path of database script file is pressed by the public packet of database, it obtains database script file and using the variable in the first parameter replacement data library script file, and execution database script file, obtains the implementing result of task.It allows task to become simpler while substantially reducing code repetition according to the present invention, saves manpower.
Description
Technical field
The present invention relates to a kind of field of computer technology more particularly to task executing method, device and servers.
Background technology
Spark is a big data processing frame around speed, ease for use and complicated analysis structure：By at data
The lower Suffle of cost (data cleansing algorithm) mode during reason promotes MapReduce (a kind of programming model) to one
A higher level；Utilize the processing capacity of internal storage data storage and near real-time, other big data treatment technologies of performance ratio
It will soon many times.
Traditional data cleaning task generally requires to develop a large amount of code to support, past when there are many cleaning task
It can become too fat to move toward engineering packet can't bear, to bring various drawbacks.Such as：1. code redundancies are high, manpower waste is serious.Work as work
There can be significant component of repeat function code when journey is prodigious, between different developers, remerge code complexity
Degree, it will waste of manpower resource significantly.2. code maintenance cost is too high.If being found when engineering operation there is leakage in some task
Hole needs to adjust, and is bound to cut off entire engineering, the feelings that most of good tasks will be caused to wait for a problem task in this way
Condition causes the serious waste of environmental resource.3. source code flexible is poor.When needing to deliver, asked if certain several task code exists
Topic, can cause project that can not integrally deliver.4. inconvenience such as code debugging, operation, transfers.Engineering becomes caused by more than task
Greatly, the debugging of code, run time can be made to become longer, Code transformations can be made to become difficult.
Invention content
The purpose of the present invention is to provide a kind of task executing method, device and servers, to solve conventional clean task
At least one of code repeats, engineering is excessive, task management is inconvenient problem.
Technical solution is as follows used by the present invention solves above-mentioned technical problem：
A kind of task executing method provided, including：The corresponding task description file of reading task, the task description text
The path of database public packet of the record for executing the task in part, the entity for indicating the task database foot
The path of this document and corresponding first parameter of the task, first parameter is for replacing the database script text
The database script file push to calculate node is had the database public by variable in part at the calculate node
Packet；The public packet of database is called according to the path of the public packet of the database, by the public packet of the database by described
The path of database script file obtains the database script file and replaces the database foot using first parameter
Variable in this document, and the database script file is executed, obtain the implementing result of the task.
Optionally, method above-mentioned, the also path of logger task configuration file in the task description file, the task
Second parameter of threshold value of the configuration file record for replacing the database script file；The method further includes：By institute
The task configuration file of stating is pushed to the calculate node；By the public packet of the database by the acquisition task configuration file
Second parameter replace the threshold value in the database script file.
Optionally, method above-mentioned obtains the database script file and replaces the number using first parameter
According to the variable in the script file of library, specifically include：According to preset data interchange language, by the task description file generated phase
The data interchange file answered extracts first parameter to replace the database script file from the data interchange file
In variable.
Optionally, method above-mentioned further includes：Whether according to the priority of the task, the required by task data
It obtains, the task, which is added to record, to be had in the task list of the one or more tasks arranged in order；According to described
The sequence in the task list of being engaged in executes after reading the task in the task list and obtains the database foot
This document simultaneously replaces the variable in the database script file using first parameter, and public according to the database
The path of packet calls the public packet of the database to execute the database script file, obtains the implementing result of the task.
Optionally, method above-mentioned, the task description file include the information, defeated of the corresponding input table of the task
Go out the information of table；It calls the public packet of the database to execute the database script file, specifically includes：It is obtained from the input table
It takes the input data of the database script and executes the database script file, the output table is added in obtained result
In.
A kind of task execution device is provided, including：Read module, for reading the corresponding task description file of task,
The path of the public packet of database for executing the task is recorded in the task description file, for indicating the task
The path of the database script file of entity and corresponding first parameter of the task, first parameter is for replacing institute
State the variable in database script file；Pushing module is used for the database script file push to calculate node, described
There is the public packet of the database at calculate node；Execution module, for calling institute according to the path of the public packet of the database
The public packet of database is stated, the path of the database script file is pressed by the public packet of the database, obtains the database
Script file simultaneously replaces the variable in the database script file using first parameter, and executes the database foot
This document obtains the implementing result of the task.
Optionally, device above-mentioned, the also path of logger task configuration file in the task description file, the task
Second parameter of threshold value of the configuration file record for replacing the database script file；The pushing module will also be described
Task configuration file is pushed to the calculate node；The execution module is also matched by the public packet of the database by the task
The path for setting file obtains second parameter in the task configuration file to replace in the database script file
Threshold value.
Optionally, device above-mentioned, the execution module is according to preset data interchange language, by task description text
Part generates corresponding data interchange file, and first parameter is extracted from the data interchange file to replace the database
Variable in script file.
Optionally, device above-mentioned further includes：Task list module, according to the priority of the task, task institute
Need whether data have obtained, the task, which is added to record, the task list of the one or more tasks arranged in order
In；According to sequence of the task in the task list acquisition is executed after reading the task in the task list
The database script file simultaneously replaces the variable in the database script file using first parameter, and according to institute
The path for stating the public packet of database calls the public packet of the database to execute the database script file, obtains the task
Implementing result.
Optionally, device above-mentioned, the task description file include the information, defeated of the corresponding input table of the task
Go out the information of table；The execution module obtains the input data of the database script from the input table and executes the data
Obtained result is added in the output table library script file.
A kind of server is provided, including：According to task execution device above-mentioned.
According to above technical scheme, it is known that task executing method, device and server of the invention at least has following excellent
Point：
Compared with prior art, task is executed based on multiple files, and multiple file independent of one anotherization can be promoted effectively
Source code flexible, and fully decoupling make code maintenance become succinctly convenient；It is huge that database text separation solved into main program
Not the drawbacks of easy care；The use of public packet allows task to become simpler while substantially reducing code repetition, saves people
Power.
Description of the drawings
Fig. 1 is a kind of flow chart of task executing method of the embodiment of the present invention；
Fig. 2 is a kind of flow chart of task executing method of the embodiment of the present invention；
Fig. 3 is a kind of block diagram of task execution device of the embodiment of the present invention；
Fig. 4 is a kind of block diagram of task execution device of the embodiment of the present invention；
Fig. 5 is a kind of file structure figure of task executing method of the embodiment of the present invention；
Fig. 6 is a kind of schematic diagram of task executing method of the embodiment of the present invention；
Fig. 7 is a kind of work flow diagram of task executing method of the embodiment of the present invention；
Fig. 8 is a kind of work flow diagram of task executing method of the embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific implementation mode
In order to keep technical problems, technical solutions and advantages to be solved clearer, clear, tie below
Drawings and examples are closed, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only
To explain the present invention, it is not intended to limit the present invention.
Embodiment one
As shown in Figure 1, a kind of task executing method is provided in one embodiment of the present of invention, including：
Step S110 reads the corresponding task description file of task, and record is for executing task in task description file
The path of the public packet of database, the entity for indicating task database script file path and task corresponding
One parameter, the first parameter are used for the variable in the script file of replacement data library.In the present embodiment, task description file can be adopted
With xml formatted files, database script file can be sql formatted files, and being carried out for task includes but not limited to that data are clear
Task is washed, the public packet of the database can be public jar packets.
Database script file push to calculate node is had the public packet of database by step S120 at calculate node.
Step S130 calls the public packet of database, by the public packet of database by number according to the path of the public packet of database
According to the path of library script file, database script file is obtained and using the change in the first parameter replacement data library script file
Amount, and database script file is executed, obtain the implementing result of task.
According to the technical solution of the present embodiment, each data cleansing task can be write as independent algorithm, use spark-
Submit submits task；The cleaning task that data source is database table structure is extracted, general public jar packets text is made
Part, developer can describe file to describe input, the output of task by the corresponding XML of configuration task.The present embodiment
These implementation features can effectively control the problems such as conventional clean task code repeats, engineering is excessive, task management is inconvenient, together
When can also substantially reduce using data source as the development cost of the cleaning task of database table structure and maintenance cost.
Embodiment two
As shown in Fig. 2, a kind of task executing method is provided in one embodiment of the present of invention, including：
Step S210 reads the corresponding task description file of task, and record is for executing task in task description file
The path of the public packet of database, the entity for indicating task database script file path and task corresponding
One parameter, the first parameter are used for the variable in the script file of replacement data library, and also logger task configuration is literary in task description file
The path of part, second parameter of the file record for the threshold value of replacement data library script file, task description in task configuration
File includes the information of the corresponding input table of task, exports the information of table.
In the present embodiment, an ancestral task is made of three parts：Task .xml files, task .sql files, task
.conf file (if no threshold value can be omitted).In task .xml files, saving task type, (sql tasks/RDD appoints
Business), input (information/file such as database, table name, type, subregion where table), output (database where table, table name,
Information/the file such as type, subregion), the time (timed task) is executed, the jar packets path of task is executed, task .conf files, appoints
Business .sql file paths etc..Task entity is preserved in task .sql files, that is, carries the sql sentences of variable.Task .conf files
In save the constant informations such as task threshold value.
Task configuration file is pushed to calculate node by database script file push to calculate node by step S220,
There is the public packet of database at calculate node.Technical solution based on the present embodiment starts main journey, it can be achieved that a kind of main program
Information in task .xml files can be read the job metadata list of memory by sequence, main program, and by task .sql files,
Task .conf file push is to each spark calculate nodes.
Whether step S230 has obtained according to the priority of task, required by task data, by task be added to record have by
In the task list of tactic one or more task；According to sequence of the task in task list, from task list
After reading task, according to preset data interchange language, by the corresponding data interchange file of task description file generated, from data
The variable that the first parameter is come in the script file of replacement data library, and the path according to the public packet of database are extracted in swap file
It calls the public packet of database to execute database script file, obtains the implementing result of task.In the present embodiment, main program according to
Job metadata describes situation (data-driven/time driving), data reach situation, task priority situation etc., by task plus
Enter to task dispatch list.
In the present embodiment, main program submits task to spark clusters and in spark cluster meters using spark-submit
Task is executed on operator node.Main program monitors spark environmental resource service conditions, and when resource allows, main program is according to task
Required parameter in metadata generates task .json files, parameter in the public jar packets reading task .json files of sql and appoints
Threshold value in business .conf files falls variable replacement in task .sql files, and generating complete task sql sentences, (task is real
Body).Corresponding database linkage information in json files (spark/gbase/mysql/ other) is obtained according to input table type,
It is thus connected to correspondence database and executes Task entity sql.According to output type, task action result is preserved to corresponding number
According in library or file.
Step S240, by the public packet of database by the second parameter obtained in task configuration file come replacement data library foot
Threshold value in this document, and obtain the input data of database script from input table and execute database script file, it will
Obtained result is added in output table.In the present embodiment, the result for the task that executes is returned to main program by public jar packets, main
Program judges the final implementing result of task.
According to the technical solution of the present embodiment, method using the present invention, compared with prior art, task independent can be with
Source code flexible, and fully decoupling are effectively promoted, code maintenance is made to become succinctly convenient；Main program is detached with algorithm and is solved
The drawbacks of main program huge not easy care；The extraction of public jar packets allows data cleansing to appoint while substantially reducing code repetition
Business becomes simpler, saves manpower.
Embodiment three
As shown in figure 3, a kind of task execution device is provided in one embodiment of the present of invention, including：
Read module 310 reads the corresponding task description file of task, and record is for executing task in task description file
The path of the public packet of database, entity for indicating task database script file path and task it is corresponding
First parameter, the first parameter are used for the variable in the script file of replacement data library.In the present embodiment, task description file can be with
Using xml formatted files, database script file can be sql formatted files, and being carried out for task includes but not limited to data
Cleaning task, the public packet of the database can be public jar packets.
Database script file push to calculate node is had database public by pushing module 320 at calculate node
Packet.
Execution module 330 is called the public packet of database according to the path of the public packet of database, is pressed by the public packet of database
The path of database script file obtains database script file and using the change in the first parameter replacement data library script file
Amount, and database script file is executed, obtain the implementing result of task.
According to the technical solution of the present embodiment, each data cleansing task can be write as independent algorithm, use spark-
Submit submits task；The cleaning task that data source is database table structure is extracted, general public jar packets text is made
Part, developer can describe file to describe input, the output of task by the corresponding XML of configuration task.The present embodiment
These implementation features can effectively control the problems such as conventional clean task code repeats, engineering is excessive, task management is inconvenient, together
When can also substantially reduce using data source as the development cost of the cleaning task of database table structure and maintenance cost.
Example IV
As shown in figure 4, a kind of task execution device is provided in one embodiment of the present of invention, including：
Read module 410 reads the corresponding task description file of task, and record is for executing task in task description file
The path of the public packet of database, entity for indicating task database script file path and task it is corresponding
First parameter, the first parameter are used for the variable in the script file of replacement data library, and also logger task configures in task description file
The path of file, second parameter of the file record for the threshold value of replacement data library script file, task are retouched in task configuration
It includes the information of the corresponding input table of task, the information of output table to state file.
In the present embodiment, an ancestral task is made of three parts：Task .xml files, task .sql files, task
.conf file (if no threshold value can be omitted).In task .xml files, saving task type, (sql tasks/RDD appoints
Business), input (information/file such as database, table name, type, subregion where table), output (database where table, table name,
Information/the file such as type, subregion), the time (timed task) is executed, the jar packets path of task is executed, task .conf files, appoints
Business .sql file paths etc..Task entity is preserved in task .sql files, that is, carries the sql sentences of variable.Task .conf files
In save the constant informations such as task threshold value.
Task configuration file is pushed to calculating section by database script file push to calculate node by pushing module 420
Point has the public packet of database at calculate node.Technical solution based on the present embodiment starts master, it can be achieved that a kind of main program
Information in task .xml files can be read the job metadata list of memory by program, main program, and task .sql is literary
Part, task .conf file push to each spark calculate nodes.
Whether task list module 430 has obtained according to the priority of task, required by task data, task has been added to
Record has in the task list of the one or more tasks arranged in order；According to sequence of the task in task list, from appointing
It is engaged in list after reading task, according to preset data interchange language, by the corresponding data exchange text of task description file generated
Part, from the variable for extracting the first parameter in data interchange file and coming in the script file of replacement data library, and according to database public affairs
The path wrapped altogether calls the public packet of database to execute database script file, obtains the implementing result of task.In the present embodiment,
Main program describes situation (data-driven/time driving) according to job metadata, data reach situation, task priority situation
Deng task is added to task scheduling list.
In the present embodiment, main program submits task to spark clusters and in spark cluster meters using spark-submit
Task is executed on operator node.Main program monitors spark environmental resource service conditions, and when resource allows, main program is according to task
Required parameter in metadata generates task .json files, parameter in the public jar packets reading task .json files of sql and appoints
Threshold value in business .conf files falls variable replacement in task .sql files, and generating complete task sql sentences, (task is real
Body).Corresponding database linkage information in json files (spark/gbase/mysql/ other) is obtained according to input table type,
It is thus connected to correspondence database and executes Task entity sql.According to output type, task action result is preserved to corresponding number
According in library or file.
Execution module 440, by the public packet of database by the second parameter obtained in task configuration file come replacement data
Threshold value in the script file of library, and obtain the input data of database script from input table and execute database script text
Obtained result is added in output table part.In the present embodiment, the result for the task that executes is returned to main journey by public jar packets
Sequence, main program judge the final implementing result of task.
According to the technical solution of the present embodiment, using method of the present invention, compared with prior art, task independent
Source code flexible, and fully decoupling can be effectively promoted, code maintenance is made to become succinctly convenient；Main program is detached into solution with algorithm
The drawbacks of main program of having determined huge not easy care；The extraction of public jar packets makes data clear while substantially reducing code repetition
The task of washing becomes simpler, saves manpower.
Embodiment five
A kind of server is provided in one embodiment of the present of invention, which includes in embodiment three or example IV
The task execution device of record.It is held it will be appreciated by those skilled in the art that above-mentioned task can be run on current server
Luggage is set, i.e., the software and/or hardware based on current server can realize each function module of above-mentioned task execution device, and
The technique effect of the task execution device of above-described embodiment may be implemented in server through this embodiment.
A concrete application example according to an embodiment of the invention is as follows：
1) it writes ancestral task and describes file, i.e., three files shown in fig. 5.
Such as lte_subject_poorquality_cell_day tasks, first according to the good task lte_ of edit model
Subject_poorquality_cell_day.xml files.The xml document describes the following contents：Task names；Execute grain
Degree, to execute daily；Indicate information, there are sql sentences for expression task；The public jar of sql are packed in mouthful class and entrance function (can be with
It dispenses, can have default value in main program)；Path of the Task entity sql files in spark calculate nodes；Task configures
Path of the file in spark calculate nodes；It is the description in task to relying on table information below；Input the information description of table；
Export the description of table information；Check figure and memory needed for calculating；Delete stale information pattern.
Secondly editor's lte_subject_poorquality_cell_day.sql files.Variable replacement rule is to use two
The title that a ' $ ' is surrounded；Threshold value Substitution Rules are the titles being surrounded using two ' # ', be can specifically include：It is defeated
Go out table alias, the alias in corresponding task .xml files is substituted for output table actual name when the public jar packets of algorithm execute；
Table partition value is exported, the alias in corresponding task .xml files is substituted for actual value when the public jar packets of algorithm execute；Thresholding
Value corresponds to the value in task .conf files, and actual value is substituted for when the public jar packets of algorithm execute；Finally, according to above-mentioned sql
Whether file has threshold value to decide whether to write lte_subject_poorquality_cell_day.conf files, this example
There are threshold values, then extract threshold value and operated to after when individual configuration file conveniently to the modification of threshold value.
2) ancestral task file is imported into main program.Main program can scan the task under had an agreement catalogue on startup,
Individually increase, the task of modification can also be individually added in the form of patch with particular interface.Main program loads ancestral task text
Task .xml files can be read memory by the process of part such as Fig. 6, main program, job metadata list be added, for generating later
It is used when task json files；By on task .conf, task .sql file push to each calculate node, sql when for task execution
The use of public jar packets.
3) task execution is waited for.With the presence of data task be only can executing for task, have spark environment need calculating
Resource could really execute task, and after all preparatory conditions are all set up, task can be submitted.
4) execution of the public jar packets of sql.Such as Fig. 7, when task is submitted, the public jar packets of sql read what main program generated
Task json files obtain task sql file paths, task conf file paths, to read the sql sentences in sql files.
The sql sentences of each task can be a plurality of, and time conditions, input, output table in these sentences, threshold value are all according to frame
Frame constrains the variable of format, needs to be replaced in sql jar packets, and replacement values are related in task json files by reading
Parameter is completed after arrangement.Sql sentences after replacing completely are the sql sentences that can directly run.At this moment according to input table pair
Type of database is answered, is established the link, sql is executed.If output terminates with the table that input is under same database, task
(insert sentences)；It is corresponding to preserve if output is file；If output is other database tables, corresponding document is generated, is held
The storing process storage of the corresponding database of row, this exemplary overall flow by task list control as shown in figure 8, wherein appointed
Business all executes.
5) last implementing result is transmitted to main program by the public jar packets of sql.
Developer's real work be 1), 2) two step.
According to the technique and scheme of the present invention, a kind of storage medium is also provided, computer program is had recorded on the storage medium,
The task execution device of embodiment above-mentioned may be implemented in the computer program after operation.
Above by reference to the preferred embodiment of the present invention has been illustrated, not thereby limit to the interest field of the present invention.This
Field technology personnel do not depart from the scope of the present invention and essence, can there are many variant scheme realize the present invention, for example as one
The feature of a embodiment can be used for another embodiment and obtain another embodiment.All institutes within the technical concept with the present invention
The all any modification, equivalent and improvement of work, should all be within the interest field of the present invention.
Claims (11)
1. a kind of task executing method, which is characterized in that including：
The task description file of reading task, record is public for executing the database of the task in the task description file
The path of packet, entity for indicating the task database script file path and the task corresponding first
Parameter；
By database script file push to calculate node, there is the public packet of database at the calculate node；According to the number
The public packet of database is called according to the path of the public packet in library, the database script file is pressed by the public packet of the database
Path, obtain database script file simultaneously replace the variable in the database script file using first parameter, with
And the database script file is executed, obtain the implementing result of the task.
2. according to the method described in claim 1, it is characterized in that, going back logger task configuration file in the task description file
Path, the second parameter of threshold value of the task configuration file record for replacing the database script file；It is described
Method further includes：
The task configuration file is pushed to the calculate node；
The path that the task configuration file is pressed by the public packet of the database, obtains described in the task configuration file
Second parameter replaces the threshold value in the database script file.
3. according to the method described in claim 1, it is characterized in that, obtaining the database script file and using described first
Parameter replaces the variable in the database script file, specifically includes：
According to preset data interchange language, by the task description file generated data interchange file, from the data exchange
First parameter is extracted in file to replace the variable in the database script file.
4. according to the method described in claim 1, it is characterized in that, further including：
Whether obtained according to the priority of the task, the required by task data, by the task be added to record have by
In the task list of tactic one or more task；According to sequence of the task in the task list, from institute
It states after reading the task in task list, executes and obtain the database script file and replace institute using first parameter
The variable in database script file is stated, and calls the public packet of database to hold according to the path of the public packet of the database
The row database script file, obtains the implementing result of the task.
5. according to the method described in claim 1, it is characterized in that, the task description file includes that the task is corresponding
It inputs the information of table, export the information of table；
It is described that the public packet of the database is called to execute the database script file, it specifically includes：
The input data of the database script is obtained from the input table and executes the database script file, by what is obtained
As a result it is added in the output table.
6. a kind of task execution device, which is characterized in that including：
Read module records in the task description file for executing for reading task corresponding task description file
State the path of the public packet of database of task, entity for indicating the task database script file path and
Corresponding first parameter of the task, first parameter are used to replace the variable in the database script file；
Pushing module, for by the database script file push to calculate node, with the number at the calculate node
According to the public packet in library；
Execution module passes through the data for calling the public packet of database according to the path of the public packet of the database
The public packet in library presses the path of the database script file, obtains the database script file and is replaced using first parameter
The variable in the database script file is changed, and executes the database script file, obtains the execution knot of the task
Fruit.
7. device according to claim 6, which is characterized in that go back logger task configuration file in the task description file
Path, the second parameter of threshold value of the task configuration file record for replacing the database script file；It is described
The task configuration file is also pushed to the calculate node by pushing module；The execution module also passes through database public affairs
Packet obtains second parameter in the task configuration file to replace the number by the path of the task configuration file altogether
According to the threshold value in the script file of library.
8. device according to claim 6, which is characterized in that
The execution module is according to preset data interchange language, by the corresponding data exchange text of the task description file generated
Part replaces the variable in the database script file from first parameter is extracted in the data interchange file.
9. device according to claim 6, which is characterized in that further include：
Whether task list module has obtained according to the priority of the task, the required by task data, the task has been added
Being added to record has in the task list of the one or more tasks arranged in order；According to the task in the task list
Sequence execute after reading the task in the task list and obtain the database script file and using described the
One parameter replaces the variable in the database script file, and calls the number according to the path of the public packet of the database
The database script file is executed according to the public packet in library, obtains the implementing result of the task.
10. device according to claim 6, which is characterized in that the task description file includes that the task corresponds to
Input table information, export table information；The execution module obtains the input of the database script from the input table
Data simultaneously execute the database script file, and obtained result is added in the output table.
11. a kind of server, which is characterized in that including：
Task execution device according to any one of claim 6 to 10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710003855.3A CN108280023B (en) | 2017-01-04 | 2017-01-04 | Task execution method and device and server |
| PCT/CN2017/118957 WO2018126964A1 (en) | 2017-01-04 | 2017-12-27 | Task execution method and apparatus and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710003855.3A CN108280023B (en) | 2017-01-04 | 2017-01-04 | Task execution method and device and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108280023A true CN108280023A (en) | 2018-07-13 |
| CN108280023B CN108280023B (en) | 2022-11-01 |
Family
ID=62789189
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710003855.3A Active CN108280023B (en) | 2017-01-04 | 2017-01-04 | Task execution method and device and server |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108280023B (en) |
| WO (1) | WO2018126964A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027196A (en) * | 2019-12-03 | 2020-04-17 | 南方电网科学研究院有限责任公司 | Simulation analysis task processing method and device for power equipment and storage medium |
| CN112540858A (en) * | 2019-09-23 | 2021-03-23 | 成都华为技术有限公司 | Task processing method, server, client and system |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
| CN115061741A (en) * | 2022-05-31 | 2022-09-16 | 北京奇艺世纪科技有限公司 | Target plug-in operation method and device, electronic equipment and storage medium |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222122A (en) * | 2019-07-26 | 2019-09-10 | 深圳市元征科技股份有限公司 | A kind of method of data synchronization and relevant device of MongoDB |
| CN110704210B (en) * | 2019-09-20 | 2023-10-10 | 天翼电子商务有限公司 | Script task calling method, system, medium and device |
| CN110990669A (en) * | 2019-10-16 | 2020-04-10 | 广州丰石科技有限公司 | DPI (deep packet inspection) analysis method and system based on rule generation |
| CN113760489B (en) * | 2020-09-21 | 2024-05-17 | 北京沃东天骏信息技术有限公司 | Resource allocation method and device |
| CN113239005B (en) * | 2021-06-02 | 2022-12-02 | 上海许继电气有限公司 | I and IV area data synchronization method and device for power monitoring system |
| CN113721824B (en) * | 2021-08-10 | 2024-05-03 | 深圳市一博科技股份有限公司 | Method for setting library path by one key of CR5000 platform |
| CN115061785A (en) * | 2022-04-15 | 2022-09-16 | 支付宝(杭州)信息技术有限公司 | Information issuing method and device, storage medium and server |
| CN117609102B (en) * | 2024-01-23 | 2024-05-28 | 云筑信息科技(成都)有限公司 | Building industry Internet counting platform system testing method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090307186A1 (en) * | 2008-06-06 | 2009-12-10 | Hitachi, Ltd. | Method and Apparatus for Database Management and Program |
| CN103678098A (en) * | 2012-09-06 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | HADOOP program testing method and system |
| CN104317928A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database |
| CN105224348A (en) * | 2014-06-11 | 2016-01-06 | 中兴通讯股份有限公司 | A kind of installation method of MySQL database and device |
| CN105808776A (en) * | 2016-03-29 | 2016-07-27 | 中国建设银行股份有限公司 | Data management system and method of distributed database |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102880546B (en) * | 2012-09-03 | 2015-06-17 | 北大方正集团有限公司 | Software integration testing method and system based on extensible markup language (XML) database |
| US20150379022A1 (en) * | 2014-06-27 | 2015-12-31 | General Electric Company | Integrating Execution of Computing Analytics within a Mapreduce Processing Environment |
| CN105487943A (en) * | 2015-12-09 | 2016-04-13 | 浪潮电子信息产业股份有限公司 | Method for automatically copying files to each node of cluster server |
-
2017
- 2017-01-04 CN CN201710003855.3A patent/CN108280023B/en active Active
- 2017-12-27 WO PCT/CN2017/118957 patent/WO2018126964A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090307186A1 (en) * | 2008-06-06 | 2009-12-10 | Hitachi, Ltd. | Method and Apparatus for Database Management and Program |
| CN103678098A (en) * | 2012-09-06 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | HADOOP program testing method and system |
| CN105224348A (en) * | 2014-06-11 | 2016-01-06 | 中兴通讯股份有限公司 | A kind of installation method of MySQL database and device |
| CN104317928A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database |
| CN105808776A (en) * | 2016-03-29 | 2016-07-27 | 中国建设银行股份有限公司 | Data management system and method of distributed database |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112540858A (en) * | 2019-09-23 | 2021-03-23 | 成都华为技术有限公司 | Task processing method, server, client and system |
| CN112540858B (en) * | 2019-09-23 | 2023-10-27 | 华为云计算技术有限公司 | Task processing method, server, client and system |
| CN111027196A (en) * | 2019-12-03 | 2020-04-17 | 南方电网科学研究院有限责任公司 | Simulation analysis task processing method and device for power equipment and storage medium |
| CN111027196B (en) * | 2019-12-03 | 2023-04-28 | 南方电网科学研究院有限责任公司 | Simulation analysis task processing method and device for power equipment and storage medium |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
| CN114489995B (en) * | 2022-02-15 | 2022-09-30 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
| CN115061741A (en) * | 2022-05-31 | 2022-09-16 | 北京奇艺世纪科技有限公司 | Target plug-in operation method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108280023B (en) | 2022-11-01 |
| WO2018126964A1 (en) | 2018-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280023A (en) | Task executing method, device and server | |
| CN111045655B (en) | Page rendering method and device, rendering server and storage medium | |
| CN111241078B (en) | Data analysis system, data analysis method and device | |
| CN111339071B (en) | Method and device for processing multi-source heterogeneous data | |
| CN107958028A (en) | Method, apparatus, storage medium and the terminal of data acquisition | |
| CN111597484B (en) | UI dynamic generation method | |
| CN108171528B (en) | Attribution method and attribution system | |
| CN109840298B (en) | Multi-information-source acquisition method and system for large-scale network data | |
| CN103309904A (en) | Method and device for generating data warehouse ETL (Extraction, Transformation and Loading) codes | |
| CN109814992A (en) | Distributed dynamic dispatching method and system for the acquisition of large scale network data | |
| CN113986241B (en) | Configuration method and device of business rules based on knowledge graph | |
| CN116089126A (en) | Method, device and equipment for acquiring data in real time and readable storage medium | |
| CN113760464A (en) | Artificial intelligence model development platform based on deep learning | |
| CN103109293A (en) | User motion processing system and method | |
| CN112379884A (en) | Spark and parallel memory computing-based process engine implementation method and system | |
| CN111782201A (en) | Method and device for realizing linkage of service codes and layout topological graph | |
| CN116560626A (en) | Data processing method, system, equipment and storage medium based on custom rules | |
| CN106096159A (en) | Distributed system behavior simulation under a kind of cloud platform analyzes the implementation method of system | |
| CN109729130A (en) | Information analysis method, service server, storage medium and device | |
| CN108563665A (en) | A kind of data processing system and method based on big data technology | |
| CN117743364A (en) | Data feature processing method, system, device and nonvolatile storage medium | |
| CN112748980B (en) | Message pushing method, device, equipment and computer readable storage medium | |
| CN108733736A (en) | Asset creation methods, devices and systems | |
| KR100836218B1 (en) | The method and apparatus for automation to manage features of product line system | |
| CN118093059A (en) | Multi-mode unstructured data processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200617 Address after: 518057 Zhongxing building, A3-01, A3-02, Nanshan District hi tech Industrial Park, Shenzhen, Guangdong Applicant after: Shenzhen ZTE Technical Service Co.,Ltd. Address before: 518000 Zhongxing building, science and technology south road, Nanshan District hi tech Industrial Park, Guangdong, Shenzhen Applicant before: ZTE Corp. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |