CN110457023A - Task creation method, apparatus, storage medium and electronic equipment - Google Patents
Task creation method, apparatus, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN110457023A CN110457023A CN201910667753.0A CN201910667753A CN110457023A CN 110457023 A CN110457023 A CN 110457023A CN 201910667753 A CN201910667753 A CN 201910667753A CN 110457023 A CN110457023 A CN 110457023A
- Authority
- CN
- China
- Prior art keywords
- data
- data processing
- processing task
- configuration item
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000012545 processing Methods 0.000 claims abstract description 185
- 238000013468 resource allocation Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012216 screening Methods 0.000 claims description 8
- 238000012546 transfer Methods 0.000 claims description 5
- 238000007726 management method Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 13
- 238000005516 engineering process Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 10
- 241001269238 Data Species 0.000 description 6
- 238000004891 communication Methods 0.000 description 3
- 241000208340 Araliaceae Species 0.000 description 2
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 2
- 235000003140 Panax quinquefolius Nutrition 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 2
- 235000008434 ginseng Nutrition 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 235000020983 fruit intake Nutrition 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This disclosure relates to a kind of task creation method, apparatus, storage medium and electronic equipment, to provide a kind of method of new creation data processing task, to more efficiently create data processing task.This method comprises: obtaining the configuration item of user's input, the configuration item includes the data command for creating data processing task；It is compiled by the data command that field language-specific DSL engine includes to the configuration item；If the compiling result to the data command is correct, the data processing task is created.
Description
Technical field
This disclosure relates to field of computer technology, and in particular, to a kind of task creation method, apparatus, storage medium and
Electronic equipment.
Background technique
With the arriving of big data era, web log file, user behavior data, social media, sensor etc. are beyond the clouds
Produce magnanimity unstructured data.Compared with traditional data, the unstructured data of magnanimity big, value density with data volume
Low, the features such as data correlation is more complicated.Therefore, how data processing task is created, to adapt to magnanimity unstructured data
It generates, is to need the problem of paying close attention to.
In the related technology, the program of creation data processing task is usually targetedly write, such mode requires to have
The software engineer of standby certain technical capability and experience writes program, and technical threshold is high, and the program write is mostly primary
Property use, without promote multiplexing ability.In addition, the relevant technologies can also be configured in independent console to create number
It according to processing task, is programmed compared to profession, such mode is reduced using threshold, professional however, there remains learning
The method of configuration.
Summary of the invention
Purpose of this disclosure is to provide a kind of task creation method, apparatus, storage medium and electronic equipments, to provide one kind
The method of new creation data processing task.
To achieve the goals above, in a first aspect, the disclosure provides a kind of task creation method, which comprises
The configuration item of user's input is obtained, the configuration item includes the data command for creating data processing task；
It is compiled by the data command that field language-specific DSL engine includes to the configuration item；
If the compiling result to the data command is correct, the data processing task is created.
It is optionally, described that the data command is compiled by field language-specific DSL engine, comprising:
The attribute information of pre-stored data relevant to the data command is obtained, the attribute information includes described
The data source information and authority information of data；
According to the data source information, data relevant to the data command are obtained from corresponding data source；
According to the authority information, the data command is executed to the data got；
If described correct to the compiling result of the data command, the data processing task is created, comprising:
If the data command runs succeeded, the data processing task is created.
Optionally, the method also includes:
The configuration item that the user is inputted by graphic user interface GUI is obtained, the configuration item includes data to be stored
Property parameters；
According to the property parameters that the configuration item includes, the attribute information of the data to be stored is generated；
According to preset data form, the attribute information of the data to be stored and the data to be stored is deposited
Storage.
Optionally, the configuration item further includes the resource allocation information and macros for creating data processing task, institute
State the creation data processing task, comprising:
The resource allocation information and macros for including according to the configuration item, create the data processing task.
Optionally, the configuration item further includes the scheduling parameter for being used to indicate the data processing task executive mode, institute
State method further include:
According to the scheduling parameter that the configuration item includes, scheduling information is generated；
According to the scheduling information, the executive mode of the data processing task is determined；
According to the executive mode, the data processing task is executed.
Optionally, the method also includes:
Instruction is checked by the log that graphic user interface GUI is inputted in response to user, obtains and appoints with the data processing
It is engaged in corresponding task daily record；
According to default screening rule, filtered out from task daily record relevant to the implementing result of the data processing task
Result data, and the result data is returned to the GUI.
Optionally, after executing the data processing task, the method also includes:
Obtain the application programming interfaces API that the user is inputted by graphic user interface GUI, wherein the API includes
The API and distributed file system of the callback interface of Relational DBMS MySQL, hypertext transfer protocol HTTP
At least one of API of Hadoop；
The implementing result of the data processing task is sent to destination application by the API.
Second aspect, the disclosure also provide a kind of task creation device, and described device includes:
Module is obtained, for obtaining the configuration item of user's input, the configuration item includes for creating data processing task
Data command；
Collector, the data command for including to the configuration item by field language-specific DSL engine are compiled
It translates；
Creation module, for creating the data processing task when the compiling result to the data command is correct.
The third aspect, the disclosure also provide a kind of computer readable storage medium, are stored thereon with computer program, special
The step of sign is, any one of first aspect the method is realized when which is executed by processor.
Fourth aspect, the disclosure also provide a kind of electronic equipment, comprising:
Memory is stored thereon with computer program；
Processor, for executing the computer program in the memory, to realize any one of first aspect institute
The step of stating method.
By the above-mentioned means, operator can input configuration item, which includes for creating data processing task
Data command, therefore can be compiled by the data command that DSL engine includes to the configuration item, if compiling is correct,
Data processing task can be automatically created.That is, user need to only set configuration item according to the scheme of the disclosure
Data processing task is automatically created according to the configuration item of user setting, simplifies the process of creation data processing task, so as to
To improve the efficiency of creation data processing task.Also, for operator, match without learning profession programming knowledge and profession
Method is set, learning cost is reduced.
Other feature and advantage of the disclosure will the following detailed description will be given in the detailed implementation section.
Detailed description of the invention
Attached drawing is and to constitute part of specification for providing further understanding of the disclosure, with following tool
Body embodiment is used to explain the disclosure together, but does not constitute the limitation to the disclosure.In the accompanying drawings:
Fig. 1 is a kind of flow chart of task creation method shown according to one exemplary embodiment of the disclosure；
Fig. 2 is that user can increase configuration item in a kind of task creation method shown according to one exemplary embodiment of the disclosure
Schematic diagram；
Fig. 3 be include matching for property parameters in a kind of task creation method shown according to one exemplary embodiment of the disclosure
Set a schematic diagram；
It includes that resource distribution is believed that Fig. 4 .1, which is in a kind of task creation method shown according to one exemplary embodiment of the disclosure,
The configuration item schematic diagram of breath；
Fig. 4 .2 be include matching for macros in a kind of task creation method shown according to one exemplary embodiment of the disclosure
Set a schematic diagram；
Fig. 5 be include matching for scheduling parameter in a kind of task creation method shown according to one exemplary embodiment of the disclosure
Set a schematic diagram；
Fig. 6 is the schematic diagram of scheduling parameter set-up mode in the related technology；
Fig. 7 is in a kind of task creation method shown according to one exemplary embodiment of the disclosure include API configuration item
Schematic diagram；
Fig. 8 is task daily record schematic diagram in the related technology；
Fig. 9 is a kind of flow chart of task creation method shown according to disclosure another exemplary embodiment；
Figure 10 is a kind of implementation block diagram of task creation method shown according to one exemplary embodiment of the disclosure；
Figure 11 is a kind of block diagram of task creation device shown according to one exemplary embodiment of the disclosure；
Figure 12 is the block diagram according to a kind of electronic equipment shown in one exemplary embodiment of the disclosure.
Specific embodiment
It is described in detail below in conjunction with specific embodiment of the attached drawing to the disclosure.It should be understood that this place is retouched
The specific embodiment stated is only used for describing and explaining the disclosure, is not limited to the disclosure.
With the arriving of big data era, web log file, user behavior data, social media, sensor etc. are beyond the clouds
Produce magnanimity unstructured data.Compared with traditional data, the unstructured data of magnanimity big, value density with data volume
Low, the features such as data correlation is more complicated.Therefore, how data processing task is created, to adapt to magnanimity unstructured data
It generates, is to need the problem of paying close attention to.
In the related technology, the program of creation data processing task is usually targetedly write, such mode requires to have
The software engineer of standby certain technical capability and experience writes program, and technical threshold is high, and the program write is mostly primary
Property use, without promote multiplexing ability.In addition, the relevant technologies can also be configured in independent console to create number
It according to processing task, is programmed compared to profession, such mode is reduced using threshold, professional however, there remains learning
The method of configuration.
Also, above two mode is first to create data processing task, then holds data processing task submission
Row, just can determine that whether the data processing task can correctly execute.Once data processing task executes mistake, it is difficult to finder
Mistake occurs from the establishing stage of data processing task, also occurs from the execution stage of data processing task.
Therefore, in order to solve the problems in the relevant technologies, more efficiently creation data processing task, the embodiment of the present disclosure are mentioned
A kind of task creation method, apparatus, storage medium and electronic equipment out.
Fig. 1 is a kind of flow chart of task creation method shown according to one exemplary embodiment of the disclosure, referring to Fig.1,
The task creation method includes:
Step S101 obtains the configuration item of user's input, which includes the data for creating data processing task
Order.
Illustratively, the configuration item that available user is inputted by graphic user interface GUI.Wherein, GUI refers to using figure
The computer operation user interface that shape mode is shown may include that text box, drop-down menu etc. are a variety of for the pre- of user's operation
The configuration item first defined.The configuration item can be used for characterizing the logic rules of creation data processing task, can be pre-defined
Alright, it is also possible to user and the operation such as is added, reduces, changing on the basis of defined configuration item according to demand to obtain
's.
For example, user can be by according to demand, voluntarily determining and needing increased configuration item, with preferably full referring to Fig. 2
Task creation demand under sufficient different scenes.Wherein, one configuration item of every increase, can choose type (i.e. Fig. 2 of the configuration item
Shown in field type), and fill in the English name (field English name i.e. shown in Fig. 2) of the configuration item.In addition, in order to make
User clearly inserts the actual parameter in the configuration item when filling in, can be as shown in Fig. 2, filling in the annotation letter of the configuration item
Breath, the content for prompting user to need to insert to the configuration item.
Step S102 is compiled the data command that configuration item includes by field language-specific DSL engine.
Step S103 creates data processing task if the compiling result to data command is correct.
Wherein, DSL refers to the computer language for being absorbed in some application domain, such as html language, sql like language
Deng.In the related technology, according to preset rule, the DSL of input can be parsed by program code by DSL engine, it is real
Now corresponding function.In the embodiments of the present disclosure, the data command of input can be any DSL, correspondingly, be drawn by DSL
The data command can be compiled by holding up, if the compiling result to the data command is correct, can pass through DSL engine pair
The data command is parsed, and to obtain the program code for creating data processing task, realizes the wound of data processing task
It builds.Illustratively, data processing task can be the task that data be inserted into, inquired, modified etc. with different data operation, this public affairs
It opens embodiment and this is not construed as limiting.
Through the above technical solutions, user need to only set configuration item, can be created automatically according to the configuration item of user setting
Data processing task is built, the process of creation data processing task is simplified, so as to improve the effect of creation data processing task
Rate.Also, for operator, without learning profession programming knowledge and speciality method, learning cost is reduced.
It, can be to creation data processing before creating data processing task also, according to the scheme of the embodiment of the present disclosure
The data command of task compiles, if later data processing task execution error, due to being compiled just to data command
The data processing task created again after really, it can be said that the establishing stage of bright data processing task does not go wrong, thus
It can quickly determine that the execution stage of data processing task there is a problem.In the case where alternatively possible, if to creation
The data command compiling of data processing task is incorrect, then can return to error message, configure for example, user is inputted by GUI
Information, then error message also can accordingly be shown on GUI, so as to quickly position mistake by error message.
In a kind of possible mode, the data command is compiled by field language-specific DSL engine, it can be with
It is the attribute information for first obtaining pre-stored data relevant to data command, which may include the data of data
Source information and authority information obtain data relevant to data command, then root from corresponding data source then according to data source information
According to authority information, data command is executed to the data got.Correspondingly, if data command runs succeeded, data are created
Processing task.
It in other words, can be according to the data source information and authority information of data, to data command in the embodiment of the present disclosure
It is compiled, to determine whether to create data processing task.Wherein, data source information is used to indicate number relevant to data command
According to data source, which may include the different data source such as Hadoop distributed file system, database.Therefore, In
After obtaining data source information, data relevant to data command can be obtained from corresponding data source according to the data source information.
For example, being stored with 100 datas of student number 1-100 in database, data command is sql like language, specifically, the number
According to order for deleting 5 datas that student number in the database is 1-5, then after obtaining data source information, can first from
5 datas of database acquisition student number 1-5.
Authority information can serve to indicate that the operating right of user couple data relevant to data command, in other words, lead to
Crossing the authority information can determine whether the user of input data order can execute the data to the relevant data of the data command
Order.For example, the authority information of 5 datas of student number 1-5 is " owner is visible and can operate " in the example above, that
After 5 datas for getting student number 1-5 from database, delete operation can be carried out to 5 data.
If data command runs succeeded, illustrate that the compiling to data command is correct, so as to create data processing
Task.For example, if successful execution deletes the delete operation of 5 data, can be created for deleting this in the example above
The data processing task of 5 datas., whereas if data manipulation executes unsuccessful, then illustrate to the compiling of data command not just
Really, so as to export corresponding error message, so that user positions mistake.
In a kind of possible mode, the attribute information of data, such as data source information and authority information, it can be and inciting somebody to action
When data are stored in data source for the first time, stored by the configuration item that user inputs.Specifically, available user passes through
The configuration item of GUI input, which includes the property parameters of data to be stored, the category that then can include according to the configuration item
Property parameter, the attribute information of data to be stored is generated, finally according to preset data form, by data to be stored and should be wait store
The attribute information of data is stored.
For example, can show GUI shown in Fig. 3 to user based on presetting under cluster scene.To which user can
To be used to indicate the parameter of data to be stored attribute by GUI input.For example, user can be with input data referring to Fig. 3
The table name of Hive table, then can be stored that data into corresponding Hive table when storing the data.Correspondingly, rear
During continuous creation data task, the table name of the Hive table can be used as the data source information of the data.Alternatively, user can be with
Privacy class is inputted, user is arranged for the rights parameters of the data.Correspondingly, in the process of subsequent creation data task
In, which can be used as the authority information of the data.
It should be understood that the property parameters of data are not limited only to data source information described above in the embodiment of the present disclosure
And authority information can also include a variety of use such as data uniform resource identifier URI, data service description for example, referring to Fig. 3
In the parameter of description data attribute, the embodiment of the present disclosure is not construed as limiting the particular content of property parameters.
It, can be according to the configuration item packet after user is used for the configuration item of characterize data property parameters by GUI input
The property parameters included generate the attribute information of data.For example, the privacy class of user's input is " C (non-sensitive number referring to Fig. 3
According to owner is visible) ", that is, the property parameters inputted are " C (nonsensitive data, owner are visible) ", can according to the property parameters
To generate corresponding attribute information, for example the attribute information generated can be " C ".As it can be seen that compared to property parameters, the category of generation
Property information it is more succinct, be more advantageous to computer-internal information storage.
It further, can be according to preset data form, by the attribute of data and the data after generating attribute information
Information is stored.Illustratively, preset data form can be Thrift format, JSON format etc., the embodiment of the present disclosure pair
This is not construed as limiting.For example, referring to Fig. 3, user can the combobox at " serializing mode " rear in the gui according to different needs
The different data format of middle selection comes storing data and the attribute information of the data.
By the above-mentioned means, the property parameters that can be inputted according to user, generate attribute information, and by data and the data
Attribute information unified storage is carried out according to preset data structure, convenient for the management to data, and when creating data task,
Data and the corresponding attribute information of data can be quickly found, the compilation process to data command is quickly carried out, to mention
The efficiency of height creation data processing task.
In a kind of possible mode, the configuration item of user's input can also include the money for creating data processing task
Source configuration information and macros, correspondingly, creation data processing task can be the resource allocation information for including according to configuration item
And macros, create data processing task.
Wherein, resource allocation information may include being used to indicate the memory source letter distributed for creation data processing task
Breath for example, resource allocation information can be the map memory size for being used to indicate creation data processing task, or is also possible to
It is used to indicate the reduce memory size, etc. of creation data processing task, the embodiment of the present disclosure is not construed as limiting this.Macros
It may include the temporal information for being used to indicate data processing task, for example, data processing task is to delete of that month all data,
It so can first determine for characterizing of that month macros, then according to the macros, create for deleting of that month all numbers
According to data processing task.
For example, can show the GUI as shown in Fig. 4 .1 and Fig. 4 .2 to user, which includes under resource allocation information
Draw the drop-down list of list and macros.In this case, user can be by the drop-down list of resource allocation information
Corresponding resource allocation information is selected, corresponding macros is selected in the drop-down list of macros, it then can be according to user
The resource allocation information and macros of selection create data processing task.For example, user wants to delete of that month all data,
It so can choose the macros in characterization " this month ", the data command of all data then deleted by text box input, and
Select resource allocation information for " map memory 100MB ", then creation deletion can work as in the map object for occupying 100MB memory
The data processing task of month all data.
By the above-mentioned means, user only need to set the configuration item including resource allocation information and macros in gui interface,
Data processing task can be automatically created according to the configuration item of user setting, simplify the process of creation data processing task, from
And the efficiency of creation data processing task can be improved.
In a kind of possible mode, the configuration item of user's input can also include being used to indicate data processing task execution
The scheduling parameter of mode, then in the embodiments of the present disclosure, the scheduling parameter that can also include according to configuration item generates scheduling letter
Breath, then according to scheduling information, determines the executive mode of data processing task, according still further to the executive mode, executes data processing
Task.
Wherein, user can include adjusting by GUI input after the configuration item that input user creates data processing task
The configuration item of parameter is spent, or GUI input can also be first passed through and include the configuration item of scheduling parameter, then input user and create data
The configuration item of processing task, the embodiment of the present disclosure are not construed as limiting this.
It should be understood that after inputting the configuration item including scheduling parameter by GUI, it can be raw by the scheduling parameter
At scheduling information stored, in this way, for the data processing task created, can call and store
Scheduling information.Certainly, in other possible situations, for the data processing task created, if user re-enters
Configuration item including dispatching scheduling parameter, the scheduling parameter that can also be re-entered according to user generate scheduling information, to realize
Scheduling to the data processing task.In this case, the stored scheduling information of the data processing task can also be updated.
Illustratively, referring to Fig. 5, user can input a variety of different scheduling parameters according to demand.Wherein, " whether dispatch "
It can be used to determine whether that the scheduling parameter inputted by user is scheduled the data processing task of creation, if selection
Be, then can according to user input specific scheduling parameter the data processing task of creation is scheduled, it is on the contrary then not according to
The specific scheduling parameter of user's input is scheduled the data processing task of creation." dispatching cycle " is determined for executing
The time cycle of data processing task, for example can be set as executing daily, executing weekly or monthly execute, etc..It is " specific
Time " is determined for data processing task, and specifically several points execute on the day of.
In addition, referring to Fig. 5, when user does not need scheduling data processing task currently, but need to keep the data processing
The current state node of task can be then arranged " temporarily when so as to the subsequent scheduling to data processing task in scheduling parameter
Stop ", then the data processing task then pauses at current state node, completed for example, data processing task is paused at creation
State node, etc..With continued reference to Fig. 5, the scheduling parameter of user's input can also include the number unsuccessfully retried, i.e. logarithm
After processing task execution failure, the number re-executed, for example, user can choose " not retrying ", retry referring to Fig. 5
" primary " retries " twice " or retries " three times ".
It should be understood that above-mentioned carried out to the content of scheduling parameter for example, being embodied in the disclosure
When, scheduling parameter can also include other content, and user can be according to different application scenarios or different demands, to GUI
The configuration item of the scheduling parameter of middle display is pre-defined, and the embodiment of the present disclosure is not construed as limiting this.
After user's input scheduling parameter, corresponding scheduling information, scheduling letter can be generated according to the scheduling parameter
Breath is the information with format for computer storage, for example, user has input use by GUI shown in fig. 5
25 scheduling parameters for dividing execution when characterize data processing task is 13 on the day of, can be according to the specific time " 25 divide when 13 "
Scheduling parameter is generated as the scheduling information of " time:1325 ".It should be understood that above-mentioned generated to according to scheduling parameter
It is that the process of scheduling information carries out for example, in the disclosure when it is implemented, for different scheduling parameters, can be by not
Same mode generates scheduling information, and the embodiment of the present disclosure is not construed as limiting this.
After generating scheduling information, the data processing task can be scheduled, i.e. basis according to the scheduling information
The executive mode of scheduling information instruction, executes the data processing task.Illustratively, data processing task is executed, can be wound
The equipment itself for building data processing task executes the data processing task, and the equipment for being also possible to create data processing task should
Data processing task is sent to other equipment and executes the data processing task.For example, the control clothes under cluster scene, in cluster
Device be engaged in after creating data processing task, the data processing task can be executed with itself, it can also be by the data processing task
Other servers being sent in cluster execute the data processing task.
By the above-mentioned means, only the configuration item including scheduling parameter can need to be set in gui interface with user, it can basis
The configuration item of user setting automatically generates scheduling information, to realize the scheduling to data processing task, compared in the related technology
For specific command row as shown in FIG. 6 of writing to the scheduling mode of data processing task, the scheme of the embodiment of the present disclosure is not necessarily to user
Learn the order line writes mode, reduces learning cost, and simplifies user and the scheduling of data processing task is set
It sets, so as to improve the dispatching efficiency to data processing task.
In a kind of possible mode, after executing data processing task, it can also obtain what user was inputted by GUI
Then the implementing result of data processing task is sent to destination application by the API by application programming interfaces API.
Wherein, the API includes the callback interface of Relational DBMS MySQL, hypertext transfer protocol
At least one of API and the API of distributed file system Hadoop of HTTP.For example, can be shown to user referring to Fig. 7
GUI including above-mentioned three kinds of API.Wherein, it if it is the API of MySQL, whether can also pre-select according to data processing task
Implementing result covering MySQL database in data with existing, and need to input the execution knot wait store the data processing task
The data table information of fruit.
If showing the GUI shown in Fig. 7 including above-mentioned three kinds of API to user, can be actually entered according to user
Configuration item determines the implementing result of data processing task is sent to destination application by which kind of API.It should be understood that
It is, if the implementing result of data processing task is sent to destination application by a variety of API, the target application journey
Sequence also correspondingly can be a variety of different application programs.
Illustratively, destination application can be the application program for triggering creation data processing task, can be it
He needs the application program of the implementing result using the data processing task.
For example, user has input the configuration item for creating data processing task in the GUI of front-end application, this is matched
Setting item includes above-mentioned data command, macros, resource allocation information, scheduling parameter and application programming interfaces API, and is somebody's turn to do
API is the API of hypertext transfer protocol HTTP.Correspondingly, back-end server can match according to data command, macros and resource
Information creating data processing task is set, and according to scheduling parameter, data processing task is scheduled, that is, is executed at the data
Reason task.After executing the data processing task, the implementing result of the data processing task can be returned to by the API
Front-end application, so that user checks.
By the above-mentioned means, user only need to set the configuration item including application programming interfaces API in gui interface
The implementing result of data processing task is sent to by destination application by the API according to the configuration item of user setting, is compared
In in such a way that complicated program is arranged to the readjustment API of implementing result, the scheme of the embodiment of the present disclosure is reduced in the related technology
The learning cost of user, convenient for the operation of user, so as to more efficiently to the implementing result of data processing task into
Row calls.
The task daily record that can be generated in the implementation procedure of data processing task may include in the task daily record a large amount of with number
According to the unrelated data of the implementing result of processing task, for example, may include under cluster scene, in task daily record big referring to Fig. 8
The data such as stack information, jvm log information, cluster processing log information are measured, user needs to have professional knowledge could be in task
Data relevant to the implementing result of data processing task are accurately found in the mass data of log, are unfavorable for novice users pair
In checking for task daily record.
Therefore, in order to solve this problem, the embodiment of the present disclosure may also respond to user and be checked by the log that GUI is inputted
Instruction obtains task daily record corresponding with data processing task, then according to screening rule is preset, filters out from task daily record
Result data relevant to the implementing result of data processing task, and the result data is returned to GUI.
Illustratively, the configuration item for checking log can be preset, which can show by way of button
Show in the gui, then when the user clicks the button when, that is, can trigger generate log check instruction.Back-end server receives this
After instruction is checked in log, available task daily record corresponding with the data processing task.
It, can be according to default screening rule, to Mission Day after obtaining task daily record corresponding with data processing task
The data of will are screened, to obtain result data relevant to the implementing result of data processing task.Illustratively, screening is preset
It is predetermined according to the actual situation that rule can be user.For example, in the data header information of data relevant to implementing result
The mark of data processing task may be carried, therefore default screening rule can be the data head according to data each in task daily record
Information filters out the data, etc. that data processing task mark is carried in data header information, and the embodiment of the present disclosure is for default sieve
The setting of choosing rule is not construed as limiting.
According to screening rule is preset, knot relevant to the implementing result of data processing task is filtered out from task daily record
After fruit data, the result data can be returned to GUI.In this way, the task daily record that user views only includes
Result data relevant to the implementing result of data processing task passes through professional knowledge in a large amount of numbers of task daily record without user
Data relevant to implementing result are searched according to middle, the learning cost of user are reduced, convenient for user's checking for task daily record.
It is illustrated below by task creation method of the disclosure another exemplary embodiment to the disclosure.Reference Fig. 9,
The task creation method includes:
Step S901, obtains the configuration item that user is inputted by GUI, which includes the attribute ginseng of data to be stored
Number.
Step S902, the property parameters for including according to configuration item generate the attribute information of data.
Step S903 stores the attribute information of data and data according to preset data form.
Step S904, obtains the configuration item that user is inputted by graphic user interface GUI, which includes data life
Enable, resource allocation information, macros, scheduling parameter and readjustment API.
In other words, in the embodiments of the present disclosure, user can by be arranged configuration item, by data command described above,
Resource allocation information, macros, scheduling parameter and readjustment API are inputted simultaneously.Certainly, in other possible situations,
The data command, resource allocation information and macros for creating data processing task can first be inputted.When creation data processing
After task, the information that creation is completed is returned to user.When user views the information and then input for dispatching in GUI
The scheduling reference of data processing task.Then after data processing task execution, the information for executing and completing also is returned to user.
When user views the information in GUI and then inputs the API for calling the implementing result of the data processing task.It should
Understand, the embodiment of the present disclosure is not construed as limiting the input sequence of each configuration item.
Step S905, the scheduling parameter for including according to configuration item generate scheduling information.
Step S906 obtains the attribute information of data relevant to data command, which includes the data of data
Source information and authority information.
Step S907 obtains data relevant to data command from corresponding data source according to data source information.
Step S908 executes data command to the data got according to authority information.
Step S909, determines whether data command runs succeeded, if it succeeds, entering step S910, otherwise enters step
Rapid S911.
Step S910 creates data processing task according to data command, resource allocation information and macros.
Step S911 returns to diagnostics to GUI, and terminates the task creation process.
Step S912, the executive mode of the data processing task according to indicated by scheduling information execute data processing task.
The implementing result of data processing task is sent to destination application by API by step S913.
The specific embodiment of above steps carries out illustrated in greater detail above, and which is not described herein again.In addition it answers
When understanding, for above method embodiment, for simple description, therefore, it is stated as a series of action combinations, but
Be those skilled in the art should understand that, the disclosure is not limited by sequence of movement as described above.Secondly, this field skill
Art personnel also should be aware of, and embodiments described above belongs to preferred embodiment, related step not necessarily this public affairs
Necessary to opening.
In a kind of possible mode, the implementation process schematic diagram of above-mentioned task creation method can be as shown in Figure 10.Ginseng
According to Figure 10, user can input above-mentioned various configuration items by the GUI of front-end application, it is then possible to by the configuration item
It is sent to back-end server.The DSL engine of back-end server can be compiled the data command that the configuration item includes, and most
Creation data processing task is exported to task scheduling engine eventually.The task scheduling engine can join according to the scheduling that configuration item includes
Several pairs of data processing tasks are scheduled.For example, referring to Fig.1 0, which can be sent to by task scheduling engine
Task execution module is executed.After task execution module executes data processing task, implementing result can be sent to knot
Implementing result can be sent to destination application by different API by fruit consumption module, the result consumption module.Meanwhile In
During task execution, the task daily record of data processing task can be collected by log management module, and send out after being screened
It send and returns to front end GUI.
Through the above technical solutions, user only need to set configuration item in gui interface, it can be according to the configuration of user setting
Item automatically creates data processing task, the process of creation data processing task is simplified, so as to improve creation data processing
The efficiency of task.Also, for operator, without learning profession programming knowledge and speciality method, study is reduced
Cost.
Based on the same inventive concept, referring to Fig.1 1, the embodiment of the present disclosure also provides a kind of task creation device 110, this
Creating device 110 of being engaged in can become some or all of electronic equipment, the task in such a way that software, hardware or both combine
Creating device 110 may include:
Module 111 is obtained, for obtaining the configuration item of user's input, the configuration item includes appointing for creating data processing
The data command of business；
Collector 112, for being carried out by field language-specific DSL engine to the data command that the configuration item includes
Compiling；
Creation module 113 is appointed for when the compiling result to the data command is correct, creating the data processing
Business.
Optionally, the collector 112 is used for:
The attribute information of pre-stored data relevant to the data command is obtained, the attribute information includes described
The data source information and authority information of data；
According to the data source information, data relevant to the data command are obtained from corresponding data source；
According to the authority information, the data command is executed to the data got；
The creation module 113 is used for when the data command runs succeeded, and creates the data processing task.
Optionally, described device 110 further include:
Attribute obtains module, the configuration item inputted for obtaining the user by graphic user interface GUI, the configuration
Item includes the property parameters of data to be stored；
First generation module, the property parameters for including according to the configuration item, generates the category of the data to be stored
Property information；
Memory module is used for according to preset data form, by the category of the data to be stored and the data to be stored
Property information is stored.
Optionally, the configuration item further includes the resource allocation information and macros for creating data processing task, institute
Resource allocation information and macros of the creation module 113 for including according to the configuration item are stated, the data processing is created and appoints
Business.
Optionally, the configuration item further includes the scheduling parameter for being used to indicate the data processing task executive mode, institute
State device 110 further include:
Second generation module, the scheduling parameter for including according to the configuration item generate scheduling information；
Determining module, for determining the executive mode of the data processing task according to the scheduling information；
Execution module, for executing the data processing task according to the executive mode.
Optionally, described device 110 further include:
Log acquisition module is obtained for checking instruction by the log that graphic user interface GUI is inputted in response to user
Task daily record corresponding with the data processing task；
Data outputting module, for filtering out from task daily record and appointing with the data processing according to screening rule is preset
The relevant result data of the implementing result of business, and the result data is returned to the GUI.
Optionally, described device 110 further include:
API obtains module, passes through graphical user circle for after executing the data processing task, obtaining the user
The application programming interfaces API of face GUI input, wherein the API includes that the readjustment of Relational DBMS MySQL connects
At least one of mouth, API of the API of hypertext transfer protocol HTTP and distributed file system Hadoop；
Sending module, for the implementing result of the data processing task to be sent to target application journey by the API
Sequence.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
By any of the above-described task creation device, user need to only set configuration item, can be according to the configuration of user setting
Item automatically creates data processing task, the process of creation data processing task is simplified, so as to improve creation data processing
The efficiency of task.Also, for operator, without learning profession programming knowledge and speciality method, study is reduced
Cost.
Based on the same inventive concept, the embodiment of the present disclosure also provides a kind of electronic equipment, comprising:
Memory is stored thereon with computer program；
Processor, for executing the computer program in the memory, to realize any of the above-described task creation side
The step of method.
In possible mode, the block diagram of the electronic equipment is shown in Fig.12.Referring to Fig.1 2, electronic equipment 1200 can
To be provided as a server.Referring to Fig.1 2, electronic equipment 1200 includes processor 1222, and quantity can be one or more
A and memory 1232, for storing the computer program that can be executed by processor 1222.The meter stored in memory 1232
Calculation machine program may include it is one or more each correspond to one group of instruction module.In addition, processor 1222 can
To be configured as executing the computer program, to execute above-mentioned task creation method.
In addition, electronic equipment 1200 can also include power supply module 1226 and communication component 1250, the power supply module 1226
It can be configured as the power management for executing electronic equipment 1200, which can be configured as realization electronic equipment
1200 communication, for example, wired or wireless communication.In addition, the electronic equipment 1200 can also include that input/output (I/O) connects
Mouth 1258.Electronic equipment 1200 can be operated based on the operating system for being stored in memory 1232, such as Windows
ServerTM, Mac OS XTM, UnixTM, LinuxTM etc..
In a further exemplary embodiment, a kind of computer readable storage medium including program instruction is additionally provided, it should
The step of above-mentioned task creation method is realized when program instruction is executed by processor.For example, the computer readable storage medium
It can be the above-mentioned memory 1232 including program instruction, above procedure instruction can be held by the processor 1222 of electronic equipment 1200
Row is to complete above-mentioned task creation method.
The preferred embodiment of the disclosure is described in detail in conjunction with attached drawing above, still, the disclosure is not limited to above-mentioned reality
The detail in mode is applied, in the range of the technology design of the disclosure, a variety of letters can be carried out to the technical solution of the disclosure
Monotropic type, these simple variants belong to the protection scope of the disclosure.
It is further to note that specific technical features described in the above specific embodiments, in not lance
In the case where shield, can be combined in any appropriate way, in order to avoid unnecessary repetition, the disclosure to it is various can
No further explanation will be given for the combination of energy.
In addition, any combination can also be carried out between a variety of different embodiments of the disclosure, as long as it is without prejudice to originally
Disclosed thought equally should be considered as disclosure disclosure of that.
Claims (10)
1. a kind of task creation method, which is characterized in that the described method includes:
The configuration item of user's input is obtained, the configuration item includes the data command for creating data processing task；
It is compiled by the data command that field language-specific DSL engine includes to the configuration item；
If the compiling result to the data command is correct, the data processing task is created.
2. the method according to claim 1, wherein it is described by field language-specific DSL engine to the number
It is compiled according to order, comprising:
The attribute information of pre-stored data relevant to the data command is obtained, the attribute information includes the data
Data source information and authority information；
According to the data source information, data relevant to the data command are obtained from corresponding data source；
According to the authority information, the data command is executed to the data got；
If described correct to the compiling result of the data command, the data processing task is created, comprising:
If the data command runs succeeded, the data processing task is created.
3. according to the method described in claim 2, it is characterized in that, the method also includes:
The configuration item that the user is inputted by graphic user interface GUI is obtained, the configuration item includes the category of data to be stored
Property parameter；
According to the property parameters that the configuration item includes, the attribute information of the data to be stored is generated；
According to preset data form, the attribute information of the data to be stored and the data to be stored is stored.
4. method according to claim 1 to 3, which is characterized in that the configuration item further includes for creating at data
The resource allocation information and macros of reason task, the creation data processing task, comprising:
The resource allocation information and macros for including according to the configuration item, create the data processing task.
5. method according to claim 1 to 3, which is characterized in that the configuration item further includes being used to indicate the number
According to processing task execution mode scheduling parameter, the method also includes:
According to the scheduling parameter that the configuration item includes, scheduling information is generated；
According to the scheduling information, the executive mode of the data processing task is determined；
According to the executive mode, the data processing task is executed.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
Instruction is checked by the log that graphic user interface GUI is inputted in response to user, is obtained and the data processing task pair
The task daily record answered；
According to default screening rule, result relevant to the implementing result of the data processing task is filtered out from task daily record
Data, and the result data is returned to the GUI.
7. according to the method described in claim 5, it is characterized in that, after executing the data processing task, the method
Further include:
Obtain the application programming interfaces API that the user is inputted by graphic user interface GUI, wherein the API includes relationship
The callback interface of type data base management system MySQL, the API of hypertext transfer protocol HTTP and distributed file system Hadoop
At least one of API；
The implementing result of the data processing task is sent to destination application by the API.
8. a kind of task creation device, which is characterized in that described device includes:
Module is obtained, for obtaining the configuration item of user's input, the configuration item includes the number for creating data processing task
According to order；
Collector, the data command for including to the configuration item by field language-specific DSL engine are compiled；
Creation module, for creating the data processing task when the compiling result to the data command is correct.
9. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is held by processor
The step of any one of claim 1-7 the method is realized when row.
10. a kind of electronic equipment characterized by comprising
Memory is stored thereon with computer program；
Processor, for executing the computer program in the memory, to realize described in any one of claim 1-7
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910667753.0A CN110457023A (en) | 2019-07-23 | 2019-07-23 | Task creation method, apparatus, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910667753.0A CN110457023A (en) | 2019-07-23 | 2019-07-23 | Task creation method, apparatus, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110457023A true CN110457023A (en) | 2019-11-15 |
Family
ID=68483103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910667753.0A Pending CN110457023A (en) | 2019-07-23 | 2019-07-23 | Task creation method, apparatus, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110457023A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254885A (en) * | 2020-02-13 | 2021-08-13 | 支付宝(杭州)信息技术有限公司 | Machine learning model protection method and device |
| CN113821515A (en) * | 2021-09-26 | 2021-12-21 | 北京慧博科技有限公司 | Method for inquiring and analyzing linkage rate based on MySql |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100005529A1 (en) * | 2008-06-30 | 2010-01-07 | Ubs Ag | Platform verification portal |
| US20110196957A1 (en) * | 2010-02-05 | 2011-08-11 | International Business Machines Corporation | Real-Time Policy Visualization by Configuration Item to Demonstrate Real-Time and Historical Interaction of Policies |
| US20110246972A1 (en) * | 2010-03-30 | 2011-10-06 | International Business Machines Corporation | Method of selecting an expression evaluation technique for domain-specific language compilation |
| CN109491651A (en) * | 2018-10-24 | 2019-03-19 | 东软集团股份有限公司 | Data preprocessing method, device, storage medium and electronic equipment |
| CN109522075A (en) * | 2018-11-09 | 2019-03-26 | 医渡云（北京）技术有限公司 | Data visualization methods of exhibiting, device, electronic equipment and computer-readable medium |
| CN109918437A (en) * | 2019-03-08 | 2019-06-21 | 北京中油瑞飞信息技术有限责任公司 | Distributed data processing method, apparatus and data assets management system |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
-
2019
- 2019-07-23 CN CN201910667753.0A patent/CN110457023A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100005529A1 (en) * | 2008-06-30 | 2010-01-07 | Ubs Ag | Platform verification portal |
| US20110196957A1 (en) * | 2010-02-05 | 2011-08-11 | International Business Machines Corporation | Real-Time Policy Visualization by Configuration Item to Demonstrate Real-Time and Historical Interaction of Policies |
| US20110246972A1 (en) * | 2010-03-30 | 2011-10-06 | International Business Machines Corporation | Method of selecting an expression evaluation technique for domain-specific language compilation |
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| CN109491651A (en) * | 2018-10-24 | 2019-03-19 | 东软集团股份有限公司 | Data preprocessing method, device, storage medium and electronic equipment |
| CN109522075A (en) * | 2018-11-09 | 2019-03-26 | 医渡云（北京）技术有限公司 | Data visualization methods of exhibiting, device, electronic equipment and computer-readable medium |
| CN109918437A (en) * | 2019-03-08 | 2019-06-21 | 北京中油瑞飞信息技术有限责任公司 | Distributed data processing method, apparatus and data assets management system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254885A (en) * | 2020-02-13 | 2021-08-13 | 支付宝(杭州)信息技术有限公司 | Machine learning model protection method and device |
| WO2021159819A1 (en) * | 2020-02-13 | 2021-08-19 | 支付宝(杭州)信息技术有限公司 | Machine learning model protection method and device |
| CN113821515A (en) * | 2021-09-26 | 2021-12-21 | 北京慧博科技有限公司 | Method for inquiring and analyzing linkage rate based on MySql |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110232162B (en) | HTML 5-based flow definition method and system | |
| CN105912474B (en) | A kind of game on-line debugging method, system and editing service end | |
| CN109460279A (en) | Forms pages development approach, device, equipment and storage medium based on small routine | |
| CN108647141A (en) | Automatic test approach, device, computer-readable medium and electronic equipment | |
| KR101637371B1 (en) | System for testing an application use for a smart device and method therefor | |
| CN105683908A (en) | Context-aware user interface section | |
| CA3050159A1 (en) | Artificial intelligence (ai) based automatic rule generation | |
| CN110633078B (en) | Method and device for automatically generating feature calculation codes | |
| CN112579455B (en) | Automatic interface testing method and device, electronic equipment and storage medium | |
| JP6568017B2 (en) | Test support apparatus and test support method | |
| CN110457023A (en) | Task creation method, apparatus, storage medium and electronic equipment | |
| Gehlot | From petri NETS to colored petri NETS: A tutorial introduction to NETS based formalism for modeling and simulation | |
| Bennett et al. | A transformation system for maintenance-turning theory into practice | |
| CN106874202A (en) | Method, device and readable storage medium storing program for executing for unit testing | |
| CN108256716A (en) | The method and apparatus of flow and/or atom flow is configured/performed based on table | |
| CN104081347A (en) | Graphical representation of an order of operations | |
| CN107577521A (en) | A kind of virtual resource management method and device | |
| WO2023160402A1 (en) | Data modeling method and apparatus, and device and storage medium | |
| CN116304211A (en) | Information interception method, device and equipment based on graph calculation and readable storage medium | |
| CN117707917A (en) | Service testing method, device, medium and product | |
| CN110544075A (en) | asset management process configuration method, device and equipment | |
| CN115904382A (en) | Code development method, system, client, server, device and storage medium | |
| Caffiau et al. | Generating interactive applications from task models: A hard challenge | |
| US12099976B2 (en) | Group eligibility criteria builder | |
| CN113971024B (en) | Interface management method, device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191115 |
|
| RJ01 | Rejection of invention patent application after publication |