CN110490451A - Task data management-control method, device and computer equipment based on hadoop - Google Patents
Task data management-control method, device and computer equipment based on hadoop Download PDFInfo
- Publication number
- CN110490451A CN110490451A CN201910754460.6A CN201910754460A CN110490451A CN 110490451 A CN110490451 A CN 110490451A CN 201910754460 A CN201910754460 A CN 201910754460A CN 110490451 A CN110490451 A CN 110490451A
- Authority
- CN
- China
- Prior art keywords
- task
- goal
- call parameter
- title
- hadoop
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 112
- 230000008569 process Effects 0.000 claims abstract description 86
- 238000003860 storage Methods 0.000 claims abstract description 17
- 230000011218 segmentation Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 10
- 238000007726 management method Methods 0.000 description 74
- 238000004519 manufacturing process Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000007639 printing Methods 0.000 description 4
- 241000208340 Araliaceae Species 0.000 description 3
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 3
- 235000003140 Panax quinquefolius Nutrition 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000005520 cutting process Methods 0.000 description 3
- 235000008434 ginseng Nutrition 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/3332—Query translation
- G06F16/3334—Selection or weighting of terms from queries, including natural language queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0633—Workflow analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/04—Manufacturing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses a kind of task data management-control method based on hadoop identifies goal task title therein and goal task call parameter this method comprises: obtaining the source information of the task process log of first task management platform；By the preamble task names and preceding sequence task call parameter progress uniformity comparison in each task record in the goal task title and goal task call parameter and preset front and back sequence task implementation list；When the record of the first task present in the front and back sequence task implementation list is with the task names of the goal task and consistent goal task call parameter, by the first task record in corresponding execute instruction of rear sequence task be sent to the second task management platform.The present invention also provides a kind of task data control device, computer equipment and computer readable storage medium based on hadoop.The present invention is capable of real-time, efficient, the accurate control of across task management platform realization task data.
Description
Technical field
The present invention relates to flow monitoring technical field more particularly to a kind of task data management-control method based on hadoop,
Device, computer equipment and computer readable storage medium.
Background technique
Currently, much can be used for carrying out production task controlling managerial role management platform is applied to the daily of enterprise
In production.Certainly, production tasks many for production task and different may execute on different task management platforms,
Accordingly, it may be desirable to which to manage and control different production task data respectively preset to complete jointly for multiple tasks management platform
Production task.For example, existing OOZIE task management platform and LINKDO task management platform are all more commonly used in production task
The management of data, but not being provided between the two task management platforms interacts the task data of two platforms
Interface, so, when production task decomposite come preceding sequence task and follow-up work be separately dispensed into OOZIE task management platform and
It, can be pretty troublesome to task data management is carried out between the two task management platforms after LINKDO task management platform.For example,
The preceding sequence task that a task management platform executes often is occurred in which because there is exception, and after relying on the preceding sequence task
Sequence task cannot but learn preamble task execution exception, then follow-up work will be behaved according to normal time, this
Sample will generate task data can not true synchronization, task action result be easy deviation the problems such as.Therefore, in the prior art, right
It can not achieve the effect that in the control that across task management platform carries out task data accurate, synchronous.
Summary of the invention
In view of this, the present invention proposes that a kind of user is set based on task data management-control method, device, the computer of hadoop
Standby and computer readable storage medium, can obtain first task management platform task process log source information, then into
Row identification, to identify goal task title and goal task call parameter；Then by the goal task title and target
Preamble task names in each task record in task call parameter and preset front and back sequence task implementation list and before
Sequence task call parameter carries out uniformity comparison；In the record of the first task present in the front and back sequence task implementation list
It, will when preamble task names and preceding sequence task call parameter are with the goal task title and consistent goal task call parameter
Corresponding execute instruction of rear sequence task in the first task record is sent to the second task management platform.Pass through this side
Formula, can across task management platform realize real-time, efficient, the accurate control of task data.
Firstly, to achieve the above object, the present invention provides a kind of task data management-control method based on hadoop, the side
Method includes:
Obtain the source information of the task process log of first task management platform；To the source information of the task process log
It is identified, identifies goal task title and goal task call parameter；By the goal task title and goal task
The preamble task names and preamble times in each task record in call parameter and preset front and back sequence task implementation list
It don't fail to parameter progress uniformity comparison；Preamble in the record of the first task present in the front and back sequence task implementation list
It, will be described when task names and preceding sequence task call parameter are with the goal task title and consistent goal task call parameter
Corresponding execute instruction of rear sequence task in first task record is sent to the second task management platform.
Optionally, the source information to the task process log identifies, identifies goal task title and mesh
The step of mark task call parameter includes: that the source information of the task process log is converted into log text information；To described
Log text information carries out sentence segmentation, obtains at least one experience table sentence；To each experience table language
Sentence carries out text identification, identifies the goal task title and goal task call parameter.
Optionally, described the step of carrying out text identification to each experience table sentence includes: according to preset
Goal task name keys identify each experience table sentence, to find out including the goal task name
The first task of title executes record sentence；Record sentence is executed to the first task according to preset call parameter format to carry out
Identification, to identify the corresponding goal task call parameter of the goal task title.
Optionally, the call parameter includes time and operating status, the call parameter format include time format and
Operating status format.
In addition, to achieve the above object, the present invention also provides a kind of task data control device based on hadoop is described
Device includes:
Module is obtained, the source information of the task process log for obtaining first task management platform；Identification module is used for
The source information of the task process log is identified, identifies goal task title and goal task call parameter；Judgement
Module, for will be in the goal task title and goal task call parameter and preset front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter in each task record carry out uniformity comparison；Sending module is used for
Preamble task names and preceding sequence task necessity ginseng in the record of the first task present in the front and back sequence task implementation list
When number is with the goal task title and consistent goal task call parameter, by the rear sequence task in first task record
Corresponding execute instruction is sent to the second task management platform.
Optionally, the identification module is also used to: the source information of the task process log is converted into log text envelope
Breath；Sentence segmentation is carried out to the log text information, obtains at least one experience table sentence；Each task is held
Row record sentence carries out text identification, identifies the goal task title and goal task call parameter.
Optionally, the identification module is also used to: being held according to preset goal task name keys to each task
Row record sentence is identified, so that finding out the first task including the goal task title executes record sentence；According to pre-
If call parameter format to the first task execute record sentence identify, to identify the goal task title
Corresponding goal task call parameter.
Optionally, the call parameter includes time and operating status, the call parameter format include time format and
Operating status format.
Further, the present invention also proposes a kind of computer equipment, and the computer equipment includes memory, processor,
The computer program that can be run on the processor is stored on the memory, the computer program is by the processor
It realizes when execution such as the step of the above-mentioned task data management-control method based on hadoop.
Further, to achieve the above object, the present invention also provides a kind of computer readable storage medium, the computers
Readable storage medium storing program for executing is stored with computer program, and the computer program can be executed by least one processor so that it is described extremely
A few processor is executed such as the step of the above-mentioned task data management-control method based on hadoop.
Compared to the prior art, task data management-control method, device, the computer proposed by the invention based on hadoop
Equipment and computer readable storage medium can obtain the source information of the task process log of first task management platform, then
It is identified, to identify goal task title and goal task call parameter；Then by the goal task title and mesh
Preamble task names in each task record in mark task call parameter and preset front and back sequence task implementation list and
Preceding sequence task call parameter carries out uniformity comparison；In the record of the first task present in the front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter with the goal task title and consistent goal task call parameter when,
The second task management platform is sent by corresponding execute instruction of rear sequence task in first task record.Pass through this side
Formula, can across task management platform realize real-time, efficient, the accurate control of task data.
Detailed description of the invention
Fig. 1 is the schematic diagram of the optional hardware structure of computer equipment one of the present invention；
Fig. 2 is the program module schematic diagram of one embodiment of task data control device the present invention is based on hadoop；
Fig. 3 is the flow diagram of one embodiment of task data management-control method the present invention is based on hadoop.
Appended drawing reference:
The object of the invention is realized, the embodiments will be further described with reference to the accompanying drawings for functional characteristics and advantage.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that described herein, specific examples are only used to explain the present invention, not
For limiting the present invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not before making creative work
Every other embodiment obtained is put, shall fall within the protection scope of the present invention.
It should be noted that the description for being related to " first ", " second " etc. in the present invention is used for description purposes only, and cannot
It is interpreted as its relative importance of indication or suggestion or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include at least one of the features.In addition, the skill between each embodiment
Art scheme can be combined with each other, but must be based on can be realized by those of ordinary skill in the art, when technical solution
Will be understood that the combination of this technical solution is not present in conjunction with there is conflicting or cannot achieve when, also not the present invention claims
Protection scope within.
As shown in fig.1, being the schematic diagram of the optional hardware structure of computer equipment 1 one of the present invention.
In the present embodiment, the computer equipment 1 may include, but be not limited only to, and company can be in communication with each other by system bus
Connect memory 11, processor 12, network interface 13.
The computer equipment 1 connects network (Fig. 1 is not marked) by network interface 13, arrives other ends by network connection
End equipment such as mobile terminal (Mobile Terminal), user equipment (User Equipment, UE), the end PC and other
Business management platform etc..The network can be intranet (Intranet), internet (Internet), global system for mobile telecommunications
System (Global System of Mobile communication, GSM), wideband code division multiple access (Wideband Code
Division Multiple Access, WCDMA), 4G network, 5G network, bluetooth (Bluetooth), Wi-Fi, speech path network
Deng wirelessly or non-wirelessly network.
It should be pointed out that Fig. 1 illustrates only the computer equipment 1 with component 11-13, it should be understood that simultaneously
All components shown realistic are not applied, the implementation that can be substituted is more or less component.
Wherein, the memory 11 includes at least a type of readable storage medium storing program for executing, and the readable storage medium storing program for executing includes
Flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory etc.), random access storage device (RAM), it is static with
Machine accesses memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable
Read memory (PROM), magnetic storage, disk, CD etc..In some embodiments, the memory 11 can be the meter
Calculate the internal storage unit of machine equipment 1, such as the hard disk or memory of the computer equipment 1.In further embodiments, described to deposit
Reservoir 11 is also possible to the External memory equipment of the computer equipment 1, such as the plug-in type that the computer equipment 1 is equipped with is hard
Disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card
(Flash Card) etc..Certainly, the memory 11 can also both include the internal storage unit of the computer equipment 1 or wrap
Include its External memory equipment.In the present embodiment, the memory 11 is installed on the behaviour of the computer equipment 1 commonly used in storage
Make system and types of applications software, such as the program code etc. of the task data control device 200 based on hadoop.In addition, institute
Stating memory 11 can be also used for temporarily storing the Various types of data that has exported or will export.
The processor 12 can be in some embodiments central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips.The processor 12 is commonly used in the control meter
The overall operation of machine equipment 1 is calculated, such as executes data interaction or the relevant control of communication and processing etc..In the present embodiment, institute
Processor 12 is stated for running the program code stored in the memory 11 or processing data, for example, operation it is described based on
Task data control device 200 of hadoop etc..
The network interface 13 may include radio network interface or wired network interface, which is commonly used in
The computer equipment 1 and other terminal devices such as mobile terminal, user equipment, the end PC and other task management platforms etc.
Between establish communication connection.
In the present embodiment, is installed in the computer equipment 1 and run the task data control device based on hadoop
When 200, when the task data control device 200 based on hadoop is run, enough obtains and take first task management platform
Task process log, is then identified, to identify goal task title and goal task call parameter；Then by the mesh
It marks in each task record in task names and goal task call parameter and preset front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter carry out uniformity comparison；Exist when in the front and back sequence task implementation list
First task record in preamble task names and preceding sequence task call parameter and the goal task title and target appoint
Don't fail to parameter it is consistent when, by the first task record in corresponding execute instruction of rear sequence task be sent to the second task pipe
Platform.In this way, can across task management platform realize real-time, efficient, the accurate control of task data.
So far, oneself is through describing the application environment of each embodiment of the present invention and the hardware configuration and function of relevant device in detail
Energy.In the following, above-mentioned application environment and relevant device will be based on, each embodiment of the invention is proposed.
Firstly, the present invention proposes a kind of task data control device 200 based on hadoop.
As shown in fig.2, being the program module of 200 1 embodiment of task data control device the present invention is based on hadoop
Figure.
In the present embodiment, the task data control device 200 based on hadoop is stored in storage including a series of
Each reality of the present invention may be implemented when the computer program instructions are executed by processor 12 in computer program instructions on device 11
Apply the task data control function based on hadoop of example.In some embodiments, it is based on the computer program instructions each section
The specific operation realized, the task data control device 200 based on hadoop can be divided into one or more modules.
For example, the task data control device 200 based on hadoop, which can be divided into, obtains module 201, identification in Fig. 2
Module 202, judgment module 203 and sending module 204.Wherein:
The acquisition module 201, the source information of the task process log for obtaining first task management platform.
Hadoop is the software frame that distributed treatment can be carried out to mass data, so, it can be based on
Hadoop technology establishes the multiple task management platform based on hadoop in the computer equipment 1 to carry out task data
Corporate management.In the present embodiment, the computer equipment 1, which is connected to, manages platform and the second task including at least first task
The multiple tasks for managing platform manage platform, due to programming between the first task management platform and the second task management platform
The difference of language or the difference of data format, so that the interaction of data cannot be carried out directly.
Specifically, the computer equipment 1 is connect by managing platform with the first task, and periodically obtains institute
State the source information for the real-time task process log that first task management platform prints.In the present embodiment, task management is flat
The operating system of platform can be monitored and record to all tasks of self-operating, then the task process of outputting standard format
The source information of log, such as the experience table including elements such as task names, time, execution states, and it is stored in operation
Subdirectory under catalogue where system, such as task daily record file.It is of course also possible to which monitoring is appointed in the instruction according to user
Business classification, the element of monitoring are defined.Therefore, the first task management platform can be preset in the mistake of the task of execution
Cheng Zhong prints task process log in real time, updates and stores under preset catalogue.Then, the acquisition module 201
Then according to preset period, such as 30s/ times, under the catalogue of the store tasks process log of first task management platform
Obtain the source information of newest task process log.In the present embodiment, the computer equipment 1 is set as possessing described first
The log read permission of task management platform, therefore, the acquisition module 201 then can read institute according to the preset period
The catalogue file that first task management platform is stored with task process log is stated, and by preset name, for example " XX month XX -appoints
The task execution log of business process log " is copied and is received.
The identification module 202 identifies goal task for identifying to the source information of the task process log
Title and goal task call parameter.
Specifically, source information of the identification module 202 first by the task process log is converted into log text information,
Then sentence segmentation is carried out to the log text information, obtains at least one experience table sentence, then appoint to each
Business executes record sentence and carries out text identification, identifies the goal task title and goal task call parameter.Wherein, described
Identification module 202 identifies each experience table sentence according to preset goal task name keys, thus
It finds out the first task including the goal task title and executes record sentence, then according to preset call parameter format to institute
It states first task execution record sentence to be identified, to identify the corresponding goal task necessity ginseng of the goal task title
Number.
In the present embodiment, not due to the programming language used or data format of different task management platforms
Together.Therefore, format of the identification module 202 first by the task process log of first task management platform printing is converted to
Text formatting.For example, the task process log of the first task management platform printing is JAVA format, then the identification mould
The task process log of the JAVA format is then converted into text according to preset JAVA- text formatting crossover tool by block 202
Format, so that the source information of the task process log to be converted to the text information of the task process log.Then, institute
Identification module 202 is stated to be identified according to text information of the preset goal task title to the task process log.At this
In embodiment, all task process before the task process log includes the first task management platform record are believed
Breath, therefore the task process log can be identified, to identify the progress information of the goal task.Firstly,
The text information of the task process log is carried out sentence segmentation by the identification module 202, wherein sentence segmentation is mainly to be
The text information of the task process log is divided into short sentence, therefore can be according to identifying the task process log
The punctuate of text information carries out short sentence cutting, to obtain the sentence of each task process log, in the present embodiment, sentence
Cutting is mainly the ascii code value for identifying the punctuation mark in the task process log, then carries out sentence segmentation, such as
When recognizing ascii code value " 0X2E ", then it is judged as fullstop, carries out line feed to come out the sentence segmentation.Then basis
The task names keyword of the preset goal task is compared with each text sentence in the task process log
It is right, to obtain the sentence including the goal task name keys.In the present embodiment, the task name of the goal task
Claiming keyword is directly the task names of the goal task.It certainly, in other embodiments, can also be according to task names
Including distinctive word, word or phrase, appoint using the distinctive word of the task names of the goal task, word or phrase as described
The keyword for title of being engaged in.That is, can be according to the goal task name keys, to identify the task process
The corresponding sentence of the goal task title in log in text information.
After the corresponding sentence of the goal task title in identifying the task process log in text information,
Then further identification includes included by each experience table of the goal task title to the identification module 202
Call parameter.In the present embodiment, the call parameter of the experience table includes: time and operating status.Therefore, may be used
Time and operating parameter in each sentence will identify that the task process log come.Specifically, identification process packet
Include: according to time format, for example " year-month-day-when-point-second " identifies the time in task execution sentence；According to operation shape
It is corresponding that state format such as " operating status: ffff " identifies the goal task title in the task process log sentence
Execution state.In this way, so that it may know the call parameter in each experience table of the goal task
It does not come out.
The judgment module 203, for by the goal task title and goal task call parameter and it is preset before
The preamble task names in each task record in postorder task execution inventory are consistent with preceding sequence task call parameter progress
Property compare.
Specifically, the computer equipment 1 presets front and back sequence task implementation list, and the front and back sequence task executes clear
Single includes the dependence of any one preceding sequence task and corresponding follow-up work in production process, and the dependence includes holding
First task the management platform, preamble task names, preamble execution status of task of sequence task before carrying, and sequence task after carrying
Second task management platform, follow-up work title and rear sequence task is corresponding executes instruction.Therefore, in the identification module
After 202 pairs of experience tables carry out preamble task names and the identification of corresponding call parameter, the judgment module 203 then may be used
With goal task title described further and goal task call parameter with it is every in preset front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter in one task record carry out uniformity comparison.In the present embodiment, institute
Judgment module 203 is stated according to the goal task title in the experience table in the front and back sequence task implementation list
In searched, the goal task title is found out according to the mode that text compares, when finding out the goal task conduct
When preamble task names, then time and operating status of the goal task in the task process log are further searched for out
Information is also to identify in such a way that text compares for time and running state information.In the present embodiment, when described
Judgment module 203 finds out the goal task when storing in the front and back sequence task implementation list as preceding sequence task, then
By in the task process log the corresponding operating status of the goal task and the front and back sequence task implementation list in make
Correspondence preamble execution status of task for the goal task of preceding sequence task is compared, and when consistent, then judges the mesh
Preceding sequence task of the mark task as the front and back sequence task implementation list, and executed completion, then it should further execute this
The corresponding follow-up work of preceding sequence task.In other embodiments, sequence task is also before each in the front and back sequence task implementation list
Including execute the time, for example, preceding sequence task when being executed between reach T hour after, then operating status reaches F, then it is assumed that this
Preceding sequence task normally completes；If executing the time not reaching T hours, and operating status reaches F, then it is assumed that the preceding sequence task is different
Often.Then follow-up work can just only be executed after preceding sequence task normally completes.Therefore, the multiple task management system can also
The execution time of the goal task in the task process log is judged.
The sending module 204, for working as in the record of first task present in the front and back sequence task implementation list
It, will when preamble task names and preceding sequence task call parameter are with the goal task title and consistent goal task call parameter
Corresponding execute instruction of rear sequence task in the first task record is sent to the second task management platform.
Specifically, when the judgment module 203 judges goal task title described in the experience table and institute
Stating running state information is respectively corresponding preamble task names and preamble task execution in the front and back sequence task implementation list
When state, then it is assumed that the preceding sequence task has been completed.Therefore, the sending module 204 then can be according to the front and back sequence task
Corresponding execute instruction of rear sequence task in first task record is sent the second task management platform by implementation list.Cause
This, the second task management platform can then start to execute the rear sequence task.
It will be recalled from above that the computer equipment 1 can obtain the source of the task process log of first task management platform
Then information is identified, to identify goal task title and goal task call parameter；Then by the goal task name
Claim and goal task call parameter is appointed with the preamble in each task record in preset front and back sequence task implementation list
Title of being engaged in and preceding sequence task call parameter carry out uniformity comparison；It is first present in the front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter and the goal task title and goal task necessity in business record are joined
When number is consistent, the second task management platform is sent by corresponding execute instruction of rear sequence task in first task record.
In this way, can across task management platform realize real-time, efficient, the accurate control of task data.
In addition, the present invention also proposes a kind of task data management-control method based on hadoop, the method is applied to calculate
Machine equipment.
As shown in fig.3, being the flow diagram of one embodiment of task data management-control method the present invention is based on hadoop.
In the present embodiment, the execution sequence of the step in flow chart shown in Fig. 3 can change according to different requirements, Mou Xiebu
Suddenly it can be omitted.
Step S500 obtains the source information of the task process log of first task management platform.
Hadoop is the software frame that distributed treatment can be carried out to mass data, so, it can be based on
Hadoop technology establishes the multiple task management platform based on hadoop in the computer equipment to carry out task data
Corporate management.In the present embodiment, the computer equipment, which is connected to, manages platform and the second task including at least first task
The multiple tasks for managing platform manage platform, due to programming between the first task management platform and the second task management platform
The difference of language or the difference of data format, so that the interaction of data cannot be carried out directly.
Specifically, the computer equipment is connect by managing platform with the first task, and periodically obtains institute
State the source information for the real-time task process log that first task management platform prints.In the present embodiment, task management is flat
The operating system of platform can be monitored and record to all tasks of self-operating, then the task process of outputting standard format
The source information of log, such as the experience table including elements such as task names, time, execution states, and it is stored in operation
Subdirectory under catalogue where system, such as task daily record file.It is of course also possible to which monitoring is appointed in the instruction according to user
Business classification, the element of monitoring are defined.Therefore, the first task management platform can be preset in the mistake of the task of execution
Cheng Zhong prints task process log in real time, updates and stores under preset catalogue.Then, the computer equipment
Then according to preset period, such as 30s/ times, under the catalogue of the store tasks process log of first task management platform
Obtain the source information of newest task process log.In the present embodiment, the computer equipment is set as possessing described first
The log read permission of task management platform, therefore, the computer equipment then can read institute according to the preset period
The catalogue file that first task management platform is stored with task process log is stated, and by preset name, for example " XX month XX -appoints
The task execution log of business process log " is copied and is received.
Step S502 identifies the source information of the task process log, identifies goal task title and target
Task call parameter.
Specifically, source information of the computer equipment first by the task process log is converted into log text information,
Then sentence segmentation is carried out to the log text information, obtains at least one experience table sentence, then appoint to each
Business executes record sentence and carries out text identification, identifies the goal task title and goal task call parameter.Wherein, described
Computer equipment identifies each experience table sentence according to preset goal task name keys, to look for
The first task including the goal task title executes record sentence out, then according to preset call parameter format to described
First task executes record sentence and is identified, to identify the corresponding goal task necessity ginseng of the goal task title
Number.
In the present embodiment, not due to the programming language used or data format of different task management platforms
Together.Therefore, format of the computer equipment first by the task process log of first task management platform printing is converted to
Text formatting.For example, the task process log of the first task management platform printing is JAVA format, then the computer
The task process log of the JAVA format is then converted into text lattice according to preset JAVA- text formatting crossover tool by equipment
Formula, so that the source information of the task process log to be converted to the text information of the task process log.Then, described
Computer equipment is identified according to text information of the preset goal task title to the task process log.In this implementation
In example, the task process log includes all task process information before the first task management platform records,
Therefore the task process log can be identified, to identify the progress information of the goal task.Firstly, described
The text information of the task process log is carried out sentence segmentation by computer equipment, wherein sentence segmentation will be primarily to will
The text information of the task process log is divided into short sentence, therefore can be according to the text for identifying the task process log
The punctuate of information carries out short sentence cutting, to obtain the sentence of each task process log, in the present embodiment, sentence segmentation
The ascii code value for mainly identifying the punctuation mark in the task process log, then carries out sentence segmentation, for example work as knowledge
When being clipped to ascii code value " 0X2E ", then it is judged as fullstop, carries out line feed to come out the sentence segmentation.Then according to default
The task names keyword of the goal task be compared with each text sentence in the task process log, from
And obtain the sentence including the goal task name keys.In the present embodiment, the task names of the goal task are closed
Key word is directly the task names of the goal task.Certainly, in other embodiments, can also include according to task names
Distinctive word, word or phrase, using the distinctive word of the task names of the goal task, word or phrase as the task name
The keyword of title.That is, can be according to the goal task name keys, to identify the task process log
The corresponding sentence of the goal task title in middle text information.
After the corresponding sentence of the goal task title in identifying the task process log in text information,
Then further identification includes included by each experience table of the goal task title to the computer equipment
Call parameter.In the present embodiment, the call parameter of the experience table includes: time and operating status.Therefore, may be used
Time and operating parameter in each sentence will identify that the task process log come.Specifically, identification process packet
Include: according to time format, for example " year-month-day-when-point-second " identifies the time in task execution sentence；According to operation shape
It is corresponding that state format such as " operating status: ffff " identifies the goal task title in the task process log sentence
Execution state.In this way, so that it may know the call parameter in each experience table of the goal task
It does not come out.
Step S504 executes the goal task title and goal task call parameter and preset front and back sequence task
The preamble task names and preceding sequence task call parameter progress uniformity comparison in each task record in inventory.
Specifically, the computer equipment presets front and back sequence task implementation list, and the front and back sequence task executes clear
Single includes the dependence of any one preceding sequence task and corresponding follow-up work in production process, and the dependence includes holding
First task the management platform, preamble task names, preamble execution status of task of sequence task before carrying, and sequence task after carrying
Second task management platform, follow-up work title and rear sequence task is corresponding executes instruction.Therefore, in the computer equipment
After carrying out preamble task names and the identification of corresponding call parameter to experience table, then it can be appointed with target described further
Before in each task record in title of being engaged in and goal task call parameter and preset front and back sequence task implementation list
Sequence task title and preceding sequence task call parameter carry out uniformity comparison.In the present embodiment, the computer equipment is according to institute
The goal task title stated in experience table is searched in the front and back sequence task implementation list, according to text
The mode of comparison finds out the goal task title, when finding out the goal task as preamble task names, then into
One step finds out time and running state information of the goal task in the task process log, for time and operation
Status information is also to be identified in such a way that text compares.In the present embodiment, when the computer equipment finds out institute
It, then will be in the task process log when stating goal task and storing in the front and back sequence task implementation list as preceding sequence task
The corresponding operating status of the goal task and the front and back sequence task implementation list in the target as preceding sequence task
The correspondence preamble execution status of task of task is compared, and when consistent, then judges the goal task as the front and back sequence
The preceding sequence task of task execution inventory, and executed completion, then it is subsequent corresponding should further to execute the preceding sequence task
Business.In other embodiments, each preceding sequence task further includes executing the time in the front and back sequence task implementation list, for example, preceding
Sequence task when being executed between reach T hours after, then operating status reaches F, then it is assumed that the preceding sequence task normally completes；If
The execution time does not reach T hours, and operating status reaches F, then it is assumed that the preamble task abnormity.Then only in preceding sequence task
Follow-up work can be just executed after normally completing.Therefore, the multiple task management system can also be in the task process log
Execution time of the goal task judged.
Step S504, the preamble task names in the record of the first task present in the front and back sequence task implementation list
When with preceding sequence task call parameter with the goal task title and consistent goal task call parameter, by the first task
Corresponding execute instruction of rear sequence task in record is sent to the second task management platform.
Specifically, when the computer equipment judges goal task title described in the experience table and described
Running state information is respectively corresponding preamble task names and preamble task execution shape in the front and back sequence task implementation list
When state, then it is assumed that the preceding sequence task has been completed.Therefore, the computer equipment can then be executed according to the front and back sequence task
Corresponding execute instruction of rear sequence task in first task record is sent the second task management platform by inventory.Therefore,
The second task management platform can then start to execute the rear sequence task.
The task data management-control method based on hadoop that the present embodiment is proposed can obtain first task management platform
Task process log source information, then identified, to identify goal task title and goal task call parameter；It connects
By each in the goal task title and goal task call parameter and preset front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter in task record carry out uniformity comparison；When the front and back sequence task executes
Preamble task names and preceding sequence task call parameter and the goal task title in the record of first task present in inventory
And goal task call parameter it is consistent when, by the first task record in corresponding execute instruction of rear sequence task be sent to
Second task management platform.In this way, can across task management platform realize the real-time, efficient, accurate of task data
Control.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, computer, clothes
Business device, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of task data management-control method based on hadoop, which is characterized in that the method includes the steps:
Obtain the source information of the task process log of first task management platform；
The source information of the task process log is identified, identifies goal task title and goal task call parameter；
By the goal task title and goal task call parameter with it is each in preset front and back sequence task implementation list
Preamble task names and preceding sequence task call parameter in task record carry out uniformity comparison；
Preamble task names and preceding sequence task in the record of the first task present in the front and back sequence task implementation list must
When wanting parameter with the goal task title and consistent goal task call parameter, by the postorder in first task record
Corresponding execute instruction of task is sent to the second task management platform.
2. the task data management-control method based on hadoop as described in claim 1, which is characterized in that described to the task
The step of source information of process log is identified, identifies goal task title and goal task call parameter include:
The source information of the task process log is converted into log text information；
Sentence segmentation is carried out to the log text information, obtains at least one experience table sentence；
Text identification is carried out to each experience table sentence, identifies that the goal task title and goal task are necessary
Parameter.
3. the task data management-control method based on hadoop as claimed in claim 2, which is characterized in that described to appoint to each
Business execution records the step of sentence carries out text identification
Each experience table sentence is identified according to preset goal task name keys, thus find out including
The first task of the goal task title executes record sentence；
It executes record sentence to the first task according to preset call parameter format to identify, to identify the mesh
Mark the corresponding goal task call parameter of task names.
4. the task data management-control method based on hadoop as claimed in claim 3, which is characterized in that the call parameter packet
Time and operating status are included, the call parameter format includes time format and operating status format.
5. a kind of task data control device based on hadoop, which is characterized in that the task data pipe based on hadoop
Controlling device includes:
Module is obtained, the source information of the task process log for obtaining first task management platform；
Identification module identifies goal task title and target for identifying to the source information of the task process log
Task call parameter；
Judgment module, for executing the goal task title and goal task call parameter and preset front and back sequence task
The preamble task names and preceding sequence task call parameter progress uniformity comparison in each task record in inventory；
Sending module, for working as the preamble task names in the record of first task present in the front and back sequence task implementation list
When with preceding sequence task call parameter with the goal task title and consistent goal task call parameter, by the first task
Corresponding execute instruction of rear sequence task in record is sent to the second task management platform.
6. the task data control device based on hadoop as claimed in claim 5, which is characterized in that the identification module is also
For:
The source information of the task process log is converted into log text information；
Sentence segmentation is carried out to the log text information, obtains at least one experience table sentence；
Text identification is carried out to each experience table sentence, identifies that the goal task title and goal task are necessary
Parameter.
7. the task data control device based on hadoop as claimed in claim 6, which is characterized in that the identification module is also
For:
Each experience table sentence is identified according to preset goal task name keys, thus find out including
The first task of the goal task title executes record sentence；
It executes record sentence to the first task according to preset call parameter format to identify, to identify the mesh
Mark the corresponding goal task call parameter of task names.
8. the task data control device based on hadoop as claimed in claim 7, which is characterized in that the call parameter packet
Time and operating status are included, the call parameter format includes time format and operating status format.
9. a kind of computer equipment, which is characterized in that the computer equipment includes memory, processor, on the memory
It is stored with the computer program that can be run on the processor, is realized such as when the computer program is executed by the processor
The step of claim 1-4 described in any item task data management-control methods based on hadoop.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer journey
Sequence, the computer program can be executed by least one processor, so that at least one described processor executes such as claim
The step of task data management-control method described in any one of 1-4 based on hadoop.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754460.6A CN110490451A (en) | 2019-08-15 | 2019-08-15 | Task data management-control method, device and computer equipment based on hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754460.6A CN110490451A (en) | 2019-08-15 | 2019-08-15 | Task data management-control method, device and computer equipment based on hadoop |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110490451A true CN110490451A (en) | 2019-11-22 |
Family
ID=68551360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910754460.6A Pending CN110490451A (en) | 2019-08-15 | 2019-08-15 | Task data management-control method, device and computer equipment based on hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110490451A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279213A (en) * | 2015-03-13 | 2016-01-27 | 中国移动通信集团广东有限公司 | Retrieval device and retrieval method for log database |
| CN108710532A (en) * | 2018-05-21 | 2018-10-26 | 平安科技（深圳）有限公司 | Across dependence implementation method, device, equipment and the storage medium of dispatching platform |
| CN110069572A (en) * | 2019-03-19 | 2019-07-30 | 深圳壹账通智能科技有限公司 | HIVE method for scheduling task, device, equipment and storage medium based on big data platform |
-
2019
- 2019-08-15 CN CN201910754460.6A patent/CN110490451A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279213A (en) * | 2015-03-13 | 2016-01-27 | 中国移动通信集团广东有限公司 | Retrieval device and retrieval method for log database |
| CN108710532A (en) * | 2018-05-21 | 2018-10-26 | 平安科技（深圳）有限公司 | Across dependence implementation method, device, equipment and the storage medium of dispatching platform |
| CN110069572A (en) * | 2019-03-19 | 2019-07-30 | 深圳壹账通智能科技有限公司 | HIVE method for scheduling task, device, equipment and storage medium based on big data platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107844634B (en) | Modeling method of multivariate general model platform, electronic equipment and computer readable storage medium | |
| CN109684047A (en) | Event-handling method, device, equipment and computer storage medium | |
| CN109936621B (en) | Information security multi-page message pushing method, device, equipment and storage medium | |
| CN108768929A (en) | The analytic method and storage medium of electronic device, reference feedback message | |
| CN110046146A (en) | The monitoring method and device of industrial equipment based on mobile edge calculations | |
| CN111342992B (en) | Method and system for processing equipment information change record | |
| CN108681504A (en) | Automated testing method, test server and computer readable storage medium | |
| CN110069925B (en) | Software monitoring method, system and computer readable storage medium | |
| CN107844468A (en) | The cross-page recognition methods of form data, electronic equipment and computer-readable recording medium | |
| CN109446515A (en) | Group information analysis method, electronic device and computer readable storage medium | |
| CN107357721B (en) | Method and device for testing system | |
| CN111580948A (en) | Task scheduling method and device and computer equipment | |
| CN110363222A (en) | Picture mask method, device, computer equipment and storage medium for model training | |
| CN107766512B (en) | Log data storage method and log data storage system | |
| CN109284331A (en) | Accreditation information acquisition method, terminal device and medium based on business datum resource | |
| CN110490451A (en) | Task data management-control method, device and computer equipment based on hadoop | |
| CN112634025A (en) | Wind control rule generation method, device, equipment and computer readable storage medium | |
| CN110515792A (en) | Monitoring method, device and computer equipment based on web edition task management platform | |
| CN109409793B (en) | Equipment full life cycle management method and related device | |
| CN110502427A (en) | Code readability inspection method, device and server | |
| CN110502538A (en) | Label of drawing a portrait generates method, system, equipment and the storage medium of logical mappings | |
| CN114238507A (en) | Data synchronization method and device based on multiple databases | |
| CN113901093A (en) | Service call log relation analysis method and system based on memory cache | |
| CN113434281A (en) | Equipment scheduling method and cloud platform | |
| CN112817953A (en) | Data verification method and device, computer equipment and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |