CN109697112A - Distributed intensive one-stop operating system and implementation method - Google Patents
Distributed intensive one-stop operating system and implementation method Download PDFInfo
- Publication number
- CN109697112A CN109697112A CN201811582185.6A CN201811582185A CN109697112A CN 109697112 A CN109697112 A CN 109697112A CN 201811582185 A CN201811582185 A CN 201811582185A CN 109697112 A CN109697112 A CN 109697112A
- Authority
- CN
- China
- Prior art keywords
- node
- module
- information
- job
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/524—Deadlock detection or avoidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5011—Pool
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/508—Monitor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Debugging And Monitoring (AREA)
- Multi Processors (AREA)
- Hardware Redundancy (AREA)
Abstract
This application involves a kind of intensive one-stop operating system of distribution, implementation method, computer equipment and storage mediums, and wherein the system includes: the operational order that client unit is used to obtain user；Service unit is used to provide data manipulation and the corresponding job instruction of response according to the operational order of the user；Execution unit is for being scheduled jobs node according to the operational order of the user；Storage unit is used for configuration information, resource information and the job information of memory node；Monitoring unit is used to monitor the resource information, job information and job logging of node.The present invention realizes so that need to only pay close attention to the realization of business for operation developer, can be in front page layout unit visualized management using network operator for operation, monitoring entire group operation situation includes the distributed type assemblies operation under clustered node monitoring resource, monitoring operation, log monitoring etc. are truly accomplished one-stop integrated management and monitored on a large scale.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of intensive one-stop operating system of distribution, reality
Existing method, computer equipment and storage medium.
Background technique
Currently, many business processings have been difficult again real with the growth of the expansion of business, data volume and system scale
When processing and analysis.Data processing and analysis under line become a part of system function, in the feelings of magnanimity business and data
Under condition, original single node is based on operating system such as Linux, and the crontab of AIX, HP-UX has carried out job scheduling not
It is able to satisfy the demand of business.
In the conventional technology, the Open Framework Quartz based on java language is widely used, and well and
Spring frame is integrated, and Cheng Liao Spring-quartz and realizes the cluster mode based on database.But due to original
Database design under Quartz cluster mode is excessively complicated, and there are CLOB, physics external key etc. is not suitable in high concurrent, height
The distributed MYSQL fragment of performance, under the system architecture for dividing table to design, and to the distribution of cluster JOB, equilibrium, the monitoring of resource,
The shortages such as unified log, instruction control are supported and are realized.
Summary of the invention
Based on this, it is necessary in view of the above technical problems, provide a kind of intensive one-stop operating system of distribution, realization
Method, computer equipment and storage medium.
A kind of intensive one-stop operating system of distribution, the system comprises:
Client unit, the client unit are used to obtain the operational order of user；
Service unit, the service unit are used to provide data manipulation and response phase according to the operational order of the user
The job instruction answered；
Execution unit, the execution unit is for being scheduled jobs node according to the operational order of the user；
Storage unit, the storage unit are used for configuration information, resource information and the job information of memory node；
Monitoring unit, the monitoring unit are used to monitor the resource information, job information and job logging of node.
The service unit includes: in one of the embodiments,
Operation increases module newly, and the operation increases module newly and is used to increase operation newly, stores data in MYSQL memory module,
And it is instructed to execution unit collection pocket transmission is newly-increased；
Operation modified module, the operation modified module are modified for operation, store data in MYSQL memory module,
And it modifies and instructs to execution unit collection pocket transmission；
Operation removing module, the operation removing module are deleted for operation, store data in MYSQL memory module,
And it deletes and instructs to execution unit collection pocket transmission；
Operation enquiry module, the operation enquiry module is for operation inquiry, account of the history in operation configuring condition, execution
Inquiry, each execution unit node of cluster operation load ready situation, operation group operation node distribution situation and
Job logging inquiry；
First communication module, the first communication module are used to broadcast all works by the watch mechanism of Zookeeper
Industry node；
Operation ignition module, the operation ignition module are used for the operation being immediately performed in ready；
Job interruption module, the job interruption module is for interrupting the operation in being carrying out；
Operation recovery module, the operation recovery module are used to restore the operation in halted state.
The execution unit includes: in one of the embodiments,
Second communication module, the second communication module are used to receive the dispatch command of service unit；
Single operation module, the single operation module are used to configure without preparatory client, do not form plan operation, needle
To specific business scenario, client temporary definition job parameter information, and operation cluster is transferred to disposably to be executed；
Timing operation module, executive plan of the timing operation module for client configuration operation, job parameter letter
Breath, jobs node load automatically on startup or dynamically add operation in running；
Resident operation module, the resident operation module is for configuring the plan that will not be executed automatically；
Resource reporting module, the resource reporting module for timing by the memory of node, operation thread pool service condition,
Data source thread pool service condition is written in REDIS caching；
Information reporting module, the information reporting module are used for by the ready operation of the node, by node dimension
In operation dimension write-in REDIS caching；
Reporting module in execution, after reporting module is for jobs node starting in the execution, periodically by the execution of node
In middle job information write-in REDIS caching.
The storage unit includes: in one of the embodiments,
MYSQL memory module, the MYSQL memory module is used to store the configuration information of operation and the execution of operation is gone through
History information；
REDIS memory module, the resource that the REDIS memory module is used to cache operation clustered node by REDIS are believed
The job information of breath and node；
Zookeeper memory module, the Zookeeper memory module are used to store the lock information of active job,
The lock information includes operation essential information, executes node, Starting Executing Time and operator.
The monitoring unit includes: in one of the embodiments,
Monitoring resource module, the monitoring resource module are used for the resource information by node active reporting, monitoring cluster
The service condition of the resources such as the memory and thread of node；
Monitoring operation module, the monitoring operation module is for monitoring resident operation either with or without executing, conventional work
Industry executes the time either with or without the maximum execution duration for being more than operation configuration；
Log monitoring module, the log monitoring module are used to monitor the output of job logging.
A kind of intensive one-stop operation implementation method of distribution, which comprises
The operational order of user is obtained by client unit；
According to the operation start instruction jobs node of the user, and load the grouping operation of the jobs node；
The node profile for obtaining storage unit storage, initializes the jobs node；
The jobs node is added scheduling cluster and is established by monitoring unit and is monitored；
The jobs node is scheduled according to the operational order of the user by execution unit.
The method also includes the operational orders according to the user to the jobs node in one of the embodiments,
The step of carrying out Automatic dispatching:
When operation timer is to after putting, the different nodes of the same operation start simultaneously at igniting；
Distributed lock based on zookeeper seizes lock immediately；
It seizes successful node then to start to execute the service logic of operation, the node of failure then abandons executing the business of operation
Logic；
Successful node is locked the information of operation, nodal information and the temporal information for starting to execute are recorded in lock simultaneously
In appearance；
Implementing result and deadline are recorded after the completion of executing and is write information into MYSQL memory module, and discharge lock.
The method also includes the operational orders according to the user to the jobs node in one of the embodiments,
The step of being dispatched manually:
User is by viewing ready operation in client modules；
By network access service unit, the virtual condition of the operation is detected by the service unit；
Firing command is sent to group operation node；
Each group operation node starts igniting after identifying the firing command and makees after receiving the firing command
Industry；
Execute described the step of Automatic dispatching is carried out to jobs node.
A kind of computer equipment can be run on a memory and on a processor including memory, processor and storage
The step of computer program, the processor realizes above-mentioned any one method when executing the computer program.
A kind of computer readable storage medium, is stored thereon with computer program, and the computer program is held by processor
The step of above-mentioned any one method is realized when row.
The intensive one-stop operational method of above-mentioned distribution, device, computer equipment and storage medium, pass through client list
Member is for obtaining the operational order of user；Service unit is used to provide data manipulation and sound according to the operational order of the user
Answer corresponding job instruction；Execution unit is for being scheduled jobs node according to the operational order of the user；Storage is single
Member is used for configuration information, resource information and the job information of memory node；Monitoring unit be used for monitor node resource information,
Job information and job logging.The present invention realizes so that the realization of business need to be only paid close attention to for operation developer, for making
Industry can be in front page layout unit visualized management using network operator, and monitoring entire group operation situation includes clustered node resource
The distribution under one-stop integrated management and monitoring on a large scale is truly accomplished in monitoring, monitoring operation, log monitoring etc.
Group operation.
Detailed description of the invention
Fig. 1 is the structural block diagram of distributed intensive one-stop operating system in one embodiment；
Fig. 2 is the structural block diagram of service unit in distributed intensive one-stop operating system in one embodiment；
Fig. 3 is the structural block diagram of execution unit in distributed intensive one-stop operating system in one embodiment；
Fig. 4 is the structural block diagram of storage unit in distributed intensive one-stop operating system in one embodiment；
Fig. 5 is the structural block diagram of monitoring unit in distributed intensive one-stop operating system in one embodiment；
Fig. 6 is the flow diagram of distributed intensive one-stop operation implementation method in one embodiment；
Fig. 7 is the flow diagram of distributed intensive one-stop operation implementation method in another embodiment；
Fig. 8 is the flow diagram of distributed intensive one-stop operation implementation method in further embodiment；
Fig. 9 is the work relationship flow chart of the client unit and service unit and storage unit in one embodiment；
Figure 10 is the work flow diagram of the execution unit in one embodiment；
Figure 11 is the internal structure chart of computer equipment in one embodiment.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood
The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, and
It is not used in restriction the application.
In one embodiment, as shown in Figure 1, providing a kind of intensive one-stop operating system 100 of distribution, this is
System includes:
Client unit 101, for obtaining the operational order of user；
Service unit 102 provides data manipulation and the corresponding job instruction of response for instruction depending on the user's operation；
Execution unit 103, is scheduled jobs node for instructing depending on the user's operation；
Storage unit 104, for the configuration information of memory node, resource information and job information；
Monitoring unit 105, for monitoring the resource information, job information and job logging of node.
In the present embodiment, it is used to obtain the operational order of user by client unit 101.Specifically, it can be PC
End is also possible to support the mobile terminal of HTML5, client is mainly concentrative operation configuration, monitoring operation, operation by browser
Operation is realized and provides one-stop service management and monitoring page function, by being configured to operation to monitoring.
Service unit 102 provides data manipulation and the corresponding job instruction of response for instruction depending on the user's operation,
Data manipulation is provided, job instruction, monitoring and inquiry, the abilities such as log query are for client use.
Execution unit 103 is for being scheduled jobs node according to the operational order of the user.Specifically, Ke Yitong
The watch event mechanism based on Zookeeper is crossed, the dispatch command of front end is passed into each jobs node, and complete
Command adapted thereto movement, correspondence are immediately performed movement, and operator can voluntarily select any one of cluster node or give
Clustered node executes immediately.
Storage unit 104 is used for configuration information, resource information and the job information of memory node, such as will configure operation
Executive plan, execute parameter, ownership grouping be stored in MYSQL.
Monitoring unit 105 is used to monitor the resource information, job information and job logging of node, specifically, Ke Yicong
The dimension of node monitors the resource and JOB situation of each node, can also monitor the JOB in clustered node with the dimension of JOB
Distribution situation, the data of monitoring by each node active reporting, and be written caching REDIS in.
The present embodiment realizes so that need to only pay close attention to the realization of business for operation developer, uses network operator for operation
Can be in front page layout unit visualized management, monitoring entire group operation situation includes clustered node monitoring resource, operation prison
The distributed type assemblies operation under one-stop integrated management and monitoring on a large scale is truly accomplished in control, log monitoring etc..
In one embodiment, as shown in Fig. 2, providing a kind of intensive one-stop operating system of distribution, wherein should
Service unit 200 in system includes:
Operation increases module 201 newly, for increasing operation newly, stores data in MYSQL memory module, and to execution unit collection
The newly-increased instruction of pocket transmission；
Operation modified module 202 is modified for operation, stores data in MYSQL memory module, and to execution unit collection
Pocket transmission modification instruction；
Operation removing module 203 is deleted for operation, stores data in MYSQL memory module, and to execution unit collection
Instruction is deleted in pocket transmission；
Operation enquiry module 204, it is each for operation inquiry, account of the history inquiry, cluster in operation configuring condition, execution
The operation of execution unit node loads ready situation, operation is inquired in the distribution situation and job logging of group operation node；
First communication module 205, for broadcasting all jobs nodes by the watch mechanism of Zookeeper；
Operation ignition module 206, the operation for being immediately performed in ready；
Job interruption module 207, for interrupting the operation in being carrying out；
Operation recovery module 208, for restoring the operation in halted state.
In the present embodiment, service unit operation is newly-increased indicates to increase an operation newly, stores data in MYSQL, and to
Execution unit (jobs node) collects the newly-increased instruction of pocket transmission, and operation modification, operation deletion is also such.Service unit provides standard
RESTFULL style http protocol for client call.
Operation inquiry includes operation inquiry, account of the history inquiry, each execution unit of cluster in operation configuring condition, execution
The JOB of node loads ready situation, JOB in inquiry class services such as distribution situation, the job logging inquiries of group operation node.
Command communication module, service unit node pass through the watch mechanism of Zookeeper, and transmitting job instruction information arrives
Jobs node cluster functions as broadcast and is notified of all jobs nodes.
Operation ignition module be used for be immediately performed one it is ready in JOB.
Job interruption module be used for interrupt one be carrying out in JOB, the next jobs node of new cluster mode can
To have the executive capability of multiple JOB, one process of multiple JOB program sharings.
Operation stopping is different from job interruption, operation stop refer to the production plan it is no longer valid (have been started up this
Subjob has executed as usual).
Operation recovery module is for restoring the operation that a production plan is in halted state.
In the present embodiment, offer data manipulation is realized, job instruction, monitoring and inquiry, the abilities such as log query are for visitor
Family end uses.
In one embodiment, as shown in figure 3, providing a kind of intensive one-stop operating system of distribution, wherein should
Execution unit 300 in system includes:
Second communication module 301, for receiving the dispatch command of service unit；
Single operation module 302 does not form plan operation, for specific business field for configuring without preparatory client
Scape, client temporary definition job parameter information, and operation cluster is transferred to disposably to be executed；
Timing operation module, for the executive plan of client configuration operation, job parameter information, jobs node is starting
Operation is dynamically added in Shi Zidong load or operation；
Resident operation module 304, for configuring the plan that will not be executed automatically；
Resource reporting module 305, for timing by the memory of node, operation thread pool service condition, data source thread pool
Service condition is written in REDIS caching；
Information reporting module 306, for being written by node dimension and operation dimension by the ready operation of the node
In REDIS caching；
Periodically job information in the execution of node is written after jobs node starting for reporting module 307 in execution
In REDIS caching.
In the present embodiment, execution unit refers to the ability that a node of job scheduling cluster has.
Specifically, command communication module is used for jobs node on startup, and the automatic Zookeeper that connects simultaneously has registered
Watch event, for receiving the dispatch command of service unit.
Single operation refers to be configured without preparatory client, plan operation is not formed, for specific business scenario, client
The information such as temporary definition job parameter, and operation cluster is transferred to disposably to be executed.
Executive plan of the timing operation module for client configuration operation, the information such as job parameter, jobs node are opening
It is loaded automatically when dynamic or dynamically adds operation in running.
Resident operation is usually to configure a plan hardly executed automatically, is such as defined as on December in 2099 31
0 point execution, when needing to be implemented, lighted a fire by client, interrupted when needing to stop.
Node resource reports refer to jobs node starting after, periodically by the memory of the node, JOB thread pool service condition,
Data source thread pool service condition is written in REDIS caching.
After JOB information reporting refers to jobs node starting, periodically by the ready JOB of the node, by node dimension
In (usually IP:PORT) and JOB dimension (JOB grouping: JOBID) write-in REDIS caching.
It is reported in execution after referring to jobs node starting, periodically by JOB information write-in REDIS caching in the execution of the node
In.
In the present embodiment, the watch event mechanism based on Zookeeper, the dispatch command of front end is passed to often
One jobs node, and command adapted thereto movement is completed, corresponding to be immediately performed movement, operator can voluntarily select appointing in cluster
What node is given clustered node and is executed immediately.
In one embodiment, as shown in figure 4, providing a kind of intensive one-stop operating system of distribution, wherein should
Storage unit 400 in system includes:
MYSQL memory module 401, for storing the configuration information of operation and the execution historical information of operation；
REDIS memory module 402 caches the resource information of operation clustered node and the operation letter of node by REDIS
Breath；
Zookeeper memory module 403, for storing the lock information of active job, the lock information includes operation
Essential information executes node, Starting Executing Time and operator.
Specifically, MYSQL memory module 401 be mainly used for store JOB configuration information, the execution historical information of operation,
REDIS memory module 402 is mainly with the resource information of REDIS caching operation clustered node, the JOB information of each node.
Zookeeper memory module 403 stores the lock information of active job, and lock content includes operation essential information, executes section
Point (ip:port), Starting Executing Time, operator (system executes automatically or manual firing).
In the present embodiment, operation allocation list has been redesigned based under the single node memorymodel of Quartz in bottom
(being only single table) introduces Zookeeper, and the bridge and distributed job communicated under cluster mode is done using Zookeeper
Lock, defines the concept of task group, the node in cluster is allowed to dispose JOB and configuration according to business scenario needs
Obstructed resource (memory and thread) so that resource performs to maximization, under new cluster mode in addition to can allow clustered node with
For machine competition lock to execute outside JOB, user voluntarily can also specify node to execute, further perfect more scenes need
It asks.
In one embodiment, as shown in figure 5, providing a kind of intensive one-stop operating system of distribution, wherein should
Monitoring unit 500 in system includes:
Monitoring resource module 501, for by the resource information of node active reporting, monitor clustered node memory and
The service condition of the resources such as thread；
Monitoring operation module 502, for monitoring resident operation either with or without executing, the conventional job execution time has not
It has more than the maximum of operation configuration and executes duration；
Log monitoring module 503, for monitoring the output of job logging.
Specifically, monitoring resource module 501 is mainly the resource information by node active reporting, monitors each node of cluster
The resources such as memory, thread service condition.Monitoring operation module 502 is mainly to monitor resident operation either with or without executing,
The conventional job execution time is either with or without the maximum execution duration for being more than operation configuration.Log monitoring module 503 is made for monitoring
The output of industry log, can define the keyword of log, and form alarm.On the one hand the monitoring information of all monitoring units has clothes
The form of business is supplied to client and is shown, on the one hand can be by the push in the form of connection alarm center progress wechat or short message
Alarm.
In the present embodiment, the resource and JOB situation of each node can be monitored from the dimension of node, it can also be with
The dimension of JOB monitors the JOB in the distribution situation of clustered node, and the data of monitoring are write by each node active reporting
Enter to cache in REDIS.The monitoring carried out to jobs node in cluster and specific operation by a variety of dimensions is realized, to operation
The execution time carried out early warning.
In one embodiment, as shown in fig. 6, providing a kind of intensive one-stop operation implementation method of distribution, with
Method is applied in the intensive one-stop operating system of distribution in above-described embodiment, this method comprises:
Step 602, the operational order of user is obtained by client unit；
Step 604, initiating task node, and the grouping operation of load operations node are instructed depending on the user's operation；
Step 606, the node profile for obtaining storage unit storage, initializes jobs node；
Step 608, jobs node is added scheduling cluster and is established by monitoring unit and monitored；
Step 610, it is instructed depending on the user's operation by execution unit and jobs node is scheduled.
Specifically, in conjunction with Fig. 9,10, client passes through the end PC browser or mobile terminal browser, access operation configuration page
Face carries out newly-increased operation to operation, modification operation, deletes Job Operations, by http protocol access operation data manipulation service,
After receiving request, MYSQL is write data into after a series of basis verifying, while modifying the monitoring of Zookeeper
The content of node, content include all information and instruction type (Add, Update, Delete) of operation.
After actuation, designated packet GROUP can be read to jobs node from MYSQL automatically by loading this node grouping JOB
JOB, GROUP can be with one or more, and initializing JOB keeps its ready, while initializing Zookeeper and connecting and build
Vertical node is monitored, for receiving dispatch command.After initializing successfully, for the newly-increased of previous step, instruction is deleted in modification, is made
Industry node will will receive instruction, and identify instruction, and judge whether the JOB grouping of this instruction is that this node is grouped it
As soon as be carry out JOB increase modification operation delete operation make its again it is ready from ready remove operation in the works, otherwise neglect
Slightly this instructs.The initialization of the basic configuration of JOB and scheduling node has just been completed substantially in this way, by execution unit according to
The operational order of user is scheduled jobs node.
It is understood that can also be come on each node of inquiry job cluster by client transactions monitoring module
JOB situation, some JOB can also be inquired in the distribution situation of clustered node, monitor entire operation by 2 dimensions
Deployment scenario.In scheduling process, the handling situations being carrying out, including operation are checked by JOB query service in executing
It executes, when executes in which node, be the information such as automatic execution or manual execution, while real-time by log services
View the execution journal of the operation.It can also check that the history of operation executes feelings after the completion of execution by historical query service
Condition and its running log.
In the present embodiment, the executive plan of configuration operation is realized, executes parameter, ownership grouping, and data are stored
In MYSQL.In addition, each node of multi-node cluster mode can load the JOB of one or more packets, jobs node quantity
Self-defining can be needed according to application.
It in one embodiment, should as shown in fig. 7, providing a kind of intensive one-stop operation implementation method of distribution
Method, which further includes the steps that instructing depending on the user's operation, carries out Automatic dispatching to jobs node:
Step 702, when operation timer is to after putting, the different nodes of the same operation start simultaneously at igniting；
Step 704, the distributed lock based on Zookeeper seizes lock immediately；
Step 706, it seizes successful node then to start to execute the service logic of operation, the node of failure is then abandoned executing work
The service logic of industry；
Step 708, successful node is locked simultaneously for the information of operation, nodal information and the temporal information note for starting execution
Record is in lock content；
Step 710, implementing result and deadline are recorded after the completion of executing and are write information into MYSQL memory module,
And discharge lock.
Specifically, in conjunction with Figure 10, the Automatic dispatching of operation, jobs node in the above-described embodiments initialize successfully it
Afterwards, when operation timer is to after putting, the different nodes of the same operation (JOBID) are (if node on different machines, takes
Between it is synchronous) start simultaneously at igniting, the distributed lock at the beginning of execution based on Zookeeper seizes lock immediately, seizes successful node
Then start the service logic of execution JOB, the node of failure then abandons this execution, waits executive plan next time, and lock is successful
Node is simultaneously by the information of JOB, and nodal information (ip and port), the information such as Starting Executing Time, which are recorded in, locks in content.It holds
After the completion of row (success or failure), records implementing result and deadline and write information into the job history table of MYSQL, and
Release lock.
It in one embodiment, should as shown in figure 8, providing a kind of intensive one-stop operation implementation method of distribution
Method further includes the steps that instructing depending on the user's operation dispatches the jobs node manually:
Step 802, user is by viewing ready operation in client modules；
Step 804, by network access service unit, the virtual condition of operation is detected by service unit；
Step 806, firing command is sent to group operation node；
Step 808, each group operation node starts igniting after identifying firing command and makees after receiving firing command
Industry；
Step 810, the step of Automatic dispatching is carried out to jobs node is executed.
Specifically, in conjunction with Figure 10, the manual scheduling of operation, the premise for dispatching certain operation by hand is that the operation is collecting
Ready in group's jobs node, user carries out being immediately performed operation, grasp by viewing some ready operation in client
Make through the operation service of network access instruction, server-side (has the node ready operation, simultaneously in the virtual condition for detecting the operation
Without in execution), then sending action instruction FIRE arrive clustered node, each group operation node receive FIRE instruct after, and
Configuration-direct process is similar, after identifying FIRE instruction, starts operation of lighting a fire, and then goes to and Automatic dispatching above igniting
The process of beginning is the same, executes complete JOB process.
It should be understood that although each step in the flow chart of Fig. 6-10 is successively shown according to the instruction of arrow,
It is these steps is not that the inevitable sequence according to arrow instruction successively executes.Unless expressly stating otherwise herein, these steps
There is no stringent sequences to limit for rapid execution, these steps can execute in other order.Moreover, in Fig. 6-10 at least
A part of step may include that perhaps these sub-steps of multiple stages or stage are not necessarily in same a period of time to multiple sub-steps
Quarter executes completion, but can execute at different times, the execution in these sub-steps or stage be sequentially also not necessarily according to
Secondary progress, but in turn or can replace at least part of the sub-step or stage of other steps or other steps
Ground executes.
In one embodiment, a kind of computer equipment is provided, internal structure chart can be as shown in figure 11.The calculating
Machine equipment includes processor, memory and the network interface connected by system bus.Wherein, the processing of the computer equipment
Device is for providing calculating and control ability.The memory of the computer equipment includes non-volatile memory medium, built-in storage.It should
Non-volatile memory medium is stored with operating system, computer program and database.The built-in storage is non-volatile memories Jie
The operation of operating system and computer program in matter provides environment.The network interface of the computer equipment is used for and external end
End passes through network connection communication.To realize a kind of intensive one-stop operation of distribution when the computer program is executed by processor
Implementation method.
It will be understood by those skilled in the art that structure shown in Figure 11, only part relevant to application scheme
The block diagram of structure, does not constitute the restriction for the computer equipment being applied thereon to application scheme, and specific computer is set
Standby may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory
And the computer program that can be run on a processor, processor are realized when executing computer program in above each embodiment of the method
The step of.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
The step in above each embodiment of the method is realized when machine program is executed by processor.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment
In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance
Shield all should be considered as described in this specification.
The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously
It cannot therefore be construed as limiting the scope of the patent.It should be pointed out that coming for those of ordinary skill in the art
It says, without departing from the concept of this application, various modifications and improvements can be made, these belong to the protection of the application
Range.Therefore, the scope of protection shall be subject to the appended claims for the application patent.
Claims (10)
1. a kind of intensive one-stop operating system of distribution, the system comprises:
Client unit, the client unit are used to obtain the operational order of user；
Service unit, the service unit is used to provide data manipulation according to the operational order of the user and response is corresponding
Job instruction；
Execution unit, the execution unit is for being scheduled jobs node according to the operational order of the user；
Storage unit, the storage unit are used for configuration information, resource information and the job information of memory node；
Monitoring unit, the monitoring unit are used to monitor the resource information, job information and job logging of node.
2. the intensive one-stop operating system of distribution according to claim 1, which is characterized in that the service unit packet
It includes:
Operation increases module newly, and the operation increases module newly for increasing operation newly, stores data in MYSQL memory module, and to
The newly-increased instruction of execution unit collection pocket transmission；
Operation modified module, the operation modified module are modified for operation, store data in MYSQL memory module, and to
Execution unit collection pocket transmission modification instruction；
Operation removing module, the operation removing module are deleted for operation, store data in MYSQL memory module, and to
Instruction is deleted in execution unit collection pocket transmission；
Operation enquiry module, the operation enquiry module are looked into for operation inquiry, account of the history in operation configuring condition, execution
Ask, the operation of each execution unit node of cluster loads ready situation, operation group operation node distribution situation and work
Industry log query；
First communication module, the first communication module are used to broadcast all operation sections by the watch mechanism of Zookeeper
Point；
Operation ignition module, the operation ignition module are used for the operation being immediately performed in ready；
Job interruption module, the job interruption module is for interrupting the operation in being carrying out；
Operation recovery module, the operation recovery module are used to restore the operation in halted state.
3. the intensive one-stop operating system of distribution according to claim 2, which is characterized in that the execution unit packet
It includes:
Second communication module, the second communication module are used to receive the dispatch command of service unit；
Single operation module, the single operation module are used to configure without preparatory client, plan operation are not formed, for tool
Body business scenario, client temporary definition job parameter information, and operation cluster is transferred to disposably to be executed；
Timing operation module, executive plan of the timing operation module for client configuration operation, job parameter information are made
Industry node loads automatically on startup or dynamically adds operation in running；
Resident operation module, the resident operation module is for configuring the plan that will not be executed automatically；
Resource reporting module, the resource reporting module is for periodically by the memory of node, operation thread pool service condition, data
Source thread pool service condition is written in REDIS caching；
Information reporting module, the information reporting module are used for by the ready operation of the node, by node dimension and work
Industry dimension is written in REDIS caching；
Reporting module in execution will periodically be made after reporting module is for jobs node starting in the execution in the execution of node
Industry information is written in REDIS caching.
4. the intensive one-stop operating system of distribution according to claim 3, which is characterized in that the storage unit packet
It includes:
MYSQL memory module, the MYSQL memory module are used to store the configuration information of operation and the execution history letter of operation
Breath；
REDIS memory module, the REDIS memory module be used for by REDIS caching operation clustered node resource information with
And the job information of node；
Zookeeper memory module, the Zookeeper memory module is used to store the lock information of active job, described
Lock information includes operation essential information, executes node, Starting Executing Time and operator.
5. the intensive one-stop operating system of distribution according to claim 4, which is characterized in that the monitoring unit packet
It includes:
Monitoring resource module, the monitoring resource module are used for the resource information by node active reporting, monitoring clustered node
Memory and the resources such as thread service condition；
Monitoring operation module, the monitoring operation module for monitoring resident operation either with or without executing, hold by conventional operation
The row time is either with or without the maximum execution duration for being more than operation configuration；
Log monitoring module, the log monitoring module are used to monitor the output of job logging.
6. a kind of intensive one-stop operation implementation method of distribution, which is characterized in that the method is applied to such as claim
Include: in the intensive one-stop operating system of the described in any item distributions of 1-5
The operational order of user is obtained by client unit；
According to the operation start instruction jobs node of the user, and load the grouping operation of the jobs node；
The node profile for obtaining storage unit storage, initializes the jobs node；
The jobs node is added scheduling cluster and is established by monitoring unit and is monitored；
The jobs node is scheduled according to the operational order of the user by execution unit.
7. the intensive one-stop operation implementation method of distribution according to claim 6, which is characterized in that the method is also
Include the steps that carrying out Automatic dispatching to the jobs node according to the operational order of the user:
When operation timer is to after putting, the different nodes of the same operation start simultaneously at igniting；
Distributed lock based on zookeeper seizes lock immediately；
It seizes successful node then to start to execute the service logic of operation, the business that the node of failure then abandons executing operation is patrolled
Volume；
It locks successful node while the information of operation, nodal information and the temporal information for starting to execute is recorded in lock content and work as
In；
Implementing result and deadline are recorded after the completion of executing and is write information into MYSQL memory module, and discharge lock.
8. the intensive one-stop operation implementation method of distribution according to claim 7, which is characterized in that the method is also
Include the steps that dispatching the jobs node manually according to the operational order of the user:
User is by viewing ready operation in client modules；
By network access service unit, the virtual condition of the operation is detected by the service unit；
Firing command is sent to group operation node；
Each group operation node starts operation of lighting a fire after receiving the firing command after identifying the firing command；
Execute described the step of Automatic dispatching is carried out to jobs node.
9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor
Calculation machine program, which is characterized in that the processor realizes any one of claim 6 to 8 institute when executing the computer program
The step of stating method.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of method described in any one of claim 6 to 8 is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811582185.6A CN109697112B (en) | 2018-12-24 | 2018-12-24 | Distributed intensive one-stop operating system and implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811582185.6A CN109697112B (en) | 2018-12-24 | 2018-12-24 | Distributed intensive one-stop operating system and implementation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109697112A true CN109697112A (en) | 2019-04-30 |
| CN109697112B CN109697112B (en) | 2023-05-16 |
Family
ID=66231928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811582185.6A Active CN109697112B (en) | 2018-12-24 | 2018-12-24 | Distributed intensive one-stop operating system and implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109697112B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112621755A (en) * | 2020-12-18 | 2021-04-09 | 浙江工业大学 | Remote monitoring system for multi-model industrial robots |
| CN113032131A (en) * | 2021-05-26 | 2021-06-25 | 天津中新智冠信息技术有限公司 | Redis-based distributed timing scheduling system and method |
| CN113065030A (en) * | 2021-01-11 | 2021-07-02 | 上海金融期货信息技术有限公司 | Operation control system based on directed acyclic graph |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1350676A (en) * | 1998-12-22 | 2002-05-22 | 电脑联合想象公司 | System for sceduling and monitoring computer processes |
| CN105022668A (en) * | 2015-04-29 | 2015-11-04 | 腾讯科技（深圳）有限公司 | Job scheduling method and system |
| US9304663B1 (en) * | 2013-04-12 | 2016-04-05 | Groupon, Inc. | Centralized, scalable, resource monitoring system |
| US20160117194A1 (en) * | 2010-08-30 | 2016-04-28 | Adobe Systems Incorporated | Methods and apparatus for resource management cluster computing |
| CN105550029A (en) * | 2015-12-24 | 2016-05-04 | 迈普通信技术股份有限公司 | Process scheduling method and device |
| CN105791354A (en) * | 2014-12-23 | 2016-07-20 | 中兴通讯股份有限公司 | Job scheduling method and cloud scheduling server |
| CN105824868A (en) * | 2015-12-24 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data processing method and distributed type database system |
| US20180048587A1 (en) * | 2016-05-16 | 2018-02-15 | Yang Bai | Port switch service |
-
2018
- 2018-12-24 CN CN201811582185.6A patent/CN109697112B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1350676A (en) * | 1998-12-22 | 2002-05-22 | 电脑联合想象公司 | System for sceduling and monitoring computer processes |
| US20160117194A1 (en) * | 2010-08-30 | 2016-04-28 | Adobe Systems Incorporated | Methods and apparatus for resource management cluster computing |
| US9304663B1 (en) * | 2013-04-12 | 2016-04-05 | Groupon, Inc. | Centralized, scalable, resource monitoring system |
| CN105791354A (en) * | 2014-12-23 | 2016-07-20 | 中兴通讯股份有限公司 | Job scheduling method and cloud scheduling server |
| CN105022668A (en) * | 2015-04-29 | 2015-11-04 | 腾讯科技（深圳）有限公司 | Job scheduling method and system |
| CN105550029A (en) * | 2015-12-24 | 2016-05-04 | 迈普通信技术股份有限公司 | Process scheduling method and device |
| CN105824868A (en) * | 2015-12-24 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data processing method and distributed type database system |
| US20180048587A1 (en) * | 2016-05-16 | 2018-02-15 | Yang Bai | Port switch service |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112621755A (en) * | 2020-12-18 | 2021-04-09 | 浙江工业大学 | Remote monitoring system for multi-model industrial robots |
| CN112621755B (en) * | 2020-12-18 | 2022-04-19 | 浙江工业大学 | Remote monitoring system for multi-model industrial robots |
| CN113065030A (en) * | 2021-01-11 | 2021-07-02 | 上海金融期货信息技术有限公司 | Operation control system based on directed acyclic graph |
| CN113032131A (en) * | 2021-05-26 | 2021-06-25 | 天津中新智冠信息技术有限公司 | Redis-based distributed timing scheduling system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109697112B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291565B (en) | Operation and maintenance visual automatic operation platform and implementation method | |
| CN110069572B (en) | HIVE task scheduling method, device, equipment and storage medium based on big data platform | |
| US7779298B2 (en) | Distributed job manager recovery | |
| CN100481021C (en) | Method, system, and apparatus for identifying unresponsive portions of a computer program | |
| US7328213B2 (en) | Transaction processing method, transaction control apparatus and program thereof | |
| CN106406993A (en) | Timed task management method and system | |
| CN109697112A (en) | Distributed intensive one-stop operating system and implementation method | |
| CN109783214A (en) | Task schedule control system | |
| CN111125444A (en) | Big data task scheduling management method, device, equipment and storage medium | |
| CN109992354A (en) | Container-treatment, device, principal server, system and storage medium | |
| CN110990432A (en) | Device and method for synchronizing distributed cache clusters across machine rooms | |
| CN110895488B (en) | Task scheduling method and device | |
| CN111371599A (en) | Cluster disaster recovery management system based on ETCD | |
| CN110138876A (en) | Task deployment method, apparatus, equipment and platform | |
| CN113760513A (en) | Distributed task scheduling method, device, equipment and medium | |
| CN108829545B (en) | Method for realizing distributed database backup | |
| WO2020063550A1 (en) | Policy decision method, apparatus and system, and storage medium, policy decision unit and cluster | |
| CN110569113A (en) | Method and system for scheduling distributed tasks and computer readable storage medium | |
| CN114356750A (en) | Test method, test device, computer equipment and storage medium | |
| CN109597837A (en) | Storage method, querying method and the relevant device of time series data | |
| CN113157411B (en) | Celery-based reliable configurable task system and device | |
| JP2005309838A (en) | Information management system and information management method, and information management sub-system therefor | |
| CN110737526A (en) | method and device for managing timed tasks under Redis-based distributed cluster | |
| CN115357198B (en) | Mounting method and device of storage volume, storage medium and electronic equipment | |
| CN116521363A (en) | Code packaging method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |