CN113626161A - Distributed multi-user data scheduling method and system - Google Patents

Distributed multi-user data scheduling method and system Download PDF

Info

Publication number
CN113626161A
CN113626161A CN202110778053.6A CN202110778053A CN113626161A CN 113626161 A CN113626161 A CN 113626161A CN 202110778053 A CN202110778053 A CN 202110778053A CN 113626161 A CN113626161 A CN 113626161A
Authority
CN
China
Prior art keywords
data
user
task
scheduling
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110778053.6A
Other languages
Chinese (zh)
Other versions
CN113626161B (en
Inventor
张鹏
唐铭浩
林祥琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Information Engineering of CAS
Original Assignee
Institute of Information Engineering of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Information Engineering of CAS filed Critical Institute of Information Engineering of CAS
Priority to CN202110778053.6A priority Critical patent/CN113626161B/en
Publication of CN113626161A publication Critical patent/CN113626161A/en
Application granted granted Critical
Publication of CN113626161B publication Critical patent/CN113626161B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a distributed multi-user data scheduling method and a system, which comprises the steps of analyzing scheduling data sent by a user A1, acquiring the routing information of a working node B bound by a user B1, and generating a scheduling task Ti(ii) a Will schedule task TiStoring the data into a corresponding task queue; parsing scheduling tasks TiAnd sort the batch execution data scheduling tasks so that user B1 retrieves the scheduling data from the worker node B's private data queue. The invention dispersedly binds different users on the scattered data scheduling working nodes, performs data scheduling among the users, adopts a strategy of asynchronous batch classified execution, and manages the working nodes by using the management nodes, thereby improving the load capacity, stability and reliability of scheduling.

Description

Distributed multi-user data scheduling method and system
Technical Field
The present invention relates to the field of network communications, and in particular, to a distributed multi-user data scheduling method and system.
Background
The distributed scheduling of the big data plays a role in starting and stopping in the data migration process and runs through the processes of data production, exchange, consumption and the like. The implementation of scheduling in computer tasks can be triggered by depending on the timing task of an operating system, but the scheduling task runs orderly in a service scene, and the characteristics of ensuring low delay, accuracy, reliability, convenience and monitoring of data scheduling are very complex problems. Distributed scheduling is developed on the basis of single-point scheduling, and the process from single-point scheduling to distributed scheduling is a qualitative process, and has many characteristics which are not possessed by single-point scheduling, such as decentralized scheduling service, elastic expansion, multilink scheduling and multi-purpose end scheduling. In the case of distributed scheduling, how to view the status of each scheduled task and the status of the scheduled nodes is an important issue. In addition, in a multi-user service scenario, how to provide a uniform data scheduling interface and meet the requirement of user scheduling data isolation is also an important issue. Therefore, it is very difficult to design a scheduling system and a scheduling method thereof that satisfy the above requirements.
Disclosure of Invention
In order to overcome the existing difficulties, the invention provides a distributed multi-user data scheduling method and a system, which utilize sixteen modules, namely a node management module, a user management module, a topology updating module, a link detection module, a task generation module, a task execution module, a task retransmission module, a data consumption module, a task monitoring module, a task reporting module, a state acquisition module, a state reporting module, a task checking module, a state checking module, a command issuing module and a command execution module, and realize the characteristics of decentralized data scheduling service, low delay, reliability, stability, user data isolation, convenient checking of scheduling tasks, manageable scheduling service working nodes and the like.
The technical scheme of the invention comprises the following steps:
a distributed multi-user data scheduling method is suitable for a network formed by a management node and a plurality of working nodes, the management node respectively generates unique working node routing information or user ID according to the information of each working node and user information, and the method comprises the following steps:
1) the working node A periodically inquires the global topology and acquires global topology information from the management node;
2) analyzing the scheduling data sent by the user A1, acquiring the data type, the scheduling data body and the priority of the scheduling data and the user ID of the user B1, and acquiring the routing information of the working node B bound by the user B1 according to the user ID of the user B1 and the global topology information;
3) Generating a scheduling task T according to the generated data scheduling task ID, the user ID of the user B1, the routing information of the working node B, the data type and the scheduling data bodyiAnd combining the priority to schedule the task TiStoring the data into a corresponding task queue, wherein i is a task number;
4) parsing scheduling tasks TiAnd sort the batch execution data scheduling tasks so that user B1 retrieves the scheduling data from the worker node B's private data queue.
Further, the interface mode of the user a1 for sending the scheduling data includes: HTTP/HTTPS RESTful interface mode.
Further, the interface URL is: v1/send _ data/{ data _ type }, where { data _ type } represents a data type; the interface HTTP header fields include: a USER-AGENT field, a LEVEL field and a DES-USER field, wherein the USER-AGENT field is the USER ID of the USER A1, the LEVEL field is the priority of data, and the DES-USER field is the USER ID of the USER B1; the scheduling data body is stored in the http request body.
Further, the rule for generating the data scheduling task ID includes: the working node number _ event stamp _ n bit random number, wherein n is more than or equal to 1.
Further, if the scheduling data is not successfully scheduled to the working node B, the execution failure task is stored in the task retransmission queue and is executed again until the task retransmission execution is successful.
Further, recording the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task, and reporting the record to the management node.
Further, reporting the server hardware state, the service index state, the network bandwidth occupation state and the command execution state of the working node to a management node; the server hardware states include: CPU utilization rate, memory utilization rate and disk utilization rate; the service index states include: the backlog quantity of the task queue and the backlog quantity of the user data queue; the network bandwidth occupation state comprises the following steps: network egress traffic speed and network ingress traffic speed.
Further, the worker node A1 immediately executes the command to obtain the non-executed command from the management node and returns the recorded command execution status to the management node
A distributed multi-user data scheduling system is suitable for a network formed by a management node and a plurality of working nodes, and comprises:
the node management module is used for generating unique working node routing information by the management node according to the information of each working node;
the user management module is used for generating a unique user ID by the management node according to the user information;
The topology updating module is used for the working node A to inquire the global topology periodically and obtain global topology information from the management node;
the task generation module is used for analyzing the scheduling data sent by the user A1, acquiring the data type, the scheduling data body and the priority of the scheduling data and the user ID of the user B1, and acquiring the routing information of the working node B bound by the user B according to the user ID of the user B1 and the global topology information; generating a scheduling task T according to the generated data scheduling task ID, the user ID of the user B1, the routing information of the working node B, the data type and the scheduling data bodyiAnd combining the priority to schedule the task TiStoring the data into a corresponding task queue, wherein i is a task number;
a task execution module for analyzing each scheduling task TiThe routing information and classified batch execution data scheduling tasks;
and the data consumption module is used for the working node A to store the user data of the user B1 in a message middleware in a separated mode, so that the user B1 can consume the data which is dispatched to the user from the working node B1 to other users.
The system further comprises a link detection module, a task retransmission module, a task monitoring module, a task reporting module, a state acquisition module, a state reporting module, a task checking module, a state checking module, a command issuing module and a command execution module; wherein the content of the first and second substances,
The link detection module is used for inquiring the topological structure by the working node A;
the task retransmission module is used for storing the execution failure task into the task retransmission queue and re-executing the execution failure task if the scheduling data is not successfully scheduled to the working node B;
the task monitoring module is used for recording the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task;
the task reporting module is used for reporting the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task to the management node;
the state acquisition module is used for acquiring the server hardware state, the service index state, the network bandwidth occupation state and the command execution state of the working node, wherein the server hardware state comprises: CPU utilization rate, memory utilization rate and disk utilization rate; the service index states include: the backlog quantity of the task queue and the backlog quantity of the user data queue; the network bandwidth occupation state comprises the following steps: network outlet traffic speed and network inlet traffic speed;
the state reporting module is used for reporting the server hardware state, the service index state and the network bandwidth occupation state of the working node to the management node;
The task viewing module is used for viewing the information and the execution progress of each data scheduling task and the consumption information of the scheduling data;
the state viewing module is used for viewing the hardware state of the system resource, the service index state and the network bandwidth occupation state of each working node through the management node;
the command issuing module is used for issuing commands to the specified working nodes through the management nodes, wherein the commands comprise: system restarting, cache data cleaning or system updating;
and the command execution module is used for downloading the unexecuted command from the management node by the working node at regular intervals, executing the unexecuted command and recording the command execution state.
The method of the invention has the following advantages:
different users are dispersedly bound on scattered data scheduling working nodes, data scheduling among the users is carried out, a strategy of asynchronous batch classified execution is adopted, and the working nodes are managed by using a management node. Compared with single-machine data scheduling, the method greatly improves the load capacity, stability and reliability of scheduling. The invention also provides a perfect method for monitoring the state of the scheduling task, which comprises the processes of generating, executing, retransmitting and the like of the scheduling task and provides a unified data scheduling task state query service through the management node.
Drawings
FIG. 1 is a system architecture diagram of the present invention.
Fig. 2 is a flow chart of data scheduling of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The technical scheme adopted by the invention mainly comprises sixteen modules: (1) a node management module: firstly, a scheduling node is divided into a management node and a working node, and the information of the working node is registered in the management node and comprises an IP address and a service port of the working node. The management node generates an identity ID and a communication configuration for each node. The communication configuration content contains the IP address and service port of the management node and the identity ID of the working node. The working node can communicate with the management node through a communication configuration. (2) A user management module: the information of the user is registered in the management node, the user is bound with one working node, a unique user ID is generated for the user, and the unique user ID is provided for the IP address and the service port information of the working node bound by the user. (3) A topology updating module: the working nodes regularly acquire the latest topological structure from the management node, and the topological structure comprises the information of all the working nodes and the information of the users bound with the working nodes. (4) A link detection module: and the working nodes inquire the topological structure, periodically send link detection packets to other working nodes and acquire the network communication state. (5) A task generation module: the user sends data to be scheduled to other users through the bound work nodes, the work nodes can generate data scheduling tasks, the priorities of the tasks are distinguished, and the tasks are stored in task queues of different levels. (6) A task execution module: the working nodes collect the data scheduling tasks, and preferentially collect the scheduling tasks with high priority. And obtaining the link of each task from the topological structure, classifying and converging the scheduling tasks according to the links, executing the converged scheduling tasks, and scheduling the data to the working node bound by the destination user. (7) And a task retransmission module: the task which fails to be executed is supported to be retransmitted, and the reliability of data scheduling is improved. (8) A data consumption module: the working node stores the destination user data in the message middleware in an isolated manner, and the destination user can consume the data dispatched to the user from the bound working node to other users. (9) A task monitoring module: and recording events generated at each stage of the scheduling task for monitoring, wherein the events comprise events such as scheduling task generation, scheduling task execution, scheduling task retransmission, scheduling data consumption and the like. (10) A task reporting module: and each working node reports the new task monitoring record to the management node at regular time. (11) A state acquisition module: the method comprises the steps of collecting the states of the working nodes, wherein the states include a hardware state of a working node system, a scheduling task queue state, a data consumption queue state, a network bandwidth use state, a command execution state and the like. (12) A state reporting module: and the working node reports the newly generated state information to the management node at regular time. (13) A task viewing module: through the management node, the information, the execution progress and the consumption information of the scheduling data of each data scheduling task can be checked. (14) A state viewing module: the management node can check the hardware state of each working node system resource, the service index state and the network bandwidth occupation state. (15) A command issuing module: the management node can issue commands to the designated working node, wherein the commands comprise commands of system restart, cache data cleaning, system update and the like. (16) A command execution module: the working node downloads the unexecuted command from the management node periodically and executes the command, and records the command execution state.
Fig. 1 shows a system architecture diagram of an embodiment of the present invention, where one server is designated as a management node and the other servers are designated as working nodes when a distributed data scheduling system is deployed. The firewalls of all nodes open a port 443 for providing services to the outside. The service port of each node of the TELNET can test whether the service exists.
The information of each working node needs to be registered on the management node, the management node generates a unique ID for the working node, the ID is four digits and is numbered from 1000, and therefore 9000 working nodes can be maximally accommodated in a single distributed scheduling cluster. The communication profile for each node is then downloaded at the management node. After each working node is configured with the communication configuration file, the working nodes can communicate with the management node to perform the processes of topology updating, task reporting, state reporting, command issuing and the like.
User information needs to be registered on a management node, a binding working node can be selected by self, then the management node generates a unique user ID for a user, the ID is 12 digits, and the generation rule is as follows: user registration year _ job node number _ user type _ four-digit serial number, such as 211000010001, represents that the user registers in 2021, selects binding 1000 job nodes, and the user type is 01, and is the user registered at the 0001 th digit. Then, the management node feeds back the user ID, the IP address of the bound working node and the service port to the user. The user can query the management node to obtain the IDs of other users. And the user carries out a data scheduling process through the bound working nodes.
The specific implementation manner of the above flows is as follows:
the topology updating process means that the working node acquires the full amount of node information and user information of each node from the management node, so that the working node can acquire the routing information, namely an IP address and a service port, of the working node bound by a target user of the data scheduling task from the global topology information.
And the task reporting process is that the working node records the processes of generating a data scheduling task, executing the data scheduling task, retransmitting the data scheduling task and the like, and then reports the records to the management node. And the user can inquire the execution flow state of the task from the management node through the data scheduling task ID. The data scheduling task ID is a unique number generated by the working node, and the generation rule is as follows: the working node number _ timestamp _ three-bit random number, and the form of the event stamp is as follows: y _ M _ D _ H _ M _ S _ F. Wherein, each symbol means: y-year, M-month, D-day, H-hour, M-minute, S-second, F-microsecond.
And (III) the state reporting process means that the working node records the server hardware state, the service index state and the network bandwidth occupation state of the node and reports the state to the management node. The server hardware state comprises CPU utilization rate, memory utilization rate and disk utilization rate. The service index state comprises the backlog quantity of the task queue and the backlog quantity of the user data queue. Network bandwidth occupation includes network egress traffic speed and network ingress traffic speed. The user can query from the management node whether there is data scheduled to him by other users.
And (IV) the command issuing process means that the working node acquires the unexecuted command instruction from the management node, immediately executes the command instruction if the unexecuted command instruction is acquired, and records the command execution state. The command execution status can be viewed through the management node. Under the distributed condition, after a single device fails, the fault phenomenon of the working node can be solved by issuing a specified command to the failed device. Commands refer to scripts written in the shell programming language.
And the data scheduling process refers to a process that a user schedules data to other users through the work nodes, and the other users acquire the data from the bound work nodes. As shown in fig. 2, when data of user a1 is scheduled to user B1, user a1 first sends the data to the bound working node a, and sends the data through HTTP/HTTPs RESTful interface. The interface URL is: the/v 1/send _ data/{ data _ type }, and { data _ type } represent data types. The HTTP header field comprises fields of USER-AGENT, LEVEL, DES-USER and the like, the USER-AGENT field is the USER ID of the USER A1, the LEVEL field is the priority of data, and the value range is as follows: 0, 1, 2, respectively, for normal, medium, and preferred, and the DES-USER field is the USER ID of USER B1. The scheduling data body is stored in the http request body, and the type is binary. The working node A queries the global topology, acquires the binding working node B routing information of the user B1, and generates the data scheduling task ID. And the data scheduling task ID, the routing information, the user ID of the user A1, the user ID of the user B1, the data type and the scheduling data body are packaged and stored in a corresponding priority task queue, and the data scheduling task ID is fed back to the user A1. And the working node A reads the tasks of the task queue in sequence according to the priority, analyzes and classifies the routing information of each task, and executes the tasks of the batch in batches according to the routing classification by using multithreading. After the data is successfully scheduled to the worker node B, the worker node B stores the scheduled data in the private data queue of user B1. And if the data is not successfully scheduled to the working node B, the working node A stores the execution failure task into a task retransmission queue, and the priority of the task retransmission queue is priority. And if the task retransmission fails, the failed task is still stored in the task retransmission queue until the task retransmission execution is successful. And after the working node B acquires the data packets sent by other working nodes, the data packets are analyzed to acquire the ID of the user receiving the data packets, and the data is cached in the data queue of the user ID. The user B1 obtains data from the working node B by means of an HTTP/HTTPs RESTful interface. The interface URL is: and/v 1/get _ data. The data packet, binary type, is obtained from the interface HTTP feedback. User B1 may parse from the packet: data scheduling task ID, routing information, user ID of user a1, user ID of user B1, data type, scheduling data body, etc.
The above examples are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications can be made without departing from the spirit and principles of the invention, and are intended to be within the scope of the invention.

Claims (10)

1. A distributed multi-user data scheduling method is suitable for a network formed by a management node and a plurality of working nodes, the management node respectively generates unique working node routing information or user ID according to the information of each working node and user information, and the method comprises the following steps:
1) the working node A periodically inquires the global topology and acquires global topology information from the management node;
2) analyzing the scheduling data sent by the user A1, acquiring the data type, the scheduling data body and the priority of the scheduling data and the user ID of the user B1, and acquiring the routing information of the working node B bound by the user B1 according to the user ID of the user B1 and the global topology information;
3) generating a scheduling task T according to the generated data scheduling task ID, the user ID of the user B1, the routing information of the working node B, the data type and the scheduling data bodyiAnd combining the priority to schedule the task T iStoring the data into a corresponding task queue, wherein i is a task number;
4) parsing scheduling tasks TiAnd sort the batch execution data scheduling tasks so that user B1 retrieves the scheduling data from the worker node B's private data queue.
2. The method of claim 1, wherein the interfacing of user a1 to send scheduling data comprises: HTTP/HTTPSRESTful interface mode.
3. The method of claim 2, wherein the interface URL is: v1/send _ data/{ data _ type }, where { data _ type } represents a data type; the interface HTTP header fields include: a USER-AGENT field, a LEVEL field and a DES-USER field, wherein the USER-AGENT field is the USER ID of the USER A1, the LEVEL field is the priority of data, and the DES-USER field is the USER ID of the USER B1; the scheduling data body is stored in the http request body.
4. The method of claim 1, wherein the rule for generating the data scheduling task ID comprises: the working node number _ event stamp _ n bit random number, wherein n is more than or equal to 1.
5. The method of claim 1, wherein if the scheduling data is not successfully scheduled to the working node B, the execution failure task is stored in the task re-transmission queue and re-executed until the task re-transmission execution is successful.
6. The method according to claim 1, characterized in that the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task are recorded and reported to the management node.
7. The method of claim 1, wherein the server hardware status, the service indicator status, the network bandwidth occupation status and the command execution status of the local work node are reported to the management node; the server hardware states include: CPU utilization rate, memory utilization rate and disk utilization rate; the service index states include: the backlog quantity of the task queue and the backlog quantity of the user data queue; the network bandwidth occupation state comprises the following steps: network egress traffic speed and network ingress traffic speed.
8. The method of claim 1 wherein worker node a1 immediately executes the get unexecuted command instruction from the management node and returns the recorded command execution status to the management node.
9. A distributed multi-user data scheduling system is suitable for a network formed by a management node and a plurality of working nodes, and comprises:
the node management module is used for generating unique working node routing information by the management node according to the information of each working node;
The user management module is used for generating a unique user ID by the management node according to the user information;
the topology updating module is used for the working node A to inquire the global topology periodically and obtain global topology information from the management node;
the task generation module is used for analyzing the scheduling data sent by the user A1 to obtain the schedulingThe data type, the scheduling data body and the priority of the data and the user ID of the user B1 are obtained, and the routing information of the working node B bound by the user B is obtained according to the user ID of the user B1 and the global topology information; generating a scheduling task T according to the generated data scheduling task ID, the user ID of the user B1, the routing information of the working node B, the data type and the scheduling data bodyiAnd combining the priority to schedule the task TiStoring the data into a corresponding task queue, wherein i is a task number;
a task execution module for analyzing each scheduling task TiThe routing information and classified batch execution data scheduling tasks;
and the data consumption module is used for the working node A to store the user data of the user B1 in a message middleware in a separated mode, so that the user B1 can consume the data which is dispatched to the user from the working node B1 to other users.
10. The system of claim 9, further comprising a link detection module, a task retransmission module, a task monitoring module, a task reporting module, a state acquisition module, a state reporting module, a task checking module, a state checking module, a command issuing module, and a command execution module; wherein the content of the first and second substances,
The link detection module is used for inquiring the topological structure by the working node A;
the task retransmission module is used for storing the execution failure task into the task retransmission queue and re-executing the execution failure task if the scheduling data is not successfully scheduled to the working node B;
the task monitoring module is used for recording the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task;
the task reporting module is used for reporting the generation process of the data scheduling task, the execution process of the data scheduling task and/or the retransmission process of the data scheduling task to the management node;
the state acquisition module is used for acquiring the server hardware state, the service index state, the network bandwidth occupation state and the command execution state of the working node, wherein the server hardware state comprises: CPU utilization rate, memory utilization rate and disk utilization rate; the service index states include: the backlog quantity of the task queue and the backlog quantity of the user data queue; the network bandwidth occupation state comprises the following steps: network outlet traffic speed and network inlet traffic speed;
the state reporting module is used for reporting the server hardware state, the service index state and the network bandwidth occupation state of the working node to the management node;
The task viewing module is used for viewing the information and the execution progress of each data scheduling task and the consumption information of the scheduling data;
the state viewing module is used for viewing the hardware state of the system resource, the service index state and the network bandwidth occupation state of each working node through the management node;
the command issuing module is used for issuing commands to the specified working nodes through the management nodes, wherein the commands comprise: system restarting, cache data cleaning or system updating;
and the command execution module is used for downloading the unexecuted command from the management node by the working node at regular intervals, executing the unexecuted command and recording the command execution state.
CN202110778053.6A 2021-07-09 2021-07-09 Distributed multi-user data scheduling method and system Active CN113626161B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110778053.6A CN113626161B (en) 2021-07-09 2021-07-09 Distributed multi-user data scheduling method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110778053.6A CN113626161B (en) 2021-07-09 2021-07-09 Distributed multi-user data scheduling method and system

Publications (2)

Publication Number Publication Date
CN113626161A true CN113626161A (en) 2021-11-09
CN113626161B CN113626161B (en) 2023-12-22

Family

ID=78379399

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110778053.6A Active CN113626161B (en) 2021-07-09 2021-07-09 Distributed multi-user data scheduling method and system

Country Status (1)

Country Link
CN (1) CN113626161B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114448816A (en) * 2021-12-30 2022-05-06 中国航空研究院 Integrated IP networking method based on heterogeneous data chain

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101309201A (en) * 2007-05-14 2008-11-19 华为技术有限公司 Route processing method, routing processor and router
CN101340331A (en) * 2007-07-06 2009-01-07 中国电信股份有限公司 Method for executing system task by idle terminal in P2P network
CN102084628A (en) * 2008-04-24 2011-06-01 厄塞勒拉特公司 A traffic manager and a method for a traffic manager
CN103279351A (en) * 2013-05-31 2013-09-04 北京高森明晨信息科技有限公司 Method and device for task scheduling
US20160142331A1 (en) * 2014-11-14 2016-05-19 Xpliant Packet scheduling using hierarchical scheduling process with priority propagation
CN106126346A (en) * 2016-07-05 2016-11-16 东北大学 A kind of large-scale distributed data collecting system and method
CN108268317A (en) * 2016-12-30 2018-07-10 华为技术有限公司 A kind of resource allocation methods and device
CN111367630A (en) * 2019-07-12 2020-07-03 北京关键科技股份有限公司 Multi-user multi-priority distributed cooperative processing method based on cloud computing

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101309201A (en) * 2007-05-14 2008-11-19 华为技术有限公司 Route processing method, routing processor and router
CN101340331A (en) * 2007-07-06 2009-01-07 中国电信股份有限公司 Method for executing system task by idle terminal in P2P network
CN102084628A (en) * 2008-04-24 2011-06-01 厄塞勒拉特公司 A traffic manager and a method for a traffic manager
CN103279351A (en) * 2013-05-31 2013-09-04 北京高森明晨信息科技有限公司 Method and device for task scheduling
US20160142331A1 (en) * 2014-11-14 2016-05-19 Xpliant Packet scheduling using hierarchical scheduling process with priority propagation
CN106126346A (en) * 2016-07-05 2016-11-16 东北大学 A kind of large-scale distributed data collecting system and method
CN108268317A (en) * 2016-12-30 2018-07-10 华为技术有限公司 A kind of resource allocation methods and device
CN111367630A (en) * 2019-07-12 2020-07-03 北京关键科技股份有限公司 Multi-user multi-priority distributed cooperative processing method based on cloud computing

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114448816A (en) * 2021-12-30 2022-05-06 中国航空研究院 Integrated IP networking method based on heterogeneous data chain
CN114448816B (en) * 2021-12-30 2023-10-10 中国航空研究院 Integrated IP networking method based on heterogeneous data chain

Also Published As

Publication number Publication date
CN113626161B (en) 2023-12-22

Similar Documents

Publication Publication Date Title
US10200251B2 (en) Methods and apparatus for accessing selectable application processing of data packets in an adaptive private network
Tolle et al. Design of an application-cooperative management system for wireless sensor networks
US7099879B2 (en) Real-time monitoring of service performance through the use of relational database calculation clusters
Chung et al. NS by Example
Chowdhury et al. Payless: A low cost network monitoring framework for software defined networks
US20030212778A1 (en) UML representation of parameter calculation expressions for service monitoring
US20060179059A1 (en) Cluster monitoring system with content-based event routing
WO2004028083A1 (en) A scheduling method for polling device data
CN103392314A (en) Systems and methods for scalable n-core statistics aggregation
CN103001891B (en) Method for promoting integral service quality of local area network
Li et al. OPTAS: Decentralized flow monitoring and scheduling for tiny tasks
CN113626161B (en) Distributed multi-user data scheduling method and system
Brunner et al. Probabilistic decentralized network management
Silva et al. Controlling network latency in mixed hadoop clusters: Do we need active queue management?
CN115314542A (en) Link tracking method and system based on Socket communication protocol
CN116723154A (en) Route distribution method and system based on load balancing
Pereira et al. The mutable consensus protocol
WO2022152230A1 (en) Information flow identification method, network chip, and network device
Shih et al. Data alignment for multiple temporal data streams without synchronized clocks on iot fusion gateway
Chen et al. Petri net modeling of the reconfigurable protocol stack for cloud computing control systems
Milosavljevic et al. Comparison of Message Queue Technologies for Highly Available Microservices in IoT
WO2014036715A1 (en) System and method for controlling real-time resource supply process based on delivery point
CN116074392B (en) Intelligent matching method and device for data stream transmission modes
Sedaghat et al. R2T-DSDN: reliable real-time distributed controller-based SDN
CN108400905A (en) A method of processing distributed storage end-to-end flux analysis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant