CN111367630A - Multi-user multi-priority distributed cooperative processing method based on cloud computing - Google Patents
Multi-user multi-priority distributed cooperative processing method based on cloud computing Download PDFInfo
- Publication number
- CN111367630A CN111367630A CN201910628210.8A CN201910628210A CN111367630A CN 111367630 A CN111367630 A CN 111367630A CN 201910628210 A CN201910628210 A CN 201910628210A CN 111367630 A CN111367630 A CN 111367630A
- Authority
- CN
- China
- Prior art keywords
- task
- priority
- tasks
- user
- subtasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 30
- 238000004458 analytical method Methods 0.000 claims abstract description 93
- 238000005457 optimization Methods 0.000 claims abstract description 24
- 230000007246 mechanism Effects 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 19
- 238000012544 monitoring process Methods 0.000 claims description 19
- 238000005516 engineering process Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 7
- 238000007726 management method Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Abstract
The invention relates to a multi-user multi-priority distributed cooperative processing method based on cloud computing, which comprises the following steps: step 1) taking users and priorities as judgment factors for forming analysis task execution sequence by codes; and 2) forming an execution sequence of the analysis tasks based on the codes, analyzing the subtasks split by each task in a distributed cluster cooperation mode, and summarizing analysis results of the subtasks. The invention relates to a multi-user multi-priority distributed cooperative processing method execution device based on cloud computing, which is used for acquiring the execution sequence of tasks according to users and priorities, splitting the tasks, distributing subtasks to each node of a cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes and realizing high-efficiency controllability of code composition analysis. And the problems that large tasks have mass data, the cooperative processing range is limited and the difficulty is high are solved.
Description
Technical Field
The invention relates to the field of software and distributed cooperation, in particular to a multi-user multi-priority distributed cooperative processing method based on cloud computing.
Background
With the advancement of the informatization process, the number of software is increased year by year, the complexity of the software is gradually increased, and the data volume of software codes required to be subjected to composition analysis during code auditing is huge, so that a cooperative processing method is required to improve the efficiency.
The common cooperative processing method at present is that the workers respectively complete corresponding tasks at first, and then perform lower-degree cooperation through video conference, shared file and other forms; or using software based on an HTTP protocol, and acquiring data such as physical information and special nodes of the object by building an application platform and using specific and non-uniform API (application program interface) forms of each software; or the object is processed in the form of an intermediate file to acquire data such as special nodes, physical information and the like of the object, and the purpose of cooperative work is achieved after the data is processed. The large task has mass data, the cooperative processing range is limited, and the difficulty is high.
In order to ensure the accuracy of code composition analysis, the number of stored components reaches the level of ten million. And the data needs to be processed under multi-user and multi-priority, and the common cooperative processing method has low efficiency and cannot meet the requirement. The project adopts a private cloud computing platform with the transverse expansion capability of the server cluster. The code analysis computing capacity and the massive code data storage capacity are improved approximately linearly by adding a computer or a virtual machine, the high concurrent processing of multiple nodes is realized, and the code search matching efficiency is increased linearly along with the nodes. Therefore, a multi-user multi-priority distributed cooperative processing method based on cloud computing is provided.
Disclosure of Invention
The invention aims at the problems provided above and provides a multi-user multi-priority distributed system processing method and device based on cloud computing. The system is used in a distributed system comprising a central server cluster which is in the cloud and is responsible for software running and a plurality of user operation and use. The method solves a series of problems in the process of multi-user, multi-priority and distributed cooperative processing, and solves the problems of mass data query, limited cooperative range and low efficiency by the multi-user and multi-priority distributed cooperative processing method.
According to one aspect of the invention, a multi-user multi-priority distributed cooperative processing method based on cloud computing is provided, which is used for improving the computing capacity and the data storage capacity of code composition analysis, and comprises the following steps:
step 1) taking users and priorities as judgment factors for forming analysis task execution sequence by codes;
the method comprises the steps that code analysis tasks are newly built by a plurality of users, code analysis requests are sent to a private cloud computing platform cluster, after the private cloud computing platform cluster receives the requests, the users and priorities are combined, relatively reasonable task execution priorities are built, then the tasks to be executed are placed into a queue, and the sequence of the tasks in the queue is used as codes to form an execution sequence of the analysis tasks.
The priority of task execution is formed by combining users and priorities, the users comprise special users and ordinary users, the priority refers to the priority which is designated when the user newly creates codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
The method for forming the execution sequence of the analysis tasks by taking the sequence of the tasks in the queue as a code comprises the following steps: for a plurality of newly-built code composition analysis tasks, determining the execution sequence of the tasks by using the priority of the tasks selected by a user and the user, putting the tasks into a queue, splitting the code composition analysis tasks into a plurality of subtasks, analyzing the plurality of subtasks in a distributed mode, and finally summarizing analysis results.
And 2) forming an execution sequence of the analysis tasks based on the codes, analyzing the subtasks split by each task in a distributed cluster cooperation mode, and summarizing an analysis result of each task.
More specifically, in the cloud computing-based multi-user multi-priority distributed cooperative processing method: and step 2) further comprises code composition analysis task splitting, cloud resource scheduling strategy optimization, multi-node parallel processing and an asynchronous control management mechanism, and the multi-node task cooperative processing capability is carried out, so that the efficient controllability of the code composition analysis process is realized.
More specifically, the splitting of the code composition analysis task refers to: and splitting the code composition analysis task into a pair of subtasks according to the sequence of the priorities from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
More specifically, the cloud source resource scheduling policy optimization refers to: the cloud resource dynamic scheduling strategy optimization is mainly realized by a resource monitoring and predicting technology and an automatic load balancing scheduling technology. The method is divided into two modules of monitoring and scheduling according to technical points. The monitoring module provides real-time state information, historical prediction information and corresponding alarm information of various resources to the scheduling module, the scheduling module selects a certain load balancing algorithm in a specified scheduling domain according to the corresponding resource state information to achieve a specified purpose, or reasonably allocates resources according to tasks submitted by users, so that the task execution of the users achieves the specified resource optimization requirement, finally executes final actions by operating the driver, returns to the corresponding state, and timely notifies the monitoring module of the update information of the resources.
More specifically, the multi-node parallel processing and asynchronous control management mechanism refers to: dividing the code composition analysis task into a plurality of subtasks, starting multithreading, then distributing the subtasks to each node of the cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability, and realizing high efficiency and controllability of code composition analysis.
According to another aspect of the present invention, there is also provided a multi-user multi-priority based distributed cooperative processing apparatus for analyzing a code, the apparatus including:
and the task priority calculating module is used for calculating the task execution sequence according to the user and the task priority specified by the user.
And the task splitting module splits the task according to the priority of task execution and available resources and splits the task into a plurality of subtasks.
And the data processing module is used for distributing a plurality of subtasks to each node of the cluster according to a cloud resource dynamic scheduling strategy, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability and realizing high-efficiency controllability of code composition analysis.
In the task priority computing module, the task execution sequence is determined by the user and the task priority specified by the user. The users comprise special users and common users, the priority refers to the priority appointed by the user when the user newly builds codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
In the task splitting module, the code composition analysis task is split into a pair of subtasks according to the sequence of the priorities of the tasks from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
In the data processing module, a plurality of subtasks are distributed to each node of a cluster according to the priority of the subtasks and a cloud resource dynamic scheduling strategy, multi-node parallel processing is carried out, asynchronous cooperation of the plurality of nodes is achieved, and finally the analysis result of each task is combined.
More specifically, in the cloud computing-based multi-user multi-priority distributed cooperative processing apparatus: the task splitting module is further configured to determine a priority for executing the subtasks, where determining the priority of the subtasks means that, when a plurality of split tasks are present, the subtask that is preferentially executed is a task with a high priority.
More specifically, in the cloud computing-based multi-user multi-priority distributed cooperative processing apparatus: the cloud resource scheduling strategy optimization is realized by a resource monitoring and predicting technology and an automatic load balancing scheduling technology. The monitoring module provides real-time state information, historical prediction information and corresponding alarm information of various resources to the scheduling module, and the scheduling module selects a certain load balancing algorithm in a specified scheduling domain according to the corresponding resource state information to achieve the specified purpose, or reasonably allocates resources according to tasks submitted by users, so that the task execution of the users achieves the specified resource optimization requirement.
The present invention has the following important inventions:
1. and multi-user multi-priority, which is used for confirming the priority of the resource usage of the code composition analysis task according to the priorities of the users and the tasks. And multiple tasks of multiple users are simultaneously executed.
2. The cloud resource dynamic scheduling strategy optimization is realized by a resource monitoring and predicting technology and an automatic load balancing scheduling technology: according to the corresponding resource state information, a certain load balancing algorithm is selected in a designated scheduling domain to achieve the designated purpose, or reasonable resource allocation is carried out according to the tasks submitted by the users, so that the task execution of the users achieves the designated resource optimization requirement. And resources are reasonably utilized.
3. The multi-node parallel processing divides the code composition analysis task into a plurality of subtasks, starts multithreading, then distributes the subtasks to each node of the cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performs multi-node parallel processing, realizes asynchronous cooperation, forms multi-node task cooperative processing capability, and realizes high efficiency and controllability of code composition analysis. The pressure of data search is reduced, and the analysis efficiency is improved.
Drawings
Embodiments of the invention will now be described with reference to the accompanying drawings, in which:
fig. 1 is a flowchart illustrating steps of a multi-user multi-priority distributed cooperative processing method based on cloud computing according to an embodiment of the present invention.
Fig. 2 is a block diagram illustrating an implementation of a multi-user multi-priority distributed cooperative processing method based on cloud computing according to an embodiment of the present invention.
FIG. 3 is a technical block diagram of a dynamic scheduling policy for private cloud resources according to the present invention
FIG. 4 is a flow chart of asynchronous cooperative processing of multi-node tasks according to the present invention
Detailed Description
An embodiment of a multi-user multi-priority distributed cooperative processing method based on cloud computing according to the present invention will be described in detail below with reference to the accompanying drawings.
The invention adopts a multi-user multi-priority distributed cooperative processing method based on cloud computing to carry out composition analysis on codes. The method comprises the steps that code analysis tasks are newly built by a plurality of users, code analysis requests are sent to a private cloud computing platform cluster, after the private cloud computing platform cluster receives the requests, user IDs and corresponding tasks are added into corresponding queues according to users and priorities, and meanwhile the number of the ongoing tasks is increased by one. And then, the multi-node task cooperative processing capability is carried out through code analysis task splitting, cloud resource scheduling strategy optimization, multi-node parallel processing and an asynchronous control management mechanism, so that the efficient controllability of the code composition analysis process is realized.
Fig. 1 is a flowchart illustrating steps of a cloud computing-based multi-user multi-priority distributed cooperative processing method for resolving a code composition analysis task, according to an embodiment of the present invention, where the method includes:
step 1) taking users and priorities as judgment factors for forming analysis task execution sequence by codes;
the method comprises the steps that code analysis tasks are newly built by a plurality of users, code analysis requests are sent to a private cloud computing platform cluster, after the private cloud computing platform cluster receives the requests, the users and priorities are combined, relatively reasonable task execution priorities are built, then the tasks to be executed are placed into a queue, and the sequence of the tasks in the queue is used as codes to form an execution sequence of the analysis tasks.
The priority of task execution is formed by combining users and priorities, the users comprise special users and ordinary users, the priority refers to the priority which is designated when the user newly creates codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
The method for forming the execution sequence of the analysis tasks by taking the sequence of the tasks in the queue as a code comprises the following steps: for a plurality of newly-built code composition analysis tasks, determining the execution sequence of the tasks by using the priority of the tasks selected by a user and the user, putting the tasks into a queue, splitting the code composition analysis tasks into a plurality of subtasks, analyzing the plurality of subtasks in a distributed mode, and finally summarizing analysis results.
And 2) forming an execution sequence of the analysis tasks based on the codes, analyzing the subtasks split by each task in a distributed cluster cooperation mode, and summarizing an analysis result of each task.
In the multi-user multi-priority distributed cooperative processing method based on cloud computing:
and step 2) further comprises code composition analysis task splitting, cloud resource scheduling strategy optimization, multi-node parallel processing and an asynchronous control management mechanism, and the multi-node task cooperative processing capability is carried out, so that the efficient controllability of the code composition analysis process is realized.
The distributed cooperative processing method based on the multi-user and multi-priority of the cloud computing comprises the following steps:
and splitting the code composition analysis task into a pair of subtasks according to the sequence of the priorities from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
The distributed cooperative processing method based on the multi-user and multi-priority of the cloud computing comprises the following steps:
dividing the code composition analysis task into a plurality of subtasks, starting multithreading, then distributing the subtasks to each node of the cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability, and realizing high efficiency and controllability of code composition analysis.
Fig. 2 is a block diagram illustrating an architecture executed by a multi-user multi-priority distributed coprocessing method based on cloud computing according to an embodiment of the present invention, wherein the apparatus is used for parsing a code composition analysis task, and the apparatus includes:
and the task priority calculating module is used for calculating the task execution sequence according to the user and the task priority specified by the user.
And the task splitting module splits the task according to the priority of task execution and available resources and splits the task into a plurality of subtasks.
And the data processing module is used for distributing a plurality of subtasks to each node of the cluster according to a cloud resource dynamic scheduling strategy, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability and realizing high-efficiency controllability of code composition analysis.
In the task priority computing module, the task execution sequence is determined by the user and the task priority specified by the user. The users comprise special users and common users, the priority refers to the priority appointed by the user when the user newly builds codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
In the task splitting module, the code composition analysis task is split into a pair of subtasks according to the sequence of the priorities of the tasks from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
In the data processing module, a plurality of subtasks are distributed to each node of a cluster according to the priority of the subtasks and a cloud resource dynamic scheduling strategy, multi-node parallel processing is carried out, asynchronous cooperation of the plurality of nodes is achieved, and finally the analysis result of each task is combined.
Next, a specific structure of the apparatus for executing the multi-user multi-priority distributed cooperative processing method based on cloud computing according to the present invention will be further described.
In the multi-user multi-priority distributed cooperative processing device based on cloud computing:
the task splitting module is further configured to determine a priority for executing the subtasks, where determining the priority of the subtasks means that, when a plurality of split tasks are present, the subtask that is preferentially executed is a task with a high priority.
In the multi-user multi-priority distributed cooperative processing device based on cloud computing:
the cloud resource scheduling strategy optimization is realized by a resource monitoring and predicting technology and an automatic load balancing scheduling technology. The monitoring module provides real-time state information, historical prediction information and corresponding alarm information of various resources to the scheduling module, and the scheduling module selects a certain load balancing algorithm in a specified scheduling domain according to the corresponding resource state information to achieve the specified purpose, or reasonably allocates resources according to tasks submitted by users, so that the task execution of the users achieves the specified resource optimization requirement.
Fig. 3 is a technical block diagram of a private cloud resource dynamic scheduling policy of the present invention. The resources of the private cloud resource dynamic scheduling policy technology framework are mainly embodied by cluster resources, and the monitoring module obtains status information of various resources from a cluster: CPU, memory, disk, network I/O, information of resources on the physical machine, and performance information monitoring of various applications. The monitoring center can make a corresponding resource early warning signal according to the configuration of an administrator, the whole load condition is pushed to the scheduling module, and meanwhile, the monitoring module also supports an interface of the scheduling module for pulling monitoring data in real time.
The scheduling module dynamically performs a load balancing algorithm, a maximum utilization rate algorithm or a performance optimization algorithm in all scheduling domains according to the type of information or the threshold value of the information by passively or actively acquiring the monitoring information of the monitoring module. There are three factors that contribute to the scheduling algorithm: early warning signals of a monitoring module, timing monitoring of a scheduling module and task requests of new users. After the execution of the scheduling algorithm is finished, a series of operations (operations of migrating, creating, suspending and modifying the configuration of the virtual machine) are formed and sent to the operation driver module in the form of operation command information to realize the deployment of the final scheduling.
And operating the driver module to uniformly package the calling interfaces of different software. The method has the main functions that after various operation requests sent by a scheduling module are received, packaged interfaces are called, a list of the requests is classified, batched and executed in parallel, the execution efficiency is improved, and finally executed results and state information are fed back to a scheduling framework and a private cloud computing system.
FIG. 4 is a multi-node task asynchronous cooperative processing flow chart of the present invention, in which a project splits a task in a multi-thread manner, allocates subtasks to each node of a cluster through a cloud resource dynamic scheduling policy optimization mechanism, performs multi-node parallel processing, realizes asynchronous cooperation of multiple nodes, forms multi-node task cooperative processing capability, and realizes efficient and controllable code composition analysis.
The code is analyzed in a distributed cooperation mode, in the analysis process, the user and the priority are used as judgment factors for analyzing the execution sequence of the task, the execution sequence of the task is firstly calculated, the task is split and divided into a plurality of subtasks, then multithreading is started, the subtasks are distributed to each node of a cluster through a cloud resource dynamic scheduling strategy optimization mechanism, multi-node parallel processing is carried out, asynchronous cooperation of the plurality of nodes is realized, the task cooperative processing capability of the plurality of nodes is formed, and efficient and controllable code composition analysis is realized. And the problem that large tasks have mass data is solved, and the cooperative processing range is limited and the difficulty is high.
It is to be understood that while the present invention has been described in conjunction with the preferred embodiments thereof, it is not intended to limit the invention to those embodiments. It will be apparent to those skilled in the art from this disclosure that many changes and modifications can be made, or equivalents modified, in the embodiments of the invention without departing from the scope of the invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.
Claims (8)
1. A multi-user multi-priority distributed cooperative processing method based on cloud computing is used for carrying out composition analysis on codes and is characterized by comprising the following steps:
step 1) taking users and priorities as judgment factors for forming analysis task execution sequence by codes;
the method comprises the steps that code analysis tasks are newly built by a plurality of users, code analysis requests are sent to a private cloud computing platform cluster, after the private cloud computing platform cluster receives the requests, the users and priorities are combined, relatively reasonable task execution priorities are built, then the tasks to be executed are placed into a queue, and the sequence of the tasks in the queue is used as codes to form an execution sequence of the analysis tasks.
The priority of task execution is formed by combining users and priorities, the users comprise special users and ordinary users, the priority refers to the priority which is designated when the user newly creates codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
The method for forming the execution sequence of the analysis tasks by taking the sequence of the tasks in the queue as a code comprises the following steps: for a plurality of newly-built code composition analysis tasks, determining the execution sequence of the tasks by using the priority of the tasks selected by a user and the user, putting the tasks into a queue, splitting the code composition analysis tasks into a plurality of subtasks, analyzing the plurality of subtasks in a distributed mode, and finally summarizing analysis results.
And 2) forming an execution sequence of the analysis tasks based on the codes, analyzing the subtasks split by each task in a distributed cluster cooperation mode, and summarizing an analysis result of each task.
2. The cloud computing-based multi-user multi-priority distributed cooperative processing method according to claim 1, wherein:
and step 2) further comprises code composition analysis task splitting, cloud resource scheduling strategy optimization, multi-node parallel processing and an asynchronous control management mechanism, and the multi-node task cooperative processing capability is carried out, so that the efficient controllability of the code composition analysis process is realized.
3. The cloud computing-based multi-user multi-priority distributed cooperative processing method according to claim 2, wherein:
and splitting the code composition analysis task, and splitting the code composition analysis task into a pair of subtasks according to the sequence of the priority from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
4. The cloud computing-based multi-user multi-priority distributed cooperative processing method according to claim 2, wherein:
dividing the code composition analysis task into a plurality of subtasks, starting multithreading, then distributing the subtasks to each node of the cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability, and realizing high efficiency and controllability of code composition analysis.
5. An execution device of a multi-user multi-priority distributed cooperative processing method based on cloud computing is used for performing composition analysis on codes, and is characterized by comprising the following steps:
and the task priority calculating module is used for calculating the task execution sequence according to the user and the task priority specified by the user.
And the task splitting module splits the task according to the priority of task execution and available resources and splits the task into a plurality of subtasks.
And the data processing module is used for distributing a plurality of subtasks to each node of the cluster according to a cloud resource dynamic scheduling strategy, performing multi-node parallel processing, realizing asynchronous cooperation of a plurality of nodes, forming multi-node task cooperative processing capability and realizing high-efficiency controllability of code composition analysis.
In the task priority computing module, the task execution sequence is determined by the user and the task priority specified by the user. The users comprise special users and common users, the priority refers to the priority appointed by the user when the user newly builds codes to form analysis tasks, and the execution sequence of the tasks is obtained through the users and the priority.
In the task splitting module, the code composition analysis task is split into a pair of subtasks according to the sequence of the priorities of the tasks from high to low. The split code composition analysis task is one or more according to the availability ratio of the resources.
In the data processing module, a plurality of subtasks are distributed to each node of a cluster according to the priority of the subtasks and a cloud resource dynamic scheduling strategy, multi-node parallel processing is carried out, asynchronous cooperation of the plurality of nodes is achieved, and finally the analysis result of each task is combined.
6. The cloud-computing-based multi-user multi-priority distributed cooperative processing method execution apparatus according to claim 5, wherein:
the task splitting module is further configured to determine a priority for executing the subtasks, where determining the priority of the subtasks means that, when a plurality of split tasks are present, the subtask that is preferentially executed is a task with a high priority.
7. The cloud-computing-based multi-user multi-priority distributed cooperative processing method execution apparatus according to claim 6, wherein:
in the multi-user multi-priority distributed cooperative processing device based on cloud computing: the cloud resource scheduling strategy optimization is realized by a resource monitoring and predicting technology and an automatic load balancing scheduling technology. The monitoring module provides real-time state information, historical prediction information and corresponding alarm information of various resources to the scheduling module, and the scheduling module selects a certain load balancing algorithm in a specified scheduling domain according to the corresponding resource state information to achieve the specified purpose, or reasonably allocates resources according to tasks submitted by users, so that the task execution of the users achieves the specified resource optimization requirement.
8. The cloud-computing-based multi-user multi-priority distributed cooperative processing method execution apparatus according to claim 6, wherein:
parsing data in a distributed collaborative manner includes: splitting a task into a plurality of subtasks, starting multithreading, distributing the subtasks to each node of a cluster through a cloud resource dynamic scheduling strategy optimization mechanism, performing multi-node parallel processing, realizing asynchronous cooperation of the plurality of nodes, and summarizing an analysis result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910628210.8A CN111367630A (en) | 2019-07-12 | 2019-07-12 | Multi-user multi-priority distributed cooperative processing method based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910628210.8A CN111367630A (en) | 2019-07-12 | 2019-07-12 | Multi-user multi-priority distributed cooperative processing method based on cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111367630A true CN111367630A (en) | 2020-07-03 |
Family
ID=71207845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910628210.8A Pending CN111367630A (en) | 2019-07-12 | 2019-07-12 | Multi-user multi-priority distributed cooperative processing method based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367630A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813554A (en) * | 2020-07-17 | 2020-10-23 | 济南浪潮数据技术有限公司 | Task scheduling processing method and device, electronic equipment and storage medium |
| CN111880843A (en) * | 2020-07-31 | 2020-11-03 | 重庆医科大学 | Biological big data analysis system and method based on Linux single command line |
| CN112101536A (en) * | 2020-08-30 | 2020-12-18 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Lightweight distributed multi-task collaboration framework |
| CN112395085A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN112540841A (en) * | 2020-12-28 | 2021-03-23 | 智慧神州(北京)科技有限公司 | Task scheduling method and device, processor and electronic equipment |
| CN112579289A (en) * | 2020-12-21 | 2021-03-30 | 中电福富信息科技有限公司 | Distributed analysis engine method and device capable of achieving intelligent scheduling |
| CN112667901A (en) * | 2020-12-31 | 2021-04-16 | 中国电子信息产业集团有限公司第六研究所 | Social media data acquisition method and system |
| CN112817732A (en) * | 2021-02-26 | 2021-05-18 | 国网电力科学研究院有限公司 | Stream data processing method and system suitable for cloud-side collaborative multi-data-center scene |
| CN113626161A (en) * | 2021-07-09 | 2021-11-09 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN114153613A (en) * | 2021-12-07 | 2022-03-08 | 杭州电子科技大学 | Distributed communication scheduling method facing edge calculation and cooperation |
| CN114756383A (en) * | 2022-06-15 | 2022-07-15 | 苏州浪潮智能科技有限公司 | Distributed computing method, system, device and storage medium |
| CN116069464A (en) * | 2022-12-19 | 2023-05-05 | 深圳计算科学研究院 | Optimization method and device based on distributed storage call data execution |
| CN117155929A (en) * | 2023-10-31 | 2023-12-01 | 浪潮电子信息产业股份有限公司 | Communication method, system, electronic equipment and readable storage medium of distributed cluster |
| CN112667901B (en) * | 2020-12-31 | 2024-04-26 | 中国电子信息产业集团有限公司第六研究所 | Social media data acquisition method and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6408277B1 (en) * | 2000-06-21 | 2002-06-18 | Banter Limited | System and method for automatic task prioritization |
| US20080172674A1 (en) * | 2006-12-08 | 2008-07-17 | Business Objects S.A. | Apparatus and method for distributed dataflow execution in a distributed environment |
| CN102402423A (en) * | 2010-09-19 | 2012-04-04 | 百度在线网络技术（北京）有限公司 | Method and equipment for performing multi-task processing in network equipment |
| CN105897837A (en) * | 2015-12-07 | 2016-08-24 | 乐视云计算有限公司 | Content distribution task submitting method and system |
| US20170161669A1 (en) * | 2015-12-07 | 2017-06-08 | Le Holdings (Beijing) Co., Ltd. | Method and system for submitting content delivery tasks |
| CN108345501A (en) * | 2017-01-24 | 2018-07-31 | 全球能源互联网研究院 | A kind of distributed resource scheduling method and system |
| US20180300174A1 (en) * | 2017-04-17 | 2018-10-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
| CN108737560A (en) * | 2018-05-31 | 2018-11-02 | 南京邮电大学 | Cloud computing task intelligent dispatching method and system, readable storage medium storing program for executing, terminal |
-
2019
- 2019-07-12 CN CN201910628210.8A patent/CN111367630A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6408277B1 (en) * | 2000-06-21 | 2002-06-18 | Banter Limited | System and method for automatic task prioritization |
| US20080172674A1 (en) * | 2006-12-08 | 2008-07-17 | Business Objects S.A. | Apparatus and method for distributed dataflow execution in a distributed environment |
| CN102402423A (en) * | 2010-09-19 | 2012-04-04 | 百度在线网络技术（北京）有限公司 | Method and equipment for performing multi-task processing in network equipment |
| CN105897837A (en) * | 2015-12-07 | 2016-08-24 | 乐视云计算有限公司 | Content distribution task submitting method and system |
| US20170161669A1 (en) * | 2015-12-07 | 2017-06-08 | Le Holdings (Beijing) Co., Ltd. | Method and system for submitting content delivery tasks |
| CN108345501A (en) * | 2017-01-24 | 2018-07-31 | 全球能源互联网研究院 | A kind of distributed resource scheduling method and system |
| US20180300174A1 (en) * | 2017-04-17 | 2018-10-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
| CN108737560A (en) * | 2018-05-31 | 2018-11-02 | 南京邮电大学 | Cloud computing task intelligent dispatching method and system, readable storage medium storing program for executing, terminal |
Non-Patent Citations (2)
| Title |
|---|
| 姜浩;: "云计算环境下海量分布式数据处理协同机制的研究" * |
| 贺智明: "云环境下基于偏好的资源公平分配策略" * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813554A (en) * | 2020-07-17 | 2020-10-23 | 济南浪潮数据技术有限公司 | Task scheduling processing method and device, electronic equipment and storage medium |
| CN111880843A (en) * | 2020-07-31 | 2020-11-03 | 重庆医科大学 | Biological big data analysis system and method based on Linux single command line |
| CN112101536A (en) * | 2020-08-30 | 2020-12-18 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Lightweight distributed multi-task collaboration framework |
| CN112395085A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN112395085B (en) * | 2020-11-05 | 2022-10-25 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN112579289A (en) * | 2020-12-21 | 2021-03-30 | 中电福富信息科技有限公司 | Distributed analysis engine method and device capable of achieving intelligent scheduling |
| CN112579289B (en) * | 2020-12-21 | 2023-06-13 | 中电福富信息科技有限公司 | Distributed analysis engine method and device capable of being intelligently scheduled |
| CN112540841A (en) * | 2020-12-28 | 2021-03-23 | 智慧神州(北京)科技有限公司 | Task scheduling method and device, processor and electronic equipment |
| CN112667901A (en) * | 2020-12-31 | 2021-04-16 | 中国电子信息产业集团有限公司第六研究所 | Social media data acquisition method and system |
| CN112667901B (en) * | 2020-12-31 | 2024-04-26 | 中国电子信息产业集团有限公司第六研究所 | Social media data acquisition method and system |
| CN112817732A (en) * | 2021-02-26 | 2021-05-18 | 国网电力科学研究院有限公司 | Stream data processing method and system suitable for cloud-side collaborative multi-data-center scene |
| CN112817732B (en) * | 2021-02-26 | 2022-11-11 | 国网电力科学研究院有限公司 | Stream data processing method and system suitable for cloud-edge collaborative multi-data-center scene |
| CN113626161A (en) * | 2021-07-09 | 2021-11-09 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN113626161B (en) * | 2021-07-09 | 2023-12-22 | 中国科学院信息工程研究所 | Distributed multi-user data scheduling method and system |
| CN114153613A (en) * | 2021-12-07 | 2022-03-08 | 杭州电子科技大学 | Distributed communication scheduling method facing edge calculation and cooperation |
| CN114756383A (en) * | 2022-06-15 | 2022-07-15 | 苏州浪潮智能科技有限公司 | Distributed computing method, system, device and storage medium |
| CN116069464B (en) * | 2022-12-19 | 2024-01-16 | 深圳计算科学研究院 | Optimization method and device based on distributed storage call data execution |
| CN116069464A (en) * | 2022-12-19 | 2023-05-05 | 深圳计算科学研究院 | Optimization method and device based on distributed storage call data execution |
| CN117155929A (en) * | 2023-10-31 | 2023-12-01 | 浪潮电子信息产业股份有限公司 | Communication method, system, electronic equipment and readable storage medium of distributed cluster |
| CN117155929B (en) * | 2023-10-31 | 2024-02-09 | 浪潮电子信息产业股份有限公司 | Communication method, system, electronic equipment and readable storage medium of distributed cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111367630A (en) | Multi-user multi-priority distributed cooperative processing method based on cloud computing | |
| Mattess et al. | Scaling mapreduce applications across hybrid clouds to meet soft deadlines | |
| US9916183B2 (en) | Scheduling mapreduce jobs in a cluster of dynamically available servers | |
| CN107025205B (en) | Method and equipment for training model in distributed system | |
| Tan et al. | Coupling task progress for mapreduce resource-aware scheduling | |
| Samal et al. | Analysis of variants in round robin algorithms for load balancing in cloud computing | |
| US10977070B2 (en) | Control system for microkernel architecture of industrial server and industrial server comprising the same | |
| Boutaba et al. | On cloud computational models and the heterogeneity challenge | |
| US20170024251A1 (en) | Scheduling method and apparatus for distributed computing system | |
| WO2019223596A1 (en) | Method, device, and apparatus for event processing, and storage medium | |
| CN109564528B (en) | System and method for computing resource allocation in distributed computing | |
| CN110569252B (en) | Data processing system and method | |
| Liu et al. | Preemptive hadoop jobs scheduling under a deadline | |
| Rupprecht et al. | SquirrelJoin: Network-aware distributed join processing with lazy partitioning | |
| CN110659278A (en) | Graph data distributed processing system based on CPU-GPU heterogeneous architecture | |
| Malekimajd et al. | Optimal map reduce job capacity allocation in cloud systems | |
| WO2024021489A1 (en) | Task scheduling method and apparatus, and kubernetes scheduler | |
| Zhang et al. | Reservation-based resource scheduling and code partition in mobile cloud computing | |
| CN104112049A (en) | P2P (peer-to-peer) architecture based cross-data-center MapReduce task scheduling system and P2P architecture based cross-data-center MapReduce task scheduling method | |
| Komarasamy et al. | A novel approach for Dynamic Load Balancing with effective Bin Packing and VM Reconfiguration in cloud | |
| Wu et al. | Dynamic acceleration of parallel applications in cloud platforms by adaptive time-slice control | |
| Teng et al. | Scheduling real-time workflow on MapReduce-based cloud | |
| CN110502337B (en) | Optimization system for shuffling stage in Hadoop MapReduce | |
| Alhussian et al. | An unfair semi-greedy real-time multiprocessor scheduling algorithm | |
| Hsiao et al. | A usage-aware scheduler for improving MapReduce performance in heterogeneous environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 102209 southeast, 6th floor, block B, national power investment Central Research Institute, South District, future science city, Changping District, Beijing Applicant after: BEIJING KEYWARE Co.,Ltd. Address before: 102208 key technology on the fourth floor of the production building of the second Pinzi Bona group, Huilongguan, Changping District, Beijing Applicant before: BEIJING KEYWARE Co.,Ltd. |
|
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200703 |