CN115543345A - Distributed computing system for power time sequence data and implementation method thereof - Google Patents
Distributed computing system for power time sequence data and implementation method thereof Download PDFInfo
- Publication number
- CN115543345A CN115543345A CN202211112639.XA CN202211112639A CN115543345A CN 115543345 A CN115543345 A CN 115543345A CN 202211112639 A CN202211112639 A CN 202211112639A CN 115543345 A CN115543345 A CN 115543345A
- Authority
- CN
- China
- Prior art keywords
- module
- subsystem
- server control
- computer
- control module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a distributed computing system for power time sequence data and an implementation method thereof, wherein the system comprises a Web management module, a data processing module and a data processing module, wherein the Web management module is used for resource monitoring, strategy configuration and operation management; the Server control module is used for distributing operation instructions, receiving and analyzing operation logs, and monitoring the resource state and load balance of the computer; the Agent calculation module is used for receiving the operation instruction, calling a calculation program for processing the power time sequence data, executing the operation, collecting the operation log and the hardware resource use information and transmitting the operation log and the hardware resource use information to the Server control module; the Web management module, the Server control module and the Agent calculation module are all deployed on a computer, and the Server control module is respectively connected with the Web management module and the Agent calculation module through message middleware. The invention can effectively improve the data calculation efficiency, expand the calculation resources, effectively monitor the task execution process, realize the effective utilization and parallel calculation of the calculation resources and improve the utilization rate and the calculation efficiency of the computer resources.
Description
Technical Field
The invention belongs to the technical field of power time sequence data processing, and particularly relates to a distributed computing system for power time sequence data and an implementation method thereof.
Background
With the development of digitalization and intellectualization in the power field, in the field of calculation and processing of time sequence data in the power production process, the data quantity to be calculated and processed is larger and larger, the business requirements are more and more complex, and the requirements on the scale and the effectiveness of data calculation are higher and higher.
For the prior art for processing electric power time sequence data, a scheduling and computing framework in a single machine mode is generally adopted for computing, and the mode has the problems of single centralization, single-node operation, no load balance, incapability of stateless and horizontal expansion and the like; various computing resources are independent of each other, and the maximum computing power of hardware resources cannot be exerted. With the increase of various computing scales, the computing expansibility of the mode has the limitation of expanding computing resources, and cannot meet the higher and higher computing requirements faced at present. Therefore, how to implement multi-resource node parallel computation in the processing of power time series data, achieve horizontal expansion of computation and fully utilize resources is a prominent problem to be solved urgently at present.
The method can distribute the calculation operation in each node in the resource pool with dynamically-expanded scale to operate through operation distributed scheduling and resource elastic expansion mechanism when the calculation traffic aiming at the power time sequence data changes rapidly, thereby providing the horizontal expansibility of calculation and effectively solving the problems.
Disclosure of Invention
In order to solve the defects in the prior art, the invention aims to provide a distributed computing system for power time series data and an implementation method thereof, which comprise the functions of configuration management, scheduling management and resource monitoring and can realize the dynamic expansion of resources and the horizontal expansion characteristic of computation in the computing process.
The invention adopts the following technical scheme.
The invention provides a distributed computing system for power time sequence data, which comprises:
the Web management module is used for resource monitoring, strategy configuration and job management;
the Server control module is used for distributing operation instructions, receiving and analyzing operation logs, and monitoring the resource state and load balance of the computer;
the Agent computing module is used for receiving the operation instruction, calling a computing program, executing the operation, collecting the operation log and the hardware resource use information and transmitting the operation log and the hardware resource use information to the Server control module;
the Web management module, the Server control module and the Agent calculation module are all arranged on a computer, the Server control module is respectively connected with the Web management module and the Agent calculation module through message middleware, data of the Web management module, the Server control module and the Agent calculation module are stored in the data storage module, and a calculation program called by the Agent calculation module is a calculation program used for processing power time sequence data.
Preferably, the Web management module further includes a resource management subsystem, a job management subsystem, and an execution history subsystem, and the resource management subsystem, the job management subsystem, and the execution history subsystem respectively set a visual interface;
the resource management subsystem is used for manual registration, information maintenance and grouping management of the computer;
the operation management subsystem is used for operation management configuration, including newly-added operation, operation start and stop and batch operation introduction;
and the execution history subsystem is used for displaying the job execution record, and packaging the job execution time, the execution state and the execution analysis result.
Preferably, the Server control module further includes an instruction sending subsystem, and the Server control module generates a new job instruction set according to the job parameters and the start-stop instruction configured by the job management subsystem of the Web management module, and distributes the job instruction through the instruction sending subsystem, and sends the job instruction to the message middleware.
Preferably, the Server control module further comprises a log analysis subsystem, and the log analysis subsystem receives the job execution log sent by the Agent calculation module through the message middleware, analyzes the job execution log, and stores the analysis result in the data storage module.
Preferably, the Server control module can be installed and deployed on a plurality of computers in a distributed mode and is used for achieving load balancing in the work of the Server control module.
Preferably, the Server control module further comprises a resource status monitoring subsystem, and the resource status monitoring subsystem comprises a status monitoring program and a resource real-time updating program;
the state monitoring program monitors the computer in real time through the program coordination service, namely the state monitoring program monitors the computer installed by the Server control module or the Agent calculation module and executes the updating logic;
and the resource real-time updating program receives the computer hardware resource use information sent by the Agent computing module through the message middleware.
Preferably, the Server control module further comprises a load balancing subsystem, the load balancing subsystem supports load balancing strategies comprising weighted polling, sequential distribution, priority according to CPU and priority according to memory, and can select and distribute the computer for executing the operation according to the operation setting strategy;
wherein, the weighted polling strategy comprises: according to the weight value set by the computer, distributing the weight values from large to small;
the sequential allocation strategy comprises: distributing according to the computer registration sequence;
the priority policy according to the CPU comprises the following steps: distributing according to the residual proportion of the computer CPU resource, wherein the higher residual proportion is distributed preferentially;
the memory priority policy comprises: and allocating according to the size of the residual proportion of the computer memory, wherein the higher residual proportion is allocated preferentially.
Preferably, the Agent computing module can be installed and deployed on a plurality of computing machines, and the Agent computing module actively registers to the Server control module through the program coordination service when being installed for the first time;
the computer with the Agent calculating module is a monitoring object of the Server control module resource state monitoring subsystem.
Preferably, the Agent computation module further comprises a computation engine scheduling subsystem, and the computation engine scheduling subsystem receives the job instruction transmitted by the message middleware and executes the job;
the calculation engine scheduling subsystem receives the instruction sent by the instruction sending subsystem of the Server control module through the message middleware, adapts the calculation engine according to the instruction information, calls and starts the calculation program and executes the operation.
Preferably, the Agent calculation module further comprises a log collection subsystem, the log collection subsystem is responsible for collecting logs generated in the job execution process and transmitting the logs to the Server control module through the message middleware, and the Server control module can analyze and store log records.
Preferably, the Agent computation module further comprises a hardware resource monitoring subsystem, wherein the hardware resource monitoring subsystem uses a hardware device monitoring service, monitors and collects hardware resource use information of the operating environment, and can transmit the hardware resource use information to the Server control module through the message middleware.
The invention also provides a distributed computing system implementation method for the power time sequence data, which comprises the following steps:
step 1, registering a computer provided with a Server control module and a computer provided with an Agent calculation module to a program coordination service;
step 2, the server control module resource state monitoring subsystem respectively judges whether the computer initiating the registration is taken over, if yes, the fourth step is carried out;
step 3, storing computer information, and taking over the computer by the Server control module;
step 4, the Agent computing module hardware resource management subsystem collects the Agent computing module operating environment hardware resource information and sends the information to the message middleware;
step 5, the server control module resource state monitoring subsystem acquires the online state of the computer of the installation Agent computing module and the use information of the hardware resource in real time through a program coordination service transmission and message middleware and updates the online state and the use information of the hardware resource to the data storage module;
step 6, the job management subsystem of the Web management module initiates a job instruction, stores the job instruction information into the data storage module, and simultaneously sends the instruction information to the message middleware;
step 7, the Server control module receives the instruction information transmitted by the message middleware, and the load balancing subsystem and the instruction sending subsystem generate new instruction information according to the instruction information and send the new instruction information to the message middleware;
step 8, the agent calculation module calculation engine scheduling subsystem receives the instruction information through the message middleware, adapts the calculation engine according to the instruction information, calls and starts a calculation program for processing the power time sequence data, and executes the operation;
step 9, the log collection subsystem monitors the operation execution process and sends the collected execution log to the message middleware;
and step 10, the server control module log analysis subsystem receives the job execution log through the message middleware, analyzes and processes the job log, and stores the processed result into the data storage module.
Preferably, the step 1 further comprises: and a state monitoring program of the resource state monitoring subsystem updates the state change of the computer in the database.
Preferably, the step 5 further comprises: and the resource management subsystem acquires the Agent computing module resource information through the data storage module and visually displays the information.
Preferably, in the step 8, when the calculation program processes the power time series data, the power time series data can be read from the time series database, and the processed result is stored in the time series database.
Preferably, the step 10 further comprises: and the execution history subsystem acquires the job execution information from the data storage module and performs visual display.
The invention also provides a terminal, which comprises a processor and a storage medium; the storage medium is to store instructions;
the processor is configured to operate in accordance with the instructions to perform the steps of the above-described distributed computing system implemented method for power timing data.
The invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above-described distributed computing system implementation method for power time series data.
Compared with the prior art, the method has the advantages that the method can effectively improve the calculation efficiency of the power time sequence data, expand and effectively monitor the calculation resources, distribute the tasks by using a reasonable technical means, monitor the task execution process, realize the effective utilization and parallel calculation of the calculation resources, and improve the utilization rate and the calculation efficiency of the computer resources.
The scheme of the invention is used for monitoring the state of the computing resource and receiving a large amount of operation from the Web management module through the Server control module, and realizes the distribution and execution of the computing operation at each resource node by linking with the Agent computing module. The Server control module and the Agent calculation module are matched with each other, distributed execution and monitoring management of calculation operation are achieved, and calculation efficiency of power time sequence data is effectively improved. The invention can realize the expansion of the existing calculation scale from a single node to a multi-node calculation cluster and realize the dynamic adjustment and horizontal expansion of the calculation resources.
Drawings
FIG. 1 is a schematic diagram of the overall architecture of a distributed computing system for power timing data in the present invention;
fig. 2 is a schematic overall flow chart of a distributed computing system implementation method for power time series data in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The embodiments described herein are only some embodiments of the invention, and not all embodiments. All other embodiments obtained by a person skilled in the art without any inventive step based on the spirit of the present invention are within the scope of the present invention.
As shown in FIG. 1, the present invention provides a distributed computing system for power timing data, the system comprising: a Web management module, a Server control module and an Agent calculation module,
the Web management module comprises a resource management subsystem, an operation management subsystem and an execution history subsystem, wherein the resource management subsystem, the operation management subsystem and the execution history subsystem are respectively provided with a visual interface, and the Web management module is used for resource monitoring, strategy configuration and operation management; the data storage module comprises a relational database and a cache data database. And the data of the Web management module, the Server control module and the Agent calculation module are stored in the data storage module.
Further, the Web management module implementation framework is divided into a back end and a front end, the back end is implemented by using a Springboot framework, and the front end is implemented by using a LayUI display framework. The back end is used for realizing business logic and providing a data interface for the outside; the front end is used for realizing a visual interface, calling the back end data interface to realize visual display and interacting with the back end.
The Server control module is connected with the Web management module through message middleware, and also comprises a log analysis subsystem, a resource state monitoring subsystem, a load balancing subsystem and an instruction sending subsystem, and is used for distributing an operation instruction, receiving and analyzing the operation log, and monitoring the resource state and the load balance of a computer;
the Agent computing module is connected and communicated with the distributed computing Server module through the message middleware, and further comprises a log acquisition subsystem, a computing engine scheduling subsystem and a hardware resource monitoring subsystem, wherein the log acquisition subsystem is used for receiving a job instruction and calling a computing program to execute a job, and can acquire a job log and hardware resource use information and transmit the job log and the hardware resource use information to the Server control module;
the Web management module, the Server control module and the Agent calculation module are all deployed on a computer, and a calculation program called by the Agent calculation module is a calculation program used for processing power time sequence data.
Specifically, a resource management subsystem in the Web management module is provided with a visual interface and is responsible for manual registration, information viewing and grouping management of computers;
the manual registration is used for performing manual registration through the resource management subsystem when the automatic registration of the computing machine fails, and editing maintenance information;
the information related to information viewing comprises the name, IP address, operating system name, operating system bit number, total hard disk space, total CPU number and online state of the computer; the computer operating system can be Windows or Linux system, and supports multi-platform operating environment
Group management involves dividing the computing machines into different types of computing resource groups based on job type.
The job management subsystem of the Web management module is responsible for job management configuration, including newly-added jobs, job start-stop and batch job introduction; adding operation according to the operation information, configuring operation types, load balancing strategies, computing resource groups and computing parameter information; the operation starting and stopping comprises starting and stopping instructions initiated by a user; the batch operation importing comprises editing batch operation information according to an importing template and importing and generating batch operations.
And the execution history subsystem of the Web management module is responsible for displaying the job execution records, and packaging the job execution time, the execution state and the execution analysis result.
Furthermore, the Server control module can be installed and deployed on a plurality of computers in a distributed mode and is used for realizing load balancing in the work of the Server control module.
And the instruction sending subsystem of the Server control module generates a new instruction set according to the operation parameters, the start-stop instruction and the marked computing resources selected by the load balancing subsystem configured by the operation management subsystem, and sends the new instruction set to the message middleware.
The Server control module also comprises a log analysis subsystem, and the log analysis subsystem receives the job execution log sent by the Agent calculation module through the message middleware, analyzes the job execution log and stores the analysis result into the data storage module.
And the resource state monitoring subsystem and the instruction sending subsystem of the Server control module are connected and communicated with the distributed Agent computing module through message middleware and are used for supporting state monitoring, task distribution and task start-stop management. The Server control module program is deployed on a plurality of computers, can perform processing load balancing on a large number of job instructions from the Web management module and a large number of logs of the Agent computing module, and adopts an MVC mode in a specific implementation mode, and a quartz framework is used at the bottom layer.
The resource state monitoring subsystem of the Server control module comprises a state monitoring program and a resource real-time updating program, wherein the state monitoring program adopts a program coordination service Zookeeper to monitor a plurality of distributed computers provided with the Server control module or the Agent computing module in real time, acquire the state change of the distributed computers in real time and execute updating logic; and the real-time resource updating program receives the computer hardware resource use information of the installation Agent computing module through the message middleware.
The load balancing subsystem of the Server control module acquires the use information of the hardware resources of the computer through the data storage module, and marks the distributed computers according to the received operation strategy configuration information, wherein the load balancing subsystem supports the load balancing strategies comprising weighted polling, sequential distribution, CPU priority and memory priority, and can select and distribute the computers for executing the operation according to the operation setting strategy;
wherein, the weighted polling strategy comprises: according to the weight value set by the computer, distributing the weight values from large to small;
the sequential allocation policy includes: distributing according to the computer registration sequence;
the priority strategy according to the CPU comprises the following steps: distributing according to the residual proportion of the computer CPU resource, wherein the higher residual proportion is distributed preferentially;
the memory priority policy comprises: and allocating according to the size of the residual proportion of the computer memory, wherein the higher residual proportion is allocated preferentially.
Furthermore, the Agent computing module can be installed and deployed on a plurality of computers, and is a computing resource part of the distributed computing system; the Agent computing module is initiatively registered to the Server control module through the program coordination service when being installed for the first time, and a computer provided with the Agent computing module is monitored by a resource state monitoring subsystem of the Server control module through the program coordination service.
Preferably, the Server control module for monitoring the computer on which the Agent computation module is installed may be installed on the same computer as the Agent computation module or may be installed on another computer.
The Agent computing module also comprises a computing engine scheduling subsystem, and the computing engine scheduling subsystem is used for receiving the job instruction transmitted by the message middleware and executing the job;
the calculation engine scheduling subsystem receives the instruction sent by the instruction sending subsystem of the Server control module through the message middleware, adapts the calculation engine according to the instruction information, calls and starts the calculation program and executes the operation.
The Agent calculation module also comprises a log collection subsystem, the log collection subsystem is responsible for collecting logs generated in the operation execution process and transmitting the logs to the Server control module through the message middleware, and the Server control module can analyze and store log records.
The Agent computing module further comprises a hardware resource monitoring subsystem, wherein the hardware resource monitoring subsystem uses a hardware device monitoring service, in the embodiment, a hardware device monitoring service Sigar is selected, and is used for monitoring and collecting hardware resource use information of the operating environment and transmitting the hardware resource use information to the Server control module through the message middleware.
As shown in fig. 2, the present invention further provides a method for implementing a distributed computing system, which is used to implement the distributed computing system. The method specifically comprises the following steps:
step 1, registering a computer provided with a Server control module and a computer provided with an Agent calculation module to a program coordination service;
step 2, the server control module resource state monitoring subsystem respectively judges whether the computer initiating the registration is taken over, if yes, the step 4 is carried out, and if not, the step 3 is carried out;
the Server control module judges whether the Agent calculation module is taken over or not through information transmitted by the program coordination service.
Step 3, storing computer information, and taking over the computer by the Server control module;
specifically, the resource monitoring subsystem of the Server control module is used for storing computer information for installing the Agent computing module and taking over the computer.
Step 4, the Agent computing module collects the operating environment hardware resource information of the Agent computing module through a hardware resource management subsystem of the Agent computing module and sends the operating environment hardware resource information to a message middleware;
step 5, the server control module resource state monitoring subsystem acquires the online state of the computer of the installation Agent computing module and the use information of the hardware resource in real time through a program coordination service transmission and message middleware and updates the online state and the use information of the hardware resource to the data storage module;
furthermore, the Web management module can acquire the hardware resource use information of the computer where the Agent module is located in the data storage module in real time and visually display the hardware resource use information.
Step 6, the job management subsystem of the Web management module initiates a job instruction, stores the job instruction information into the data storage module, and simultaneously sends the instruction information to the message middleware;
specifically, the job instruction information sent by the job management subsystem includes a job ID, an Agent calculation resource group, a load balancing policy, a start time, an end time, a job type, and a job parameter.
7, the Server control module receives the operation instruction information transmitted by the message middleware, and the load balancing subsystem and the instruction sending subsystem generate new instruction information according to the instruction information and send the new instruction information to the message middleware;
specifically, generating new instruction information further includes: and selecting a matched Agent computer according to the load balancing strategy and the Agent computing resource group in the received instruction information and the hardware resource use condition of the Agent computing resource group computer, and packaging the Agent computer information and the original instruction information into new instruction information.
And the hardware resource use condition of the Agent computing resource group computer is obtained through the step 4.
Furthermore, the command sending subsystem sends new command information to the message middleware according to the matched Agent computer information.
Step 8, the agent calculation module calculation engine scheduling subsystem receives the instruction information through the message middleware, adapts the calculation engine according to the instruction information, calls and starts a calculation program for processing the power time sequence data, and executes the operation;
the calculation program is a program package which comprises a formula and calculation logic and is used for calculating indexes and is used for processing power time sequence data, the calculation scheduling engine is responsible for starting the corresponding calculation program, the calculation program is classified according to a programming language, and different calculation programs depend on different calculation engines to be scheduled and executed.
When the calculation program processes the power time sequence data, the power time sequence data can be read from the time sequence database, and the processed result is stored in the time sequence database.
Step 9, the log collection subsystem monitors the operation execution process and sends the collected execution log to the message middleware;
and the log collection subsystem monitors the job execution process and sends the monitored job execution change information to the message middleware. The job execution change information includes job execution start time, end time, execution result status, and execution failure time log.
And step 10, the log analysis subsystem of the Server control module receives the job execution log through the message middleware, analyzes and processes the job log, and stores the processed result into the data storage module.
The Server control module comprises a log analysis subsystem, a data storage module and a Web management module, wherein the log analysis subsystem of the Server control module receives job execution log information, analyzes and processes job execution progress and state information, and stores results into the data storage module so that an execution history subsystem of the Web management module can acquire the results and display the results through a visual interface.
The term is defined as:
sigar: system Information collector And reporting tool.
Compared with the prior art, the method has the advantages that the data computing efficiency can be effectively improved, the computing resources can be expanded and effectively monitored, the tasks are distributed by using a reasonable technical means, the task execution process is monitored, the computing resources are effectively utilized and calculated in parallel, and the method has the advantage of improving the utilization rate and the computing efficiency of the computer resources.
The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.
The computer-readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as a punch card or an in-groove protruding structure with instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (18)
1. A distributed computing system for power timing data, the system comprising:
the Web management module is used for resource monitoring, strategy configuration and job management;
the Server control module is used for distributing operation instructions, receiving and analyzing operation logs, and monitoring the resource state and load balance of the computer;
the Agent calculation module is used for receiving the operation instruction, calling a calculation program to execute the operation, collecting the operation log and the hardware resource use information and transmitting the operation log and the hardware resource use information to the Server control module;
the Web management module, the Server control module and the Agent calculation module are all deployed on a computer, the Server control module is connected with the Web management module and the Agent calculation module through message middleware respectively, data of the Web management module, the Server control module and the Agent calculation module are stored in the data storage module, and a calculation program called by the Agent calculation module is a calculation program used for processing power time sequence data.
2. The distributed computing system for power timing data of claim 1,
the Web management module also comprises a resource management subsystem, an operation management subsystem and an execution history subsystem, and the resource management subsystem, the operation management subsystem and the execution history subsystem are respectively provided with a visual interface;
the resource management subsystem is used for manual registration, information maintenance and grouping management of the computer;
the operation management subsystem is used for operation management configuration, including newly-added operation, operation start and stop and batch operation import;
and the execution history subsystem is used for displaying the job execution record, and packaging the job execution time, the execution state and the execution analysis result.
3. The distributed computing system for power timing data of claim 2,
the Server control module also comprises an instruction sending subsystem, and the Server control module generates a new operation instruction set according to operation parameters and start-stop instructions configured by the operation management subsystem of the Web management module, distributes operation instructions through the instruction sending subsystem and sends the operation instructions to the message middleware.
4. The distributed computing system for power timing data of claim 1,
the Server control module also comprises a log analysis subsystem, the log analysis subsystem receives the job execution log sent by the Agent calculation module through the message middleware, analyzes the job execution log and stores the analysis result in the data storage module.
5. The distributed computing system for power timing data of claim 3,
the Server control module can be installed and deployed on a plurality of computers in a distributed mode and is used for achieving load balancing in the work of the Server control module.
6. The distributed computing system for power timing data according to claim 5,
the Server control module also comprises a resource state monitoring subsystem, wherein the resource state monitoring subsystem comprises a state monitoring program and a resource real-time updating program;
the state monitoring program monitors the computer in real time through the program coordination service, namely the state monitoring program monitors the computer installed by the Server control module or the Agent calculation module and executes the updating logic;
and the resource real-time updating program receives the computer hardware resource use information sent by the Agent computing module through the message middleware.
7. The distributed computing system for power timing data according to claim 3,
the Server control module also comprises a load balancing subsystem which supports load balancing strategies comprising weighted polling, sequential distribution, CPU priority and memory priority and can select and distribute computers for executing the operation according to the operation setting strategy;
wherein, the weighted polling strategy comprises: according to the weight value set by the computer, distributing the weight values from large to small;
the sequential allocation policy includes: distributing according to the computer registration sequence;
the priority policy according to the CPU comprises the following steps: distributing according to the residual proportion of the computer CPU resource, wherein the higher residual proportion is distributed preferentially;
the memory priority policy comprises: and allocating according to the size of the residual proportion of the computer memory, wherein the higher residual proportion is allocated preferentially.
8. The distributed computing system for power timing data according to claim 1,
the Agent computing module can be installed and deployed on a plurality of computing machines, and actively registers to the Server control module through the program coordination service when the Agent computing module is installed for the first time;
and the computer provided with the Agent computing module is a monitoring object of the resource state monitoring subsystem of the Server control module.
9. The distributed computing system for power timing data of claim 3,
the Agent computing module also comprises a computing engine scheduling subsystem, and the computing engine scheduling subsystem receives the job instruction transmitted by the message middleware and executes the job;
the calculation engine scheduling subsystem receives the instruction sent by the instruction sending subsystem of the Server control module through the message middleware, adapts the calculation engine according to the instruction information, calls and starts the calculation program and executes the operation.
10. The distributed computing system for power timing data according to claim 9,
the Agent calculation module also comprises a log collection subsystem, the log collection subsystem is responsible for collecting logs generated in the operation execution process and transmitting the logs to the Server control module through the message middleware, and the Server control module can analyze and store log records.
11. The distributed computing system for power timing data according to claim 9,
the Agent calculation module also comprises a hardware resource monitoring subsystem, the hardware resource monitoring subsystem uses hardware equipment monitoring service, monitors and collects the use information of the hardware resources of the operating environment, and can transmit the use information to the Server control module through the message middleware.
12. A method for implementing a distributed computing system for power time series data according to any one of claims 1 to 11, the method comprising the steps of:
step 1, registering a computer provided with a Server control module and a computer provided with an Agent calculation module to a program coordination service;
step 2, the server control module resource state monitoring subsystem respectively judges whether the computer initiating the registration is taken over, if yes, the fourth step is carried out;
step 3, storing computer information, and taking over the computer by the Server control module;
step 4, the Agent computing module hardware resource management subsystem collects the Agent computing module operating environment hardware resource information and sends the information to the message middleware;
step 5, the server control module resource state monitoring subsystem acquires the online state of the computer of the installation Agent computing module and the use information of the hardware resource in real time through a program coordination service transmission and message middleware and updates the online state and the use information of the hardware resource to the data storage module;
step 6, the job management subsystem of the Web management module initiates a job instruction, stores the job instruction information into the data storage module, and simultaneously sends the instruction information to the message middleware;
step 7, the Server control module receives the instruction information transmitted by the message middleware, and the load balancing subsystem and the instruction sending subsystem generate new instruction information according to the instruction information and send the new instruction information to the message middleware;
8, the agent calculation module calculation engine scheduling subsystem receives the instruction information through the message middleware, adapts the calculation engine according to the instruction information, calls and starts a calculation program for processing the power time sequence data, and executes the operation;
step 9, the log collection subsystem monitors the job execution process and sends the collected execution log to the message middleware;
and step 10, the server control module log analysis subsystem receives the job execution log through the message middleware, analyzes and processes the job log, and stores the processed result into the data storage module.
13. The distributed computing system implemented method for power timing data of claim 12,
the step 1 further comprises: and the state monitoring program of the resource state monitoring subsystem updates the state change of the computer in the database.
14. The distributed computing system implemented method for power timing data of claim 12,
the step 5 further comprises: and the resource management subsystem acquires the resource information of the Agent calculation module through the data storage module and visually displays the resource information.
15. The distributed computing system implemented method for power timing data of claim 12,
in the step 8, when the calculation program processes the power time sequence data, the power time sequence data can be read from the time sequence database, and the processed result is stored in the time sequence database.
16. The distributed computing system implemented method for power timing data of claim 12,
the step 10 further comprises: and the execution history subsystem acquires the job execution information from the data storage module and performs visual display.
17. A terminal comprising a processor and a storage medium; the method is characterized in that:
the storage medium is to store instructions;
the processor is configured to operate in accordance with the instructions to perform the steps of the method according to any one of claims 12 to 16.
18. Computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 12 to 16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211112639.XA CN115543345A (en) | 2022-09-13 | 2022-09-13 | Distributed computing system for power time sequence data and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211112639.XA CN115543345A (en) | 2022-09-13 | 2022-09-13 | Distributed computing system for power time sequence data and implementation method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115543345A true CN115543345A (en) | 2022-12-30 |
Family
ID=84728536
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211112639.XA Pending CN115543345A (en) | 2022-09-13 | 2022-09-13 | Distributed computing system for power time sequence data and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115543345A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107765A (en) * | 2023-04-14 | 2023-05-12 | 中国科学院长春光学精密机械与物理研究所 | Target range data processing system |
-
2022
- 2022-09-13 CN CN202211112639.XA patent/CN115543345A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107765A (en) * | 2023-04-14 | 2023-05-12 | 中国科学院长春光学精密机械与物理研究所 | Target range data processing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8265973B2 (en) | Analytic-based scaling of information technology resources | |
| CN110597626B (en) | Method, device and system for allocating resources and tasks in distributed system | |
| CN104022902A (en) | Method and system of monitoring server cluster | |
| US20180011742A1 (en) | Job scheduling management | |
| CN110796343A (en) | Intelligent dispatching method, device and system | |
| Trihinas et al. | Monitoring elastically adaptive multi-cloud services | |
| CN104731580A (en) | Automation operation and maintenance system based on Karaf and ActiveMQ and implement method thereof | |
| CN113742031A (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN105592122A (en) | Cloud platform monitoring method and cloud platform monitoring system | |
| US9195535B2 (en) | Hotspot identification | |
| CN108874623A (en) | Distributed type assemblies method for monitoring performance, device, equipment, system and storage medium | |
| US10255127B2 (en) | Optimized diagnostic data collection driven by a ticketing system | |
| CN115543345A (en) | Distributed computing system for power time sequence data and implementation method thereof | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN110727563A (en) | Cloud service alarm method and device for preset customer | |
| CN114168297A (en) | Method, device, equipment and medium for scheduling collection tasks | |
| CN109213743B (en) | Data query method and device | |
| US12050505B2 (en) | Systems and methods for automatically applying configuration changes to computing clusters | |
| CN112817687A (en) | Data synchronization method and device | |
| JP2019219743A (en) | Load test system | |
| CN115102730A (en) | Integrated monitoring method for multiple devices | |
| CN110247802B (en) | Resource configuration method and device for cloud service single-machine environment | |
| CN113672472A (en) | Disk monitoring method and device | |
| JP2018032245A (en) | Computer system and resource control method | |
| CN110796474A (en) | Automatic adjustment method and device for activity operation and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |