CN111694671B

CN111694671B - Big data component management method, device, server, electronic equipment and system

Info

Publication number: CN111694671B
Application number: CN202010537575.2A
Authority: CN
Inventors: 赵宇; 徐寅斐; 侯雪峰
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2020-06-12
Filing date: 2020-06-12
Publication date: 2023-09-01
Anticipated expiration: 2040-06-12
Also published as: CN111694671A

Abstract

The embodiment of the invention provides a big data component management method, a device, a server, electronic equipment and a system, when the requirement of executing a task to be executed of a specified big data component exists, the server can acquire a corresponding operation instruction, and because the task execution specification and the task execution strategy of each big data component are preset on the server, the task execution specification and the task execution strategy of the task to be executed can be read according to the task name of the task to be executed of the specified big data component in the operation instruction, and the task execution specification and the task execution strategy agree with the information related to task execution such as scripts, execution sequences and the like required by task execution, the electronic equipment in the equipment cluster can automatically execute the task to be executed without manual operation, thereby improving the efficiency of managing the big data component.

Description

Big data component management method, device, server, electronic equipment and system

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a method, an apparatus, a server, an electronic device, and a system for managing big data components.

Background

With the development of big data and AI (Artificial Intelligence ) technology, big data technology has shown increasingly important value in applications of various industries. Big data technology has long relied on open source system and tool implementations, where open source systems and tools are commonly referred to as big data components, there are about 100 or more big data components currently available, there are at least 30, and big data components are mostly deployed in distributed clusters (e.g., elastosearch (an open source distributed search and data analysis engine), hadoop (an open source software framework for storing data and running applications on a cluster of commercial devices), one cluster needs to deploy hundreds or even thousands of electronic devices, and for big data component management, management personnel need to manually perform on each electronic device separately.

The big data component management mainly refers to management of various tasks in a life cycle of big data component use, and comprises execution of tasks such as installation and deployment, configuration change, start and stop operation and the like. In general, any task needs to be executed through a series of processes, for example, a large data component needs to be downloaded, copied and deployed by all electronic devices of a cluster, configured by all electronic devices of a cluster, and started by all electronic devices of a cluster. The usual task execution mode is to execute each operation manually, which results in extremely low efficiency of large data component management.

Disclosure of Invention

The embodiment of the invention aims to provide a big data component management method, a big data component management device, a big data component management server, electronic equipment and a big data component management system, so that big data component management efficiency is improved. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a big data component management method, applied to a server, where the method includes:

acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component;

according to the task name, reading a task execution specification of a task to be executed from task execution specifications of all preset big data components, and reading a task execution policy of the task to be executed from task execution policies of all preset big data components;

and according to the read task execution strategy, issuing the read task execution specification to the equipment cluster, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

Optionally, the step of acquiring the operation instruction includes:

and receiving an operation instruction input by a user through an application program interface.

Optionally, according to the task name, reading a task execution specification of a task to be executed from task execution specifications of preset big data components, and reading a task execution policy of the task to be executed from task execution policies of preset big data components, including:

And reading a task execution specification and a task execution strategy of a task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

Optionally, the database stores task execution specifications of execution steps of different tasks of each big data component in a buffer queue mode;

according to the task name, reading a task execution specification and a task execution strategy of a task to be executed from a database, wherein the task execution specification and the task execution strategy comprise the following steps:

according to task names, sequentially reading task execution specifications of execution steps of the tasks to be executed from a database according to the storage sequence of the cache queue, and reading task execution strategies of the tasks to be executed from the database according to the task names;

and issuing the read task execution specification to the device cluster according to the read task execution strategy, wherein the step comprises the following steps:

according to the read task execution strategy, issuing the task execution specification of each execution step of the read task to be executed to the equipment cluster in a buffer queue mode, so that the electronic equipment in the equipment cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

Optionally, the step of issuing the read task execution specification to the device cluster according to the read task execution policy includes:

and according to the read task execution strategy, issuing the read task execution specification to the equipment cluster through a remote procedure call protocol, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

Optionally, the operation instruction further includes the number of electronic devices;

selecting a plurality of electronic devices from a device cluster according to the number of the electronic devices;

and issuing the read task execution specification into a plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

Optionally, after the step of issuing the read task execution specification to the device cluster according to the read task execution policy, the method further includes:

receiving a task execution result fed back by the equipment cluster;

according to the task execution result, adjusting a task execution strategy;

and according to the adjusted task execution strategy, issuing the task execution specification to the equipment cluster.

In a second aspect, an embodiment of the present invention provides a method for managing big data components, which is applied to an electronic device in a device cluster, where the method includes:

receiving a task execution specification issued by a server, wherein the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from the task execution specification of each preset big data component and according to a task execution strategy of the task to be executed read from the task execution strategy of each preset big data component;

and executing the task to be executed according to the task execution specification.

Optionally, the step of receiving the task execution specification issued by the server includes:

receiving a task execution specification of each execution step of a task to be executed, which is sent by a server in a cache queue mode;

according to the task execution specification, executing the task to be executed comprises the following steps:

and sequentially executing the execution steps of the task to be executed according to the task execution specification of the execution steps of the task to be executed.

In a third aspect, an embodiment of the present invention provides a big data component management apparatus, applied to a server, including:

the acquisition module is used for acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component;

The reading module is used for reading the task execution specification of the task to be executed from the task execution specification of each preset big data component according to the task name, and reading the task execution policy of the task to be executed from the task execution policy of each preset big data component;

and the sending module is used for sending the read task execution specification to the equipment cluster according to the read task execution strategy so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

Optionally, the acquiring module is specifically configured to: and receiving an operation instruction input by a user through an application program interface.

Optionally, the reading module is specifically configured to: and reading a task execution specification and a task execution strategy of a task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

the reading module is specifically used for: according to task names, sequentially reading task execution specifications of execution steps of the tasks to be executed from a database according to the storage sequence of the cache queue, and reading task execution strategies of the tasks to be executed from the database according to the task names;

The sending module is specifically configured to: according to the read task execution strategy, issuing the task execution specification of each execution step of the read task to be executed to the equipment cluster in a buffer queue mode, so that the electronic equipment in the equipment cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

Optionally, the sending module is specifically configured to: and according to the read task execution strategy, issuing the read task execution specification to the equipment cluster through a remote procedure call protocol, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

the sending module is specifically configured to: selecting a plurality of electronic devices from a device cluster according to the number of the electronic devices; and issuing the read task execution specification into a plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

Optionally, the apparatus further comprises:

the receiving module is used for receiving a task execution result fed back by the equipment cluster;

the adjusting module is used for adjusting the task execution strategy according to the task execution result;

And the sending module is also used for sending the task execution specification to the equipment cluster according to the adjusted task execution strategy.

In a fourth aspect, an embodiment of the present invention provides a big data component management apparatus, which is applied to an electronic device in a device cluster, and the apparatus includes:

the receiving module is used for receiving a task execution specification issued by the server, wherein the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from the task execution specification of each preset big data component and according to a task execution strategy of the task to be executed read from the task execution strategy of each preset big data component;

and the execution module is used for executing the task to be executed according to the task execution specification.

Optionally, the receiving module is specifically configured to: receiving a task execution specification of each execution step of a task to be executed, which is sent by a server in a cache queue mode;

the execution module is specifically used for: and sequentially executing the execution steps of the task to be executed according to the task execution specification of the execution steps of the task to be executed.

In a fifth aspect, an embodiment of the present invention provides a server, which is characterized by including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

A memory for storing a computer program;

and a processor, configured to implement the method provided in the first aspect of the embodiment of the present invention when executing the computer program stored in the memory.

In a sixth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and a processor, configured to implement the method provided in the second aspect of the embodiment of the present invention when executing the computer program stored in the memory.

In a seventh aspect, an embodiment of the present invention provides a computer readable storage medium, in which a computer program is stored, which when executed by a processor implements the method provided in the first aspect of the embodiment of the present invention, or the method provided in the second aspect.

In an eighth aspect, embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method provided by the first aspect of the embodiments of the present invention, or the method provided by the second aspect.

In a ninth aspect, an embodiment of the present invention provides a big data component management system, where the system includes a server and a device cluster, where the device cluster is composed of a plurality of electronic devices;

the server is used for acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component; according to the task name, reading a task execution specification of a task to be executed from task execution specifications of all preset big data components, and reading a task execution policy of the task to be executed from task execution policies of all preset big data components; according to the read task execution strategy, issuing the read task execution specification to the equipment cluster;

and the electronic equipment in the equipment cluster is used for executing the task to be executed according to the task execution specification.

According to the big data component management method, device, server, electronic equipment and system provided by the embodiment of the invention, the server reads the task execution specification of the task to be executed from the task execution specification of each preset big data component according to the task name of the task to be executed of the specified big data component in the operation instruction by acquiring the operation instruction, reads the task execution policy of the task to be executed from the task execution policy of each preset big data component, and issues the read task execution specification to the equipment cluster according to the read task execution policy, so that the electronic equipment in the equipment cluster execute the task to be executed according to the task execution specification.

When a task to be executed needs to execute a specified big data component, the server acquires a corresponding operation instruction, and because the task execution specification and the task execution strategy of each big data component are preset on the server, the task execution specification and the task execution strategy of the task to be executed can be read according to the task name of the task to be executed of the specified big data component in the operation instruction, and the task execution specification and the task execution strategy agree on the information related to task execution such as scripts, execution sequences and the like required by task execution, the electronic equipment in the equipment cluster can automatically execute the task to be executed without manual operation, thereby improving the efficiency of managing the big data component.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a big data component management method applied to a server according to an embodiment of the invention;

FIG. 2 is a schematic diagram of a system architecture according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating a big data component management method applied to a server according to another embodiment of the present invention;

fig. 4 is a flow chart of a big data component management method applied to an electronic device in a device cluster according to an embodiment of the present invention;

FIG. 5 is a schematic diagram illustrating data flow transmission of each module in the big data management system according to the embodiment of the present invention;

FIG. 6 is a flow chart of a big data component management method applied to a big data component management system according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a big data component management device applied to a server according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a big data component management apparatus applied to an electronic device in a device cluster according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a server according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 11 is a schematic structural diagram of a big data component management system according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order to improve the efficiency of big data component management, the embodiment of the invention provides a big data component management method, a device, a server, electronic equipment and a system.

First, the big data component management method provided by the embodiment of the invention is described. The big data component management method is applied to a big data component management system, the system comprises a server and a device cluster, the device cluster is composed of a plurality of electronic devices, the server refers to devices with data processing capacity, data storage capacity and control functions, and the electronic devices refer to devices for specifically executing tasks of the big data components. The big data component management method provided by the embodiment of the invention is respectively introduced from several angles of interaction among the server side, the equipment cluster side and the big data component management system.

As shown in fig. 1, the method for managing big data components provided by the embodiment of the present invention is applied to the server, and may include the following steps.

S101, acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component.

S102, according to task names, reading task execution specifications of tasks to be executed from task execution specifications of preset big data components, and reading task execution strategies of the tasks to be executed from task execution strategies of preset big data components.

S103, according to the read task execution strategy, issuing the read task execution specification to the equipment cluster, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

When the embodiment of the invention is applied, when the requirement of executing the task to be executed of the designated big data component exists, the server can acquire the corresponding operation instruction, and because the task execution specification and the task execution strategy of each big data component are preset on the server, the task execution specification and the task execution strategy of the task to be executed can be read according to the task name of the task to be executed of the designated big data component in the operation instruction, the task execution specification and the task execution strategy agree on the information related to task execution such as scripts, execution sequences and the like required by task execution, and the electronic equipment in the equipment cluster can automatically execute the task to be executed without manual operation, thereby improving the efficiency of managing the big data component.

In the embodiment of the invention, a system architecture of a C/S (Client/Server) mode is adopted, namely one Server and a plurality of clients are deployed in the system. The server refers to equipment with a control and management function at the back end, and a management module is generally deployed; a client refers to each electronic device of a large cluster of data devices, or a software program deployed on each electronic device. The server and the client may not be deployed together, so long as the network is reachable, as shown in fig. 2, where the server may manage the deployment situation of one device cluster, and of course, one server may also manage multiple device clusters, where communications may be performed between the server and the client through a specified communication protocol.

The main work of the server is as follows: managing task arrangement of different big data components, interacting with a database, providing an application program Interface for external systems and front end UI (User Interface) calls, issuing task execution specifications according to a plan or User trigger by a certain task execution strategy, collecting task execution results reported by a client side and the like. The main work of the client is as follows: receiving a task execution specification issued by the server, executing a task according to the task execution specification, reporting a task execution result to the server and the like.

When there is a task requirement for executing the specified big data component, the server acquires a corresponding operation instruction, wherein the operation instruction indicates which big data component needs to execute what task, namely, the operation instruction comprises a task name of a task to be executed, which is input for the specified big data component, for example, the electronic device in the device cluster is required to execute the task of installing the elastic search component, and the operation instruction acquired by the server comprises the task name of installing the elastic search component.

The operation instruction can be automatically generated when the task execution requirement exists, or can be input by a user when the task execution requirement exists. Optionally, S101 may specifically be: and receiving an operation instruction input by a user through an application program interface.

The server provides an application program interface for the external system and the front end UI, and when a user has a task requirement of executing a specified big data component, the user can input an operation instruction in the front end UI, and the operation instruction is transmitted to the server through the application program interface.

The server is responsible for the access of the big data components and the task execution arrangement, namely the task execution flow in the whole life cycle of the big data components is arranged in the server in advance, and the task execution specification and the task execution strategy of each big data component are preset in the server locally. The task execution specification includes address information required for task execution, information about large data component executable scripts, information about executable system commands, and the like, for example, a resource package address of a large data component, an operation and maintenance management script address, an operation and maintenance management script, and the like. The operation and maintenance management script is a script provided by the big data component in an open source or a custom script of an administrator. The task execution strategy defines an execution mode when the equipment cluster executes the big data component, and characterizes that when one big data component is executed, all the electronic equipment in the equipment cluster execute, randomly select one electronic equipment to execute, designate one electronic equipment to execute, execute a plurality of electronic equipment in parallel or execute a plurality of electronic equipment in series. Corresponding task execution specifications and task execution strategies can be preset for different tasks of each big data component, and specifically, the corresponding relation among task names, task execution specifications and task execution strategies can be recorded according to a mapping table, and after the operation instruction is acquired, the server can read the task execution specifications and task execution strategies of the tasks to be executed according to the task names of the tasks to be executed of the big data components specified in the operation instruction.

Alternatively, S102 may specifically be: and reading a task execution specification and a task execution strategy of a task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

The server may be provided with a database, where the database is configured to store task execution specifications and task execution policies preset for different tasks of each big data component, and after the task execution specifications and task execution policies preset for different tasks of each big data component are preset, the administrator stores the set task execution specifications and task execution policies in the database, and after obtaining the operation instruction, the server may find the task execution specifications and task execution policies of the task to be executed from the server according to the task name of the task to be executed. Of course, the database may be distributed with the server.

Optionally, the database stores task execution specifications of execution steps of different tasks of each big data component in a buffer queue mode. Correspondingly, the step of reading the task execution specification and the task execution policy of the task to be executed from the database according to the task name may specifically be: according to the task names, the task execution specifications of all execution steps of the task to be executed are sequentially read from the database according to the storage sequence of the cache queue, and according to the task names, the task execution strategy of the task to be executed is read from the database. S103 may specifically be: according to the read task execution strategy, issuing the task execution specification of each execution step of the read task to be executed to the equipment cluster in a buffer queue mode, so that the electronic equipment in the equipment cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

In general, task execution of one big data component is realized through sequential execution of one step, when task execution specifications of different tasks of each big data component are stored in a database, the task execution specifications of each execution step of different tasks of each big data component can be stored in a buffer queue mode, the task execution specifications of each execution step stored in the buffer queue mode meet a first-in first-out principle, namely, when the task execution specifications of each execution step are stored in sequence according to the execution sequence, when the task execution specifications are transmitted, the task execution specifications of each execution step are transmitted to an equipment cluster in sequence according to the execution sequence in the buffer queue mode, and the equipment cluster can sequentially execute each execution step according to the received sequence. If a certain execution step is not successfully executed, the server can be informed, the server can sequentially send the execution steps after the execution step, the server does not need to send the task execution specifications of all the execution steps from the beginning, the pressure of data transmission and task execution is reduced, and the running stability of the whole system is ensured.

Because the task execution policy defines an execution mode when the device cluster executes the big data component, for example, the task execution policy agrees that all electronic devices in the device cluster execute and install the elastic search component, the server can issue a task execution specification to the device cluster according to the task execution policy, and because the task execution specification includes address information required by the task execution, related information of an executable script of the big data component, related information of an executable system command and the like, the electronic devices in the device cluster can execute related operations (operations such as big data component configuration change, installation deployment, start stop and the like) of a task to be executed according to the task execution specification. That is, the big data component can be managed in the form of a plug-in by only realizing some task arrangement, and the big data component itself is not required to be modified. When the server issues the task execution specification, the server can monitor the health condition of the device cluster and send the task execution specification to the healthy electronic device.

Optionally, S103 may specifically be: and according to the read task execution strategy, issuing the read task execution specification to the equipment cluster through a remote procedure call protocol, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

The network between the server and the device cluster is reachable, so that the server can send the task execution specification to the device cluster through a specific network protocol, and the specific network protocol used can be an RPC (Remote Procedure Call ) protocol, which is a protocol that requests services from a remote computer program through the network without knowing the underlying network technology. The protocol allows a program running on one computer to call a subroutine of another computer without the programmer having to additionally program this interaction. The RPC protocol makes it easier to develop applications including network distributed multiprogramms. Of course, the server and the device cluster may also use protocols such as HTTP (HyperText Transfer Protocol ) and TCP (Transmission Control Protocol, transmission control protocol) for bidirectional communication.

Optionally, the operation instructions may further include the number of electronic devices. Correspondingly, S103 may specifically be: selecting a plurality of electronic devices from a device cluster according to the number of the electronic devices; and issuing the read task execution specification into a plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

The user can also specify the number of electronic devices for executing tasks of the big data component, the number of electronic devices for executing tasks is limited in the operation instruction, after the server acquires the operation instruction, firstly, the server selects the corresponding number of electronic devices from the device cluster according to the number of electronic devices, for example, 10 electronic devices are selected if the input number of electronic devices is 10, and then, the read task execution specification is issued to the electronic devices according to the task execution policy.

The electronic equipment can have no business logic, and after receiving the task execution specification issued by the server, a new process is created according to the task execution specification, and the task to be executed is executed by using the process.

Based on the embodiment shown in fig. 1, the embodiment of the invention also provides a big data component management method, which is applied to the server, and as shown in fig. 3, the method can comprise the following steps.

S301, acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component.

S302, according to task names, reading task execution specifications of tasks to be executed from task execution specifications of preset big data components, and reading task execution strategies of the tasks to be executed from task execution strategies of preset big data components.

S303, issuing the read task execution specification to the equipment cluster according to the read task execution strategy, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

S304, receiving a task execution result fed back by the device cluster.

S305, adjusting a task execution strategy according to the task execution result.

S306, issuing the task execution specification to the device cluster according to the adjusted task execution strategy.

After the electronic devices in the device cluster complete the task to be executed, the task execution results are fed back to the server, wherein the task execution results comprise successful task execution, task execution failure, task execution efficiency and the like, after the server receives the task execution results, the server knows which electronic devices successfully execute the task to be executed and which electronic devices fail to execute the task, and the task execution strategy can be adaptively adjusted according to the task execution results, so that the electronic devices executing the task can achieve the purposes of successfully executing the task, improving the task execution efficiency and the like after issuing the task execution specification to the device cluster according to the adjusted task execution strategy.

As shown in fig. 4, the method for managing big data components provided by the embodiment of the present invention is applied to the electronic device in the device cluster, and may include the following steps.

S401, receiving a task execution specification issued by a server, wherein the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from task execution specifications of preset big data components and according to task execution strategies of the task to be executed read from task execution strategies of the preset big data components.

S402, executing the task to be executed according to the task execution specification.

According to the embodiment shown in fig. 1, the server reads the task execution specification and the task execution policy of the task to be executed of the specified big data component, and issues the read task execution specification to the device cluster according to the read task execution policy, after the electronic device in the device cluster receives the task execution specification, because the task execution specification includes address information required by task execution, related information of an executable script of the big data component, related information of an executable system command and the like, the electronic device in the device cluster can execute related operations (operations such as big data construction configuration change, installation deployment, start stop and the like) of the task to be executed according to the task execution specification. The electronic devices in the device cluster can automatically execute tasks to be executed without manual operation, so that the management efficiency of the big data components is improved.

Alternatively, S401 may specifically be: and receiving a task execution specification of each execution step of the task to be executed, which is sent by the server in a buffer queue mode. S402 may specifically be: and sequentially executing the execution steps of the task to be executed according to the task execution specification of the execution steps of the task to be executed.

As described above, the server may send the task execution specification of each execution step of the task to be executed in a buffer queue, and then the device cluster may sequentially execute each execution step according to the received sequence. If a certain execution step is not successfully executed, the server can be informed, the server can sequentially send the execution steps after the execution step, the server does not need to send the task execution specifications of all the execution steps from the beginning, the pressure of data transmission and task execution is reduced, and the running stability of the whole system is ensured.

Based on the above embodiments, in the big data management system provided by the embodiments of the present invention, the server mainly includes a task management module, a policy module, and a task issuing module, and the electronic devices in the device cluster mainly include a task execution module, as shown in fig. 5, which is a data flow schematic diagram of each module. The main functions of the task management module are as follows: and setting an access specification for the big data component, namely registering task information of some big data components, including a resource packet address, an operation and maintenance management script and the like of the big data component. The main workflow of the task management module comprises: receiving task names set in a front end or a system; managing the scheduling task; the task execution specification of the big data component is provided or the task execution specification customized by an administrator is received. The main functions of the policy module are: and recording a task execution strategy of each big data component, wherein the task execution strategy comprises all electronic equipment execution of equipment clusters, random electronic equipment execution, appointed electronic equipment execution, parallel execution or serial execution of a plurality of electronic equipment and the like. The main workflow of the policy module includes: determining a task execution strategy according to the task content; adjusting a task execution strategy according to the task execution condition; and maintaining a task execution strategy. The main functions of the task issuing module are as follows: and maintaining the network and health conditions of all the electronic devices of the device cluster. The main workflow of the task issuing module comprises the following steps: task encapsulation; maintaining a task cache queue; tracking a health condition of the electronic device; and the task execution specification issues and receives feedback of the task execution result. The main functions of the task execution module are: and obtaining and executing the subprocesses according to the tasks issued by the server, and returning task execution results to the server after the execution is completed. The main workflow of the task issuing module comprises the following steps: executing the task to be executed according to the task execution specification; and reporting the task execution result to the server. After the policy module obtains the task execution policy, the task execution policy and the task execution specification can be stored in the task cache queue, and then the task issuing module reads the task execution specification from the task cache queue for issuing. Similarly, when the task is issued, the task issuing module can store the task execution specification into the task cache queue, the task execution module reads the task execution specification from the task cache queue for execution, and when the task execution module returns the task execution result, the task issuing module can store the task execution result into the task cache queue, and the task issuing module reads the task execution result from the task cache queue. The data transmission protocol between the task issuing module and the task executing module can adopt RPC protocol.

In order to facilitate understanding, the following describes an example of the present invention, and a big data component management method according to an embodiment of the present invention is shown in fig. 6.

An administrator can preset task execution specifications of some big data components in a server, including task execution authority, paths, task execution script acquisition addresses, task execution script execution commands and the like, and can set task execution strategies, such as execution of all electronic devices in a device cluster, execution of one electronic device at random, parallel execution or serial execution of a plurality of electronic devices and the like, and finally record the task execution specifications and the task execution strategies in a database of the server.

Specific task execution is triggered by a user, for example, the user needs to install an elastic search component in a device cluster of 10 pieces of electronic equipment, the user selects the number of the electronic equipment, and inputs a task name of installing the elastic search component through an API (Application Programming Interface ), and for the requirement, namely, the task execution specification of the task is read from a database and issued to clients of the 10 pieces of electronic equipment through an RPC protocol, wherein the task name is the task of all issuing and installing the selected 10 pieces of electronic equipment.

After the client side of the 10 pieces of electronic equipment obtains the task execution specification, queuing is firstly carried out, if no task is currently being executed, the task is started to be executed according to the task execution specification, namely, a process is established to execute the task, the successful execution can report the task execution result of successful execution to the server through the RPC protocol, and the task is not successfully retried.

After issuing the task execution specification, the server can circularly wait for the task execution results of all 10 pieces of electronic equipment, judge whether the execution is successful, if the task execution results reported by the 10 pieces of electronic equipment are all successful, the server can save the results and return the saved results to the user, and if the electronic equipment is in task execution, the server can always wait for the successful execution results.

Corresponding to the above method embodiment, the embodiment of the present invention provides a big data component management device, which is applied to a server, as shown in fig. 7, and the device may include:

an obtaining module 710, configured to obtain an operation instruction, where the operation instruction includes a task name of a task to be executed, which is input for a specified big data component;

the reading module 720 is configured to read a task execution specification of a task to be executed from task execution specifications of preset big data components according to task names, and read a task execution policy of the task to be executed from task execution policies of preset big data components;

And the sending module 730 is configured to issue the read task execution specification to the device cluster according to the read task execution policy, so that the electronic device in the device cluster executes the task to be executed according to the task execution specification.

Optionally, the acquiring module 710 may specifically be configured to: and receiving an operation instruction input by a user through an application program interface.

Optionally, the reading module 720 may specifically be configured to: and reading a task execution specification and a task execution strategy of a task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

the reading module 720 may specifically be configured to: according to task names, sequentially reading task execution specifications of execution steps of the tasks to be executed from a database according to the storage sequence of the cache queue, and reading task execution strategies of the tasks to be executed from the database according to the task names;

the sending module 730 may specifically be configured to: according to the read task execution strategy, issuing the task execution specification of each execution step of the read task to be executed to the equipment cluster in a buffer queue mode, so that the electronic equipment in the equipment cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

Optionally, the sending module 730 may specifically be configured to: and according to the read task execution strategy, issuing the read task execution specification to the equipment cluster through a remote procedure call protocol, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

Optionally, the operation instruction may further include the number of electronic devices;

the sending module 730 may specifically be configured to: selecting a plurality of electronic devices from a device cluster according to the number of the electronic devices; and issuing the read task execution specification into a plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

Optionally, the apparatus may further include:

the sending module 730 may be further configured to issue a task execution specification to the device cluster according to the adjusted task execution policy.

The embodiment of the invention provides a big data component management device, which is applied to electronic equipment in an equipment cluster, as shown in fig. 8, and the device can comprise:

the receiving module 810 is configured to receive a task execution specification issued by a server, where the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from task execution specifications of preset big data components, and according to a task execution policy of the task to be executed read from task execution policies of preset big data components;

the execution module 820 is configured to execute a task to be executed according to a task execution specification.

Optionally, the receiving module 810 may specifically be configured to: receiving a task execution specification of each execution step of a task to be executed, which is sent by a server in a cache queue mode;

the execution module 820 may specifically be configured to: and sequentially executing the execution steps of the task to be executed according to the task execution specification of the execution steps of the task to be executed.

By applying the embodiment of the invention, the server reads the task execution specification and the task execution strategy of the task to be executed of the specified big data component, and issues the read task execution specification to the equipment cluster according to the read task execution strategy, and after the electronic equipment in the equipment cluster receives the task execution specification, the electronic equipment in the equipment cluster can execute related operations (operations such as big data component configuration change, installation deployment, start and stop, and the like) of the task to be executed according to the task execution specification because the task execution specification comprises address information required by the task execution, related information of the executable script of the big data component, related information of the executable system command, and the like. The electronic devices in the device cluster can automatically execute tasks to be executed without manual operation, so that the management efficiency of the big data components is improved.

The embodiment of the invention also provides a server, as shown in fig. 9, which comprises a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902 and the memory 903 complete communication with each other through the communication bus 904;

a memory 903 for storing a computer program;

the processor 901 is configured to implement the big data component management method applied to the server provided in the embodiment of the present invention when executing the computer program stored in the memory 903.

The embodiment of the invention also provides an electronic device, as shown in fig. 10, which comprises a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 complete communication with each other through the communication bus 1004;

a memory 1003 for storing a computer program;

the processor 1001 is configured to implement the big data component management method applied to the electronic devices in the device cluster according to the embodiment of the present invention when executing the computer program stored in the memory 1003.

The communication bus mentioned by the above server may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the server and other devices.

The Memory may include RAM (Random Access Memory ) or NVM (Non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a CPU (Central Processing Unit ), NP (Network Processor, network processor), etc.; but also DSP (Digital Signal Processor ), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the big data component management method applied to the server provided by the embodiment of the invention is realized.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the big data component management method applied to the electronic equipment in the equipment cluster provided by the embodiment of the invention is realized.

The embodiment of the invention also provides a computer program product containing instructions, which when run on a computer, cause the computer to execute the big data component management method applied to the server.

The embodiment of the invention also provides a computer program product containing instructions, which when run on a computer, cause the computer to execute the big data component management method applied to the electronic equipment in the equipment cluster.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, DSL (Digital Subscriber Line, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD (Digital Versatile Disc, digital versatile Disk)), or a semiconductor medium (e.g., an SSD (Solid State Disk)), or the like.

The embodiment of the invention also provides a big data component management system, as shown in fig. 11, which comprises a server 1110 and a device cluster 1120, wherein the device cluster is composed of a plurality of electronic devices 1121;

a server 1110, configured to obtain an operation instruction, where the operation instruction includes a task name of a task to be executed, which is input for a specified big data component; according to the task name, reading a task execution specification of a task to be executed from task execution specifications of all preset big data components, and reading a task execution policy of the task to be executed from task execution policies of all preset big data components; issuing the read task execution specification to the device cluster 1120 according to the read task execution policy;

the electronic devices 1121 in the device cluster 1120 are configured to execute tasks to be executed according to the task execution specification.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus, server, electronic device, storage medium, machine-readable storage medium, computer program product containing instructions, big data component management system embodiments, the description is relatively simple as it is substantially similar to the method embodiments, where relevant see the section description of the method embodiments.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A big data component management method, applied to a server, the method comprising:

according to the task names, reading task execution specifications of the tasks to be executed from preset task execution specifications of all big data components, and reading task execution strategies of the tasks to be executed from preset task execution strategies of all big data components; wherein the task execution specification includes: the method comprises the steps that address information, relevant information of a big data component executable script and relevant information of an executable system command are required when a task is executed, and a task execution strategy is used for limiting an execution mode when a device cluster executes the big data component; the execution mode comprises the following steps: all electronic devices in the device cluster execute, or randomly select one electronic device to execute, or designate one electronic device to execute, or execute a plurality of electronic devices in parallel, or execute a plurality of electronic devices in series;

And according to the read task execution strategy, issuing the read task execution specification to a device cluster, so that the electronic device in the device cluster executes the task to be executed according to the task execution specification.

2. The method of claim 1, wherein the acquiring the operation instruction comprises:

3. The method according to claim 1, wherein the reading the task execution specification of the task to be executed from the task execution specification of each big data component set in advance according to the task name, and the reading the task execution policy of the task to be executed from the task execution policy of each big data component set in advance includes:

and reading the task execution specification and the task execution strategy of the task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

4. A method according to claim 3, wherein the database stores task execution specifications of execution steps of different tasks of each big data component in a buffer queue;

The reading the task execution specification and the task execution strategy of the task to be executed from the database according to the task name comprises the following steps:

according to the task names, sequentially reading task execution specifications of execution steps of the tasks to be executed from the database according to the storage sequence of the cache queue, and reading task execution strategies of the tasks to be executed from the database according to the task names;

the issuing the read task execution specification to the device cluster according to the read task execution policy includes:

and according to the read task execution strategy, issuing the read task execution specification of each execution step of the task to be executed to a device cluster in the form of the cache queue, so that the electronic device in the device cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

5. The method of claim 1, wherein issuing the read task execution specification to a device cluster according to the read task execution policy comprises:

and according to the read task execution strategy, issuing the read task execution specification to a device cluster through a remote procedure call protocol, so that the electronic device in the device cluster executes the task to be executed according to the task execution specification.

6. The method of claim 1, wherein the operational instructions further comprise a number of electronic devices;

and issuing the read task execution specification to the plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

7. The method of claim 1, wherein after said issuing the read task execution specification to a device cluster in accordance with the read task execution policy, the method further comprises:

receiving a task execution result fed back by the equipment cluster;

according to the task execution result, adjusting the task execution strategy;

8. A big data component management method, applied to an electronic device in a device cluster, the method comprising:

Receiving a task execution specification issued by a server, wherein the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from preset task execution specifications of all big data components and according to a task execution strategy of the task to be executed read from preset task execution strategies of all big data components; the task execution specification includes: the method comprises the steps that address information, relevant information of a big data component executable script and relevant information of an executable system command are required when a task is executed, and a task execution strategy is used for limiting an execution mode when a device cluster executes the big data component; the execution mode comprises the following steps: all electronic devices in the device cluster execute, or randomly select one electronic device to execute, or designate one electronic device to execute, or execute a plurality of electronic devices in parallel, or execute a plurality of electronic devices in series;

9. The method of claim 8, wherein receiving the task execution specification issued by the server comprises:

the executing the task to be executed according to the task execution specification includes:

10. A big data component management apparatus, characterized by being applied to a server, the apparatus comprising:

the reading module is used for reading the task execution specification of the task to be executed from the task execution specification of each preset big data component according to the task name, and reading the task execution policy of the task to be executed from the task execution policy of each preset big data component; wherein the task execution specification includes: the method comprises the steps that address information, relevant information of a big data component executable script and relevant information of an executable system command are required when a task is executed, and a task execution strategy is used for limiting an execution mode when a device cluster executes the big data component; the execution mode comprises the following steps: all electronic devices in the device cluster execute, or randomly select one electronic device to execute, or designate one electronic device to execute, or execute a plurality of electronic devices in parallel, or execute a plurality of electronic devices in series;

And the sending module is used for sending the read task execution specification to the equipment cluster according to the read task execution policy, so that the electronic equipment in the equipment cluster executes the task to be executed according to the task execution specification.

11. The apparatus of claim 10, wherein the obtaining module is specifically configured to: and receiving an operation instruction input by a user through an application program interface.

12. The apparatus according to claim 10, wherein the reading module is specifically configured to: and reading the task execution specification and the task execution strategy of the task to be executed from a database according to the task name, wherein the task execution specification and the task execution strategy preset for different tasks of each big data component are stored in the database.

13. The apparatus of claim 12, wherein the database stores task execution specifications for each execution step of different tasks for each big data component in a cache queue;

the reading module is specifically configured to: according to the task names, sequentially reading task execution specifications of execution steps of the tasks to be executed from the database according to the storage sequence of the cache queue, and reading task execution strategies of the tasks to be executed from the database according to the task names;

The sending module is specifically configured to: and according to the read task execution strategy, issuing the read task execution specification of each execution step of the task to be executed to a device cluster in the form of the cache queue, so that the electronic device in the device cluster sequentially executes each execution step of the task to be executed according to the task execution specification of each execution step of the task to be executed.

14. The apparatus according to claim 10, wherein the sending module is specifically configured to: and according to the read task execution strategy, issuing the read task execution specification to a device cluster through a remote procedure call protocol, so that the electronic device in the device cluster executes the task to be executed according to the task execution specification.

15. The apparatus of claim 10, wherein the operational instructions further comprise an electronic device number;

the sending module is specifically configured to: selecting a plurality of electronic devices from a device cluster according to the number of the electronic devices; and issuing the read task execution specification to the plurality of electronic devices according to the read task execution policy, so that each electronic device executes the task to be executed according to the task execution specification.

16. The apparatus of claim 10, wherein the apparatus further comprises:

the adjustment module is used for adjusting the task execution strategy according to the task execution result;

the sending module is further configured to issue the task execution specification to the device cluster according to the adjusted task execution policy.

17. A big data component management apparatus for application to an electronic device in a cluster of devices, the apparatus comprising:

the receiving module is used for receiving a task execution specification issued by a server, wherein the task execution specification is issued by the server according to a task name of a task to be executed of a specified big data component, read from the task execution specification of each preset big data component and according to a task execution strategy of the task to be executed read from the task execution strategy of each preset big data component; the task execution specification includes: the method comprises the steps that address information, relevant information of a big data component executable script and relevant information of an executable system command are required when a task is executed, and a task execution strategy is used for limiting an execution mode when a device cluster executes the big data component; the execution mode comprises the following steps: all electronic devices in the device cluster execute, or randomly select one electronic device to execute, or designate one electronic device to execute, or execute a plurality of electronic devices in parallel, or execute a plurality of electronic devices in series;

18. The apparatus according to claim 17, wherein the receiving module is specifically configured to: receiving a task execution specification of each execution step of a task to be executed, which is sent by a server in a cache queue mode;

the execution module is specifically configured to: and sequentially executing the execution steps of the task to be executed according to the task execution specification of the execution steps of the task to be executed.

19. The server is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor being adapted to implement the method of any of claims 1-7 when executing a computer program stored on a memory.

20. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus;

The memory is used for storing a computer program;

the processor being adapted to implement the method of claim 8 or 9 when executing a computer program stored on a memory.

21. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1-7 or the method of claim 8 or 9.

22. The big data component management system is characterized by comprising a server and a device cluster, wherein the device cluster consists of a plurality of electronic devices;

the server is used for acquiring an operation instruction, wherein the operation instruction comprises a task name of a task to be executed, which is input for a specified big data component; according to the task names, reading task execution specifications of the tasks to be executed from preset task execution specifications of all big data components, and reading task execution strategies of the tasks to be executed from preset task execution strategies of all big data components; issuing the read task execution specification to the equipment cluster according to the read task execution strategy; wherein the task execution specification includes: the method comprises the steps that address information, relevant information of a big data component executable script and relevant information of an executable system command are required when a task is executed, and a task execution strategy is used for limiting an execution mode when a device cluster executes the big data component; the execution mode comprises the following steps: all electronic devices in the device cluster execute, or randomly select one electronic device to execute, or designate one electronic device to execute, or execute a plurality of electronic devices in parallel, or execute a plurality of electronic devices in series;