CN112463144B - Distributed storage command line service method, system, terminal and storage medium - Google Patents
Distributed storage command line service method, system, terminal and storage medium Download PDFInfo
- Publication number
- CN112463144B CN112463144B CN202011392165.XA CN202011392165A CN112463144B CN 112463144 B CN112463144 B CN 112463144B CN 202011392165 A CN202011392165 A CN 202011392165A CN 112463144 B CN112463144 B CN 112463144B
- Authority
- CN
- China
- Prior art keywords
- command line
- node
- command
- monitoring
- task queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012544 monitoring process Methods 0.000 claims abstract description 87
- 238000012545 processing Methods 0.000 claims abstract description 37
- 238000012795 verification Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 9
- 230000002085 persistent effect Effects 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 5
- 230000007246 mechanism Effects 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 12
- 230000008878 coupling Effects 0.000 description 7
- 238000010168 coupling process Methods 0.000 description 7
- 238000005859 coupling reaction Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000004044 response Effects 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Hardware Redundancy (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a command line service method, a system, a terminal and a storage medium for distributed storage, comprising the following steps: converting the received user command line request into a command code according to a preset format, verifying the command code, and taking the verified command code as a message event to occur to a task queue; identifying monitoring nodes from a storage system cluster, establishing a subscription relationship between a main monitoring node and other nodes, and starting timing synchronization threads on all the monitoring nodes to monitor and backup the task queue; and sequentially extracting the message events from the task queue, analyzing the message events and sending the analyzed message events to a target node for processing. The invention not only can make the command line function self-adaptively coincide with the distributed storage mechanism, but also is beneficial to the load balance among the storage cluster nodes, reduces the load pressure of the cluster main node to a certain extent, and improves the concurrent processing capability of the system.
Description
Technical Field
The invention relates to the technical field of distributed storage systems, in particular to a command line service method, a command line service system, a command line service terminal and a command line service storage medium for distributed storage.
Background
GUI (graphical user interface), CLI (command line interface) have been the standard for various storage systems or clusters as a common interface format for large cluster systems. On a large-scale storage system, a product configuration item which is tightly attached to a LINUX server operating system and is matched for a CLI (common line interface) of storage software is already a product configuration item which is marked by each manufacturer and is also an indispensable part for storage.
The current CLI interface is based on an operating system and is presented in a script form, the corresponding CLI name and command are transmitted to the inside of the system, and then the system execution result is transmitted and presented to a user. The method can be widely applied to a single-machine system, but is poor in performance on a large-scale storage cluster, and has the following main problems:
RPC remote command call is complex to realize, always acts on a cluster main node, and when a large number of CLI requests are issued simultaneously, the pressure of the main node is too high, the system load is too high, and stable operation of the cluster is not facilitated.
CLI command realizes lack of management and service functions, and realizes specific logic by stacking command definition codes when adding new commands, so that the commands are redundant and tedious; and the command form is difficult to remember, a help manual is often required to be maintained, and a detailed real operation guide is provided to guide the user to use, so that the cost of hands is high.
From the programming realization perspective, the coupling degree between modules such as parameter definition, help document realization, parameter dependency relationship constraint, authority management, exception handling mechanism and the like in CLI realization is higher, which is not beneficial to the software capability and quality construction of high cohesion and low coupling.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides a method, a system, a terminal and a storage medium for servicing a command line of distributed storage, so as to solve the above-mentioned technical problems.
In a first aspect, the present invention provides a command line service method for distributed storage, including:
converting the received user command line request into a command code according to a preset format, verifying the command code, and taking the verified command code as a message event to occur to a task queue;
identifying monitoring nodes from a storage system cluster, establishing a subscription relationship between a main monitoring node and other nodes, and starting timing synchronization threads on all the monitoring nodes to monitor and backup the task queue;
and sequentially extracting the message events from the task queue, analyzing the message events and sending the analyzed message events to a target node for processing.
Further, the converting the received user command line request into a command code according to a preset format, and verifying the command code includes:
loading a command line template library file which has finished the definition configuration and corresponding command codes of various preset command line commands;
converting the user command line request into a command code by a matching search method;
checking whether the user name authority requested by the user command line is matched with the command code authority, and prompting an error code without authority if the user name authority is not matched with the command code authority;
and verifying the validity of the issued command line parameters according to the command codes and the parameter dependence configuration in the command line template library file.
Further, the identifying a monitoring node in the slave storage system cluster, establishing a subscription relationship between the master monitoring node and other nodes, and monitoring and backing up the task queue by starting a timing synchronization thread on all monitoring nodes includes:
initializing the task queue at each node of the cluster;
acquiring all node information of a cluster, and marking monitoring nodes, wherein the monitoring nodes comprise a main monitoring node and a sub monitoring node;
starting timing synchronization threads on all monitoring nodes, monitoring data changes of a task queue by the timing synchronization threads, and backing up updated data to the local monitoring nodes if data updating is monitored;
and if the fault monitoring node exists, loading the queue data in the local backup configuration to the switched target monitoring node memory for processing.
Further, the sequentially extracting the message events from the task queue, analyzing the message events, and sending the analyzed message events to the target node for processing includes:
establishing a plurality of working threads to lock the task queue;
the multiple working threads sequentially extract the message events from the task queue and analyze the request content of the command line from the extracted message events;
calculating the waiting time of the command line request according to the command line request content, judging whether the command line request is overdue according to the waiting time, and outputting a request overtime error code if the command line request is overdue;
acquiring a target node of the command line request according to the content of the command line request, and sending the command line request to a service queue of the target node for waiting the target node to process;
the target node analyzes the command line request and executes corresponding service logic at the current node according to the command of the command line request, backfills an execution result to a result record field, marks an information ID of the record field and then sends the record field to the main service node;
and the main service node sorts the record fields according to the marks of the record fields and sends the sorting result to the user.
In a second aspect, the present invention provides a command line service system for distributed storage, including:
the definition analysis unit is configured to convert the received user command line request into a command code according to a preset format, verify the command code and take the verified command code as a message event to a task queue;
the service subscription unit is configured and used for identifying the monitoring nodes from the storage system cluster, establishing a subscription relation between the main monitoring node and other nodes, and starting timing synchronous threads on all the monitoring nodes to monitor and backup the task queue;
and the service processing unit is configured to sequentially extract the message events from the task queue, analyze the message events and send the analyzed message events to a target node for processing.
Further, the definition parsing unit includes:
the file loading module is used for loading the command line template library file which is defined and configured and the preset corresponding command codes of various command line commands;
the command conversion module is configured to convert the user command line request into a command code by a matching search method;
the authority verification module is configured to verify whether the user name authority requested by the user command line is matched with the command code authority, and if not, a no-authority error code is prompted;
and the legal verification module is configured and used for verifying the validity of the issued command line parameters according to the command codes and the parameter dependence configuration in the command line template library file.
Further, the service subscription unit includes:
the queue processing module is configured to initialize the task queue at each node of the cluster;
the node marking module is configured to acquire all node information of the cluster and mark monitoring nodes, wherein the monitoring nodes comprise a main monitoring node and a sub monitoring node;
the queue monitoring module is configured to start timing synchronization threads on all monitoring nodes, wherein the timing synchronization threads monitor data changes of the task queue, and if data updating is monitored, the updated data is backed up to the local monitoring nodes;
and the fault switching module is configured to load the queue data in the local backup configuration to the switched target monitoring node memory for processing if the fault monitoring node exists.
Further, the service processing unit includes:
the thread establishing module is configured to establish a plurality of working threads to lock the task queue;
the event extraction module is configured for sequentially extracting the message events from the task queue by the multiple working threads and analyzing the command line request content from the extracted message events;
the timeout verification module is configured to calculate the waiting time of the command line request according to the command line request content, judge whether the command line request is timeout according to the waiting time, and output a request timeout error code if the command line request is timeout;
the request sending module is configured to acquire a target node of the command line request according to the content of the command line request, and send the command line request to a service queue of the target node to wait for the target node to process;
the result recording module is configured for a target node to analyze the command line request and execute corresponding service logic at the current node according to the command of the command line request, the target node backfills an execution result to a result recording field, and the recording field marks an information ID and then sends the information ID to the main service node;
and the result sorting module is configured for sorting the record fields by the main service node according to the marks of the record fields and sending the sorting result to the user.
In a third aspect, a terminal is provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program which,
the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.
In a fourth aspect, a computer storage medium is provided having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the above aspects.
The beneficial effect of the invention is that,
the command line service method, the system, the terminal and the storage medium for distributed storage, provided by the invention, not only can make the function of the command line adaptively matched with a distributed storage mechanism, but also are beneficial to storing load balance among cluster nodes, thereby reducing the load pressure of the cluster main nodes to a certain extent and improving the concurrent processing capacity of the system; meanwhile, for the development framework, the command line interface and the realization of the service are realized, the coupling degree among CLI definition, realization and data transmission in the development process is reduced, and the integrity and the usability of the whole development framework are facilitated.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
FIG. 2 is a schematic flow diagram of a definition resolution process of a method of one embodiment of the invention.
FIG. 3 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
GUI (Graphical User Interface): a graphical user interface, a computer operating user interface displayed graphically, is an interface display format for human-to-computer communications that allows a user to manipulate on-screen icons or menu options using an input device such as a mouse to select commands, invoke files, launch programs, or perform other daily tasks.
CLI (Command-Line Interface): the command line interface is the most widely used user interface before the popularization of the graphic user interface, generally does not support a mouse, and a user inputs an instruction through a keyboard and a computer executes the instruction after receiving the instruction.
RPC (Remote Procedure Call): remote procedure calls, as do local functions, to invoke remote functions. Generally understood to be a function or method (which may be collectively referred to as a service) that is invoked from one machine (client) by means of parameter passing on to another machine (server) and gets the result back.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention. The execution subject in fig. 1 may be a command line service system for distributed storage.
As shown in fig. 1, the method includes:
and step 130, sequentially extracting the message events from the task queue, analyzing the message events and sending the analyzed message events to a target node for processing.
Specifically, the distributed storage command line service method includes:
s1, as shown in figure 2, converting a received user command line request into a command code according to a preset format, verifying the command code, and taking the verified command code as a message event to occur to a task queue.
And Step1, loading all CLI template library files of which the definition configuration is completed, completing the initialization operation of all CLI command codes, and waiting for receiving a CLI command request from a user.
Step2: after receiving the CLI command of the user, obtaining the command code of the command through pattern matching, and then turning to the next step.
Step3: and checking whether the user has the request authority of the command according to the definition of the user name authority and the command code authority, if the user passes the check, turning to the next step, and otherwise, throwing a no-authority error code.
Step4: and issuing the validity of the CLI parameter according to the command code and the parameter dependent configuration verification, if the command code and the parameter dependent configuration verification pass, turning to the next step, and if the command code and the parameter dependent configuration verification do not pass, throwing no command error code.
Step5: and disassembling the CLI character string, and defining and packaging the CLI character string into a CLI service message event ismCliiItem in a JSON format according to an agreed format.
Step6: and sending the generated isomCliiltem message event to an isomCliiQueue queue of a CLI service subscription module for further processing by the next module.
Generally, the ismClitem definition data structure is shown in Table 1:
table 1 ismCliItem definition data structure table
The CliCode three-segment eight-bit digital code is specifically defined as follows:
a first stage: one bit, identifying the general class of business operations, from 0-9, 10 classes,
presetting 0-hardware; 1- -the system; 2- -storing; the rest reserves the extension.
And a second stage: three bits, identification subsystem service module codes, from 001 to 999, 1000, such as an alarm subsystem, a distributed file storage subsystem, a distributed block storage subsystem, a distributed object storage subsystem, a cache subsystem, a monitoring subsystem, a cluster service subsystem and the like.
A third stage: and four bits are used for identifying specific service function codes of each subsystem, wherein the specific service functions are selected from 0001-9999 and 10000, such as creation, deletion, query and the like of the logical volume of the distributed block storage subsystem.
For example, in a complete example, the command codes of the logical volume creation function of the distributed block storage are: 2 0031001
S2, identifying the monitoring nodes from the storage system cluster, establishing a subscription relationship between the main monitoring node and other nodes, and starting timing synchronization threads on all the monitoring nodes to monitor and backup the task queue.
Step1: each node of the cluster initializes a command line service message queue ismCliQueue.
Step2: all node information of the cluster is obtained, and MON nodes (monitoring nodes) including a main MON node and ordinary nodes are marked. (deep binding with distributed storage MON node)
Step3: and establishing a subscription relation between the main MON node and all other nodes, and turning to the next step.
Step4: starting timing synchronization threads on all MON nodes of the cluster, monitoring the data change of the ismCliquue, and turning to the next step if the data change of the ismCliquue is found; otherwise, the monitoring is continuously kept.
Step5: persisting (keeping copies locally, failover use) the ismCliQueue data queue across all MON nodes; and when node fault switching occurs, automatically loading queue data from the persistent configuration to a memory for processing.
Step6: judging len (ismCliquue) + len (newData) > =1024, and if yes, expanding the capacity of the ismCliquue queue by one time.
And S3, sequentially extracting the message events from the task queue, analyzing the message events and sending the analyzed message events to a target node for processing.
Step1: after startup, several worker threads (default 3, not more than 64) are established according to the configuration.
Step2: and after the current working thread is started according to the scheduling test, locking the command line service queue ismCliQueue.
Step3: and taking out a command operation message event ismCliiItem from the ismCliQueue, analyzing the corresponding CLI request content, and then turning to the next step.
Step4: acquiring message data according to the analysis result, judging whether the request fails according to expiration time (the problem that the network is abnormal or the node fails easily to fail), if the expiration time is exceeded, throwing out a system request timeout error code and requesting to retry a prompt slightly, and removing the ismCliItem from the ismCliQueue; otherwise, the next step is carried out.
Step5: obtaining the dstNode information of the CLI request, judging whether the current node is the same as the dstNode, if not, sending the ismClitem to a command line service queue of the dstNode, and processing by the dstNode; otherwise, if the nodes are the same, the next step is carried out.
Step6: analyzing the ismCliiItem, executing corresponding business logic at the current node according to the CliCode, backfilling an execution Result to an ismCliiItem Result field, marking fields such as State/End _ time and the like, returning the ismCliiItem Result according to the MsgId and sending the ismCliItem Result to the ismCliQueue of the main sysgt-cli service node, then deleting the completed ismCliCli item from the queue by the dstNode, and then turning to the next step.
Step7: and the master node ismCliQueue sorts the result data sent back by the dstNode nodes (corresponding to ALL nodes if the dstNode nodes are ALL) according to msgId, and displays the result back to the user.
Step8: the command line service queue ismCliQueue is released.
In summary, the CLI definition parsing unit is responsible for data processing and logic verification of functions such as basic concept definition and constraint of the command line; the CLI service subscription unit is responsible for establishing a CLI service subscription relation between a master node and a slave node, packaging CLI request data or finished result response data and message data, maintaining the CLI request data or finished result response data in an ismClientQueue queue, performing dequeue and enqueue operation to realize data request and response, providing timing synchronization thread active backup queue data and providing abnormal fault-tolerant capability; the CLI service processing unit is responsible for decoding, updating and packaging message events in a command line service queue ismCliQueue, and actively pushing response data to a main node service queue; the three units are organically combined and interacted to form a command line servitization method in the distributed storage cluster.
As shown in fig. 2, the system provided in this embodiment is composed of a definition parsing unit, a service subscription unit, and a service processing unit, and is deployed in a distributed storage cluster. The user communicates and interacts with the invention through the shell and the cluster network. The three units in the system of the present invention communicate and interact data through asynchronous message channels. Specifically, the system comprises:
the system comprises a definition analysis unit, a task queue and a data processing unit, wherein the definition analysis unit is configured to convert a received user command line request into a command code according to a preset format, verify the command code and take the verified command code as a message event to be generated in the task queue;
the service subscription unit is configured and used for identifying the monitoring nodes from the storage system cluster, establishing a subscription relation between the main monitoring node and other nodes, and starting timing synchronous threads on all the monitoring nodes to monitor and backup the task queue;
and the service processing unit is configured to sequentially extract the message events from the task queue, analyze the message events and send the analyzed message events to a target node for processing.
Optionally, as an embodiment of the present invention, the definition parsing unit includes:
the file loading module is used for loading the command line template library file which is defined and configured and the preset corresponding command codes of various command line commands;
the command conversion module is configured to convert the user command line request into a command code by a matching search method;
the authority verification module is configured to verify whether the user name authority requested by the user command line is matched with the command code authority, and if not, a no-authority error code is prompted;
and the legal verification module is configured and used for verifying the validity of the issued command line parameters according to the command codes and the parameter dependence configuration in the command line template library file.
Optionally, as an embodiment of the present invention, the service subscription unit includes:
the queue processing module is configured to initialize the task queue at each node of the cluster;
the node marking module is configured for acquiring all node information of the cluster and marking the monitoring nodes, wherein the monitoring nodes comprise a main monitoring node and a sub monitoring node;
the queue monitoring module is configured to start timing synchronization threads on all monitoring nodes, wherein the timing synchronization threads monitor data changes of the task queue, and if data updating is monitored, the updated data is backed up to the local monitoring nodes;
and the fault switching module is configured to load the queue data in the local backup configuration to the switched target monitoring node memory for processing if the fault monitoring node exists.
Optionally, as an embodiment of the present invention, the service processing unit includes:
the thread establishing module is configured to establish a plurality of working threads to lock the task queue;
the event extraction module is configured for sequentially extracting the message events from the task queue by the multiple working threads and analyzing the command line request content from the extracted message events;
the timeout verification module is configured to calculate the waiting time of the command line request according to the command line request content, judge whether the command line request is out of date according to the waiting time, and output a request timeout error code if the command line request is out of date;
the request sending module is configured to acquire a target node of the command line request according to the content of the command line request, and send the command line request to a service queue of the target node to wait for the target node to process;
the result recording module is configured for a target node to analyze the command line request and execute corresponding service logic at the current node according to the command of the command line request, the target node backfills an execution result to a result recording field, and the recording field marks an information ID and then sends the information ID to the main service node;
and the result sorting module is configured for sorting the record fields by the main service node according to the marks of the record fields and sending the sorting result to the user.
Fig. 4 is a schematic structural diagram of a terminal 400 according to an embodiment of the present invention, where the terminal 400 may be used to execute the command line service method of distributed storage according to the embodiment of the present invention.
Among them, the terminal 400 may include: a processor 410, a memory 420, and a communication unit 430. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 420 may be used for storing instructions executed by the processor 410, and the memory 420 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 420, when executed by processor 410, enable terminal 400 to perform some or all of the steps in the method embodiments described below.
The processor 410 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 420 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 410 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 430, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Therefore, the invention not only can make the command line function self-adaptively coincide with the distributed storage mechanism, but also is beneficial to the load balance among the storage cluster nodes, reduces the load pressure of the cluster main node to a certain extent, and improves the concurrent processing capacity of the system; meanwhile, for the development framework, the command line interface and the implementation of the service are realized, the coupling degree among definition, implementation and data transmission of the CLI in the development process is reduced, and the integrity and the usability of the whole development framework are facilitated.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some interfaces, indirect coupling or communication connection of systems or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A command line servicing method for distributed storage, comprising:
converting a received user command line request into a command code according to a preset format, verifying the command code, and taking the verified command code as a message event to occur to a task queue;
identifying monitoring nodes from a storage system cluster, establishing a subscription relationship between a main monitoring node and other nodes, and starting timing synchronization threads on all the monitoring nodes to monitor and backup the task queue;
sequentially extracting message events from the task queue, analyzing the message events and sending the analyzed message events to a target node for processing;
the method for identifying the monitoring nodes in the slave storage system cluster, establishing a subscription relationship between the master monitoring node and other nodes, and monitoring and backing up the task queue by starting timing synchronization threads on all the monitoring nodes comprises the following steps:
initializing the task queue at each node of the cluster;
acquiring all node information of a cluster, and marking monitoring nodes, wherein the monitoring nodes comprise a main monitoring node and a sub monitoring node;
starting timing synchronous threads on all the monitoring nodes, monitoring data change of a task queue by the timing synchronous threads, if data updating is monitored, persisting the updated task queue data among all the monitoring nodes, and locally keeping task queue data copies at the monitoring nodes monitoring the task queue;
and if the fault monitoring node exists, loading the persistent queue data corresponding to the fault monitoring node to the switched target monitoring node memory for processing.
2. The method of claim 1, wherein converting the received user command line request into a command code according to a preset format and verifying the command code comprises:
loading a command line template library file which has finished the definition configuration and corresponding command codes of various preset command line commands;
converting the user command line request into a command code by a matching search method;
checking whether the user name authority requested by the user command line is matched with the command code authority, and prompting an error code without authority if the user name authority is not matched with the command code authority;
and verifying the validity of the issued command line parameters according to the command codes and the parameter dependence configuration in the command line template library file.
3. The method of claim 1, wherein the extracting message events from the task queue in sequence, parsing the message events, and sending the parsed message events to a target node for processing comprises:
establishing a plurality of working threads to lock the task queue;
the multiple working threads sequentially extract the message events from the task queue and analyze the request content of the command line from the extracted message events;
calculating the waiting time of the command line request according to the command line request content, judging whether the command line request is overdue according to the waiting time, and outputting a request overtime error code if the command line request is overdue;
acquiring a target node of the command line request according to the content of the command line request, and sending the command line request to a service queue of the target node to wait for the target node to process;
the target node analyzes the command line request and executes corresponding service logic at the current node according to the command of the command line request, backfills an execution result to a result recording field, marks an information ID (identity) on the recording field and then sends the recording field to the main service node;
and the main service node sorts the record fields according to the marks of the record fields and sends the sorting result to the user.
4. A command line servicing system for distributed storage, comprising:
the system comprises a definition analysis unit, a task queue and a data processing unit, wherein the definition analysis unit is configured to convert a received user command line request into a command code according to a preset format, verify the command code and take the verified command code as a message event to be generated to the task queue;
the service subscription unit is configured and used for identifying the monitoring nodes from the storage system cluster, establishing a subscription relation between the main monitoring node and other nodes, and starting timing synchronous threads on all the monitoring nodes to monitor and backup the task queue;
the service processing unit is configured to sequentially extract the message events from the task queue, analyze the message events and send the analyzed message events to a target node for processing;
the service subscription unit includes:
the queue processing module is configured to initialize the task queue at each node of the cluster;
the node marking module is configured to acquire all node information of the cluster and mark monitoring nodes, wherein the monitoring nodes comprise a main monitoring node and a sub monitoring node;
the queue monitoring module is configured to start timing synchronization threads on all monitoring nodes, wherein the timing synchronization threads monitor data changes of a task queue, if data updating is monitored, the updated task queue data is persisted among all the monitoring nodes, and a task queue data copy is locally kept at the monitoring nodes monitoring the task queue;
and the fault switching module is configured to load the persistent queue data corresponding to the fault monitoring node into a switched target monitoring node memory for processing if the fault monitoring node exists.
5. The system of claim 4, wherein the definition parsing unit comprises:
the file loading module is used for loading the command line template library file which is defined and configured and the preset corresponding command codes of various command line commands;
the command conversion module is configured to convert the user command line request into a command code by a matching search method;
the authority verification module is configured for verifying whether the authority of the user name requested by the user command line is matched with the authority of the command code or not, and prompting an error code without authority if the authority is not matched with the authority of the command code;
and the legal verification module is configured and used for verifying the validity of the issued command line parameters according to the command codes and the parameter dependence configuration in the command line template library file.
6. The system of claim 4, wherein the service processing unit comprises:
the thread establishing module is configured to establish a plurality of working threads to lock the task queue;
the event extraction module is configured for sequentially extracting the message events from the task queue by the multiple working threads and analyzing the command line request content from the extracted message events;
the timeout verification module is configured to calculate the waiting time of the command line request according to the command line request content, judge whether the command line request is out of date according to the waiting time, and output a request timeout error code if the command line request is out of date;
the request sending module is configured to acquire a target node of the command line request according to the content of the command line request, and send the command line request to a service queue of the target node to wait for the target node to process;
the result recording module is configured for a target node to analyze the command line request and execute corresponding service logic at the current node according to the command of the command line request, the target node backfills an execution result to a result recording field, and the recording field marks an information ID and then sends the information ID to the main service node;
and the result sorting module is configured for sorting the record fields by the main service node according to the marks of the record fields and sending the sorting result to the user.
7. A terminal, comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any one of claims 1-3.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011392165.XA CN112463144B (en) | 2020-12-02 | 2020-12-02 | Distributed storage command line service method, system, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011392165.XA CN112463144B (en) | 2020-12-02 | 2020-12-02 | Distributed storage command line service method, system, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112463144A CN112463144A (en) | 2021-03-09 |
| CN112463144B true CN112463144B (en) | 2022-11-11 |
Family
ID=74806739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011392165.XA Active CN112463144B (en) | 2020-12-02 | 2020-12-02 | Distributed storage command line service method, system, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463144B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113098969B (en) * | 2021-04-09 | 2022-12-20 | 薪得付信息技术(上海)有限公司 | Data distribution method, device and system and electronic equipment |
| CN113722125B (en) * | 2021-09-08 | 2024-01-26 | 浙江中控技术股份有限公司 | Data processing method and system based on distributed subscription and release network |
| CN113810228A (en) * | 2021-09-13 | 2021-12-17 | 中国人民银行清算总中心 | Message queue channel resetting method and device |
| CN113923095A (en) * | 2021-09-30 | 2022-01-11 | 济南浪潮数据技术有限公司 | Cluster message forwarding method, system and storage medium |
| CN114138825A (en) * | 2021-11-24 | 2022-03-04 | 聚好看科技股份有限公司 | Server and method for providing data query service for application program |
| CN114063936B (en) * | 2022-01-18 | 2022-03-22 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium for optimizing timing task |
| CN114428682B (en) * | 2022-01-23 | 2023-11-03 | 苏州浪潮智能科技有限公司 | Overtime task processing method, overtime task processing system, storage medium and storage device |
| CN115174296B (en) * | 2022-05-30 | 2024-03-22 | 青岛海尔科技有限公司 | Equipment function access method and device, storage medium and electronic device |
| CN115086099B (en) * | 2022-06-07 | 2023-12-29 | 浙江浩瀚能源科技有限公司 | Data processing method, device, storage medium and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101110703A (en) * | 2007-06-28 | 2008-01-23 | 中兴通讯股份有限公司 | Made-to-order command line implementing method and system |
| CN102857355A (en) * | 2011-06-27 | 2013-01-02 | 中兴通讯股份有限公司 | Method and device for information adaptation |
| CN106375462A (en) * | 2016-09-13 | 2017-02-01 | 北京百度网讯科技有限公司 | Method and device for realizing message persistence in distributed message system |
| CN106528166A (en) * | 2016-11-22 | 2017-03-22 | 郑州云海信息技术有限公司 | Command line interface processing system and method |
| CN111562922A (en) * | 2020-04-29 | 2020-08-21 | 北京中大唯信科技有限公司 | Method, system and electronic equipment for modularizing command line program and cloud-end method and system |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6980987B2 (en) * | 2002-06-28 | 2005-12-27 | Alto Technology Resources, Inc. | Graphical user interface-relational database access system for a robotic archive |
-
2020
- 2020-12-02 CN CN202011392165.XA patent/CN112463144B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101110703A (en) * | 2007-06-28 | 2008-01-23 | 中兴通讯股份有限公司 | Made-to-order command line implementing method and system |
| CN102857355A (en) * | 2011-06-27 | 2013-01-02 | 中兴通讯股份有限公司 | Method and device for information adaptation |
| CN106375462A (en) * | 2016-09-13 | 2017-02-01 | 北京百度网讯科技有限公司 | Method and device for realizing message persistence in distributed message system |
| CN106528166A (en) * | 2016-11-22 | 2017-03-22 | 郑州云海信息技术有限公司 | Command line interface processing system and method |
| CN111562922A (en) * | 2020-04-29 | 2020-08-21 | 北京中大唯信科技有限公司 | Method, system and electronic equipment for modularizing command line program and cloud-end method and system |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
Non-Patent Citations (1)
| Title |
|---|
| 多层次命令行解析框架研究与设计;叶文晖等;《计算机工程与设计》;20070823(第16期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112463144A (en) | 2021-03-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112463144B (en) | Distributed storage command line service method, system, terminal and storage medium | |
| KR102493449B1 (en) | Edge computing test methods, devices, electronic devices and computer-readable media | |
| CN108108297B (en) | Method and device for automatic testing | |
| US11176030B2 (en) | Conducting automated software testing using centralized controller and distributed test host servers | |
| US9092230B2 (en) | Configuration of componentized software applications | |
| US20200348921A1 (en) | Microservice update system | |
| CN107241315B (en) | Access method and device of bank gateway interface and computer readable storage medium | |
| CN110222119B (en) | Data conversion synchronization method, equipment and storage medium for heterogeneous database | |
| US10223248B2 (en) | Conducting automated software testing using centralized controller and distributed test host servers | |
| US9442822B2 (en) | Providing a visual representation of a sub-set of a visual program | |
| CN112486466B (en) | Method for realizing quick universal basic framework based on micro-service architecture | |
| WO2017020459A1 (en) | Method and apparatus for configuring plugin package for host | |
| CN111538659B (en) | Interface testing method, system, electronic equipment and storage medium of business scene | |
| CN112099843A (en) | Code hosting platform management method and device, computer equipment and storage medium | |
| CN114356521A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN113296795A (en) | Application deployment method, device, equipment, storage medium and program product | |
| CN110688305B (en) | Test environment synchronization method, device, medium and electronic equipment | |
| CN112445860B (en) | Method and device for processing distributed transaction | |
| CN111258618A (en) | File configuration method and device, computer equipment and storage medium | |
| CN114443294B (en) | Big data service component deployment method, system, terminal and storage medium | |
| CN115934272A (en) | Online batch task processing method and device | |
| CN112115118B (en) | Database pressure measurement optimization method and device, storage medium and electronic equipment | |
| CN111708568A (en) | Component development decoupling method and terminal | |
| WO2003003244A1 (en) | Method of rapidly eliminating different information in databases | |
| CN115334152B (en) | Method for submitting structured machine learning calculation task to calculation cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |