CN117149512A - Metadata operation processing method and device - Google Patents

Metadata operation processing method and device Download PDF

Info

Publication number
CN117149512A
CN117149512A CN202210562829.5A CN202210562829A CN117149512A CN 117149512 A CN117149512 A CN 117149512A CN 202210562829 A CN202210562829 A CN 202210562829A CN 117149512 A CN117149512 A CN 117149512A
Authority
CN
China
Prior art keywords
metadata
log
state machine
server
servers
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210562829.5A
Other languages
Chinese (zh)
Inventor
李海波
吴建超
张晓�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Wodong Tianjun Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN202210562829.5A priority Critical patent/CN117149512A/en
Publication of CN117149512A publication Critical patent/CN117149512A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2358—Change logging, detection, and notification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
    • G06F9/4498—Finite state machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80—Database-specific techniques

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a metadata operation processing method and device, and relates to the technical field of data processing. One embodiment of the method comprises the following steps: receiving a call request of a client to a metadata interface, analyzing the call request to obtain operation parameters of metadata, and generating an operation log based on the operation parameters and the metadata; after the operation log is stored, the operation log is sent to the slave servers to be stored, so that a storage result returned by each slave server is received, and the server duty ratio of which the storage result is successful is calculated; and in response to the duty ratio being greater than or equal to the preset duty ratio, submitting the operation log to a state machine so that the state machine operates the metadata according to the operation parameters, and returning an operation result to the client. The operation log of the embodiment needs to be sent to all servers, and after most servers are successfully stored, the operation log is submitted to a state machine to update metadata, so that the condition that the operation log is lost due to downtime or disk loss of the subsequent servers is avoided.

Description

Metadata operation processing method and device
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a metadata operation processing method and apparatus.
Background
As enterprises have evolved into digital intelligence, the demands of online analytical processing systems (Online Analytical Processing, OLAP for short) have increased explosively, and many OLAP components have emerged to meet such demands, such as Kylin (an independent intellectual property operating system), guided (an efficient data query system), and ClickHouse (an open-source columnar database, mainly used in the field of data analysis).
Metadata management is an important part in OLAP, and metadata refers to the information of fragments and copies of a cluster, information of libraries, tables, account numbers and the like created by users, information of data files and the like, and metadata management is to manage creation, change, deletion and access of the metadata. However, metadata management currently has some defects, such as low availability caused by the fact that consistency of data in a master server and a slave server cannot be guaranteed, high management cost and excessive operation timeout and fluctuation caused by limited throughput.
Disclosure of Invention
In view of this, embodiments of the present invention provide a metadata operation processing method and apparatus, which at least can solve the problems of low availability, high cost and limited throughput of metadata management in the prior art.
To achieve the above object, according to one aspect of the embodiments of the present invention, there is provided a metadata manipulation processing method including:
Receiving a call request of a client to a metadata interface, analyzing the call request to obtain operation parameters of metadata, and generating an operation log based on the operation parameters and the metadata;
after the operation log is stored, the operation log is sent to a slave server for storage, so that a storage result returned by each slave server is received, and the server duty ratio of which the storage result is successful is calculated;
responding to the fact that the duty ratio is larger than or equal to a preset duty ratio, submitting the operation log to a state machine, enabling the state machine to operate metadata according to the operation parameters, and returning an operation result to a client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
Optionally, the operation parameters include an operation type and operation content, the operation content includes a key name and a key value of metadata, and the state machine operates the metadata according to the operation parameters, and the method includes:
the state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
When the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
And when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
Optionally, the state machine operates metadata according to the operation parameters, and further includes:
updating the state of the metadata, wherein when the operation type is newly added or modified, the state comprises the latest version number of the metadata; and determining the latest historical version number of the metadata, and taking the sum of the latest historical version number and a first preset value as the latest version number of the metadata.
Optionally, the operation log further includes a log number; the sum of the current maximum log number and a second preset value is used as the current log number;
the sending the operation log to the slave server for storage further comprises:
when the slave server stores the operation log, calculating the difference value between the log number and the second preset value, and determining the maximum log number of other operation logs which are stored currently;
Determining an operation log located between the maximum log number and the log number as a missing log in response to the maximum log number being less than the difference value;
and sending a request for acquiring the missing log to a main server so as to receive the missing log returned by the main server, and storing the missing log and the operation log according to the sequence from the small log number to the large log number.
Optionally, the master server and the slave server save the operation log to the disk, and the method further includes:
in response to a restarting operation on a target server, the operation log of the metadata is fetched in batches from the disk of the target server, or in response to the disk does not exist or the disk content is empty, the operation log of the metadata is fetched in batches from the disk of other servers;
and submitting the operation logs to the state machine one by one according to the sequence of log numbers from small to large, so that the state machine updates the metadata according to the operation logs to obtain the metadata of the latest version.
Optionally, the calculating the server duty ratio with the saved result being successful includes:
counting the number of the successful slave servers as a storage result;
calculating the ratio of the number of the slave servers to the total number of the slave servers; or (b)
And accumulating the sum of the number of the slave servers and the number of the master servers, and calculating the ratio of the accumulated value to the total number of the servers.
To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided a metadata operation processing apparatus, a main server including a processing module, a log module, and a state machine:
the processing module is used for receiving a calling request of the client to the metadata interface and analyzing the calling request to obtain operation parameters of metadata; after the operation log is stored, the operation log is sent to a slave server to be stored, a storage result returned by each slave server is received, the server duty ratio of which the storage result is successful is calculated, and the operation log is submitted to a state machine in response to the duty ratio being greater than or equal to a preset duty ratio;
the log module is used for generating an operation log based on the operation parameters and the metadata and storing the operation log;
the state machine is used for operating the metadata according to the operation parameters and returning an operation result to the client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
Optionally, the operation parameters include an operation type and operation content, the operation content includes a key name and a key value of metadata, and the state machine is configured to:
The state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
When the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
And when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
Optionally, the state machine is further configured to:
updating the state of the metadata, wherein when the operation type is newly added or modified, the state comprises the latest version number of the metadata; and determining the latest historical version number of the metadata, and taking the sum of the latest historical version number and a first preset value as the latest version number of the metadata.
Optionally, the log module is further configured to: the operation log also comprises a log number; the sum of the current maximum log number and a second preset value is used as the current log number;
A log module of the slave server for:
when the slave server stores the operation log, calculating the difference value between the log number and the second preset value, and determining the maximum log number of other operation logs which are stored currently;
determining an operation log located between the maximum log number and the log number as a missing log in response to the maximum log number being less than the difference value;
and sending a request for acquiring the missing log to a main server so as to receive the missing log returned by the main server, and storing the missing log and the operation log according to the sequence from the small log number to the large log number.
Optionally, the master server and the slave server store operation logs to a disk, and the log module is further configured to:
in response to a restarting operation on a target server, the operation log of the metadata is fetched in batches from the disk of the target server, or in response to the disk does not exist or the disk content is empty, the operation log of the metadata is fetched in batches from the disk of other servers;
and submitting the operation logs to the state machine one by one according to the sequence of log numbers from small to large, so that the state machine updates the metadata according to the operation logs to obtain the metadata of the latest version.
Optionally, the processing module is configured to:
counting the number of the successful slave servers as a storage result;
calculating the ratio of the number of the slave servers to the total number of the slave servers; or (b)
And accumulating the sum of the number of the slave servers and the number of the master servers, and calculating the ratio of the accumulated value to the total number of the servers.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a metadata operation processing electronic device.
The electronic equipment of the embodiment of the invention comprises: one or more processors; and a storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement any of the metadata operation processing methods described above.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements any one of the above-described metadata manipulation processing methods.
According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: the metadata operation log needs to be sent to all servers and persisted to the disk, and after most servers are successfully saved, the operation log is submitted to a state machine for metadata updating, so that the operation log is ensured to be saved in a server cluster, and the condition that the metadata operation log is lost due to downtime or disk loss of the subsequent servers is avoided.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic flow diagram of a metadata operation processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an overall implementation flow;
FIG. 3 is a flow chart of a specific metadata manipulation processing method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of main modules of a metadata operation processing apparatus according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It is noted that embodiments of the application and features of the embodiments may be combined with each other without conflict. The technical scheme of the application obtains, stores, uses, processes and the like the data, which all meet the relevant regulations of national laws and regulations.
A distributed metadata management system should have several features: high availability, high throughput, and low latency. High availability means that the metadata stored in the system is high in reliability and cannot be lost, and normal use is not affected when part of the servers are down. High throughput means that the larger the number of logs per second the system can handle, the better the same amount of data is written each time. Low latency means that the response time per write to the log can be very low and smooth.
Current OLAP typically utilizes third party components to store metadata, such as Kylin uses HBase (a distributed, column-oriented open source database), drague uses MySQL (a relational database management system), clickHouse uses ZooKeeper (a distributed, open source distributed application coordination service), but each has some limitations:
1. the primary and backup synchronization modes of MySQL cannot guarantee the consistency of data, if the data is not synchronized to the backup server after the data of the primary server is persistence, the data is lost if the primary server is down at the moment, and then the service cannot be provided.
2. The HBase involves more modules, resulting in higher cost and resource occupation for managing metadata operations and dimensions. Kylin stores metadata in HBase because data is stored in HBase, and storing metadata with HBase for other OLAP systems increases system complexity and operation and maintenance costs.
3. The ZooKeeper has performance bottleneck, throughput limitation, queuing and overtime during high-frequency operation, and is limited by JVM (Java Virtual Machine ), so that delay fluctuation is large, the problem can affect the expandability of an OLAP system, and ultra-large-scale clusters can not be deployed.
In order to solve the three problems, the server cluster is constructed, the log of adding, deleting and changing metadata is durable in most servers of the cluster each time, and when a certain server is down, metadata operation logs can be found out through other servers, so that high availability of metadata is ensured, and services can be continuously provided. Meanwhile, the performance of the scheme is superior to that of other components, higher throughput and lower and more stable delay are provided, and the metadata system is enabled to run faster and more stably.
The words related to the scheme are explained as follows:
Metadata (Metadata): also called intermediate data and relay data, which are data (data about data) describing data, and information mainly describing data attribute (property) for supporting functions such as indicating storage location, history data, resource searching, file recording, etc.
OLAP: a software technology enables analysts to quickly, consistently and interactively observe information from various aspects to achieve the goal of deep understanding data, wherein Kylin, druid and ClickHouse are all open source components of OLAP.
The processing module is used for: and sending logs to all servers, and judging whether more than half of the servers are successfully returned.
And (3) log: and (3) operating a record of the metadata, and obtaining the latest version of the metadata according to the information recorded by the log.
State machine StateMachine: the latest version number of the metadata is saved, and the log is submitted to a state machine after being written to a half of the server.
Referring to fig. 1, a main flowchart of a metadata operation processing method provided by an embodiment of the present invention is shown, including the following steps:
s101: the method comprises the steps that a main server receives a call request of a client to a metadata interface, analyzes the call request to obtain operation parameters of metadata, and generates an operation log based on the operation parameters and the metadata;
S102: after the operation log is stored, the operation log is sent to a slave server for storage, so that a storage result returned by each slave server is received, and the server duty ratio of which the storage result is successful is calculated;
s103: responding to the fact that the duty ratio is larger than or equal to a preset duty ratio, submitting the operation log to a state machine, enabling the state machine to operate metadata according to the operation parameters, and returning operation result information to a client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
In the above embodiment, for step S101, a server cluster is provided in this embodiment, where the server cluster includes a master server and a plurality of slave servers, and is configured to store an operation log for metadata, and each server includes a state machine, where the state machine is configured to store metadata, and subsequent data reading may be performed from the state machine.
The main server is provided with a metadata interface, and a user can initiate a call request for the metadata interface through the client to realize creation, modification and deletion operations for metadata, so that the call request needs to carry operation types so as to execute different processing logics according to different operation types. When the user needs to operate the metadata, the client can connect to the main server based on TCP (Transmission Control Protocol ), and create, modify and delete the metadata, which needs to connect to the main server, and read the metadata from the state machine.
After receiving a call request for a certain metadata transmitted by a client, a main server analyzes the request to obtain operation parameters for the metadata, wherein the operation parameters comprise an operation type and operation content, the operation content is different according to different operation types, but is in a key-value form (key name-key value), and a subsequent state machine performs adding and deleting operations on the metadata by executing processing logic corresponding to the operation type.
The main server generates an operation log for the metadata based on the metadata to be processed and the parsed operation parameters; the operation log comprises a log number, an operation type of metadata and operation content. The memory of the main server is provided with a circulation array for storing the latest log number, so that the current maximum log number can be determined through the circulation data, and the maximum log number plus a second preset value (e.g. 1) is used as the log number of the operation log.
For step S102, after generating the metadata operation log, if the metadata log is created, the master server saves the operation log in its own disk, and after the operation log is saved, the master server sends the operation log to other slave servers, and after receiving the operation log sent by the master server, the slave server saves the operation log in its own disk, and after executing, returns the saved result information to the master server.
Further, the log numbering is used for maintaining the continuity of the log in the disk and preventing the metadata and the log from missing, such as the first log operation and the second log operation for the metadata A. When the operation log with the log number is stored from the server, if the maximum log number of other operation logs stored in the disk is found to be unequal to the current log number-the second preset value, the log between the maximum log number and the current log number is the missing log, and the logs with middle missing need to be acquired from the main server, specifically, a request for acquiring the missing log is sent to the main server so as to receive the missing log returned by the main server.
A server cluster typically consists of an odd number of servers equal to or greater than 3, with one and only one master server, and the remainder being slave servers, log generation being possible only at the master server, but metadata oplogs can be read from the master and slave servers. In actual operation, some slave servers may fail to save the log to the disk due to network abnormality, disk failure and the like, so that the save result returned from the server needs to be checked in a circulating manner:
1) When the storage result is the successful server number, including the number of slave servers and one master server, which is greater than or equal to half of the number of all servers in the server cluster, log storage success is confirmed, for example, more than 2 servers need to be stored successfully, and more than 3 servers need to be stored successfully.
2) And when the number of the successful slave servers is greater than or equal to half of the number of all the slave servers as a result of the storage, confirming that the log storage is successful. Since the master server generally writes successfully, more than half of the servers here include only slave servers, e.g., 3 servers require more than 1 slave server to write successfully, and 5 servers require 2 and more slave servers to write successfully.
3) But if 1) or 2) is not satisfied within a prescribed time (e.g., 1 minute), the operation metadata failure result is returned to the client.
By the method, most servers can be ensured to store the operation log, compared with the method that all servers are confirmed to be successfully stored and then follow-up operation is continued, the time delay for returning operation result information of the client can be properly shortened, and meanwhile throughput of the server cluster is improved.
For step S103, by storing the operation log to the master server and the slave server as described above, it is ensured that the operation log is stored in the disk of most servers. Subsequently, when a certain server is restarted, the operation log of the metadata can be fetched in batches from the disk of the server, but when the disk of the server is lost (aiming at downtime and fault scenes) or the disk content is empty (aiming at capacity expansion scenes), the operation log of the metadata can also be fetched in batches from the disk of other servers. And then, in order to ensure that the logs in the server correspond to the metadata in the state machine, the operation logs which are called in batches are submitted to the state machine one by one according to the sequence of the log numbers from small to large, so that the state machine updates the metadata according to the operation logs to obtain the metadata of the latest version, and the subsequent reading performance is better.
The main server includes a state machine, as shown in fig. 2, after determining that most servers write the operation log successfully, the main server submits the operation log to its own state machine, so that the state machine determines corresponding operation logic according to the operation type, and further performs corresponding operation on metadata based on the operation content according to the operation logic, thereby obtaining updated metadata. Whether the state machine is updated successfully or not, the operation log written into the disk is not affected. Specifically:
1. the operation type is creation, the scene of creating the metadata can be creation library, table or account, the operation content is key and value of the metadata to be created, and the metadata is created according to the key and value, such as metadata A is created;
2. the operation type is deleting, the operation content can be a key and a value of metadata to be deleted, or can only contain a key, and then the corresponding metadata can be directly deleted according to the key, for example, the metadata B is deleted according to the key of the metadata B;
3. the operation type is modification, the operation content is a key and a value of metadata to be modified, the value is a modification value, and after the original value (namely, a target value) is determined according to the key, the target value is replaced by the value in the operation content, for example, name three in metadata C is replaced by Lifour.
The state machine comprises a hash table for metadata keys, and in order to optimize memory occupation and reduce capacity expansion time, the hash table is provided with a plurality of sub-buckets, each bucket is a small hash table and can only contain a section of hash value, so that the memory copy cost in capacity expansion of the hash table is reduced, the card time is reduced, and the stability of time delay of inquiring metadata is ensured. If all metadata are stored in a hash table, when the capacity of the hash table is insufficient and the capacity needs to be expanded, a large memory is usually reapplied, the existing data are copied into a new memory, and the process is to be locked, wherein the locking time depends on the size of the new memory and the old memory, so that the memory size and the capacity expansion time of each barrel can be effectively reduced through barrel separation, the read-write delay is reduced, and the stability of the delay is ensured. Considering that metadata generally consists of Key and Value, for example, the Key is "table1", the Value is "Create table table1 …", hash processing can be performed on the metadata Key to obtain a hash Value, whether the hash Value exists in the hash table is judged, and different operations are executed according to the judgment result.
1. For the creation operation, if the hash value exists in the hash table, the metadata history is shown to exist, the creation is impossible, and a creation failure result is returned to the client;
2. For the deleting and modifying operation, if the hash value does not exist in the hash table, the metadata is deleted before, the modifying or deleting operation cannot be continued at this time, and a deleting failure result or a modifying failure result is returned to the client.
In addition, each update of metadata in the state machine generates a version number, which is preferably represented by an integer, such as 1 initially, 2 once, 3 once again, and so on, each time by adding a first predetermined value. Metadata of each version can be reserved in the state machine, or only metadata of the latest version can be reserved, and parameters carried by subsequent metadata reading requests are different according to different reservation modes:
1. only the metadata of the latest version is reserved, the request can only contain the key of the metadata, and the state machine reads the corresponding value according to the metadata key and returns the value to the client;
2. the metadata of each version or the preset number of metadata (for example, only 5 metadata are reserved) nearest to the current version are reserved, the request needs to contain version numbers besides metadata keys, a state machine firstly reads the value of each corresponding version number according to the metadata keys, then determines corresponding metadata according to the version numbers and returns the corresponding metadata to the client.
It should be noted that the core component of each server is composed of three parts, namely a processing module, a state machine and a log module. The log module refers to logs of metadata operations, such as creating, modifying and deleting metadata, each log has a number, and is responsible for writing the operation log into a disk, and can acquire the log according to the log number. The processing module is responsible for communicating with other servers, such as sending logs and receiving save result information, and determining whether the logs of most servers are successfully written. The state machine is used for storing the latest version of the metadata, and after the log writing is successful, the metadata operation log is put into the state machine, and the metadata operation log is obtained from the state machine when the metadata is read.
Thus, after the master server submits the metadata operation log to its state machine, the slave server resubmits the metadata operation log to the state machine, and if the slave server does not submit the operation log to the state machine, the subsequent client cannot read the latest metadata from the state machine, so that the write log operation can only be accessed by the master server, but for metadata reading, the write log operation can be accessed from the master server or the slave server.
According to the method provided by the embodiment, the metadata operation log needs to be sent to all servers and persisted to the disk, and after most servers are successfully stored, the operation log is submitted to the state machine for metadata updating, so that the operation log is ensured to be stored in a server cluster for multiple times, and the condition that the metadata operation log is lost due to the fact that the subsequent servers are down or the disk is lost is avoided.
Referring to fig. 3, a flowchart of a metadata operation processing method according to an embodiment of the present invention is shown, including the following steps:
s301: the method comprises the steps that a main server receives a call request of a client to a metadata interface, analyzes the call request to obtain operation parameters of metadata, generates a log number, and generates an operation log based on the operation parameters and the log number; the operation parameters comprise operation types and operation contents, wherein the operation contents comprise key names and key values of metadata;
s302: after the operation log is stored, the operation log is sent to a slave server for storage;
s303: when the slave server stores the operation log, calculating the difference value between the log number and a second preset value, and determining the maximum log number of other operation logs which are currently stored;
s304: determining an operation log located between the maximum log number and the log number as a missing log in response to the maximum log number being less than the difference value;
s305: sending a request for acquiring the missing log to a main server to receive the missing log returned by the main server, and storing the missing log and the operation log according to the sequence from small log numbers to large log numbers;
S306: the main server receives the preservation result returned by each slave server, and calculates the preservation result as the successful server duty ratio;
s307: submitting the operation log to a state machine in response to the duty cycle being greater than or equal to a preset duty cycle; wherein, the state machine is used for reading the metadata from the state machine and returning when processing the read request of the metadata;
s308: the state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
s309: when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
S310: when the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
S311: and when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
Although some steps may be changed in actual operation, the changed method cannot ensure the reliability and consistency of the data, and some may have complexity higher than that of the scheme. For example, the client may only obtain the source data from the primary server, and may ensure that the obtained source data is up-to-date, but may reduce the throughput of the primary server. For example, if continuity of log numbers from the server is not guaranteed, complexity of guaranteeing integrity of logs by the background thread is greatly increased, and delay of reading the latest logs is increased because the latest logs are not stored in the memory. For example, if only one hash table is set in the state machine, it is more time-consuming to apply for a new memory block and copy existing data to the memory block during capacity expansion, and more space is wasted.
Compared with the prior art, the method provided by the scheme has at least the following beneficial effects:
1. the server cluster is only used for storing metadata operation logs, specific metadata still needs to be stored in a state machine StateMachine of the server, and the metadata in the StateMachine needs to be updated by relying on the metadata operation logs;
2. the server cluster comprises a master server and a plurality of slave servers, the metadata operation log is required to be sent to all servers to be persisted to the disk, and most servers are required to be successfully saved, so that the operation log is ensured to be saved in multiple parts in the server cluster, and the condition that the metadata operation log is lost due to the fact that some subsequent servers are down or the disk is lost is avoided;
3. When the server is down and restarted, the metadata operation log is triggered to be read from the disk of the server (the disk is lost or needs to be read from other server disks when the disk is empty) and rolled back again to obtain the metadata of the latest version, so that the condition that the metadata in the state machine does not correspond to the metadata operation log in the server is avoided;
4. each operation log has a log number, when the slave server stores the current operation log, if the numbers of other operation logs in the disk are found to be discontinuous with the current stored number, the missing log needs to be pulled from the master server.
Referring to fig. 4, a schematic diagram of main modules of a metadata operation processing apparatus 400 provided by an embodiment of the present invention is shown, where a main server includes a processing module, a log module, and a state machine:
the processing module 401 is configured to receive a call request from a client to a metadata interface, and parse the call request to obtain an operation parameter for metadata; after the operation log is stored, the operation log is sent to a slave server to be stored, a storage result returned by each slave server is received, the server duty ratio of which the storage result is successful is calculated, and the operation log is submitted to a state machine in response to the duty ratio being greater than or equal to a preset duty ratio;
A log module 402, configured to generate and store an operation log based on the operation parameters and metadata;
the state machine 403 is configured to operate the metadata according to the operation parameters, and return an operation result to the client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
In the embodiment of the present invention, the operation parameters include an operation type and an operation content, the operation content includes a key name and a key value of metadata, and the state machine 403 is configured to:
the state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
When the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
And when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
In the embodiment of the present invention, the state machine 403 is further configured to:
updating the state of the metadata, wherein when the operation type is newly added or modified, the state comprises the latest version number of the metadata; and determining the latest historical version number of the metadata, and taking the sum of the latest historical version number and a first preset value as the latest version number of the metadata.
In the embodiment of the present invention, the log module 402 is further configured to: the operation log also comprises a log number; the sum of the current maximum log number and a second preset value is used as the current log number;
a log module of the slave server for:
when the slave server stores the operation log, calculating the difference value between the log number and the second preset value, and determining the maximum log number of other operation logs which are stored currently;
determining an operation log located between the maximum log number and the log number as a missing log in response to the maximum log number being less than the difference value;
and sending a request for acquiring the missing log to a main server so as to receive the missing log returned by the main server, and storing the missing log and the operation log according to the sequence from the small log number to the large log number.
In the embodiment of the present invention, the master server and the slave server store the operation log to the disk, and the log module 402 is further configured to:
in response to a restarting operation on a target server, the operation log of the metadata is fetched in batches from the disk of the target server, or in response to the disk does not exist or the disk content is empty, the operation log of the metadata is fetched in batches from the disk of other servers;
and submitting the operation logs to the state machine one by one according to the sequence of log numbers from small to large, so that the state machine updates the metadata according to the operation logs to obtain the metadata of the latest version.
In the embodiment of the present invention, the processing module 401 is configured to:
counting the number of the successful slave servers as a storage result;
calculating the ratio of the number of the slave servers to the total number of the slave servers; or (b)
And accumulating the sum of the number of the slave servers and the number of the master servers, and calculating the ratio of the accumulated value to the total number of the servers.
In addition, the implementation of the apparatus in the embodiments of the present invention has been described in detail in the above method, so that the description is not repeated here.
Fig. 5 shows an exemplary system architecture 500, including terminal devices 501, 502, 503, a network 504, and a server 505 (by way of example only), to which embodiments of the invention may be applied.
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, are installed with various communication client applications, and a user may interact with the server 505 through the network 504 using the terminal devices 501, 502, 503 to receive or transmit messages, etc.
The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The server 505 may be a server providing various services, and it should be noted that the method provided by the embodiment of the present invention is generally performed by the server 505, and accordingly, the apparatus is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 6 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a processing module, a log module, and a state machine. The names of these modules do not in some way constitute a limitation of the module itself, for example, a processing module may also be described as a "consistency module".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer-readable medium carries one or more programs which, when executed by a device, cause the device to perform any of the metadata manipulation methods described above.
According to the technical scheme provided by the embodiment of the invention, at least the following beneficial effects exist:
1. the server cluster is only used for storing metadata operation logs, specific metadata still needs to be stored in a state machine StateMachine of the server, and the metadata in the StateMachine needs to be updated by relying on the metadata operation logs;
2. the server cluster comprises a master server and a plurality of slave servers, the metadata operation log is required to be sent to all servers to be persisted to the disk, and most servers are required to be successfully saved, so that the operation log is ensured to be saved in multiple parts in the server cluster, and the condition that the metadata operation log is lost due to the fact that some subsequent servers are down or the disk is lost is avoided;
3. When the server is down and restarted, the metadata operation log is triggered to be read from the disk of the server (the disk is lost or needs to be read from other server disks when the disk is empty) and rolled back again to obtain the metadata of the latest version, so that the condition that the metadata in the state machine does not correspond to the metadata operation log in the server is avoided;
4. each operation log has a log number, when the slave server stores the current operation log, if the numbers of other operation logs in the disk are found to be discontinuous with the current stored number, the missing log needs to be pulled from the master server.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims (10)

1. A metadata manipulation processing method, characterized by comprising:
receiving a call request of a client to a metadata interface, analyzing the call request to obtain operation parameters of metadata, and generating an operation log based on the operation parameters and the metadata;
After the operation log is stored, the operation log is sent to a slave server for storage, so that a storage result returned by each slave server is received, and the server duty ratio of which the storage result is successful is calculated;
responding to the fact that the duty ratio is larger than or equal to a preset duty ratio, submitting the operation log to a state machine, enabling the state machine to operate metadata according to the operation parameters, and returning an operation result to a client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
2. The method of claim 1, wherein the operating parameters include an operation type and an operation content, the operation content including a key name and a key value of metadata, the state machine operating the metadata according to the operating parameters, comprising:
the state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
When the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
And when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
3. The method of claim 2, wherein the state machine operates metadata in accordance with the operating parameters, further comprising:
updating the state of the metadata, wherein when the operation type is newly added or modified, the state comprises the latest version number of the metadata; and determining the latest historical version number of the metadata, and taking the sum of the latest historical version number and a first preset value as the latest version number of the metadata.
4. A method according to any one of claims 1-3, wherein the oplog further comprises a log number; the sum of the current maximum log number and a second preset value is used as the current log number;
the sending the operation log to the slave server for storage further comprises:
when the slave server stores the operation log, calculating the difference value between the log number and the second preset value, and determining the maximum log number of other operation logs which are stored currently;
Determining an operation log located between the maximum log number and the log number as a missing log in response to the maximum log number being less than the difference value;
and sending a request for acquiring the missing log to a main server so as to receive the missing log returned by the main server, and storing the missing log and the operation log according to the sequence from the small log number to the large log number.
5. The method of claim 4, wherein the master server and the slave server save the operation log to disk, the method further comprising:
in response to a restarting operation on a target server, the operation log of the metadata is fetched in batches from the disk of the target server, or in response to the disk does not exist or the disk content is empty, the operation log of the metadata is fetched in batches from the disk of other servers;
and submitting the operation logs to the state machine one by one according to the sequence of log numbers from small to large, so that the state machine updates the metadata according to the operation logs to obtain the metadata of the latest version.
6. The method of claim 1, wherein the calculating a server duty cycle at which the saved result is successful comprises:
counting the number of the successful slave servers as a storage result;
Calculating the ratio of the number of the slave servers to the total number of the slave servers; or (b)
And accumulating the sum of the number of the slave servers and the number of the master servers, and calculating the ratio of the accumulated value to the total number of the servers.
7. A metadata operation processing device, wherein a main server comprises a communication module, a log module and a state machine:
the communication module is used for receiving a calling request of the client to the metadata interface and analyzing the calling request to obtain operation parameters of metadata; after the operation log is stored, the operation log is sent to a slave server to be stored, a storage result returned by each slave server is received, the server duty ratio of which the storage result is successful is calculated, and the operation log is submitted to a state machine in response to the duty ratio being greater than or equal to a preset duty ratio;
the log module is used for generating an operation log based on the operation parameters and the metadata and storing the operation log;
the state machine is used for operating the metadata according to the operation parameters and returning an operation result to the client; wherein the state machine is configured to read and return metadata from the state machine when processing a read request for the metadata.
8. The apparatus of claim 7, wherein the operation parameters include an operation type and an operation content, the operation content including a key name and a key value of the metadata, the state machine to:
The state machine carries out hash processing on the key value of the metadata to obtain a hash value, and inquires whether the hash value exists in a hash table; wherein the state machine comprises a hash table of metadata key values;
when the operation type is creation, if the query result is present, returning a creation failure result to the client, otherwise, creating metadata according to the key name and the key value; or (b)
When the operation type is deleting, if the query result is not present, returning a deleting failure result to the client, otherwise deleting the metadata corresponding to the key name; or (b)
And when the operation type is modification, if the query result is not present, returning a modification failure result to the client, otherwise, determining a target key value corresponding to the key name, and replacing the target key value with the key value.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
CN202210562829.5A 2022-05-23 2022-05-23 Metadata operation processing method and device Pending CN117149512A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210562829.5A CN117149512A (en) 2022-05-23 2022-05-23 Metadata operation processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210562829.5A CN117149512A (en) 2022-05-23 2022-05-23 Metadata operation processing method and device

Publications (1)

Publication Number Publication Date
CN117149512A true CN117149512A (en) 2023-12-01

Family

ID=88885401

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210562829.5A Pending CN117149512A (en) 2022-05-23 2022-05-23 Metadata operation processing method and device

Country Status (1)

Country Link
CN (1) CN117149512A (en)

Similar Documents

Publication Publication Date Title
CN109493076B (en) Kafka message unique consumption method, system, server and storage medium
CN108363813B (en) Data storage method, device and system
KR20200027413A (en) Method, device and system for storing data
US10187255B2 (en) Centralized configuration data in a distributed file system
CN111881116A (en) Data migration method, data migration system, computer system, and storage medium
CN111177159B (en) Data processing system and method and data updating equipment
CN109614439A (en) Method of data synchronization, device, electronic equipment and storage medium
CN114900449B (en) Resource information management method, system and device
CN109831323B (en) Server information management method, management system and server
WO2023073547A1 (en) Efficient creation of secondary database system
WO2024103898A1 (en) Database cluster management method and apparatus
US11321374B2 (en) External storage of unstructured database objects
CN113127430B (en) Mirror image information processing method, mirror image information processing device, computer readable medium and electronic equipment
CN117950850A (en) Data transmission method, device, electronic equipment and computer readable medium
CN116226151A (en) Method and device for storing, reading and deleting data
CN114969165B (en) Data query request processing method, device, equipment and storage medium
CN112115206A (en) Method and device for processing object storage metadata
CN117149512A (en) Metadata operation processing method and device
CN115454666A (en) Data synchronization method and device among message queue clusters
CN114925078A (en) Data updating method, system, electronic device and storage medium
CN114756173A (en) Method, system, device and computer readable medium for file merging
CN113568892A (en) Method and equipment for carrying out data query on data source based on memory calculation
CN110851192B (en) Method and device for responding to degraded switch configuration
CN113761049A (en) Data synchronization method and device under read-write separation
US11386051B2 (en) Automatic intelligent hybrid business intelligence platform service

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination