CN110196835B

CN110196835B - Metadata processing method, metadata processing device and electronic equipment

Info

Publication number: CN110196835B
Application number: CN201910452885.1A
Authority: CN
Inventors: 葛凯凯; 潘晓东
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-05-28
Filing date: 2019-05-28
Publication date: 2023-10-13
Anticipated expiration: 2039-05-28
Also published as: CN110196835A

Abstract

The disclosure relates to the technical field of data processing, and provides a metadata processing method, a metadata processing device, a computer readable storage medium for realizing the metadata processing method and electronic equipment. Wherein the method comprises the following steps: responding to a pre-written log type operation request of metadata, and determining a metadata service thread and a log service thread; processing the metadata according to the operation request through the metadata service thread, and acquiring a log record obtained by processing the metadata; and processing the log record through the log service thread to finish log processing corresponding to the operation request. The technical scheme can improve the data processing efficiency of the metadata server, is favorable for improving the concurrency of metadata access, and further improves the overall performance of the distributed file storage system.

Description

Metadata processing method, metadata processing device and electronic equipment

Technical Field

The disclosure relates to the technical field of data processing, in particular to a metadata processing method, a metadata processing device and electronic equipment for realizing the metadata processing method.

Background

To ensure the reliability of the metadata operations, the metadata operations may be recorded in a log-before-write (writehead) manner. Taking a MetaData server (MetaData Server, abbreviated as MDS) in a distributed file storage system Ceph as an example, the data processing method is generated after the MDS receives the operation request of the type.

Currently, after the MDS in the Ceph receives the operation request passing through the above type, the corresponding metadata processing is generally completed based on a single thread. Specifically, for an operation request of the above type, a series of tasks such as receiving the operation request based on a single thread, searching for metadata cache, modifying and processing corresponding metadata, generating a log record, writing the log record into a write cache area, and refreshing the log write cache area to an Object storage device (Object-based Storage Device, abbreviated as OSD) after the log write cache area is full are performed. And after the above-mentioned respective processes are completed, processing of the next operation request of the above-mentioned type is continued.

However, in the metadata processing method provided by the above-described related art, the data processing efficiency of the metadata server is to be improved for the operation request of the above-described type.

It should be noted that the information disclosed in the foregoing background section is only for enhancing understanding of the background of the present disclosure.

Disclosure of Invention

The present disclosure provides a metadata processing method, a metadata processing device, a computer storage medium, and an electronic apparatus, and further shortens a processing period of metadata to at least a certain extent, thereby improving data processing efficiency of a metadata server.

Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.

According to one aspect of the present disclosure, there is provided a metadata processing method, including: determining a metadata service thread and a Log service thread in response to an operation request of a Write Ahead Log (WAL) for metadata; processing the metadata according to the operation request through the metadata service thread, and acquiring a log record obtained by processing the metadata; and processing the log record through the log service thread to finish log processing corresponding to the operation request.

According to one aspect of the present disclosure, there is provided a metadata processing apparatus including a service thread determination module configured to determine a metadata service thread and a log service thread in response to a pre-written log-type operation request for metadata; the log thread processing module is configured to process the metadata according to the operation request through the metadata service thread and acquire log records obtained by processing the metadata; and the log thread processing module is configured to process the log record through the log service thread so as to complete log processing corresponding to the operation request.

In some embodiments of the present disclosure, based on the foregoing solution, the data processing flow corresponding to the operation request includes: a metadata request stage, a log write cache stage and a log write cache region refreshing stage; the service thread determining module is specifically configured to: determining a metadata service thread for processing the metadata request phase; determining a first journaling service thread for processing the journaling write cache phase; and determining a second log service thread for processing the log write cache flush phase.

In some embodiments of the present disclosure, based on the foregoing solution, the metadata thread processing module is specifically configured to: and in the metadata request stage, processing the metadata according to the operation request by the metadata service thread, and acquiring a log record obtained by processing the metadata.

In some embodiments of the present disclosure, based on the foregoing solution, the metadata thread processing module includes: a search unit, a copy-on-write unit, a modification unit, and a receiving unit.

Wherein, in the metadata service thread, the searching unit is configured to: searching target metadata in a metadata cache according to a current operation request in the metadata service thread, and acquiring a log record obtained by searching the target metadata; the copy-on-write unit is configured to: copy On Write (COW) is carried out On the target metadata to obtain Copy metadata, and log records obtained by copying the target metadata when writing are obtained; the modification unit is configured to: modifying the copy metadata according to the current operation request, and acquiring a log record obtained by modifying the copy metadata; the current operation request is a pre-written log type operation request; and the receiving unit is configured to: waiting for receiving and processing a next operation request, wherein the next operation request is a pre-written log type operation request.

In some embodiments of the disclosure, based on the foregoing solution, the metadata thread processing module further includes: and a transmitting unit.

Wherein the transmitting unit is configured to: and sending the log record to a log processing linked list for processing of the first log service thread.

In some embodiments of the disclosure, based on the foregoing solution, the log processing linked list includes a plurality of sub-linked list structures, where the sub-linked list structures form nodes of the log processing linked list; the transmitting unit is specifically configured to: and determining a target node for storing the log record in the log processing linked list, and sending the log record to a linked list tail of a sub-linked list structure corresponding to the target node.

In some embodiments of the present disclosure, based on the foregoing scheme, the transmitting unit is specifically configured to: and packaging the log record and the callback function of the log record into a log object, and sending the log object to a linked list tail of a sub-linked list structure corresponding to the target node.

In some embodiments of the disclosure, based on the foregoing solution, the metadata thread processing module further includes: and a serialization unit.

Wherein the serialization unit is configured to: and before the sending unit sends the log object to the linked list tail of the sub-linked list structure corresponding to the target node, carrying out serialization processing on the log object.

In some embodiments of the disclosure, based on the foregoing solution, the log thread processing module includes: a first logging sub-module and a second logging sub-module.

Wherein the first log sub-module is configured to: in the log writing cache stage, the obtained log record is stored in a log writing cache area through the first log service thread; the second log sub-module is configured to: and in the refreshing stage of the log writing buffer area, refreshing the data stored in the log writing buffer area to an object storage device through the second log service thread.

In some embodiments of the present disclosure, based on the foregoing scheme, the log write cache comprises at least two caches; wherein, the first log sub-module includes: an acquisition unit and a commit unit.

Wherein, in the first log service thread, the acquiring unit is configured to: in the first log service thread, a first log record is obtained from the log processing linked list; the above-mentioned acquisition unit is further configured to: acquiring an unfilled storage area from the log writing cache area as a first target cache area; the submitting unit is configured to: adding the first log record to the first target cache region; and the acquisition unit is further configured to acquire and process a second log record from the log processing linked list.

In some embodiments of the present disclosure, based on the foregoing scheme, the above-described acquisition unit is specifically configured to: and acquiring a log object from the log processing linked list, analyzing the log object and determining the first log record.

In some embodiments of the disclosure, based on the foregoing scheme, the second log submodule is configured to: in the second log service thread, monitoring whether the log writing buffer area has a full buffer area or not; and in response to monitoring a second target buffer area which is full, refreshing the log stored in the second target buffer area to the OSD.

According to an aspect of the present disclosure, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of processing metadata as described in the first aspect above.

According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of processing metadata as described in the first aspect above via execution of the executable instructions.

As can be seen from the above technical solutions, the metadata processing method, the metadata processing apparatus, the computer storage medium, and the electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

in the technical solutions provided in some embodiments of the present disclosure, there is no association between the log processing portion and the processing of metadata in the process of processing metadata through the WAL operation mode. Therefore, the technical scheme responds to the operation request of recording the metadata operation in the WAL mode, and determines the metadata service thread and the log service thread. Meanwhile, the data processing flow corresponding to the operation request is subjected to granularity subdivision to obtain a plurality of stages, so that metadata is processed in stages through a plurality of service threads.

On the one hand, compared with the prior art scheme based on single-thread metadata processing, the technical scheme provided by the disclosure processes the data processing flow corresponding to the operation request through multi-thread cooperation, and each thread processes the corresponding phasing by pipeline. Therefore, the period of processing the metadata according to the operation request of the pre-writing log can be shortened, and the data processing efficiency of the metadata server request can be improved.

On the other hand, the metadata processing flow corresponding to the operation request in the pre-written log mode is subjected to granularity subdivision, so that a log processing stage and a metadata processing stage are separated and are respectively processed by different service threads. Therefore, the metadata access concurrency can be improved, and the overall performance of the distributed file storage system can be improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort. In the drawings:

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the technical solution of embodiments of the present disclosure may be applied;

FIG. 2 is a flow chart illustrating a method of processing metadata in an exemplary embodiment of the present disclosure;

FIG. 3 shows a flow diagram of a method of processing metadata according to another embodiment of the present disclosure;

FIG. 4 illustrates a flow diagram of a method of multithreading pipelined processing of metadata in accordance with one embodiment of the present disclosure;

FIG. 5 illustrates a flow diagram of a method of metadata request phase according to an embodiment of the present disclosure;

FIG. 6 illustrates a schematic diagram of a log processing linked list according to an embodiment of the present disclosure;

FIG. 7 illustrates a flow diagram of a method of multithreading pipelined processing of metadata in accordance with another embodiment of the present disclosure;

FIG. 8 illustrates a flow diagram of a method of a log write caching stage according to an embodiment of the present disclosure;

FIG. 9 is a diagram illustrating a metadata manipulation service flow in a Ceph System according to one embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a log write caching service in a Ceph system according to an embodiment of the disclosure;

FIG. 11 illustrates a format for serializing log records according to an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of a log refresh service flow in a Ceph system according to one embodiment of the disclosure;

FIG. 13 is a schematic diagram of single threaded processing metadata in the prior art;

FIG. 14 illustrates a schematic diagram of multi-threaded processing of metadata in accordance with an embodiment of the present disclosure;

FIG. 15 illustrates a schematic diagram of metadata processing efficiency in an exemplary embodiment in accordance with the present disclosure;

FIG. 16 is a schematic diagram showing the structure of a processing apparatus for metadata in an exemplary embodiment of the present disclosure; the method comprises the steps of,

fig. 17 shows a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the disclosed aspects may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present disclosure may be applied.

As shown in fig. 1, the system architecture 100 may include a client 101, a server 102, an OSD 103, and a network 104.

Wherein, the client 101 may be a terminal device with a display screen, such as a portable computer, a desktop computer, etc.; the server 102 may be a distributed computer cluster, and is configured to manage metadata and process an operation request of the metadata; OSD 103 is used to store a log obtained by processing metadata according to the above operation request; the network 104 is a medium used to provide communication links between the client 101 and the server 102, and between the server 102 and the OSD 103.

The network 104 may include various connection types, such as a wired communication link, a wireless communication link, and the like, and in the embodiments of the present disclosure, the network 104 between the client 101 and the server 102 may be a wired communication link, for example, a communication link may be provided through a serial port connection line, or a wireless communication link, and a communication link may be provided through a wireless network; the network 104 between the server 102 and the OSD103 may be a wireless communication link, in particular a mobile network.

It should be understood that the number of terminals, networks and servers in fig. 1 is merely illustrative. There may be any number of terminals, networks, and servers, as desired for implementation. The server 102 may be an MDS, for controlling interaction between the client 101 and the OSD 103. The OSD103 includes a CPU, a memory, a network, and a disk system. The main functions of OSD include data storage and secure access.

In one embodiment of the present disclosure, a user sends a pre-written log type operation request (i.e., an operation request that needs to record metadata operations in a WAL manner) to a server 102 through a client 101, and after the server 102 receives the pre-written log type operation request, determines a metadata service thread and a log service thread; and processing the metadata according to the operation request through the metadata service thread, acquiring a log record obtained by processing the metadata, and processing the log record through the log service thread to complete log processing corresponding to the operation request. Thereby completing the data processing corresponding to the operation request in stages.

The technical scheme of the embodiment of the disclosure has the following technical effects: on the one hand, compared with the prior art scheme based on single-thread metadata processing, the technical scheme provided by the disclosure processes the data processing flow corresponding to the operation request through multi-thread cooperation, and each thread processes the corresponding phasing by pipeline. Therefore, the period of processing the metadata according to the operation request of the pre-writing log can be shortened, and the data processing efficiency of the metadata server request can be improved. On the other hand, the metadata processing flow corresponding to the operation request in the pre-written log mode is subjected to granularity subdivision, so that a log processing stage and a metadata processing stage are separated and are respectively processed by different service threads. Therefore, the metadata access concurrency can be improved, and the overall performance of the distributed file storage system can be improved.

It should be noted that, the data processing method provided in the embodiment of the present disclosure is generally executed by the server 102, and accordingly, the processing device for metadata is generally disposed in the server 102. However, in other embodiments of the present disclosure, the terminal may also have a similar function as the server, thereby performing the data processing scheme provided by the embodiments of the present disclosure.

The inventors found that in the related art in this field, in response to receiving an operation request of the above type, the metadata processing section and the log processing section are sequentially performed by a single thread. That is, after the log corresponding to the current operation request is processed, the metadata corresponding to the next operation request can be processed. However, there is no association between the log processing corresponding to the current operation request and the metadata processing corresponding to the next operation request. It can be seen that in the related art, the metadata processing period is long, resulting in lower data processing efficiency of the metadata server.

In an embodiment of the present disclosure, a metadata processing method is provided that overcomes, at least to some extent, the drawbacks of the prior art. The execution body of the metadata processing method provided in this embodiment may be a device with a computing processing function, and may be, for example, any metadata server MDS in a distributed file storage system.

Fig. 2 shows a flow diagram of a method of processing metadata in an exemplary embodiment of the present disclosure. Referring to fig. 2, the metadata processing method provided in this embodiment includes:

Step S210, responding to a pre-written log type operation request of metadata, and determining a metadata service thread and a log service thread;

step S220, processing the metadata according to the operation request through the metadata service thread, and obtaining a log record obtained by processing the metadata; the method comprises the steps of,

step S230, processing the log record by the log service thread to complete log processing corresponding to the operation request.

In the technical solution provided in the embodiment shown in fig. 2, there is no association between the log processing portion and the metadata processing during the metadata processing by the WAL operation mode. Therefore, the technical scheme responds to the operation request of recording the metadata operation in the WAL mode, and determines the metadata service thread and the log service thread. Meanwhile, the data processing flow corresponding to the operation request is subjected to granularity subdivision to obtain a plurality of stages, so that metadata is processed in stages through a plurality of service threads. On the one hand, the period of processing the metadata according to the operation request of the pre-writing log can be shortened, and the data processing efficiency of the metadata server request can be improved. On the other hand, the metadata access concurrency can be improved, and the overall performance of the distributed file storage system is improved.

The following describes in detail the implementation of the individual steps of the embodiment shown in fig. 2:

in an exemplary embodiment, in response to the operation request for the pre-written log of metadata, a metadata service thread and a log service thread are determined in step S210.

The log service thread may be one service thread or a plurality of service threads. The metadata service thread may be one service thread or a plurality of service threads. Illustratively, the data processing flow corresponding to the operation request is subjected to granularity subdivision, and a plurality of stages are determined, so that the thread number of the metadata service thread/log service thread can be adapted to the stage number.

In the technical scheme provided by the embodiment, the MDS performs granularity subdivision on the data processing flow corresponding to the operation request, so as to obtain three stages, namely a metadata request stage, a log writing cache stage and a log writing cache region refreshing stage. Then adaptively, in response to the operation request of the pre-written log type for metadata, a metadata service thread, a first log service thread, and a second log service thread are determined in step S210. That is, in this embodiment, the metadata service thread is one thread, and the log service thread is two service threads.

Fig. 3 is a flow chart illustrating a metadata processing method according to another embodiment of the present disclosure. Referring to fig. 3, the metadata processing method provided in this embodiment includes step S310 and step S320.

In step S310, determining a metadata service thread for processing the metadata request phase; determining a first journaling service thread for processing the journaling write cache phase; and determining a second log service thread for processing the log write cache flush phase.

In an exemplary embodiment, as a specific implementation manner of step S210, service threads for processing the three phases respectively are determined in step S310, and the operation request is processed with multithreading coordination. Therefore, the period of processing the metadata according to the operation request of the pre-writing log type is shortened, and the data processing efficiency of the metadata server is improved.

In step S320, in the metadata request stage, processing, by the metadata service thread, the metadata according to the operation request, and obtaining a log record obtained by processing the metadata; in the log writing cache stage, the obtained log record is stored in a log writing cache area through the first log service thread; and refreshing the data stored in the log write buffer to an OSD through the second log service thread in the log write buffer refreshing stage.

In an exemplary embodiment, as a specific implementation manner of step S220 and step S230, in step S320, the metadata service thread, the first log service thread, and the second log service thread may respectively pipeline the metadata request phase, the log write cache phase, and the log write cache flush phase.

In the technical solution provided in the embodiment shown in fig. 3, the metadata service thread, the first log service thread and the second log service thread can respectively process the corresponding stages in a pipelined manner, so that the period of processing metadata according to the operation request of pre-writing the log is further shortened, the concurrency of metadata access is improved, and the overall performance of the distributed file storage system is improved.

In an exemplary embodiment, FIG. 4 shows a flow diagram of a method of multithreading pipelined processing of metadata in accordance with an embodiment of the present disclosure.

Referring to fig. 4, an aspect specifically illustrates pipelining the metadata request phase based on metadata service threads, including step S410 and step S420. In step S410, the metadata is processed according to the current operation request 40, and the generated log record is submitted to the log processing linked list 41 (specifically stored at node S _i In (3) a semiconductor device; in step S420, the next operation request is processed based on the metadata service thread (in turn, the target log generated by this process is specifically stored in the node S _i+1 In (c) a). The metadata operation corresponding to the next operation request includes: metadata operations through pre-written log records.

On the other hand, fig. 4 specifically shows that, based on the first log service thread, the log write cache stage is pipelined, including step S430 and step S440. For the log record generated by the current operation request, in step S430, the first log service thread is used to process the node S of the linked list 41 from the log _i The target log is obtained and added to the log write buffer 42. In step S440, the next target log is processed, illustratively, at node S of log processing chain 41 _i+1 Is provided.

Compared with the prior art, the method provided by the embodiment does not need to wait for the log write cache flow after the metadata processing flow is finished, but directly processes the next operation request. Therefore, the technical scheme provided by the embodiment can effectively shorten the metadata processing period, and is beneficial to accelerating the metadata processing efficiency.

In an exemplary embodiment, FIG. 5 shows a flow diagram of a method of metadata request phase according to an embodiment of the present disclosure. Referring to fig. 5, the method provided by this embodiment includes the steps of:

step S510, in the metadata service thread, searching target metadata in a metadata cache according to a current operation request, and obtaining a log record obtained by searching the target metadata;

step S520, performing COW on the target metadata to obtain copy metadata, and obtaining a log record obtained by copying the target metadata during writing;

step S530, modifying the copy metadata according to the current operation request, and obtaining a log record obtained by modifying the copy metadata;

step S540, the log record is sent to a log processing linked list for processing of the first log service thread; the method comprises the steps of,

step S550, waiting to receive and process a next operation request, where the next operation request is a pre-written log type operation request.

In an exemplary embodiment, the metadata service thread completes the processing of the metadata request phase of the current operation request through steps S510-S540. Since the generated log record is to be further processed by the first log service thread and the metadata service thread does not need to wait for further processing of the log record, the metadata service thread may perform step S550, waiting for receiving and processing the next operation request to pipeline the metadata request phase.

In an exemplary embodiment, since the metadata service thread and the first log service thread are respectively adopted to respectively pipeline the corresponding stages, in order to make the log records submitted by the metadata service thread accurately acquired by the first log service thread, the log records corresponding to different operation requests can be stored through a linked list structure. Referring to FIG. 4, log records generated corresponding to different operation requests are stored in nodes S of the log processing chain table 41 _i-1 Node S _i Node S _i+1 Is a kind of medium.

Illustratively, the different operation requests described above include: creating a catalog "mkdir", deleting a catalog "rmdir", creating a new "create", deleting "delete", and opening "open".

In an exemplary embodiment, a producer consumer model is employed to manage a log processing linked list. The producer consumer mode refers to that the producer continuously produces data and sends the data to the cache, and the consumer continuously fetches the data from the cache for processing.

By way of example, FIG. 6 illustrates a schematic diagram of a log processing linked list in accordance with an embodiment of the present disclosure.

Referring to fig. 6, the nodes of the log processing linked list are composed of a plurality of log segments 61 (log 0, log 1, log 2, …). Wherein each log segment 61 is the same size for adaptation to the underlying storage object. For example, to satisfy the object size of RADOS bottom store, the log segment size is set to 4MB.

Illustratively, each log segment 61log contains at least one log record 60. In the technical solution provided in this embodiment, the log segment segments are used as logical units to determine a log processing linked list, and further, each log segment is allocated with a sub-linked list (i.e., a log processing linked list sub-linked list including at least one log record).

For example, when the metadata service thread generates the log records, the log records 60 must be sequentially added to the linked list of the corresponding log segments 61 (i.e. the sub-linked list of the log processing link) because of the write-order relationship between the log records 60, so that the log segments 61 are sequentially generated, for example, the log segment1 must be generated after the log segment0 is fully stored.

In an exemplary embodiment, the specific implementation manner of sending the log record to the log processing linked list provided in step S540 may be: and determining a target node for storing the log record in the log processing linked list, and sending the log record to a linked list tail of a sub-linked list structure corresponding to the target node.

In an exemplary embodiment, after the target log is submitted to the log processing linked list through the metadata service thread, the log records are sequentially fetched from the log processing linked list through the first log service thread, and added to the log write buffer in an additional form.

Illustratively, FIG. 7 shows a flow diagram of a method of multithreading pipelined processing of metadata in accordance with another embodiment of the present disclosure. And in particular to the description of the specific implementation of the first log service thread based on the embodiment shown in fig. 4. Referring to fig. 7, the log write buffer 42 includes at least two write buffers, and in this embodiment, the log write buffer 42 includes a first buffer 421 and a second buffer 422. Wherein, the buffer occupancy is shaded, the first buffer 421 is an unfinished buffer, and the second buffer 422 is a full buffer.

Compared with the prior art, when the write buffer is full, the log in the write buffer needs to be submitted to the OSD before the log is submitted to the write buffer, which results in an excessively long data processing period. Meanwhile, the log accumulation in the log processing linked list is caused, and the system performance is affected.

In the technical scheme provided by the embodiment, at least two write buffer areas are arranged for alternately receiving log data. When one of the write buffers is full, the log is committed to the other, less full write buffer. Therefore, the technical scheme provided by the embodiment not only can further shorten the metadata processing period and improve the metadata processing efficiency, but also can avoid influencing the system performance due to log accumulation in the log processing linked list.

In an exemplary embodiment, the specific implementation of step S540 may be: and in the first log service thread, packaging the log record and the callback function of the log record into a log object, and sending the log object to a linked list tail of a sub-linked list structure corresponding to the target node. And in the second log service thread, acquiring a first log object from the log processing linked list, analyzing the first log object and determining a first log record.

FIG. 8 illustrates a flow diagram of a method of a log write caching stage according to an embodiment of the present disclosure. Referring to fig. 8, the method provided by this embodiment includes: step S810-step S830.

In step S810, in the first log service thread, a first log record is obtained from the log processing linked list; in step S820, an unfilled storage area is obtained from the log write cache area as a first target cache area; in step S830, the first log record is added to the first target cache area; and, in step S840, a second log record is obtained and processed from the log processing linked list.

In an exemplary embodiment, steps S810-S840 are explained in connection with fig. 7 as follows: by the first log service thread, an unfilled first buffer 421 is determined as a first target buffer in the log write buffer 42. Further, by the first log service thread, the first log record is obtained from the log processing linked list 41, and submitted to the first target cache area (i.e. the first cache area 421).

In an exemplary embodiment, before step S820, the present solution further includes serializing the log record, so that the data members of the log record class object are converted into a buffer list, thereby facilitating the transmission of the network.

In an exemplary embodiment, before step S820, the present solution further includes locating a write location of the log write cache. Specifically, the position of the positioning writing can be recorded and stored at the position after the last time of writing and caching, so that the processing accuracy of the log data is ensured.

In an exemplary embodiment, the first journaling service thread completes submitting the first journaling record to the journaling write cache through steps S810-S820. Since the log record is flushed to OSD after the log write buffer is full and the first log service thread does not need to wait for the processing of the second log service thread, the first log service thread may execute step S840 to process the log write buffer stage in a pipeline manner.

In an exemplary embodiment, referring to fig. 7, after the log write buffer is full, the present solution further includes submitting, by the second log service thread, the log record in the second buffer 422 to the OSD.

By way of example, through the second log service thread, a full buffer area is obtained from the log write buffer area as a second target buffer area, and the log stored in the second target buffer area is refreshed to the object storage device OSD.

Illustratively, in order to adapt to the object size of the RADOS bottom layer storage, when the data in each buffer area in the log write buffer area in the Ceph system reaches 4MB, the log needs to be refreshed on the OSD. In an exemplary embodiment, it may be determined whether the write buffer is full by appending two parameters, the write buffer location after the write is completed and the location after the last refresh. For example, the write cache location is write_pos, the location after the last refresh is flush_pos, the object size in RADOS is object_size, write_obj_id indicates the id number of the object where the cache is located, and flush_obj_id indicates the id number of the last refreshed object. Specific:

the write_obj_id=write_pos/object_size, flush_obj_id=flush_pos/object_size, and whether the write buffer is full or not can be known by judging whether the write_obj_id and flush_obj_id are equal, so as to determine whether the log record of the write buffer needs to be refreshed to the OSD through the second log service thread.

In the following exemplary embodiments, a method for processing metadata in a Ceph system will be specifically described as an example.

Based on the foregoing embodiment, in the present technical solution, the granularity of the operation request operation flow is subdivided into a plurality of stages, and each stage is pipelined by different threads, so as to reduce the data processing period, and further improve the MDS processing efficiency. Specifically, the operation flow of MDS on metadata in the existing Ceph system is divided into three stages: a metadata processing stage (log-free processing), a log writing to a cache stage, and a cache full log flushing to an OSD stage. The following describes three embodiments of the present invention.

It should be noted that, the operation request according to the following embodiments is a pre-written log type operation request for metadata, that is, an operation request for recording metadata operations by WAL.

In the MDS, metadata processing stages are pipelined through a metadata service thread:

fig. 9 shows a metadata operation service flow diagram in a Ceph system according to an embodiment of the present disclosure. Referring to fig. 9, the method includes steps S910 to S950.

In step S910, a current operation request is received, and a metadata service thread and a log service thread are determined;

In step S920, the metadata is processed according to the operation request by the metadata service thread, and a log record LogEvent obtained by processing the metadata is obtained;

in step S930, encapsulating the log record LogEvent to obtain a penningevent object;

in step S940, a node identification Logsegment corresponding to the current operation request is determined in the log processing linked list _i And inserts the PendingEvent object into the node Logsegment _i A linked list tail of the corresponding sub linked list; the method comprises the steps of,

in step S950, the next operation request is waited for.

In an exemplary embodiment, the log service thread determined in step S910 is specifically: the commit service thread sub_thread and the log flush service thread flush_thread will be described in detail in the embodiments shown in fig. 10 to 12.

In an exemplary embodiment, in step S930, the function class PendingEvent may be used to encapsulate the produced log record into a PendingEvent object, so as to send the encapsulated log record to the log processing linked list.

Wherein, the detailed data structure of the above-mentioned PendingEvent-like is as follows in Table 1:

TABLE 1

Member name	Member type	Member meaning
			le	LogEvent*	Log recording
fin	Context*	Callback function after log record is written to OSD
			flush	Bool	Marking this log as to whether or not to write to OSD

Illustratively, the encapsulation process is: callback functions Context (which are used to convert clean metadata into dirty labels) that need to be performed after writing the log record, on OSD are combined into pair pairs (i.e., pendingEvent objects). Wherein each pair is a number of a segment (segment _i ) And a log record linked list. Further, the PendingEvent object is committed to the log processing linked list xio_events. Wherein, the log processing linked list xio_events is managed by a map container in a standard template library (Standard Template Library, STL for short).

And after completing the metadata operation corresponding to the current operation request through step S910-step S940, submitting the log record to the log processing chain, and indicating that the metadata operation corresponding to the current operation request is completed. Further in step S950, the next operation request is waited for. Through a pipelining processing mode, the metadata processing period can be shortened, and the data processing efficiency of MDS is improved.

In the MDS, a log write cache stage is pipelined through a first log service thread:

In an exemplary embodiment, the producer consumer model is employed to manage the log processing linked list described above. Specifically, the log processing linked list in the producer consumer mode is realized by adopting the function class LogXio.

The detailed data structure of the above functional log xio is shown in table 2 below:

TABLE 2

Illustratively, after log events are encapsulated as objects and deposited into log processing linked list xio_events, the consumer function may be performed by submitting service thread submit_thread (i.e., the first log service thread described in the above embodiment) to continuously fetch encapsulated PendingEvent objects from log processing linked list xio_events.

The cross service thread sub_thread determines whether a log record is stored in the log processing linked list xio_events according to a condition variable sub_cond.

Fig. 10 shows a schematic diagram of a log write caching service in a Ceph system according to an embodiment of the disclosure. Referring to fig. 10, the method includes steps S1010 to S1052.

In step S1010, the Submit service thread_thread is turned on;

in step S1020, detecting whether the penodingevent object arrives in the log processing linked list xio_events through the condition variable subset_cond;

In step S1030, the log record LogEvent is parsed according to the PengdingEvent object;

in step S1040, serializing the obtained log record log event;

in step S1050, judging whether the first write buffer area is full or not through flag [0] identification;

if flag [0] is not equal to 1, indicating that the first write buffer is not full, step S1051 is performed: writing the log record LogEvent into a first cache region write_buf [0];

if flag [0] is equal to 1, indicating that the first write buffer is full, step S1052 is executed: the log record LogEvent is written to the second buffer write_buf [1].

In an exemplary embodiment, after it is determined in step S1020 that the penodingevent object arrives in the log processing linked list xio_events, log records are sequentially fetched from the log processing linked list xio_events in the order of log record generation, and then steps S1030-S1050 are performed to add the log records to the log write cache.

The log write cache includes a first cache region and a second cache region, so as to work alternately through the two cache regions, so as to basically ensure that the MDS log module can write the log uninterruptedly, where the write_pos identifies the offset position of the cache, and the flag identifies whether the two write caches are full, and the full value is 1. Where cache fullness means that the size can mean that the swap inventory reaches 4MB.

In an exemplary embodiment, FIG. 11 illustrates a format for serializing log records according to an embodiment of the present disclosure. Referring to fig. 11, in step S1040, by serializing the log records, the data members of the log record class object may be converted into a buffer list, including the Size of each log record and the log record content stamp, type, metablob, so as to facilitate the network transmission.

In an exemplary embodiment, a TwoWriteBuf-like dual write cache structure is employed to implement the log. Wherein, the detailed data structure of the TwoWriteBuf-like data is shown in the following table 3:

TABLE 3 Table 3

Member name	Member type	Member meaning
			write_buf[2]	Bufferlist	Two 4 MB-sized log write caches
flag[2]	uint64_t	Sign of whether two caches are full
			write_pos[2]	uint64_t	Position in cache after writing log
flush_thread	FlushThread	Cache full flush to OSD service thread

In the MDS, the log refreshing is pipelined to an OSD stage through a second log service thread:

the flushing operation of the log is completed using the log flush service thread flush_thread (and the second log service thread in the above embodiment). Judging whether a first buffer area in the log writing buffer area is full or not through judging a flag [0] mark, and refreshing the log full of the first buffer area/the second buffer area to an OSD through OSDC if the first buffer area/the second buffer area is full.

Fig. 12 shows a schematic diagram of a log refresh service flow in a Ceph system according to an embodiment of the present disclosure. Referring to fig. 12, the method includes steps S1210 to S1270.

In step S1210, a log flush service thread flush_thread is started;

in step S1220, determining whether the first write buffer is full by determining whether the flag [0] flag is equal to 1;

if flag [0] is equal to 1, indicating that the first write buffer is full, then step S1230, step S1250 and step S1270 are performed. Specifically:

in step S1230, the write buffer location write_pos [0] of the first write buffer write_buf [0] is processed;

in step S1250, the flag [0] of the first write buffer write_buf [0] is set to 0, indicating that the first write buffer write_buf [0] is not full;

in step S1270, the log record LogEvent in the first write buffer write_buf [0] is flushed to the OSD.

If flag [0] is not equal to 1, indicating that the first write buffer is not full, then step S1240, step S1260 and step S1270 are performed. Specifically:

in step S1240, the write buffer location write_pos [1] of the second write buffer write_buf [1] is processed;

in step S1260, the flag [1] of the second write buffer area write_buf [1] is set to 0, indicating that the flag [0] of the second write buffer area write_buf [0] is not full;

In step S1270, the log record LogEvent in the second write buffer write_buf [1] is flushed to the OSD.

In the technical solutions provided in the embodiments shown in fig. 10 to fig. 12, the log processing portion corresponding to the operation request of the pre-write log is completed. By separating the log processing section and the metadata processing section, and processing by different service threads, respectively. The metadata access concurrency is improved, and the overall performance of the distributed file storage system Ceph is improved.

Fig. 13 shows a schematic diagram of single-threaded processing metadata in the related art, and fig. 14 shows a schematic diagram of multi-threaded processing metadata in an embodiment according to the present disclosure.

Referring to fig. 13 and 14, fig. 13 is a flow chart of processing an operation request in a conventional Ceph system, which is completed based on a single thread. Specifically, the metadata processing flow (including the first stage, the second stage, and the third stage) in the current operation request (request 1) is ended, and then the next request (request 2) is processed. It can be seen that each request processing cycle is longer, resulting in a lower system performance. Fig. 14 is a schematic diagram of a method according to the present disclosure, in which metadata processing flows corresponding to a pre-written log type operation request are subjected to granularity subdivision to obtain a plurality of processing stages, so that metadata is processed in stages by a plurality of service threads, and stages corresponding to pipelined processing in each thread are processed, so that metadata processing efficiency is effectively improved, and meanwhile, resource utilization of a system is advantageously improved.

Fig. 15 illustrates a schematic diagram of metadata processing efficiency in an exemplary embodiment according to the present disclosure. Referring to fig. 15, the technical scheme is adopted to influence different metadata operations (such as creating a directory "mkdir", deleting a directory "rmdir", creating a new "create", deleting a "delete", and opening "open") in a unit time (such as within 5 minutes). Wherein "origin" represents the number of operation requests per unit time before the present technical solution is adopted, and "process_opt" represents the number of operation requests per unit time after the present technical solution is adopted.

As can be seen from the figure, the processing speed of the first four metadata operations is improved by 4.09%, 16.1%, 9.81% and 13.11%, respectively, because the first four operations modify metadata, and thus related log generation occurs. But the open operation does not need to generate a log because the metadata is not modified, so the optimization strategy has no impact on the open operation.

In an exemplary embodiment, the specific implementation of the data statistics may be: testing is performed by writing a mini-application test program using the interface functions of the file portable operating system interface (Portable Operating System Interface of UNIX, POSIX for short) provided in the libpepfs library, and incrementing a statistics count at the acceptance request of the MDS source code.

The metadata processing method provided by the technical scheme can improve the concurrency of MDS metadata access, thereby improving the overall performance of Cephfs.

Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments are implemented as a computer program executed by a CPU. The above-described functions defined by the above-described methods provided by the present disclosure are performed when the computer program is executed by a CPU. The program may be stored in a computer readable storage medium, which may be a read-only memory, a magnetic disk or an optical disk, etc.

Furthermore, it should be noted that the above-described figures are merely illustrative of the processes involved in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.

The following describes embodiments of an interface adaptation system of the present disclosure that may be used to perform the above-described i.e. adaptation method of the present disclosure.

Fig. 16 illustrates a schematic structural diagram of a processing apparatus of metadata in an exemplary embodiment of the present disclosure. As shown in fig. 16, the metadata processing apparatus 1600 includes: a service thread determination module 1601, a metadata thread processing module 1602, and a log thread processing module 1603.

Wherein, the service thread determining module 1601 is configured to determine a metadata service thread and a log service thread in response to a pre-written log type operation request for metadata; the log thread processing module 1602 is configured to process the metadata according to the operation request by the metadata service thread, and obtain a log record obtained by processing the metadata; and the log thread processing module 1603 is configured to process the log record through the log service thread to complete log processing corresponding to the operation request.

In some embodiments of the present disclosure, based on the foregoing solution, the data processing flow corresponding to the operation request includes: a metadata request stage, a log write cache stage and a log write cache refreshing stage.

The service thread determining module 1601 is specifically configured to: determining a metadata service thread for processing the metadata request phase; determining a first journaling service thread for processing the journaling write cache phase; and determining a second log service thread for processing the log write cache flush phase.

In some embodiments of the present disclosure, based on the foregoing, the metadata thread processing module 1602 is specifically configured to: and in the metadata request stage, processing the metadata according to the operation request by the metadata service thread, and acquiring a log record obtained by processing the metadata.

In some embodiments of the present disclosure, based on the foregoing, the metadata thread processing module 1602 includes: a search unit, a copy-on-write unit, a modification unit, and a receiving unit.

Wherein, in the metadata service thread, the searching unit is configured to: and in the metadata service thread, searching target metadata in a metadata cache according to the current operation request, and acquiring a log record obtained by searching the target metadata.

The copy-on-write unit is configured to: and performing COW on the target metadata to obtain copy metadata, and obtaining log records obtained by copying the target metadata during writing.

The modification unit is configured to: modifying the copy metadata according to the current operation request, and acquiring a log record obtained by modifying the copy metadata; the current operation request is a pre-written log type operation request.

And the receiving unit is configured to: waiting for receiving and processing a next operation request, wherein the next operation request is a pre-written log type operation request.

In some embodiments of the present disclosure, based on the foregoing scheme, the metadata thread processing module 1602 further includes: and a transmitting unit.

In some embodiments of the disclosure, based on the foregoing solution, the log processing linked list includes a plurality of sub-linked list structures, and the sub-linked list structures form nodes of the log processing linked list.

Wherein the transmitting unit is specifically configured to: and determining a target node for storing the log record in the log processing linked list, and sending the log record to a linked list tail of a sub-linked list structure corresponding to the target node.

In some embodiments of the present disclosure, based on the foregoing scheme, the metadata thread processing module 1602 further includes: and a serialization unit.

In some embodiments of the present disclosure, based on the foregoing solution, the log thread processing module 1603 includes: a first logging sub-module and a second logging sub-module.

The specific details of each unit in the above-mentioned metadata processing apparatus are already described in detail in the corresponding metadata processing method, and thus will not be described here again.

Fig. 17 shows a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure.

It should be noted that the computer system 1700 of the electronic device shown in fig. 17 is only an example, and should not impose any limitation on the functions and application scope of the embodiments of the present disclosure.

As shown in fig. 17, the computer system 1700 includes a central processing unit (Central Processing Unit, CPU) 1701, which can execute various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) 1702 or a program loaded from a storage portion 1708 into a random access Memory (Random Access Memory, RAM) 1703. In the RAM 1703, various programs and data required for system operation are also stored. The CPU 1701, ROM 1702, and RAM 1703 are connected to each other through a bus 1704. An Input/Output (I/O) interface 1705 is also connected to the bus 1704.

The following components are connected to the I/O interface 1705: an input section 1706 including a keyboard, a mouse, and the like; an output portion 1707 including a Cathode Ray Tube (CRT), a liquid crystal display (Liquid Crystal Display, LCD), and a speaker, etc.; a storage portion 1708 including a hard disk or the like; and a communication section 1709 including a network interface card such as a LAN (Local Area Network ) card, a modem, or the like. The communication section 1709 performs communication processing via a network such as the internet. The driver 1710 is also connected to the I/O interface 1705 as needed. A removable medium 1711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 1710, so that a computer program read therefrom is installed into the storage portion 1708 as needed.

In particular, according to embodiments of the present disclosure, the processes described below with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network via the communication portion 1709 and/or installed from the removable medium 811. When executed by a Central Processing Unit (CPU) 1701, performs various functions defined in the system of the present application.

It should be noted that, the computer readable medium shown in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware, and the described units may also be provided in a processor. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.

As another aspect, the present application also provides a computer-readable medium that may be contained in the electronic device described in the above embodiment; or may exist alone without being incorporated into the electronic device. The computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the methods described in the above embodiments.

For example, the electronic device may implement the method as shown in fig. 2: step S210, responding to a pre-written log type operation request of metadata, and determining a metadata service thread and a log service thread; step S220, processing the metadata according to the operation request through the metadata service thread, and obtaining a log record obtained by processing the metadata; and step S230, processing the log record by the log service thread to complete the log processing corresponding to the operation request.

As another example, the electronic device may implement the steps shown in fig. 3 or fig. 5 or any of fig. 8 to fig. 10.

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of processing metadata, the method comprising:

responding to a pre-written log type operation request of metadata, and determining a metadata service thread and a log service thread;

processing the metadata according to the operation request through the metadata service thread, and acquiring a log record obtained by processing the metadata;

processing the log record through the log service thread to finish log processing corresponding to the operation request;

the method further comprises the steps of: determining a target node for storing the log record in a log processing linked list, packaging the log record and a callback function of the log record into a log object, carrying out serialization processing on the log object, and sending the log object to a linked list tail of a sub-linked list structure corresponding to the target node for processing by a first log service thread; the first log service thread is a service thread for processing a log write cache stage, and the log processing linked list comprises a plurality of sub-linked list structures, and the sub-linked list structures form nodes of the log processing linked list.

2. The method for processing metadata according to claim 1, wherein the data processing procedure corresponding to the operation request includes: a metadata request stage, a log write cache stage and a log write cache region refreshing stage;

wherein the determining the metadata service thread and the log service thread comprises:

determining a metadata service thread for processing the metadata request phase;

determining a first journaling service thread for processing the journaling write cache phase;

and determining a second log service thread for processing the log write cache flush stage.

3. The method for processing metadata according to claim 2, wherein said processing the metadata according to the operation request by the metadata service thread and obtaining a log record obtained by processing the metadata comprises:

and in the metadata request stage, processing the metadata according to the operation request by the metadata service thread, and acquiring a log record obtained by processing the metadata.

4. A method for processing metadata according to claim 2 or 3, wherein said processing, by said metadata service thread, said metadata according to said operation request and obtaining a log record obtained by processing said metadata, comprises:

Searching target metadata in a metadata cache according to a current operation request in the metadata service thread, and acquiring a log record obtained by searching the target metadata;

copying the target metadata when writing to obtain copy metadata, and obtaining a log record obtained by copying the target metadata when writing;

modifying the copy metadata according to the current operation request, and acquiring a log record obtained by modifying the copy metadata;

waiting for receiving and processing a next operation request, wherein the next operation request is a pre-written log type operation request.

5. The method of processing metadata according to claim 1, wherein the processing the log record by the log service thread includes:

in the log writing cache stage, the obtained log record is stored in a log writing cache area through the first log service thread;

and in the refreshing stage of the log writing buffer area, refreshing the data stored in the log writing buffer area to the object storage equipment through a second log service thread, wherein the second log service thread is used for processing the refreshing stage of the log writing buffer area.

6. The method according to claim 5, wherein the log writing buffer comprises at least two buffers;

the storing, by the first log service thread, the obtained log record in the log write cache area in the log write cache stage includes:

in the first log service thread, a first log record is obtained from the log processing linked list;

acquiring an unfilled storage area from the log writing cache area as a first target cache area;

adding the first log record to the first target cache region;

and acquiring and processing a second log record from the log processing linked list.

7. The method of claim 6, wherein the obtaining the first log record from the log processing linked list includes:

and acquiring a log object from the log processing linked list, analyzing the log object and determining the first log record.

8. The method according to claim 6 or 7, wherein in the step of flushing the log write buffer, the data stored in the log write buffer is flushed to the object storage device by the second log service thread, comprising:

In the second log service thread, monitoring whether the log writing buffer area has a full buffer area or not;

and in response to monitoring a second target cache region which is full, refreshing the log stored in the second target cache region to the object storage device.

9. A device for processing metadata, the device comprising:

a service thread determination module configured to determine a metadata service thread and a log service thread in response to a pre-written log operation request for metadata;

the log thread processing module is configured to process the metadata according to the operation request through the metadata service thread and acquire log records obtained by processing the metadata;

the log thread processing module is configured to process the log record through the log service thread so as to complete log processing corresponding to the operation request;

a transmission unit configured to: determining a target node for storing the log record in a log processing linked list, packaging the log record and a callback function of the log record into a log object, carrying out serialization processing on the log object, and sending the log object to a linked list tail of a sub-linked list structure corresponding to the target node for processing by a first log service thread; the first log service thread is a service thread for processing a log write cache stage, and the log processing linked list comprises a plurality of sub-linked list structures, and the sub-linked list structures form nodes of the log processing linked list.

10. A computer storage medium, characterized in that it has stored thereon a computer program which, when executed by a processor, implements a method of processing metadata according to any of claims 1 to 8.

11. An electronic device, the electronic device comprising:

one or more processors;

storage means for storing one or more programs which when executed by the one or more processors cause the one or more processors to implement the method of processing metadata as claimed in any one of claims 1 to 8.