CN114528267A - Method, apparatus, gateway, medium, and product for global storage data consistency - Google Patents

Method, apparatus, gateway, medium, and product for global storage data consistency Download PDF

Info

Publication number
CN114528267A
CN114528267A CN202210155165.0A CN202210155165A CN114528267A CN 114528267 A CN114528267 A CN 114528267A CN 202210155165 A CN202210155165 A CN 202210155165A CN 114528267 A CN114528267 A CN 114528267A
Authority
CN
China
Prior art keywords
metadata
preset
operation logs
logs
global
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210155165.0A
Other languages
Chinese (zh)
Inventor
鲁莎莎
钟丽波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Eisoo Information Technology Co Ltd
Original Assignee
Shanghai Eisoo Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Eisoo Information Technology Co Ltd filed Critical Shanghai Eisoo Information Technology Co Ltd
Priority to CN202210155165.0A priority Critical patent/CN114528267A/en
Publication of CN114528267A publication Critical patent/CN114528267A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method, a device, a gateway, a medium and a product for consistency of global storage data. The method comprises the following steps: acquiring a preset number of operation logs, wherein the operation logs correspond to metadata operations; adding file creation operation logs included in the operation logs in a preset number into a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue; and synchronizing the metadata corresponding to other operation logs except the file creation operation log in the preset number of operation logs to a global memory according to the sequence. According to the method, the efficiency of updating the cache metadata to the global memory can be improved by adding the file creation operation logs into the preset queue.

Description

Method, apparatus, gateway, medium, and product for global storage data consistency
Technical Field
The embodiment of the invention relates to the technical field of gateway data storage, in particular to a method, a device, a gateway, a medium and a product for global storage data consistency.
Background
With the development of information technology, unstructured data of enterprises grows sharply. With the increase of data, the back-end storage pressure is increased, and the performance of the unstructured storage device under mass data is greatly reduced. For this reason, cache needs to be provided through the local storage gateway to speed up the storage access performance, so the data consistency of updating the cache data to the global storage has to be guaranteed.
In order to ensure the data consistency of the cache data updated to the global storage, the prior art generally records a file operation log by a storage gateway, and then sequentially updates the metadata operation in the cache to the global storage according to the file operation log.
However, the prior art is slow to update metadata operations in a cache to global storage.
Disclosure of Invention
The invention provides a method, a device, a gateway, a medium and a product for consistency of global storage data, which aim to solve the problem that the speed of updating metadata operation in a storage gateway cache to global storage is slow in the prior art.
According to an aspect of the present invention, there is provided a method for global storage data consistency, including:
acquiring a preset number of operation logs, wherein the operation logs correspond to metadata operations;
adding file creation operation logs included in the operation logs in a preset number into a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue;
and synchronizing the metadata corresponding to other operation logs in the preset number of operation logs except the file creation operation log to a global memory according to the sequence.
According to another aspect of the present invention, there is provided an apparatus for global storage data consistency, comprising:
the acquisition module is used for acquiring a preset number of operation logs, and the operation logs correspond to metadata operations;
the first synchronization module is used for adding the file creation operation logs included in the preset number of operation logs into a preset queue and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue;
and the second synchronization module is used for synchronizing the metadata corresponding to other operation logs except the file creation operation log in the preset number of operation logs to a global memory according to the sequence.
According to another aspect of the present invention, there is provided a storage gateway, comprising: at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform a method of global storage data coherency as described in any of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to perform a method for global storage data consistency according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the file creation operation log is added into the preset queue, so that the problem that the speed of updating the metadata operation in the storage gateway cache to the global storage is low is solved, and the beneficial effect of improving the efficiency of updating the cache metadata to the global storage is achieved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for global storage data consistency according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a method for global storage data consistency according to a second embodiment of the present invention;
FIG. 3 is a flowchart illustrating a method for global storage data consistency according to an exemplary embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a flow of a metadata operation corresponding to a synchronization operation log to a global memory according to an exemplary embodiment of the present invention;
fig. 5 is a schematic structural diagram of an apparatus for global storage data consistency according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of a storage gateway of a method for global storage data consistency according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. It should be understood that the various steps recited in the method embodiments of the present invention may be performed in a different order and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the invention is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It is noted that references to "a", "an", and "the" modifications in the present invention are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that reference to "one or more" unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present invention are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Example one
Fig. 1 is a flowchart of a method for global storage data consistency according to an embodiment of the present invention, where the method is applicable to a case where cache metadata in a storage gateway is updated to a global storage for global storage, and the method may be executed by a device for global storage data consistency, where the device may be implemented by software and/or hardware and is generally integrated on the storage gateway, and in this embodiment, the storage gateway includes, but is not limited to, a NAS gateway.
As shown in fig. 1, a method for global storage data consistency according to an embodiment of the present invention includes the following steps:
and S110, acquiring a preset number of operation logs.
Wherein the operation log corresponds to metadata operations. The number of the acquired operation logs can be set by self, and the preset number can be any integer larger than 1.
In this embodiment, the manner of obtaining the operation logs is not particularly limited, and the multiple operation logs may be obtained in any feasible manner, for example, one feasible manner is to pull the multiple operation logs from the metadata cache.
Wherein, the metadata cache can be persisted to various NoSQL databases, such as MongoDB, RocksDB, Redis, etc. Wherein, the persistence of the operation log can adopt a message queue, a database or other components capable of guaranteeing the sequence.
S120, adding the file creation operation logs included in the preset number of operation logs into a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue.
The preset number of operation logs may include multiple operation logs. Because the metadata cache is of a tree structure, the operation logs with small influence on the tree structure can be added into the preset queue, and the file creation operation log can be used as the operation log with small influence on the tree structure.
In this embodiment, before synchronizing the metadata operations corresponding to a preset number of operation logs to the global memory, a queue may be created in advance, the operation logs having a smaller influence on the tree structure are added to the queue, the operation logs having a smaller influence on the tree structure are obtained from the queue, and the metadata operations corresponding to the operation logs having a smaller influence on the tree structure are synchronized to the global memory according to the obtaining sequence of the logs.
The global memory can perform global storage on the cache metadata, and the global memory can be any one of a storage device, an unstructured data management system and cloud storage.
Further, synchronizing the metadata operation corresponding to the file creation operation log to a global memory from the preset queue includes: and synchronizing a plurality of metadata operations corresponding to a plurality of file creation operation logs in the preset queue to a global memory simultaneously in a concurrent manner through a plurality of working threads in a thread pool.
In this embodiment, a plurality of metadata operations corresponding to a plurality of file creation operation logs may be concurrently synchronized to the global memory for global storage by creating the thread pool.
In an embodiment, the number of the working threads in the thread pool may be dynamically configured, all the working threads may obtain the file creation operation log from the preset queue, and perform corresponding data updating operations according to the file creation operation log, and all the working threads in the thread pool may concurrently update the metadata operations corresponding to the plurality of file creation operation logs to the global memory. When the file creation operation log is acquired from the preset queue, the preset queue can be locked to prevent repeated file creation operation logs from being acquired.
In this embodiment, synchronizing the metadata corresponding to the file creation operation log to the global memory may include: updating the metadata cache in the global memory according to the complete cache path corresponding to the metadata operation meeting the preset condition and the metadata operation; and adding an operation log corresponding to the metadata operation which does not accord with the preset condition into a preset data structure.
Wherein the preset conditions include: nodes of the metadata operation do not include nodes existing in a preset data structure, and a parent path of the metadata operation does not exist in the preset data structure. Illustratively, when the nodes of the metadata operation include A, B, C three nodes, if one of A, B, C three nodes exists in the preset data structure, it is characterized that the metadata operation does not satisfy the preset condition, and an operation log corresponding to the metadata operation needs to be added to the preset data structure; if A, B, C none of the three nodes exist in the preset data structure, continuing to find a parent path upwards in the metadata cache, if none of the parent paths of the metadata operation exist in the preset data structure, representing that the metadata operation meets the preset condition, splicing a complete cache path, and updating the metadata in the global memory through the spliced complete cache path and the metadata operation.
S130, synchronizing the metadata corresponding to other operation logs in the preset number of operation logs except the file creation operation log to a global memory according to the sequence.
The metadata operations corresponding to the other operation logs may include operations that have a large influence on the tree structure, such as directory Create, Rename, and Remove.
In this embodiment, the metadata operations corresponding to other operation logs may be synchronized to the global memory according to the arrangement order of the operation logs.
It can be understood that the process of synchronizing the metadata operations corresponding to the other operation logs to the global memory is the same as the process of synchronizing the metadata operations corresponding to the file creation operation logs to the global memory in step S120, and specifically may include the following processes: judging whether nodes of the metadata operation include nodes existing in a preset data structure or not, and if so, adding an operation log corresponding to the metadata operation into the preset data structure; if not, finding a father path upwards in the metadata cache, judging whether the father path exists in a preset data structure, and if so, adding an operation log corresponding to the metadata operation into the preset data structure; if not, determining whether a complete cache path can be spliced or not, and if the complete cache path can be spliced, updating the metadata in the global memory through the spliced complete cache path and the metadata operation; and if the complete cache path cannot be spliced, continuously returning to search the parent path upwards in the metadata cache until the complete cache path is spliced.
It should be noted that the execution of S120 and S130 does not distinguish the order, and may also be executed simultaneously.
The method for the consistency of the global storage data, provided by the embodiment of the invention, comprises the steps of firstly obtaining a preset number of operation logs; then adding file creation operation logs included in the operation logs of the preset number into a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue; and finally, synchronizing the metadata corresponding to other operation logs in the preset number of operation logs except the file creation operation log to a global memory according to the sequence. According to the method, the efficiency of updating the cache metadata to the global memory can be improved by adding the file creation operation logs to the preset queue.
Example two
Fig. 2 is a schematic flowchart of a method for global storage data consistency according to a second embodiment of the present invention, where the second embodiment is optimized based on the foregoing embodiments. In this embodiment, a preset data structure is further introduced. Please refer to the first embodiment for a detailed description of the present embodiment.
As shown in fig. 2, a method for global storage data consistency according to a second embodiment of the present invention includes the following steps:
s210, obtaining a preset number of operation logs.
S220, adding the file creation operation logs included in the operation logs in a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue.
And S230, synchronizing the metadata corresponding to other operation logs in the preset number of operation logs except the file creation operation log to a global memory according to the sequence.
S240, adding the operation logs which fail to be synchronized in the preset number of operation logs into a preset data structure, synchronizing the metadata operation corresponding to the operation logs which fail to be synchronized in the preset data structure to a global memory after the preset number of operation logs are processed, and deleting the operation logs corresponding to the metadata operation which succeeds in synchronization from the preset data structure.
In one embodiment, if a certain oplog fails for some reason during the processing, the certain oplog may be added to the preset data structure. The preset data structure may be a preset data structure with a search function.
When other operation logs are processed subsequently, if the parent path of the metadata operation corresponding to the operation log is found in the preset data structure, the operation log may be added to the preset data structure.
It should be noted that, adding the preset data structure may make a certain oplog execute failure only to affect other data updates of its subtree, and the bypass data can still be updated to the global memory.
On one hand, in the method for consistency of global storage data provided by the second embodiment of the present invention, the file creation operation logs are added to the preset queue, and metadata operations corresponding to the file creation operation logs can be concurrently executed by introducing the thread pool, so that efficiency of updating cache metadata to a global storage can be improved; on the other hand, by introducing the preset data structure, when the metadata operation corresponding to a certain operation log fails to synchronize, the synchronization-limited sub-tree can be skipped, so that the bypass data is not affected.
The embodiment of the invention provides a specific implementation mode on the basis of the technical scheme of each embodiment.
As a specific implementation manner of this embodiment, fig. 3 is a schematic flowchart of a method for global storage data consistency according to an exemplary embodiment of the present invention, and as shown in fig. 3, the method may include the following steps:
s101, pulling N operation logs at a time.
S102, pulling out the first unprocessed operation log in the N operation logs in sequence.
S103, judging whether the operation log is File Create.
If yes, executing S107; if not, S104 is executed.
And S104, updating the metadata operation corresponding to the operation log into a global memory.
And S105, judging whether the processing of the N taken logs is finished.
If yes, executing S102; if not, go to step S106.
S106, sequentially processing the DelayMap, namely the operation logs in the preset data structure, deleting the operation logs which are successfully processed from the DelayMap, and continuously remaining the operation logs which are still failed to be processed in the DelayMap.
And S107, adding the operation log into a queue.
And S108, taking the operation logs from the queue, setting up another thread pool in the program, dynamically configuring worker numbers in the thread pool, and operating the metadata corresponding to the synchronous operation logs of a plurality of workers to a global memory.
Further, fig. 4 is a schematic flowchart of a process of operating metadata corresponding to a synchronization operation log to a global memory according to an exemplary embodiment of the present invention, where steps of operating the metadata corresponding to the synchronization operation log to the global memory are shown in fig. 4, and the specific steps are as follows:
s201, searching whether the entry influencing the node exists in the DelayMap.
If yes, executing S207; if not, go to S202.
S202, searching up the parent path in the MdCache, namely the metadata cache.
S203, searching whether the entry in the DelayMap has influence on the parent path.
If yes, executing S207; if not, S204 is executed.
And S204, judging whether the paths are spliced completely.
If yes, go to S205; if not, returning to execute S202-S204 again.
And S205, updating the metadata in the global memory through the spliced complete path and the corresponding metadata operation.
S206, judging whether the updating operation is successful.
If the success is achieved, the operation is ended; if the failure occurs, S207 is executed.
And S207, adding the operation log into the DelayMap.
According to the method, the File Create operation is plugged into the thread pool and executed concurrently, and the updating speed of the cache metadata is improved. Through the decomposition of the tree structure, if a certain metadata fails to be updated, all metadata updating operations of the subtree are skipped, so that the updating exception does not block the whole data updating, and the efficiency of updating the cache data into the global storage is improved.
EXAMPLE III
Fig. 5 is a schematic structural diagram of an apparatus for global storage data consistency according to a third embodiment of the present invention, where the apparatus may be adapted to update cache metadata in a storage gateway to a global memory for global storage, where the apparatus may be implemented by software and/or hardware and is generally integrated on the storage gateway.
As shown in fig. 5, the apparatus includes: an acquisition module 110, a first synchronization module 120, and a second synchronization module 130.
An obtaining module 110, configured to obtain a preset number of operation logs, where the operation logs correspond to metadata operations;
a first synchronization module 120, configured to add file creation operation logs included in the preset number of operation logs into a preset queue, and synchronize metadata operations corresponding to the file creation operation logs to a global memory from the preset queue;
a second synchronization module 130, configured to synchronize, according to a sequence, metadata corresponding to each of the other operation logs, to a global memory, the other operation logs, except the file creation operation log, in the preset number of operation logs.
In this embodiment, the apparatus first obtains a preset number of operation logs through the obtaining module 110; then, adding the file creation operation logs included in the preset number of operation logs into a preset queue through a first synchronization module 120, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue; and finally, synchronizing the metadata corresponding to each of the other operation logs in the preset number of operation logs, except for the file creation operation log, to a global memory according to a sequence through a second synchronization module 130.
The embodiment provides a device for data consistency of global storage, which can improve the efficiency of updating cache metadata to a global storage.
Further, the global memory includes one of a storage device, an unstructured data management system, and cloud storage.
Further, the first synchronization module 120 is specifically configured to: and synchronizing a plurality of metadata operations corresponding to a plurality of file creation operation logs in the preset queue to a global memory simultaneously in a concurrent manner through a plurality of working threads in a thread pool.
The device further comprises a processing module, configured to add the operation logs, which fail to be synchronized, in the preset number of operation logs into a preset data structure, synchronize the metadata operations corresponding to the operation logs, which fail to be synchronized, in the preset data structure to a global memory after the preset number of operation logs are processed, and delete the operation logs corresponding to the metadata operations, which succeed in synchronization, from the preset data structure.
Further, synchronizing the metadata operations corresponding to the oplogs to the global memory comprises: if the metadata operation corresponding to the operation log meets the preset condition, determining a complete cache path corresponding to the metadata operation meeting the preset condition; updating a cache in a global memory according to the complete cache path and the metadata operation; wherein the preset conditions include: nodes of the metadata operation do not include nodes existing in a preset data structure, and a parent path of the metadata operation does not exist in the preset data structure.
Further, the device adds an operation log corresponding to the metadata operation which does not meet the preset condition into the preset data structure.
The device for global storage data consistency can execute the method for global storage data consistency provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
FIG. 6 illustrates a schematic structural diagram of a storage gateway 10 that may be used to implement an embodiment of the present invention. As shown in fig. 6, the storage gateway 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the gateway 10 can also be stored. The processor 11, the ROM12, and the RAM 13 are connected to each other by a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in storage gateway 10 are connected to I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
Processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. Processor 11 performs the various methods and processes described above, such as a method of global storage data coherency.
In some embodiments, a method of global storage data consistency may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto storage gateway 10 via ROM12 and/or communication unit 19. When loaded into RAM 13 and executed by processor 11, a computer program may perform one or more of the steps of a method of global storage data consistency described above. Alternatively, in other embodiments, the processor 11 may be configured by any other suitable means (e.g., by way of firmware) to perform a method of global storage data coherency.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method of global storage data consistency, the method comprising:
acquiring a preset number of operation logs, wherein the operation logs correspond to metadata operations;
adding file creation operation logs included in the operation logs in a preset number into a preset queue, and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue;
and synchronizing the metadata corresponding to other operation logs except the file creation operation log in the preset number of operation logs to a global memory according to the sequence.
2. The method of claim 1, wherein the global memory comprises one of a storage device, an unstructured data management system, and cloud storage.
3. The method of claim 1, wherein synchronizing metadata operations corresponding to the file creation operation log from the preset queue to a global memory comprises:
and synchronizing a plurality of metadata operations corresponding to a plurality of file creation operation logs in the preset queue to a global memory simultaneously in a concurrent manner through a plurality of working threads in a thread pool.
4. The method of claim 1, further comprising:
and adding the operation logs which fail to be synchronized in the preset number of operation logs into a preset data structure, synchronizing the metadata operation corresponding to the operation logs which fail to be synchronized in the preset data structure to a global memory after the preset number of operation logs are processed, and deleting the operation logs corresponding to the metadata operation which succeeds in synchronization from the preset data structure.
5. The method of claim 1, wherein synchronizing metadata operations corresponding to oplogs to global memory comprises:
if the metadata operation corresponding to the operation log meets the preset condition, determining a complete cache path corresponding to the metadata operation meeting the preset condition;
updating a cache in a global memory according to the complete cache path and the metadata operation;
wherein the preset conditions include: nodes of the metadata operation do not include nodes existing in a preset data structure, and a parent path of the metadata operation does not exist in the preset data structure.
6. The method of claim 5, further comprising:
and adding an operation log corresponding to the metadata operation which does not accord with the preset condition into the preset data structure.
7. An apparatus for globally storing data consistency, the apparatus comprising:
the acquisition module is used for acquiring a preset number of operation logs, and the operation logs correspond to metadata operations;
the first synchronization module is used for adding the file creation operation logs included in the preset number of operation logs into a preset queue and synchronizing metadata operations corresponding to the file creation operation logs to a global memory from the preset queue;
and the second synchronization module is used for synchronizing the metadata corresponding to other operation logs except the file creation operation log in the preset number of operation logs to a global memory according to the sequence.
8. A storage gateway, wherein the storage gateway comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of global storage data consistency of any of claims 1-6.
9. A computer-readable storage medium having stored thereon computer instructions for causing a processor to execute a method for global storage data consistency according to any one of claims 1-6.
10. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of global storage data consistency according to any one of claims 1-6.
CN202210155165.0A 2022-02-21 2022-02-21 Method, apparatus, gateway, medium, and product for global storage data consistency Pending CN114528267A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210155165.0A CN114528267A (en) 2022-02-21 2022-02-21 Method, apparatus, gateway, medium, and product for global storage data consistency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210155165.0A CN114528267A (en) 2022-02-21 2022-02-21 Method, apparatus, gateway, medium, and product for global storage data consistency

Publications (1)

Publication Number Publication Date
CN114528267A true CN114528267A (en) 2022-05-24

Family

ID=81624994

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210155165.0A Pending CN114528267A (en) 2022-02-21 2022-02-21 Method, apparatus, gateway, medium, and product for global storage data consistency

Country Status (1)

Country Link
CN (1) CN114528267A (en)

Similar Documents

Publication Publication Date Title
CN112925811B (en) Method, apparatus, device, storage medium and program product for data processing
WO2023103432A1 (en) Code file editing method and apparatus, electronic device, and storage medium
CN115454971A (en) Data migration method and device, electronic equipment and storage medium
CN114817223A (en) Service data extraction method and device, electronic equipment and storage medium
CN114528267A (en) Method, apparatus, gateway, medium, and product for global storage data consistency
CN114691781A (en) Data synchronization method, system, device, equipment and medium
CN115480704A (en) Method, device and equipment for constructing data block group migration mapping table and storage medium
CN114968950A (en) Task processing method and device, electronic equipment and medium
CN115168291A (en) Hierarchical directory implementation method and device, electronic equipment and storage medium
CN114417070A (en) Method, device and equipment for converging data authority and storage medium
CN116775171B (en) Architecture switching method and device, electronic equipment and storage medium
CN114780021B (en) Copy repairing method and device, electronic equipment and storage medium
CN117331902A (en) Dynamic path monitoring method, device, equipment and storage medium
CN111258954B (en) Data migration method, device, equipment and storage medium
CN116881368A (en) Data synchronization method and device, electronic equipment and storage medium
CN117370354A (en) Snapshot and query method and device of metadata tree and electronic equipment
CN114416881A (en) Real-time synchronization method, device, equipment and medium for multi-source data
CN115237968A (en) Node management method, device, equipment and storage medium in database system
CN118093048A (en) Multi-module starting management method, device, equipment and medium
CN115408547A (en) Dictionary tree construction method, device, equipment and storage medium
CN118012897A (en) Heterogeneous database grammar conversion method, device, equipment and storage medium
CN117931805A (en) Data processing method and device, electronic equipment and storage medium
CN115687529A (en) Data synchronization method and device, electronic equipment and storage medium
CN115774702A (en) Distributed database implementation method and device, electronic equipment and storage medium
CN115587091A (en) Data storage method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination