CN115639966A

CN115639966A - Data writing method and device, terminal equipment and storage medium

Info

Publication number: CN115639966A
Application number: CN202211369660.8A
Authority: CN
Inventors: 何进东; 王瀚
Original assignee: Shanghai Eisoo Information Technology Co Ltd
Current assignee: Shanghai Eisoo Information Technology Co Ltd
Priority date: 2022-11-03
Filing date: 2022-11-03
Publication date: 2023-01-24

Abstract

The invention discloses a data writing method, a data writing device, terminal equipment and a storage medium. The method comprises the following steps: receiving write data and data write operation information written by a client of a first storage system; storing data contained in the write data into a data storage area; writing the data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area; the pre-written log file is sent to a server of a second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, converts the first metadata into second metadata and writes the second metadata into a second metadata storage area, the problems of long mutual access time and low efficiency among storage systems based on different storage modes are solved, and the access time consumption among the storage systems of different storage modes is short, the resource occupancy rate is low and the access efficiency is high.

Description

Data writing method and device, terminal equipment and storage medium

Technical Field

The present invention relates to the field of data storage technologies, and in particular, to a data writing method and apparatus, a terminal device, and a storage medium.

Background

Data storage can be divided into three types according to different storage interfaces: file storage, block storage, and object storage. Although the file storage and the object storage have similarities, data cannot be directly accessed and migrated to each other due to different protocols used by the two.

For example, a Network File System (NFS) protocol and a Simple Storage Service (S3) protocol are used to access a File Storage System and an object Storage System, respectively. In general, a method of data inter-access between the NFS protocol and the S3 protocol is to convert NFS metadata/S3 metadata into S3/NFS metadata when reading the metadata. With this approach, each data exchange may undergo a metadata transformation calculation, resulting in inefficient data access between the NFS protocol and the S3 protocol.

Disclosure of Invention

The invention provides a data writing method, a data writing device, terminal equipment and a storage medium, which are used for solving the problems of long time, low efficiency and high resource occupancy rate of data exchange among storage systems based on different storage modes, so that the exchange efficiency among the storage systems with different storage modes is improved, and the resource occupancy rate is reduced.

According to an aspect of the present invention, there is provided a data writing method applied to a server of a first storage system, the method including:

receiving write data and data write operation information written by a client of the first storage system;

storing data contained in the write data into a data storage area;

writing the data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area;

and sending the pre-written log file to a server of a second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, and converts the first metadata into second metadata to write the second metadata into a second metadata storage area.

According to another aspect of the present invention, there is provided a data writing method applied to a server of a second storage system, including:

analyzing a received pre-written log file sent by a server of a first storage system to obtain a data operation log;

performing data operation playback on the data operation log to obtain first metadata;

and performing metadata format conversion on the first metadata to obtain second metadata, and writing the second metadata into the second metadata storage area.

According to another aspect of the present invention, there is provided a data writing apparatus integrated in a server of a first storage system, the apparatus including:

the receiving module is used for receiving write-in data and data write-in operation information written by a client of the first storage system;

the data storage module is used for storing the data contained in the written data into a data storage area;

the first metadata writing module is used for writing the data writing operation information into a pre-written log file in a local pre-written log system and writing first metadata corresponding to the data writing operation information into a first metadata storage area;

the log file sending module is used for sending the pre-written log file to a server of a second storage system; and enabling the server side of the second storage system to perform data operation playback according to the pre-written log file, and converting the first metadata into second metadata and writing the second metadata into a second metadata storage area.

According to another aspect of the present invention, there is provided a data writing apparatus integrated in a server of a second storage system, including:

the log file analysis module is used for analyzing the received pre-written log file sent by the server of the first storage system to obtain a data operation log;

the data operation playback module is used for performing data operation playback on the data operation log to obtain first metadata;

and the second metadata writing module is used for performing metadata format conversion on the first metadata to obtain second metadata and writing the second metadata into the second metadata storage area.

According to another aspect of the present invention, there is provided a terminal device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data writing method according to any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data writing method according to any one of the embodiments of the present invention when the computer instructions are executed.

The invention provides a data writing method, a data writing device, terminal equipment and a storage medium, wherein the method comprises the following steps: receiving write data and data write operation information written by a client of a first storage system; storing data contained in the write data into a data storage area; writing the data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area; and sending the pre-written log file to a server of the second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, and converts the first metadata into second metadata to write the second metadata into a second metadata storage area. By completing the conversion and storage of the metadata in the data writing stage, no additional metadata conversion is needed in the data reading stage, the problems of long time and low efficiency of mutual access between storage systems based on different storage modes are solved, and the beneficial effects of short time consumption, low resource occupancy rate and high access efficiency of the access between the storage systems of different storage modes are achieved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1A is a flowchart of a data writing method according to an embodiment of the present invention;

FIG. 1B is a data structure diagram of metadata for a file storage system;

FIG. 1C is a data structure diagram of metadata for an object storage system;

FIG. 1D is a schematic diagram of an interworking system of a first storage system and a second storage system;

FIG. 2 is a flowchart of a data writing method according to an embodiment of the present invention;

fig. 3A is a flowchart of a data writing method according to a third embodiment of the present invention;

FIG. 3B is a schematic diagram of a data structure of a pre-write log file;

FIG. 3C is a flow diagram of data operation playback of a pre-write log file;

fig. 4 is a schematic structural diagram of a data writing device according to a fourth embodiment of the present invention;

fig. 5 is a schematic structural diagram of a data writing device according to a fifth embodiment of the present invention;

fig. 6 is a schematic structural diagram of a terminal device implementing the data writing method according to the embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1A is a flowchart of a data writing method according to an embodiment of the present invention, where the method is applicable to data writing between storage systems based on different storage manners, so as to facilitate data access between the storage systems based on different storage manners, and the method may be executed by a data writing device, where the data writing device may be implemented in a form of hardware and/or software, and the data writing device may be configured in a server of a first storage system.

The system for data exchange access with the first storage system is a second storage system, the first storage system and the second storage system are systems based on different storage modes, for example, the first storage system is an object storage system, and the second storage system is a file storage system; or the first storage system is a file storage system, and the second file storage system is an object storage system; or the first storage system and the second storage system can be other two storage systems based on different storage modes. The different storage modes of the first storage system and the second storage system are mainly reflected in that the data structures of the metadata are different. FIG. 1B is a data structure diagram of metadata for a file storage system. FIG. 1C is a data structure diagram of metadata for an object storage system. As can be seen from fig. 1B and 1C, a data structure of a tree directory is used in the file storage system to store metadata; metadata is stored in the object storage system using a data structure of key-value stores. If the object storage system directly analyzes the metadata stored in the file storage system, or the file storage system directly analyzes the metadata stored in the object storage system, a large amount of computing resources are consumed, and the efficiency of reading files or objects is low. In view of the foregoing problems, embodiments of the present invention provide a data writing method, which completes metadata conversion and storage in a data writing stage, so that no additional metadata conversion is needed in a data reading stage.

Fig. 1D is a schematic structural diagram of an interworking system of a first storage system and a second storage system. As shown in fig. 1D, the interworking system includes: a user layer, a semantic layer, a metadata layer, and a data layer. For example, the first storage system is a file storage system, the second storage system is an object storage system, and the user layer includes: a client of the file storage system and a client of the object storage system; the semantic layer comprises: the server side of the file storage system and the server side of the object storage system are used for receiving write-in data written in by a client side connected through a network; the metadata layer includes: a file metadata storage area and a pre-written log system which are used for storing file metadata and arranged under a server side of the file storage system, and a pre-written log system and an object metadata storage area which are used for storing object metadata and arranged under a server side of the object storage system; the data layer includes: at least one data storage area for storing data contained in write data (write file or write object). The data between the first storage system and the second storage system can be accessed mutually, the first storage system can access the data uploaded by the second storage system, and the second storage system can also access the data uploaded by the first storage system.

As shown in fig. 1A, the method includes:

and S110, receiving write data and data write operation information written by the client of the first storage system.

If the first storage system is an object storage system, the written data is a written object; and if the first storage system is a file storage system, the written data is a written file. An Object-Based Storage System refers to a Storage System that stores data Based on an Object Storage manner, that is, a key-value Storage System understood in the general sense, such as a Simple Storage Service (S3) Storage System. The File storage System refers to a storage System that stores data based on a File storage method, such as a Network File System (NFS).

The data writing operation information refers to information corresponding to an operation of writing data into the first storage system, and the type of the written data is different according to different storage modes of the first storage system. The data writing operation information may specifically include new operation information, deletion operation information, update operation information, additional writing operation information, and overwrite operation information. For example, the data writing operation information may be generated when the write data is written to the server through the client of the first storage system, and sent to the server of the first storage system.

For example, if the first storage system is an object storage system, the server of the first storage system receives, through the network, the write object and the object write operation information written by the client of the first storage system. If the first storage system is a file storage system, the server side of the first storage system receives the written file and the file writing operation information written by the client side of the first storage system through the network.

If the first storage system is the network file storage system NFS, the server of the network file system needs to be mounted on the mount point directory of the Client of the production environment through the Client (NFS Client) of the network file system on the Client of the production environment. The client of the network file system writes file data into a local file system directory (namely, a mounting point directory), and the file data is sent to the server of the network file system through a network.

And S120, storing the data contained in the written data into a data storage area.

In which storage data in any data storage system can be divided into data and metadata. Data refers to actual data; metadata refers to information used to describe attributes of data to support functions such as indicating storage locations, history data, resource lookups, and file records. The data storage area is an area for storing data.

Specifically, after receiving the write data, the server of the first storage system splits the write data into data and metadata, and directly stores the data in the data storage area.

S130, writing the data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area.

The pre-Write Log system is a system for storing a pre-Write Log file (WAL), and the pre-Write Log file is a standard file for implementing a Transaction Log (Transaction Journal) in a database technology. Firstly, because the database file is randomly written, the WAL file is sequentially written, and the sequential writing performance is higher than that of random writing, the writing efficiency of the metadata is improved by a pre-writing log system. Secondly, the WAL file records the historical version of the data, the value of the historical version can be read, and even the state is rolled back to a certain historical version, so that the metadata can be obtained by performing data operation playback through the pre-written log file.

The first metadata may be considered metadata stored based on a data structure of the first storage system. The first metadata storage area refers to an area for storing metadata in a server of the first storage system. And determining the type of the first metadata according to the storage mode of the first file storage system. For example, if the first storage system is a file storage system, the first metadata is file metadata; if the first storage system is an object storage system, the first metadata is object metadata.

Specifically, when receiving data write operation information corresponding to write data, a server of the first storage system writes the data write operation information into a local pre-write log system, and the data write operation information is stored in the pre-write log system in a form of a pre-write log file, so that the data write operation information is conveniently copied to a server of the second storage system. In addition, the first metadata corresponding to the data writing operation information generated during data writing is directly written into the first metadata storage area of the first storage system.

S140, sending the pre-written log file to a server of a second storage system; and the server side of the second storage system performs data operation playback according to the pre-written log file, converts the first metadata into second metadata and writes the second metadata into a second metadata storage area.

Wherein the second metadata may be considered metadata that employs a data structure of the second storage system. The second metadata storage area refers to an area for storing metadata in a server of the second storage system. And determining the type of the second metadata according to the storage mode of the second file storage system. For example, if the second storage system is a file storage system, the second metadata is file metadata; if the second storage system is an object storage system, the second metadata is object metadata.

In order to solve the problem in the prior art, when a second storage system reads first metadata stored in a first storage system, the first metadata needs to be converted into second metadata; in the embodiment of the invention, when writing data into first metadata, the first metadata is written into a first metadata storage area, and the first metadata is further converted into second metadata to be written into a second metadata storage area.

Specifically, after the server of the first storage system writes the data write operation information into the pre-written log file in the pre-written log system of the local storage system, the pre-written log file is sent to the server of the second storage system. After receiving the pre-written log file, the server of the second storage system may perform data operation playback according to the pre-written log file to obtain first metadata, and convert the first metadata into second metadata and write the second metadata into a local second metadata storage area, as with the server of the first storage system.

It should be noted that this step and the step of "writing the first metadata corresponding to the data writing operation information into the first metadata storage area" are executed independently, and the execution sequence is not sequential, or may be executed simultaneously.

The step can realize that the first metadata stored in the first metadata storage area of the first storage system is consistent with the second metadata stored in the second metadata storage area of the second storage system. The second storage system can read the metadata from the local second metadata storage area and the data from the data storage area when reading the write data stored in the first storage system. Because the metadata conversion and the replication of the metadata are completed in the writing stage, the data conversion of the metadata is not needed in the reading stage, the time consumption of the reading process is short, the resource occupancy rate is low, and the efficiency of mutual data access between storage systems based on different storage modes is further effectively improved.

It can be understood that the data writing method provided by the present invention can also be applied to a server of a second storage system, and the data included in the written data is stored in the data storage area, and the data writing operation information is written into the pre-written log file in the local pre-written log system; performing data operation playback according to the pre-written log file based on a pre-written log system, and writing second metadata written with data into a second metadata storage area; and sending the pre-written log file to a server of the first storage system, so that the server of the first storage system performs data operation playback according to the pre-written log file, converts the second metadata into the first metadata and writes the first metadata into the first metadata storage area, and thus mutual access between the first storage system and the second storage system is realized.

According to the technical scheme of the embodiment of the invention, the write-in data and the data write-in operation information written by the client of the first storage system are received; storing data contained in the write data in a data storage area; writing the data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area; and sending the pre-written log file to a server of the second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, and converts the first metadata into second metadata to write the second metadata into a second metadata storage area. By completing the conversion and storage of the metadata in the data writing stage, no additional metadata conversion is needed in the data reading stage, the problems of long time and low efficiency of mutual access between storage systems based on different storage modes are solved, and the beneficial effects of short time consumption, low resource occupancy rate and high access efficiency of the access between the storage systems of different storage modes are achieved.

Example two

Fig. 2 is a flowchart of a data writing method according to a second embodiment of the present invention, and this embodiment further details "writing data writing operation information into a pre-written log file in a local pre-written log system, and writing first metadata corresponding to the data writing operation information into a first metadata storage area" in step S130 of the foregoing embodiment. As shown in fig. 2, the method includes:

s210, receiving write data and data write operation information written by a client of the first storage system.

And S220, storing the data contained in the written data into a data storage area.

And S230, writing the first metadata corresponding to the data writing operation information into the first metadata storage area.

S240, after the size of the pre-written log file in the pre-written log system reaches a preset threshold value, cutting the pre-written log file to obtain a blank pre-written log file and a fully written pre-written log file.

The blank pre-written log file is a blank pre-written log file generated after the pre-written log file is cut, and is used for receiving new data write-in operation information. The written-full pre-written log file is a pre-written log file which is cut and then written with a full part, and is used for persisting to a local hard disk of the first storage system. The preset threshold corresponding to the cutting may be set according to actual requirements, for example, the preset threshold may be 60% of the file capacity.

Specifically, the initial pre-written log file is empty, and the received data write operation information is written into the pre-written log file. And after the size of the pre-written log file reaches a preset threshold value, cutting the pre-written log file to obtain a new blank pre-written log file and a fully written pre-written log file.

And S250, storing the fully written pre-written log file into a local hard disk of a server of the first storage system.

Specifically, the written-full pre-written log file is stored in a local hard disk of a server of the first storage system, so that persistent storage of data write operation stored in the written-full pre-written log file is realized.

And S260, writing the data writing operation information into the blank pre-written log file.

Specifically, the received data write operation information is written into a blank pre-written log file, and if the blank pre-written log file reaches a preset threshold, the cutting is continued to obtain a new blank pre-written log file for writing the received new data write operation information.

S270, the fully written pre-written log file stored in the local hard disk is asynchronously sent to a server of the second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, and converts the first metadata into second metadata which is written into a second metadata storage area.

Specifically, the written pre-written log file stored in the local hard disk is sent to the server of the second storage system in an asynchronous manner. And starting an operation playback program by a server of the second storage system, performing data operation playback according to the received pre-written log file (namely, the pre-written log file is fully written), converting the first metadata into second metadata, and writing the second metadata into a second metadata storage area.

In the step, the second metadata corresponding to the first metadata written in the data is stored in the second storage system in a background asynchronous mode, the second metadata stored in the second storage system can be accessed by a server of the second storage system, and no extra burden is brought to the server of the first storage system and the server of the second storage system due to the written metadata.

According to the technical scheme of the embodiment of the invention, the write-in data and the data write-in operation information written by the client of the first storage system are received; storing data contained in the write data into a data storage area; writing first metadata corresponding to the data writing operation information into a first metadata storage area; cutting the pre-written log file to obtain a blank pre-written log file and a fully written pre-written log file after the file size of the pre-written log file in the pre-written log system reaches a preset threshold value; storing the fully written pre-written log file into a local hard disk of a server of a first storage system; writing the data writing operation information into a blank pre-written log file; the written pre-written log file stored in the local hard disk is asynchronously sent to the server of the second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, converts the first metadata into second metadata and writes the second metadata into a second metadata storage area, the time consumption of access between the storage systems with two different storage modes is short, the resource occupancy rate is low, the access efficiency is high, and the metadata conversion and copy are performed in a background asynchronous mode, so that the metadata stored in the storage systems with the two different storage modes are kept uniform, and no extra burden is brought to the server of the first storage system and the server of the second storage system.

EXAMPLE III

Fig. 3A is a flowchart of a data writing method according to a third embodiment of the present invention, where this embodiment is applicable to data writing between storage systems based on different storage manners, so as to facilitate data access between the storage systems based on different storage manners, and the method may be executed by a data writing device, where the data writing device may be implemented in a form of hardware and/or software, and the data writing device may be configured in a server of a second storage system.

As shown in fig. 3A, the method includes:

s310, analyzing the received pre-written log file sent by the server side of the first storage system to obtain a data operation log.

The pre-written log file, namely the WAL file, mainly includes: a Data Segment information Area (Segment Info) and a Data Area (Data Area). The data segment information area is mainly used for storing attribute information of the pre-written log file; the data area mainly stores data write operation information, such as object write operation information or file write operation information. The data operation log stores a log of data writing operation information.

Specifically, a server of the second storage system receives a pre-written log file sent by the server of the first storage system, wherein the pre-written log file comprises a pre-written log file; and the client of the first storage system writes data to generate data writing operation information, and analyzes the pre-written log file to obtain a data operation log.

In order to ensure the data consistency and data integrity of the written data in the writing process, before obtaining the log entry of the data area to obtain the data operation log, the information stored in the data segment information area of the pre-written log file needs to be verified.

It should be noted that the server of the first storage system may write the data write operation information into a pre-written log file obtained in a local pre-written log system, and send the pre-written log file to the server of the second storage system; or the server of the first storage system may cut the pre-written log file to obtain a blank pre-written log file and a fully written pre-written log file after the file size of the pre-written log file in the pre-written log system reaches a preset threshold, store the fully written pre-written log file in the local hard disk of the server of the first storage system, and send the fully written pre-written log file stored in the local hard disk to the server of the second storage system. Therefore, the pre-written log file sent by the server of the first storage system and received by the server of the second storage system may be a pre-written log file or a fully written pre-written log file.

And S320, performing data operation playback on the data operation log to obtain first metadata.

Each data operation log can be stored in a data area of the pre-written log file by adopting a data structure of a log entry LogEntry.

Specifically, the server side of the second storage system starts an operation playback program, deserializes the log entry through an agreed data structure to obtain a recognizable data operation log, and performs data operation playback on the data operation log to obtain first metadata corresponding to the data operation of the written data.

S330, performing metadata format conversion on the first metadata to obtain second metadata, and writing the second metadata into a second metadata storage area.

Specifically, because the metadata of the first storage system and the metadata of the second storage system have different data structures, the first metadata under the protocol of the first storage system needs to be converted into the second metadata under the protocol of the second storage system, and then the second metadata is written into the second metadata storage area. By completing the conversion and storage of the metadata in the data writing stage, no additional metadata conversion is needed in the data reading stage, and the access time consumption between the storage systems with two different storage modes is short, the resource occupancy rate is low, and the access efficiency is high.

For example, if the first storage system is a file storage system and the second storage system is an object storage system, the first metadata is file metadata and the second metadata is object metadata, and the file metadata of the tree directory structure is converted into object metadata of key value storage and then written into the second metadata storage area.

If the first storage system is an object storage system and the second storage system is a file storage system, the first metadata is object metadata, the second metadata is file metadata, and the object metadata such as a bucket name and an object name are converted into file metadata of a tree directory structure and then written into a second metadata storage area.

According to the technical scheme of the embodiment of the invention, a data operation log is obtained by analyzing a received pre-written log file sent by a server of a first storage system; carrying out data operation playback on the data operation log to obtain first metadata; and performing metadata format conversion on the first metadata to obtain second metadata, and writing the second metadata into a second metadata storage area. By completing the conversion and storage of the metadata in the data writing stage, no additional metadata conversion is needed in the data reading stage, the problems of long time and low efficiency of mutual access between storage systems based on different storage modes are solved, and the beneficial effects of short time consumption, low resource occupancy rate and high access efficiency of the access between the storage systems of different storage modes are achieved.

In an optional embodiment, parsing the received pre-written log file sent by the server of the first storage system to obtain a data operation log includes:

analyzing a received pre-written log file sent by a server of a first storage system to obtain a data segment information area and a data area;

verifying the data segment information stored in the data segment information area;

if the log entry passes the verification, acquiring the log entry stored in the data area of the pre-written log file; the log entry includes: a data length field and a log data area;

and reading the data operation log in the log data area based on the data length field.

Fig. 3B is a schematic diagram of a data structure of a pre-write log file. As shown in fig. 3B, the pre-written log file includes: a data segment information area and a data area; the data segment information area mainly stores the version number, the index number, the log entry data volume and the check code of the pre-written log file. The target check code may be, for example, a check code calculated based on an algorithm such as a hash algorithm or a crc32 code, and the algorithm of the check code is not limited in the embodiment of the present invention.

The data area mainly stores each piece of data writing operation information in the form of a log entry LogEntry, and the data writing operation information may include: object write operation information or file write operation information. The log entry LogEntry includes a data length field EntryLength and a log data area EntryData; the data length field is used for describing the length of the log data area; the log data area may be a custom area for logging data operations.

Specifically, the server side of the second storage system receives the pre-written log file sent by the server side of the first storage system, and analyzes the pre-written log file based on the data structure of the pre-written log file to obtain a data segment information area and a data area; and verifying the field information of the pre-written log file stored in the data segment information area to ensure the data consistency and the data integrity of the pre-written log file. If the verification fails, the analysis process of the pre-written log file is ended, and the data writing error is shown. If the verification is passed, acquiring the log entry stored in the data area of the pre-written log file, analyzing the data length field EntryLength contained in the log entry, and reading the data operation log contained in the log data area in the log entry according to the data length field EntryLength.

In an optional embodiment, verifying the data segment information stored in the data segment information area includes:

verifying whether the version number supports data operation playback;

verifying whether the index number is the index number of the first pre-written log file or whether the index number is continuous with the index number corresponding to the pre-written log file subjected to data operation playback;

verifying whether the pre-written log file corresponding to the index number is not subjected to data operation playback;

and verifying whether the check code is the same as the check code calculated by the pre-written log file.

Specifically, firstly, judging whether the version number of the pre-written log WAL file supports data operation playback, secondly, ensuring the data consistency of the pre-written log WAL file, and ensuring that the index number of the pre-written log WAL file is continuous and the data operation process must be played back according to the index number sequence and the data operation sequence, so that whether the index number is the index number of the first pre-written log file or whether the index number is continuous with the index number corresponding to the last pre-written log file subjected to data operation playback needs to be verified; it is also necessary to ensure that the pre-written log WAL file is not repeatedly subjected to data operation playback, and therefore, it is verified whether the pre-written log file corresponding to the index number is not subjected to data operation playback.

In addition, the data integrity of the pre-written log WAL file needs to be ensured, so that whether the check code stored in the data segment information area of the pre-written log WAL file is the same as the check code calculated by a predefined check algorithm needs to be verified, if so, the data of the pre-written log WAL file is complete, and the data operation playback of the WAL file is continuously executed; if the difference is not, the data of the pre-written log WAL file is incomplete, a writing error such as packet loss and the like may occur, and the data operation playback cannot be performed by using the WAL file.

In one particular example, fig. 3C is a flow diagram of data operation playback of a pre-written log file. As shown in fig. 3C, after receiving the pre-written log file sent by the server of the first storage system, the server of the second storage system analyzes the pre-written log file to obtain a data segment information area and a data area; and judging whether the version number contained in the data segment information area of the pre-written log WAL file supports data operation playback, and if not, ending the data operation playback process of the pre-written log file. If so, continuing the data operation playback process of the pre-written log file, and judging whether the pre-written log file is processed according to the index number. If the pre-written log file is processed, judging whether all the pre-written log files are analyzed, and if not, returning to receive the pre-written log files; and if so, ending the data operation playback process of the pre-written log file. If the pre-written log file is not processed, continuing the data operation playback process of the pre-written log file, and judging whether the pre-written log file is the first pre-written log file or is continuous with the previous pre-written log file for data operation playback according to the index number so as to ensure that the pre-written log file continuously executes data operation playback according to the index sequence and ensure the consistency of data; and if the data are inconsistent, the pre-written log file is not processed, and the data operation playback process of the pre-written log file is ended. If the data are consistent, continuing the data operation playback process of the pre-written log file, and judging whether the check codes of the pre-written log file are matched so as to ensure the integrity of the data. If not, the pre-written log file is not processed, and the data operation playback process of the pre-written log file is ended; if the data in the log data area is matched with the metadata in the metadata storage area, deserializing the data in the log data area to obtain a data operation log, continuously performing data operation playback on the data operation log to obtain metadata, and writing the metadata into the metadata storage area; finishing data operation playback on the current pre-written log file, and recording the index number of the pre-written log file currently subjected to data operation playback so as to continuously play back the subsequent pre-written log file and ensure the consistency of data; continuously judging whether all the pre-written log files are analyzed, and if not, returning to receive the pre-written log files; and if so, ending the data operation playback process of the pre-written log file.

Optionally, before performing metadata format conversion on the first metadata to obtain second metadata, the method further includes:

shielding first metadata corresponding to data operation which is not supported by a second storage system in a data operation log;

or converting the data operation which is not supported by the second storage system in the data operation log into the data operation which is supported by the second storage system, and determining the first metadata corresponding to the data operation which is supported by the second storage system.

Specifically, because the object storage manner of the object storage system does not support a part of operations of a part of file storage systems, it is necessary to mask first metadata corresponding to data operations that are not supported by the second storage system in the data operation log, or after converting the data operations that are not supported by the second storage system in the data operation log into data operations that are supported by the second storage system, determine first metadata corresponding to the data operations that are supported by the second storage system, and then perform metadata format conversion on the first metadata to obtain second metadata, and store the second metadata in the second metadata storage area.

For example, if the first storage system is a file storage system, the second storage system is an object storage system; the object storage system does not support write operations such as additional write and overwrite, and file metadata corresponding to the additional write data operation or the overwrite write data operation needs to be converted. The object storage system does not support the data updating operation, the data updating operation needs to be divided into a deleting operation and a newly-built operation, after first metadata corresponding to the deleting operation and first metadata corresponding to the newly-built operation are obtained, metadata format conversion is carried out, second metadata corresponding to the deleting operation and second metadata corresponding to the newly-built operation are obtained, and the second metadata are stored in a second metadata storage area.

Example four

Fig. 4 is a schematic structural diagram of a data writing device according to a fourth embodiment of the present invention. The apparatus may be integrated with a server of the first storage system, as shown in fig. 4, and includes: the system comprises a receiving module 410, a data storage module 420, a first metadata writing module 430 and a log file sending module 440;

the receiving module 410 is configured to receive write data and data write operation information written by a client of the first storage system;

a data storage module 420, configured to store data included in the write data in a data storage area;

a first metadata writing module 430, configured to write the data writing operation information into a pre-written log file in a local pre-written log system, and write first metadata corresponding to the data writing operation information into a first metadata storage area;

the log file sending module 440 is configured to send the pre-written log file to a server of a second storage system; and enabling the server side of the second storage system to perform data operation playback according to the pre-written log file, and converting the first metadata into second metadata and writing the second metadata into a second metadata storage area.

The data writing device provided by the embodiment of the invention receives the written data and the data writing operation information written by the client of the first storage system; storing data contained in the write data into a data storage area; writing the data writing operation information into a pre-written log file in a local pre-written log system; writing first metadata corresponding to the data writing operation information into a first metadata storage area; the pre-written log file is sent to a server of the second storage system, so that the server of the second storage system performs data operation playback according to the pre-written log file, converts the first metadata into second metadata and writes the second metadata into a second metadata storage area, and the beneficial effects of short time consumption, low resource occupancy rate and high access efficiency of access among the storage systems with different storage modes are achieved.

Optionally, the first metadata writing module 430 includes:

the log file cutting unit is used for cutting the pre-written log file to obtain a blank pre-written log file and a fully written pre-written log file after the file size of the pre-written log file reaches a preset threshold value;

the log file storage unit is used for storing the fully written pre-written log file into a local hard disk of a server of the first storage system;

and the information writing unit is used for writing the data writing operation information into the blank pre-written log file.

Optionally, the log file sending module 440 is configured to:

and asynchronously sending the fully written pre-written log file stored in the local hard disk to a server of the second storage system.

Optionally, the first storage system is a file storage system, and the second storage system is an object storage system;

or, the first storage system is an object storage system, and the second storage system is a file storage system.

The data writing device provided by the embodiment of the invention can execute the data writing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE five

Fig. 5 is a schematic structural diagram of a data writing device according to a fifth embodiment of the present invention. The apparatus may be integrated with a server of the first storage system, as shown in fig. 5, and includes: a log file parsing module 510, a data operation playback module 530 and a second metadata writing module 540;

a log file analyzing module 510, configured to analyze a received pre-written log file sent by a server of the first storage system to obtain a data operation log;

a data operation playback module 520, configured to perform data operation playback on the data operation log to obtain first metadata;

a second metadata writing module 530, configured to perform metadata format conversion on the first metadata to obtain second metadata, and write the second metadata into the second metadata storage area.

The data writing device provided by the embodiment of the invention obtains the data operation log by analyzing the received pre-written log file sent by the server of the first storage system; carrying out data operation playback on the data operation log to obtain first metadata; and performing metadata format conversion on the first metadata to obtain second metadata, and writing the second metadata into a second metadata storage area. By completing the conversion and storage of the metadata in the data writing stage, no additional metadata conversion is needed in the data reading stage, the problems of long time and low efficiency of mutual access between storage systems based on different storage modes are solved, and the beneficial effects of short time consumption, low resource occupancy rate and high access efficiency of the access between the storage systems of different storage modes are achieved.

Optionally, the log file parsing module 510 includes:

the analysis subunit is used for analyzing the pre-written log file to obtain a data segment information area and a data area;

the verification subunit is used for verifying the data segment information stored in the data segment information area;

an obtaining subunit, configured to obtain, if the verification passes, a log entry stored in a data area of the pre-written log file; the log entry includes: a data length field and a log data area;

and the reading subunit is used for reading the data operation log in the log data area based on the data length field.

Optionally, the data segment information stored in the data segment information area includes: a target version number, a target index number and a target log check code; the verification subunit is specifically configured to:

verifying whether the version number supports data operation playback;

Optionally, the apparatus further comprises:

the shielding module is used for shielding the first metadata corresponding to the data operation which is not supported by the second storage system in the data operation log before the metadata format conversion is carried out on the first metadata to obtain the second metadata;

and the operation conversion module is used for converting the data operation which is not supported by the second storage system in the data operation log into the data operation which is supported by the second storage system, and determining first metadata corresponding to the data operation which is supported by the second storage system.

EXAMPLE six

Fig. 6 shows a schematic structural diagram of a terminal device 10 that can be used to implement an embodiment of the invention. Terminal devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The terminal device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 6, the terminal device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, where the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the terminal device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A plurality of components in the terminal device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the terminal device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the data writing method.

In some embodiments, the data writing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed on the terminal device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data writing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data writing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here may be implemented on a terminal device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the terminal device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data writing method is applied to a server of a first storage system, and the method comprises the following steps:

storing data contained in the write data into a data storage area;

2. The method of claim 1, wherein writing the data write operation information to a pre-write log file in a local pre-write log system comprises:

after the file size of the pre-written log file reaches a preset threshold value, cutting the pre-written log file to obtain a blank pre-written log file and a fully written pre-written log file;

storing the fully written pre-written log file into a local hard disk of a server of the first storage system;

and writing the data writing operation information into the blank pre-written log file.

3. The method of claim 2, wherein sending the pre-written log file to a server of a second storage system comprises:

4. The method of claim 1, wherein the first storage system is a file storage system and the second storage system is an object storage system;

5. A data writing method, applied to a server of a second storage system, the method comprising:

analyzing a received pre-written log file sent by a server of the first storage system to obtain a data operation log;

6. The method of claim 5, wherein the parsing the received pre-written log file sent by the server of the first storage system to obtain the data operation log comprises:

if the verification is passed, acquiring the log entries stored in the data area of the pre-written log file; the log entry includes: a data length field and a log data area;

and reading a data operation log in the log data area based on the data length field.

7. The method of claim 6, wherein the data segment information stored in the data segment information area comprises: the version number, the index number and the check code of the pre-written log file; the verifying the data segment information stored in the data segment information area includes:

verifying whether the version number supports data operation playback;

verifying whether the index number is the index number of the first pre-written log file or whether the index number is continuous with the index number corresponding to the last pre-written log file subjected to data operation playback;

8. The method according to claim 5, wherein before the converting the first metadata into the second metadata, further comprising:

shielding first metadata corresponding to data operation which is not supported by the second storage system in the data operation log;

9. A data writing apparatus, integrated in a server of a first storage system, the apparatus comprising:

10. A data writing apparatus integrated with a server of a second storage system, the apparatus comprising:

11. A terminal device, characterized in that the terminal device comprises:

at least one processor; and

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data writing method of any one of claims 1-8.

12. A computer-readable storage medium storing computer instructions for causing a processor to implement the data writing method according to any one of claims 1 to 8 when executed.