CN108459824B - Data modification and rewriting method and device - Google Patents

Data modification and rewriting method and device Download PDF

Info

Publication number
CN108459824B
CN108459824B CN201711376932.6A CN201711376932A CN108459824B CN 108459824 B CN108459824 B CN 108459824B CN 201711376932 A CN201711376932 A CN 201711376932A CN 108459824 B CN108459824 B CN 108459824B
Authority
CN
China
Prior art keywords
disk
data
fragment
location
modification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711376932.6A
Other languages
Chinese (zh)
Other versions
CN108459824A (en
Inventor
汪正洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Huawei Cloud Computing Technology Co ltd
Original Assignee
Xian Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xian Huawei Technologies Co Ltd filed Critical Xian Huawei Technologies Co Ltd
Priority to CN201711376932.6A priority Critical patent/CN108459824B/en
Publication of CN108459824A publication Critical patent/CN108459824A/en
Application granted granted Critical
Publication of CN108459824B publication Critical patent/CN108459824B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0617—Improving the reliability of storage systems in relation to availability
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0658—Controller construction arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2211/00—Indexing scheme relating to details of data-processing equipment not covered by groups G06F3/00 - G06F13/00
    • G06F2211/10—Indexing scheme relating to G06F11/10
    • G06F2211/1002—Indexing scheme relating to G06F11/1076
    • G06F2211/104—Metadata, i.e. metadata associated with RAID systems with parity

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A data modification and rewriting method and device. The method comprises the following steps: the metadata server receives a modification writing request sent by a client, wherein the modification writing request carries modification data, and the modification data is used for replacing original data stored at a first position in a first disk; the metadata server determines a second location capable of storing the modified data, the second location being located on the first disk or the second disk; the metadata server sends the modification writing request and second identification information of a second position to the OSD; and after receiving the response information fed back by the OSD, the metadata server modifies the first identification information of the first position in the file metadata into the second identification information, and the client can read the modified data according to the second identification information. In the mode, the modified data and the original data are stored in different positions, the data modification process can be executed in the process of reading the original data, the service is not easy to interrupt, and the service processing efficiency is improved.

Description

Data modification and rewriting method and device
Technical Field
The embodiment of the invention relates to the technical field of data storage, in particular to a data modification and rewriting method and device.
Background
In order to improve the utilization rate of the hard disk, the current data storage method is generally distributed storage, that is, a client dispersedly writes file data into different storage areas (e.g., disks). After the client writes in the file data, the file data can be modified, and the data modification and rewriting process of the current file distributed system is mainly based on a two-stage submission mechanism. Fig. 1 is a schematic diagram illustrating a data modification process in a current file distribution system (DFS), as shown in fig. 1, in a first stage, a Metadata Server (MDS) sends a prefix request to an OSD of an object store of a disk, and then receives a prefix response returned by the OSD. The purpose of the first stage is to determine that the disk is ready, that is, to notify the OSD to suspend the read/write operation on the disk, for example, if other clients need to read the file data on the disk, the OSD does not respond to other clients to prepare for modifying the file data on the disk. And in the second stage, after the disk is ready to work, the metadata server sends a commit request to the OSD and then receives a commit response returned by the OSD. The second phase functions to formally commit the modified data. The specific process is that the OSD releases the file data (hereinafter referred to as original data) on the disk, the metadata server stores the modified data in the cache and informs the storage position of the OSD cache, and the OSD writes the modified data in the cache into the disk. After the modified data is successfully written, the metadata server can respond to the requests of other clients for reading the modified data so as to maintain the normal operation of other client services.
As can be seen from the above description, the current data modification process is complex, and service interruption is easy to occur, thereby causing a problem of low service processing efficiency.
Disclosure of Invention
The embodiment of the invention provides a data modification method and a data modification device, which are used for solving the problems that the data modification process in the existing distributed file system is complex, and service interruption is easy to occur, so that the service processing efficiency is low.
In a first aspect, an embodiment of the present invention provides a data modification and rewriting method, where the method includes: the metadata server receives a modification writing request sent by a client, wherein the modification writing request carries modification data, and the modification data is used for replacing original data stored at a first position in a first disk; the metadata server determining a second location capable of storing the modified data, the second location being located on the first disk or a second disk; the metadata server sends the modification writing request and second identification information of the second position to an Object Storage Device (OSD), wherein the OSD is a control device for writing the modification data into the second position; the metadata server receives response information fed back by the OSD, wherein the response information is used for indicating that the OSD successfully writes the modified data into the second position; and the metadata server modifies the first identification information of the first position stored in the file metadata into the second identification information, so that the client can read the modified data according to the second identification information.
In the embodiment of the present invention, the modified data and the original data are stored in different locations, for example, the modified data and the original data may be stored in different locations in the same disk or may be stored in different disks. When the original data is stored at the first position, the first identification information of the first position is stored in the file metadata, the client can read the original data through the first identification information, and after the modified data is successfully stored at the second position and the first identification information in the file metadata is modified into the second identification information, the client can continue to read the modified data according to the second identification information. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
In one possible design, the first disk includes a plurality of segments, and the second location and the first location are located in the same segment or different segments on the first disk, respectively; if the second position and the first position are located in the same fragment on the first disk respectively, the first identification information includes a disk identifier of the first disk, a fragment identifier of the same fragment, and a first storage address of the first position in the same fragment; the second identification information comprises a disk identification of the first disk, a fragment identification of the same fragment and a second storage address of a second position in the same fragment; or if the first location is located in a first segment on the first disk, the second location is located in a second segment on the first disk, and the first identification information includes a disk identifier of the first disk, a segment identifier of the first segment, and a first storage address of the first location in the first segment; the second identification information includes a disk identification of the first disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
In the embodiment of the present invention, one disk includes a plurality of fragments, and the modified data and the original data may be stored in different fragments of the first disk, or may be located in the same fragment of the first disk. The identification information of the storage location of the modified data (or the original data) may be a disk identification, a fragment identification, a storage address of the modified data (or the original data) in the fragment, and the like. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
In one possible design, the second memory address is an address offset from the first memory address.
In the embodiment of the present invention, the address offset may be used to determine the specific storage location of the modified data more accurately, so as to facilitate reading, and certainly, the second storage address may further include other parameters, which are only examples, and this is not limited in the embodiment of the present invention.
In one possible design, the first location is in a first slice on the first disk and the second location is in a second slice on the second disk; the first identification information comprises a disk identification of the first disk, a fragment identification of the first fragment and a first storage address of a first position in the first fragment; the second identification information includes a disk identification of the second disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
In the embodiment of the invention, one disk comprises a plurality of fragments, and the modified data and the original data can be respectively stored in different fragments of different disks. The identification information of the storage location of the modified data (or the original data) may be a disk identification, a fragment identification, a storage address of the modified data (or the original data) in the fragment, and the like. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
In one possible design, the metadata server sends a release request to the OSD, the release request for releasing the original data stored at the first location; and the metadata server receives response information which is fed back by the OSD and used for indicating the successful release.
In an embodiment of the present invention, after the file metadata is modified, the metadata server may send a release request to the OSD to cause the OSD to delete the original data stored at the first location in order to store other data. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
In a possible design, when the metadata server sends the modification request and the second identification information of the second location to the OSD, N pieces of sub-modification data may be obtained by processing the modification data; the metadata server respectively sends the N pieces of sub-modification data and the second identification information to OSD (on screen display) corresponding to the N disks in the disk array, so that each OSD respectively writes the received sub-modification data into a storage position corresponding to the second identification information in the corresponding disk; the disk array comprises N stacked disks, and the storage positions corresponding to the second identification information in each disk are the same; n is an integer greater than or equal to 2; when receiving the response information fed back by the OSD, the metadata server specifically includes: and collecting the response information fed back by each OSD.
In the embodiment of the present invention, the disk array includes N stacked disks, each disk corresponds to one OSD, the metadata server may process the modified data to obtain N pieces of sub-modified data, and respectively send the N pieces of sub-modified data and the second identification information to the OSD corresponding to each disk in the disk array, and after each OSD respectively writes the received sub-modified data in the storage location corresponding to the second identification information in the corresponding disk, the metadata server may collect response information fed back by each OSD. In this way, after writing the sub-modification data into each disk in the disk array, the metadata server can determine that the sub-modification data in each disk is successfully written, which is helpful for improving the accuracy of the data modification process.
In a second aspect, an embodiment of the present invention further provides a data modification and rewriting apparatus. The data modification and rewriting device has the function of realizing the metadata server in the method design. These functions may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more units corresponding to the above functions.
In one possible design, the specific structure of the data modification and rewriting apparatus may include a receiving unit, a processing unit, and a transmitting unit. These units may perform the respective functions in the method provided by the first aspect or any one of the possible designs of the first aspect.
In a third aspect, an embodiment of the present invention further provides a data modification and rewriting apparatus. The data modification and rewriting device has the function of realizing the metadata server in the method design. These functions may be implemented by hardware. The data modification and rewriting apparatus includes: a memory for storing computer executable program code; a processor coupled with the memory. Wherein the program code stored in the memory includes instructions that, when invoked and executed by the processor, cause the data modification apparatus to perform the method performed by the metadata server of the first aspect or any one of the possible designs of the first aspect.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer is caused to execute the data modification writing method according to the first aspect.
In a fifth aspect, an embodiment of the present invention further provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the data modification writing method according to the first aspect.
In the embodiment of the present invention, the modified data and the original data are stored in different locations, for example, the modified data and the original data may be stored in different locations in the same disk or may be stored in different disks. When the original data is stored at the first position, the first identification information of the first position is stored in the file metadata, the client can read the original data through the first identification information, and after the modified data is successfully stored at the second position and the first identification information in the file metadata is modified into the second identification information, the client can continue to read the modified data according to the second identification information. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
Drawings
FIG. 1 is a diagram illustrating a data modification write process of a distributed file system according to the prior art;
fig. 2 is a schematic structural diagram of a disk array according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an EC redundancy scheme provided by an embodiment of the present invention;
fig. 4 is a system architecture of a distributed file system according to an embodiment of the present invention;
fig. 5 is a flowchart of a data modification and rewrite method according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating a data modification process according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating another data modification process according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a data modification and rewriting apparatus according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of another data modification and rewriting apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
Hereinafter, some terms in the present invention will be explained to facilitate understanding by those skilled in the art.
(1) A disk array and a technology commonly used in a distributed file system DFS are disk array (RAID) technologies, fig. 2 shows a structure diagram of a disk array, and as shown in fig. 2, 3 disks are logically regarded as one disk, the disk may be divided into different stripes, that is, a portion of each stripe falling on a single hard disk is called an object or a shard (shard), the stripe in fig. 2 includes 3 shards, that is, shards 1 to 3, the different shards are located on different disks, that is, the shard 1 is located on the disk 1, the shard 2 is located on the disk 2, and the shard 3 is located on the disk 3. When the client writes file data on the disk array, the file data can be dispersedly stored in 3 fragments of the disk array. The specific process is that the metadata server receives a write request sent by a client, divides the write request into three sub-write requests, then sends the three sub-write requests to the OSD corresponding to each disk in the disk array, namely, the three sub-write requests are sent to the OSD1-3, and each OSD writes file data corresponding to the received sub-write request into a fragment on the disk corresponding to each OSD.
(2) In an actual operation process, after receiving a write request sent by a client, a metadata server may divide file data into different data blocks, for example, the data blocks are divided into data blocks 1 to 3, the metadata server may store the data block 1 in the fragments 1 to 3 in a disk array, and store the data blocks 2 and 3 in other fragments in the disk array, that is, the data block 1 in the file data is stored in the fragment 1, and the data blocks 2 and 3 are stored in other fragments in the disk array.
Generally, when file metadata is modified, for example, data block 1 is modified, data in segments 1-3 in segment 1 may all be modified, and when data block 1 is read, data in segments 1-3 in segment 1 are read in parallel to obtain data block 1.
(3) The fragmentation, the storage unit on the disk, and generally a disk may be divided into a plurality of fragmentations, and the size of the storage space of each fragmentation may be customized, for example, 1G, 2G, and so on.
The shards are defined based on the physical disks, and the stripes are defined based on the logical disks, that is, when a plurality of physical disks are logically regarded as one logical disk, the logical disk is divided into a plurality of stripes, and the part of one stripe falling on each physical disk is one shard.
(4) An Erasure Code (EC) redundancy mode, a data storage manner of a distributed file system, may be used for data recovery. In the EC redundancy mode, some of the different slices on one stripe in the disk array may be used to store file data, while other slices are used to store parity data. For example, fig. 3 shows a schematic diagram of data modification and rewriting in a 4+2 redundancy mode, such as fig. 3, where 4 slices are used to store file data and 2 slices are used to store parity data, where slices 1-4 are used to store file data and slices 5-6 are used to store parity data. When the file data in the segment 1 is lost, the file data in the segment 1 can be restored through the check data in the segments 5-6. In the process of data recovery through check data, a precondition is that the number of fragments required for data recovery does not exceed the number of redundancies (i.e. the number of fragments of check data). For example, as shown in fig. 3, before the data is modified, the original data is stored in the fragments 1-6, and when the original data is modified, if the modified data of the fragments 1-3 is not successfully written into the disks 1-3, for example, the server where the disks 1-3 are located is reset, and the fragments 4-6 are successfully modified. At this time, since the number of unmodified slices (3) exceeds the number of redundant slices (2), the modified data on the slices 1-3 cannot be recovered. When the client reads the modified data, the storage position of the modified data, namely the fragments 1-6, is determined, but the modified data on the fragments 1-3 cannot be recovered, so that the modified data read by the client is not satisfactory, namely, data loss is easily caused in the process of modifying and writing the data.
(5) The metadata server, an important device in the distributed file system, is responsible for managing files such as file metadata, and may play a role in coordinating clients and disk arrays. There may be only one metadata server or multiple metadata servers in the distributed file system, and each metadata server may be responsible for different disk arrays and clients. The plurality of metadata servers form a service layer, and the service layer is responsible for processing service requests sent by the client, such as read requests, write requests and the like.
(6) The object storage device is used for processing a control device of read and write requests, namely, data is written into a disk or data is read from the disk. Each object storage device is connected with one disk and is responsible for data writing, reading and the like of the disk.
(7) The distributed file system stores basic attribute information of file data stored each time in the file metadata, such as file names, storage location information and the like, so that the file metadata can be understood as a directory of data storage and stored in a metadata server, and the storage location of the file data can be clearly known through the file metadata, so that the file metadata is convenient for a client to read.
(8) The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship, unless otherwise specified. Also, the terms "first," "second," and the like in the description of the present invention are used for descriptive purposes only and are not intended to indicate or imply relative importance nor order to be construed.
To better describe the data modification writing method in the embodiment of the present invention, an application scenario of the embodiment of the present invention, that is, an architecture diagram of a distributed file system, is described first, and fig. 4 shows an architecture diagram of a distributed file system provided in the embodiment of the present invention, as shown in fig. 4, the distributed file system includes: client (client), metadata server, persistence layer (persistence layer). The client can be terminal equipment such as a desktop computer, a portable computer, a network server, a palm computer, a mobile phone and a tablet computer. The metadata server is used to coordinate the client and the persistence layer. The persistence layer includes a plurality of disks (or a plurality of disk arrays) for storing data. If the client sends a write request to the metadata server, the metadata server sends the write request to an OSD (not shown in the figure) corresponding to each disk in the persistence layer, so that the OSD writes file data corresponding to the write request into the corresponding disk. If the client sends a read request to the metadata server, the metadata server determines the storage position of the file data corresponding to the read request in the file metadata, informs the client of the storage position, and the client reads the file data from the persistence layer according to the storage position.
The data modification and rewriting process of the current file distribution system is mainly based on a two-stage submission mechanism. The first stage is that the metadata server sends a prepare request to the object storage of the disk by OSD, and then receives a prepare response returned by the OSD. The purpose of the first stage is to determine that the disk is ready, that is, to notify the OSD to suspend the read/write operation on the disk, for example, if other clients need to read the file data on the disk, the OSD does not respond to other clients to prepare for modifying the file data on the disk. And in the second stage, after the disk is ready to work, the metadata server sends a commit request to the OSD and then receives a commit response returned by the OSD. The second phase functions to formally commit the modified data. The specific process is that the OSD releases the file data (hereinafter referred to as original data) on the disk, the metadata server stores the modified data in the cache and informs the storage position of the OSD cache, and the OSD writes the modified data in the cache into the disk. After the modified data is successfully written, the metadata server can respond to the requests of other clients for reading the modified data so as to maintain the normal operation of other client services. As can be seen from the above description, the current data modification process is complex, and service interruption is likely to occur, thereby causing a problem of low service processing efficiency.
In order to solve the technical problem, an embodiment of the present invention provides a data modification and rewriting method. In the method, a metadata server receives a modification writing request sent by a client, wherein the modification writing request carries modification data, and the modification data is used for replacing original data stored at a first position in a first disk; the metadata server determining a second location capable of storing the modified data, the second location being located on the first disk or a second disk; the metadata server sends the modification writing request and second identification information of the second position to an Object Storage Device (OSD), wherein the OSD is a control device for writing the modification data into the second position; the metadata server receives response information fed back by the OSD, wherein the response information is used for indicating that the OSD successfully writes the modified data into the second position; and the metadata server modifies the first identification information of the first position stored in the file metadata into the second identification information, so that the client can read the modified data according to the second identification information. In the embodiment of the present invention, the modified data and the original data are stored in different locations, for example, the modified data and the original data may be stored in different locations in the same disk or may be stored in different disks. When the original data is stored at the first position, the first identification information of the first position is stored in the file metadata, the client can read the original data through the first identification information, and after the modified data is successfully stored at the second position and the first identification information in the file metadata is modified into the second identification information, the client can continue to read the modified data according to the second identification information. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
Fig. 5 is a flowchart illustrating a method for modifying data according to an embodiment of the present invention. The method may be applied to a metadata server of the system architecture shown in fig. 4. As shown in fig. 5, the flow of the method is described as follows:
s501: the metadata server receives a modification write request sent by a client, wherein the modification write request carries modification data, and the modification data is used for replacing original data stored in a first position on a first disk.
As mentioned above, in the process of writing file data (hereinafter referred to as raw data), the raw data is stored in the fragments of the respective disks of the disk array, and after the raw data is stored in the disk array, the file metadata in the metadata server records the file name of the file data, and the storage location of the raw data, for example, the raw data with the file name of file 1 recorded in the file metadata is stored in fragment 1 on disk 1, fragment 2 on disk 2, and fragment 3 on disk 3, respectively. There are various forms of file metadata, and table 1 is an example.
Figure BDA0001514854330000061
TABLE 1
In embodiments of the present invention, file data (raw data) may be divided into different data blocks, such as data blocks 1-3, each of which is stored in a separate disk array in a distributed manner. Table 2 shows another file metadata in which storage locations of data blocks 1 to 3 of file a are recorded, respectively, as shown in table 2.
Figure BDA0001514854330000062
Figure BDA0001514854330000071
TABLE 2
After the metadata server receives a modification write request sent by the client, the modification write request may carry identification information of original data corresponding to the modification data, for example, the modification write request carries file a, that is, the modification data is used to replace the original data with file name of file a, or the modification write request carries identification information of file a and data block 1, that is, the modification data is used to replace data block 1 in file a. Thus, when the metadata server receives a modify write request, it is clear which original data the modified data is to replace.
S502: the metadata server determines a second location where the modified data can be stored.
In the embodiment of the present invention, the file metadata records a current storage state in each disk in the disk array, for example, a remaining storage space of each disk. After receiving the modified data sent by the client, the metadata server may determine a second location capable of storing the modified data in the disk array, where the second location may be located in the same disk as the original data or in a different disk, and will be described below separately.
In the first case, the second location and the first location are located on the same slice of the same disk.
Continuing with the example of Table 1, the storage locations of the original data are: 1-3 magnetic disks and 1-3 sub-slices. If the remaining storage space of the shards 1-3 is larger than the storage space required for modifying the data, e.g., the shard 1 is 1G, and the original data occupies 1-100M, the metadata server may determine that the second location is on the shards 1-3 on the disks 1-3.
In the second case, the second location and the first location are located on different slices of the same disk.
Continuing with the example of Table 1, the storage locations of the original data are: 1-3 magnetic disks and 1-3 sub-slices. If the remaining storage space of the segments 1-3 is less than the storage space required to modify the data, the metadata server may determine a second location in the other segments in the disks 1-3 where the modified data can be stored, e.g., the second location may be in the segment 4 in the disk 1, the segment 5 in the disk 2, or the segment 6 in the disk 3.
In a third case, the second location and the first location are located on different disks.
Continuing with the example of Table 1, the original data storage locations are: 1-3 magnetic disks and 1-3 sub-slices. If the remaining storage space of all of the slices in disks 1-3 is less than the storage space needed to modify the data, the metadata server may determine a second location on the other disks, e.g., the second location is in slice 4-6 of disks 4-6.
The above three cases are described by taking a fragment as a storage unit for each disk in the disk array, that is, the fragment on each disk is divided in advance, for example, each disk is divided into 3 fragments. In the embodiment of the present invention, the storage units on one disk may also be plogs, that is, the storage units in each disk do not need to be divided in advance, but instead, plogs are established to store data when the plogs are needed, for example, only 1 plog is currently in one disk, when the storage space of the plog is full and other data needs to be stored in the disk, one plog may be established again in the remaining storage space on the disk, and then other data is written into the reestablished plog, where the size of the storage space of the plog may be customized to 1G, for example. S402 may also be applied to the case where the storage unit on one disk is plog, which will be described separately below.
In the first case, the second location and the first location are located on the same plog on the same disk.
For example, the original data is stored in plog1 of disk 1, plog2 of disk 2, and plog3 of disk 3, and if the remaining storage space in plog1-3 is sufficient to store the modified data, the second location is determined in plog1-3, i.e., the first location and the second location are located in the same plog. The specific implementation mode is that after receiving a modification write request sent by a client, a metadata server processes the modification write request to obtain 3 additional write (apendlog) requests, each appndlog carries child modification data, the 3 appndlog requests and identification information of plog1-3 are respectively sent to OSD1-3, and OSD1-3 respectively additionally writes the child modification data to plog 1-3.
In the second case, the second location and the first location are located on different plogs on the same disk.
Fig. 6 shows a schematic diagram of a data modification process according to an embodiment of the present invention, in fig. 6, each disk in the disk array takes plog as a storage unit, original data is stored in plog1 of disk 1, plog2 of disk 2, and plog3 of disk 3, and after a metadata server receives a modification write request sent by a client, if there is not enough storage space in plog1-3 to store modified data, the metadata server sends a plog creation (createplog) request to OSD1-3, respectively, where the createplog request is used to create plog, that is, create plog4 on disk 1, create plog5 on disk 2, create plog6 on disk 3, that is, plog4-6 is used to store modified data, that is, a second location and a first location are located in different plogs in the same disk. The metadata server processes the modification writing request to obtain 3 apppenddplog requests, each apppenddplog carries child modification data, the 3 apppenddplog requests and identification information of plog4-6 are respectively sent to OSD1-3, and the OSD1-3 respectively stores the received child modification data in plog 4-6.
In a third case, the second location and the first location are located on different disks.
For example, the original data is stored in plog1 in disk 1, plog2 in disk 2, and plog3 in disk 3, and when the metadata server determines that there is not enough storage space in disks 1-3 to store the modified data, it may send a plog creation request (the original data is stored in OSD 1-3) to OSD4-6, the plog creation request is used to newly create plog in disks 4-6, and OSD4-6 stores the modified data in the newly created plog in disks 4-6.
S503: the metadata server sends the modification write request and second identification information of the second location to the OSD.
In an embodiment of the present invention, after the metadata server determines the second location capable of storing the modification data, the modification write request and the second identification information of the second location may be sent to the OSD, so that the OSD writes the modification data in the second location.
It should be noted that, as mentioned above, one OSD is responsible for one disk, so that after the metadata server determines the second location, the metadata server may send a modification request to the OSD corresponding to the disk where the second location is located, and if the second location is located in N disks, the metadata server may process the modification data to obtain N pieces of sub-modification data; and the metadata server respectively sends the N pieces of sub-modification data and the second identification information to OSD (on screen display) corresponding to the N disks in the disk array, so that each OSD respectively writes the received sub-modification data into a storage position corresponding to the second identification information in the corresponding disk.
For example, the original data storage locations are: the disk 1-3 and the slice 1-3, and the second position determined by the metadata server is located in the slice 4-6 of the disk 4-6, the metadata server may send the slice identification information of the slice 4 and the first sub-modification data to the OSD4 corresponding to the disk 4, send the slice identification information of the slice 5 and the second sub-modification data to the OSD5 corresponding to the disk 5, send the slice identification information of the slice 6 and the third sub-modification data to the OSD6 corresponding to the disk 6, and the OSDs 4-6 write the respectively received sub-modification data into the corresponding slices.
S504: and the metadata server receives response information fed back by the OSD.
In the embodiment of the present invention, after the OSD successfully writes the modification data into the second location, response information indicating that the modification data is successfully written may be fed back to the metadata server. As mentioned above, the modification data is divided into a plurality of sub-modification data stored in different disks, and each disk corresponds to one OSD, so that the metadata server can receive the response information fed back by each disk.
In the embodiment of the present invention, the metadata server may also use the aforementioned EC redundancy mode when storing the modified data, for example, the segment 1 in the disk 1, the segment 2 in the disk 2 are used to store the modified data, and the segment 3 in the disk 3 is used to store the check data. After the OSD1-3 successfully writes the modification data and the verification data into the slices 1-3, respectively, the OSD1-3 may feed back response information to the metadata server, and after the metadata server has collected the response information fed back by each OSD, the file metadata may be modified.
Continuing with the previous example, the metadata server receives response information fed back by the OSDs 1-3, respectively. If the metadata server does not receive response information fed back by one of the OSDs within a preset time, it indicates that the OSD does not successfully write the modified data into the disk, for example, a disk failure, or a disk reset, and in this case, the metadata server may re-determine other locations where the modified data can be stored.
Fig. 7 is a schematic diagram illustrating a data modification process according to an embodiment of the present invention. As shown in fig. 7, after receiving a modification write request sent by a client, a metadata server processes the modification write request into 3 apendlog requests, and sends the 3 apendlog requests to OSD1-3 capable of storing modification data, and if the metadata server does not receive response information fed back by OSD3 within a preset time, it can be determined that a disk 3 is failed, so in this case, the metadata server can redetermine other disks capable of storing modification data, such as disks 4-6, then send a createplog request to OSD4-6, newly create plogs in disks 4-6, then send apendlog requests to OSD4-6, that is, the modification data is dispersedly stored in the newly created plogs in disks 4-6.
S505: the metadata server modifies the file metadata.
In the embodiment of the invention, the storage position information of the original data is recorded in the file metadata, so that the client can conveniently read the original data according to the storage position information. After the modified data is successfully written into the disk, if the file metadata is not modified, the original data is still read by the client, so the metadata server needs to modify the file metadata, and for the three cases mentioned in S402 above for determining the second position, the modification manner for the file metadata is different in each case, which will be described separately below.
For the first case, since the original data and the modified data are stored in the same slice (or plog) of the same disk, the first identification information of the first position of the original data recorded in the file metadata is a disk identification, a slice identification, an offset identification. The file metadata comprises disk identification: 1-3 of a disk, and fragment identification: 1-3 of fragments and offset identification: 1-10M, 1-10M and 1-10M. When the modified data is successfully written, modifying the first identification information in the file metadata into second identification information, namely the file metadata comprises a disk identification: 1-3 of a disk, and fragment identification: 1-3 of fragments and offset identification: 11-20M, 11-20M and 11-20M. As another example, the original data is stored in disk 1-3, plog1-3, offsets 1-10M, and modified data is stored in disk 1-3, plog1-3, offsets 11-20M, and 11-20M.
In this case, when the metadata server modifies the file metadata, the disk identifier and the fragment identifier do not need to be modified, and only the offset identifier needs to be modified.
For the second case described above, the original data and the modified data are stored in different slices (or plogs) of the same disk. For example, before modification, the file metadata includes a disk identifier: 1-3 of a disk, and fragment identification: 1-3 of fragments and offset identification: 1-10M, 1-10M and 1-10M. When the modified data is successfully written, modifying the first identification information in the file metadata into second identification information, namely the file metadata comprises a disk identification: 1-3 of a disk, and fragment identification: 4-6 of fragments and offset identification: 1-10M, 1-10M and 1-10M. As another example, the storage locations for the raw data are disks 1-3, plog1-3, offsets 1-10M, 1-10M. The storage locations for the modified data are disks 1-3, plog1-3, offsets 11-20M, 11-20M. Table 3 lists the modified file metadata, and Table 3 exemplifies that the file data of file A is not divided into different data blocks.
Figure BDA0001514854330000091
TABLE 3
In this case, when modifying the file metadata, the metadata server only needs to modify the fragment identifier (plog identifier) and the offset identifier.
For the third case described above, the original and modified data are stored in different slices (or plogs) on different disks. For example, before modification, the file metadata includes a disk identifier: 1-3 of a disk, and fragment identification: 1-3 of fragments and offset identification: 1-10M, 1-10M and 1-10M. When the modified data is successfully written, modifying the first identification information in the file metadata into second identification information, namely the file metadata comprises a disk identification: 4-6 of the disk, and fragment identification: 4-6 of fragments and offset identification: 1-10M, 1-10M and 1-10M. As another example, the storage locations for the raw data are disks 1-3, plog1-3, offsets 1-10M, 1-10M. The storage locations for the modified data are disks 4-6, plog4-6, offsets 11-20M, 11-20M.
In this case, the metadata server may modify the disk identifier, the fragment identifier (or plog identifier), and the offset identifier when modifying the file metadata.
It should be noted that, as shown in fig. 3, in the prior art, since the modified data and the original data are stored in the same location, when the original data in the slices 1-6 is modified, the modified data is required to replace the original data stored in the slices 1-6, and if the data in the slices 4-6 is successfully modified, but the data in the slices 1-3 is not necessarily successfully modified, not only the original data is destroyed, but also the modified data cannot be recovered. However, in the method provided in the embodiment of the present application, because the original data and the modified data are stored in different locations, if the modified data are not successfully written into the disk, the metadata server does not modify the file metadata, and the client can continue to read the original data according to the file metadata, that is, the original data are not damaged.
In the embodiment of the present invention, after the metadata server modifies the file metadata, the client may read the modified data according to the modified file metadata, that is, read the modified data according to the second identification information of the second location. The client that reads the modification data may be a client that modifies file data, or may be another client, for example, after the client 1 modifies a file, the client 2 may read the modification data and execute a service.
In the embodiment of the present invention, after the file metadata is modified, the metadata server may send a release request to the OSD, where the release request is used to determine whether to store the original data at the first location, and since the first identification information of the first location recorded in the file metadata is modified into the second identification information, the original data stored at the first location is not read by the client, the OSD may release the original data stored at the first location, and after the release is successful, send response information indicating that the release is successful to the metadata server, where the first location may be used to store other data.
As can be seen from the above description, in the embodiment of the present invention, the modified data and the original data are stored in different locations, for example, the modified data and the original data may be stored in different locations in the same disk or may be stored in different disks. When the original data is stored at the first position, the first identification information of the first position is stored in the file metadata, the client can read the original data through the first identification information, and after the modified data is successfully stored at the second position and the first identification information in the file metadata is modified into the second identification information, the client can continue to read the modified data according to the second identification information. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
The following describes the apparatus provided by the embodiment of the present invention with reference to the drawings.
Fig. 8 shows a schematic configuration of a data modification and rewriting apparatus 800. The data modification writing apparatus 800 may implement the functions of the metadata server referred to above. The data modification writing apparatus 800 may include a receiving unit 801, a processing unit 802, and a transmitting unit 803. Among other things, the receiving unit 801 may be used to perform S501, S504 in the embodiment shown in fig. 5, and/or other processes for supporting the techniques described herein. Processing unit 802 may be used to perform S502, S505 in the embodiment shown in fig. 5, and/or other processes for supporting the techniques described herein. The sending unit 803 may be used to perform S503 in the embodiment shown in fig. 4, and/or other processes for supporting the techniques described herein. All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.
It should be noted that, in the embodiment of the present invention, the data modification writing device is shown in the form of a functional unit. As used herein, without limitation, the term "unit" may refer to an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.
In a simple embodiment, it will be appreciated by those skilled in the art that the data modification writing apparatus 800 may also be implemented by a structure as shown in fig. 9.
As shown in fig. 9, the data modification and rewriting apparatus 900 may include: memory 901, processor 902, and bus 903. The processor 901 and the processor 902 may be connected by a bus 903. Wherein, the memory 901 is used for storing computer executable instructions, and when the data modification writing device 900 is in operation, the processor 902 executes the computer executable instructions stored in the memory 901, so as to make the data modification writing device 900 execute the data modification writing method provided by the embodiment shown in fig. 5. For a specific data modification writing method, reference may be made to the above description and the related description in the drawings, and details are not repeated here.
In the embodiment of the present invention, the processor 902 may be a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on chip (SoC), a Central Processing Unit (CPU), a Network Processor (NP), a digital signal processing circuit (DSP), a Microcontroller (MCU), or a Programmable Logic Device (PLD) or other integrated chips. The bus 803 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 903 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 9, but this does not indicate only one bus or one type of bus.
In summary, the embodiments of the present invention provide a data modification method and apparatus. In the method, a metadata server receives a modification writing request sent by a client, wherein the modification writing request carries modification data, and the modification data is used for replacing original data stored at a first position in a first disk; the metadata server determining a second location capable of storing the modified data, the second location being located on the first disk or a second disk; the metadata server sends the modification writing request and second identification information of the second position to an Object Storage Device (OSD), wherein the OSD is a control device for writing the modification data into the second position; the metadata server receives response information fed back by the OSD, wherein the response information is used for indicating that the OSD successfully writes the modified data into the second position; and the metadata server modifies the first identification information of the first position stored in the file metadata into the second identification information, so that the client can read the modified data according to the second identification information. In the embodiment of the present invention, the modified data and the original data are stored in different locations, for example, the modified data and the original data may be stored in different locations in the same disk or may be stored in different disks. When the original data is stored at the first position, the first identification information of the first position is stored in the file metadata, the client can read the original data through the first identification information, and after the modified data is successfully stored at the second position and the first identification information in the file metadata is modified into the second identification information, the client can continue to read the modified data according to the second identification information. In this way, the modified data and the original data are stored in different positions, and a data modification process can be executed in the process of reading the original data, i.e. the service is not easy to interrupt, which is beneficial to improving the service processing efficiency.
In the above-described embodiments of the invention, may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above embodiments are only used to describe the technical solutions of the embodiments of the present invention in detail, but the above description of the embodiments is only used to help understanding the method and the core idea of the embodiments of the present invention, and should not be construed as limiting the present application. Those skilled in the art can easily conceive of changes or substitutions within the technical scope of the embodiments of the present invention, and all such changes or substitutions are intended to be included within the scope of the embodiments of the present invention.

Claims (12)

1. A data modification method, comprising:
the metadata server receives a modification writing request sent by a client, wherein the modification writing request carries modification data, and the modification data is used for replacing original data stored at a first position in a first disk;
the metadata server determining a second location capable of storing the modified data, the second location being located on the first disk or a second disk;
the metadata server sends the modification writing request and second identification information of the second position to an Object Storage Device (OSD), wherein the OSD is a control device for writing the modification data into the second position;
the metadata server receives response information fed back by the OSD, wherein the response information is used for indicating that the OSD successfully writes the modified data into the second position;
the metadata server modifies first identification information of the first position stored in file metadata into second identification information so that the client can read the modified data according to the second identification information;
the metadata server sending a release request to the OSD, the release request being for releasing the original data stored at the first location;
and the metadata server receives response information which is fed back by the OSD and used for indicating the successful release.
2. The method of claim 1, wherein the first disk comprises a plurality of partitions, and the second location and the first location are located in a same partition or in different partitions on the first disk, respectively;
if the second position and the first position are located in the same fragment on the first disk respectively, the first identification information includes a disk identifier of the first disk, a fragment identifier of the same fragment, and a first storage address of the first position in the same fragment; the second identification information comprises a disk identification of the first disk, a fragment identification of the same fragment and a second storage address of a second position in the same fragment; or
If the first location is located in a first fragment on the first disk, the second location is located in a second fragment on the first disk, and the first identification information includes a disk identification of the first disk, a fragment identification of the first fragment, and a first storage address of the first location in the first fragment; the second identification information includes a disk identification of the first disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
3. The method of claim 2, wherein the second memory address is an address offset from the first memory address.
4. The method of claim 1, wherein the first location is in a first slice on the first disk and the second location is in a second slice on the second disk;
the first identification information comprises a disk identification of the first disk, a fragment identification of the first fragment and a first storage address of a first position in the first fragment; the second identification information includes a disk identification of the second disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
5. The method of claim 1, wherein the metadata server sends the modification request and the second identification information of the second location to the OSD, comprising:
the metadata server processes the modified data to obtain N pieces of sub-modified data;
the metadata server respectively sends the N pieces of sub-modification data and the second identification information to OSD (on screen display) corresponding to the N disks in the disk array, so that each OSD respectively writes the received sub-modification data into a storage position corresponding to the second identification information in the corresponding disk; the disk array comprises N stacked disks, and the storage positions corresponding to the second identification information in each disk are the same; n is an integer greater than or equal to 2;
the metadata server receives response information fed back by the OSD, and the response information comprises:
and the metadata server collects the response information fed back by each OSD.
6. A data modification apparatus, comprising:
a receiving unit, configured to receive a modification write request sent by a client, where the modification write request carries modification data, and the modification data is used to replace original data stored in a first location in a first disk;
a processing unit, configured to determine a second location capable of storing the modified data, where the second location is located on the first disk or a second disk;
a sending unit, configured to send the modification write request and the second identification information of the second location to an object storage device OSD, where the OSD is a control device configured to write the modification data into the second location;
the receiving unit is further configured to receive response information fed back by the OSD, where the response information is used to indicate that the OSD successfully writes the modification data into the second location;
the processing unit is further configured to modify first identification information of the first location stored in file metadata to the second identification information, so that the client can read the modified data according to the second identification information;
the sending unit is further configured to: sending a release request to the OSD, the release request being for releasing the original data stored at the first location;
the receiving unit is further configured to receive response information, which is fed back by the OSD and used for indicating that the release is successful.
7. The apparatus of claim 6, wherein the first disk comprises a plurality of partitions, and the second location and the first location are located in a same partition or in different partitions on the first disk;
if the second position and the first position are located in the same fragment on the first disk respectively, the first identification information includes a disk identifier of the first disk, a fragment identifier of the same fragment, and a first storage address of the first position in the same fragment; the second identification information comprises a disk identification of the first disk, a fragment identification of the same fragment and a second storage address of a second position in the same fragment; or
If the first location is located in a first fragment on the first disk, the second location is located in a second fragment on the first disk, and the first identification information includes a disk identification of the first disk, a fragment identification of the first fragment, and a first storage address of the first location in the first fragment; the second identification information includes a disk identification of the first disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
8. The apparatus of claim 7, wherein the second memory address is an address offset from the first memory address.
9. The apparatus of claim 6, wherein the first location is located in a first slice on the first disk and the second location is located in a second slice on the second disk;
the first identification information comprises a disk identification of the first disk, a fragment identification of the first fragment and a first storage address of a first position in the first fragment; the second identification information includes a disk identification of the second disk, a fragment identification of the second fragment, and a second storage address of a second location in the second fragment.
10. The apparatus of claim 6, wherein the sending unit, when being configured to send the modification request and the second identification information of the second location to the OSD, is specifically configured to:
processing the modified data to obtain N pieces of modified data;
respectively sending the N pieces of sub-modification data and the second identification information to OSD (on screen display) corresponding to the N disks in the disk array, so that each OSD respectively writes the received sub-modification data into a storage position corresponding to the second identification information in the corresponding disk; the disk array comprises N stacked disks, and the storage positions corresponding to the second identification information in each disk are the same; n is an integer greater than or equal to 2;
when the receiving unit is configured to receive the response information fed back by the OSD, the receiving unit is specifically configured to:
and collecting the response information fed back by each OSD.
11. A data modification apparatus, comprising: a processor and a memory;
the memory is used for storing computer execution instructions;
the processor, configured to call and execute the instructions stored in the memory, to cause the data modification apparatus to perform the method according to any one of claims 1 to 5.
12. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1-5.
CN201711376932.6A 2017-12-19 2017-12-19 Data modification and rewriting method and device Active CN108459824B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711376932.6A CN108459824B (en) 2017-12-19 2017-12-19 Data modification and rewriting method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711376932.6A CN108459824B (en) 2017-12-19 2017-12-19 Data modification and rewriting method and device

Publications (2)

Publication Number Publication Date
CN108459824A CN108459824A (en) 2018-08-28
CN108459824B true CN108459824B (en) 2021-05-04

Family

ID=63221166

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711376932.6A Active CN108459824B (en) 2017-12-19 2017-12-19 Data modification and rewriting method and device

Country Status (1)

Country Link
CN (1) CN108459824B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109002266A (en) * 2018-08-29 2018-12-14 郑州云海信息技术有限公司 A method of promoting metadata reliability in traditional RAID group
CN110764949A (en) * 2019-09-29 2020-02-07 北京浪潮数据技术有限公司 Hard disk replacement method, hard disk replacement device, and storage medium
CN113535068B (en) * 2020-04-21 2024-09-17 华为技术有限公司 Data reading method and system
CN112866409B (en) * 2021-02-08 2022-10-11 深圳市杉岩数据技术有限公司 Object transmission and collection method based on object storage fragment uploading and terminal equipment
CN113553215B (en) * 2021-06-25 2024-10-18 济南浪潮数据技术有限公司 Erasure code data recovery optimization method and device based on environment information
CN113630441B (en) * 2021-07-12 2023-12-08 中国科学院国家天文台 Low-power consumption distributed storage system and terminal equipment
CN114301790B (en) * 2021-12-02 2024-02-13 无线生活(北京)信息技术有限公司 Data transmission method and device
CN117950597B (en) * 2024-03-22 2024-06-18 浙江大华技术股份有限公司 Data modification writing method, data modification writing device, and computer storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02247717A (en) * 1989-03-22 1990-10-03 Matsushita Electric Ind Co Ltd Method and device for duplicating auxiliary storage medium
JP2006119688A (en) * 2004-10-19 2006-05-11 Hitachi Ltd System and method for controlling update of storage device
CN102520883A (en) * 2011-12-12 2012-06-27 杭州华三通信技术有限公司 Data access method and device thereof
CN103176909A (en) * 2011-12-26 2013-06-26 中国银联股份有限公司 Service processing method and service processing system
CN104407806A (en) * 2014-10-09 2015-03-11 杭州华为企业通信技术有限公司 Method and device for revising hard disk information of redundant array group of independent disk (RAID)
CN105022590A (en) * 2014-04-30 2015-11-04 腾讯科技（深圳）有限公司 Shared memory data access method and apparatus
CN106030501A (en) * 2014-09-30 2016-10-12 株式会社日立制作所 Distributed storage system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02247717A (en) * 1989-03-22 1990-10-03 Matsushita Electric Ind Co Ltd Method and device for duplicating auxiliary storage medium
JP2006119688A (en) * 2004-10-19 2006-05-11 Hitachi Ltd System and method for controlling update of storage device
CN102520883A (en) * 2011-12-12 2012-06-27 杭州华三通信技术有限公司 Data access method and device thereof
CN103176909A (en) * 2011-12-26 2013-06-26 中国银联股份有限公司 Service processing method and service processing system
CN105022590A (en) * 2014-04-30 2015-11-04 腾讯科技（深圳）有限公司 Shared memory data access method and apparatus
CN106030501A (en) * 2014-09-30 2016-10-12 株式会社日立制作所 Distributed storage system
CN104407806A (en) * 2014-10-09 2015-03-11 杭州华为企业通信技术有限公司 Method and device for revising hard disk information of redundant array group of independent disk (RAID)

Also Published As

Publication number Publication date
CN108459824A (en) 2018-08-28

Similar Documents

Publication Publication Date Title
CN108459824B (en) Data modification and rewriting method and device
US11281531B2 (en) Serial storage node processing of data functions
US20210117441A1 (en) Data replication system
US10467246B2 (en) Content-based replication of data in scale out system
US10127233B2 (en) Data processing method and device in distributed file storage system
US9740416B2 (en) Essential metadata replication
US20190245919A1 (en) Method and apparatus for information processing, server and computer readable medium
US11321291B2 (en) Persistent version control for data transfer between heterogeneous data stores
KR101231563B1 (en) Real time data replication
US10146786B2 (en) Managing deduplication in a data storage system using a Bloomier filter data dictionary
US10331625B2 (en) Managing sequential data store
US20150213100A1 (en) Data synchronization method and system
CN108733311B (en) Method and apparatus for managing storage system
US11221785B2 (en) Managing replication state for deleted objects
WO2015010327A1 (en) Data sending method, data receiving method and storage device
US11567899B2 (en) Managing dependent delete operations among data stores
US10997031B2 (en) System and method for log metadata automatic recovery on dual controller storage system
CN104794119A (en) Middleware message storage and transmission method and system
US10860224B2 (en) Method and system for delivering message in storage system
US10990324B2 (en) Storage node processing of predefined data functions
US11210236B2 (en) Managing global counters using local delta counters
US11055018B2 (en) Parallel storage node processing of data functions
US11314593B2 (en) Storage node processing of data functions using overlapping symbols
KR20160062683A (en) COMPUTING SYSTEM WITH heterogeneous storage AND METHOD OF OPERATION THEREOF
CN116662603B (en) Time shaft control method and system based on kafka, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20220214

Address after: 550003 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an new area, Guiyang City, Guizhou Province

Patentee after: Huawei Cloud Computing Technologies Co.,Ltd.

Address before: No. 127, Jinye Road, high tech Zone, Xi'an, Shaanxi 710075

Patentee before: Huawei Technologies Co.,Ltd, Xi'an

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20221209

Address after: 518129 Huawei Headquarters Office Building 101, Wankecheng Community, Bantian Street, Gangqu District, Shenzhen, Guangdong

Patentee after: Shenzhen Huawei Cloud Computing Technology Co.,Ltd.

Address before: 550003 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an new area, Guiyang City, Guizhou Province

Patentee before: Huawei Cloud Computing Technologies Co.,Ltd.