CN109634526B - Data operation method based on object storage and related device - Google Patents
Data operation method based on object storage and related device Download PDFInfo
- Publication number
- CN109634526B CN109634526B CN201811513259.0A CN201811513259A CN109634526B CN 109634526 B CN109634526 B CN 109634526B CN 201811513259 A CN201811513259 A CN 201811513259A CN 109634526 B CN109634526 B CN 109634526B
- Authority
- CN
- China
- Prior art keywords
- request
- operation sub
- version number
- object storage
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0634—Configuration or reconfiguration of storage systems by changing the state or mode of one or more devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention discloses a data operation method based on object storage, which can receive a plurality of operation sub-requests and target version numbers corresponding to the operation sub-requests, wherein the operation requests corresponding to target objects are directly divided into the plurality of operation sub-requests through erasure codes at a client, and then whether the number of the current version numbers of the corresponding target objects in a plurality of object storage units, which are the same as the target version numbers, is greater than the preset number is judged; greater than means that the operation sub-request can be executed at this time; and after the operation sub-request is successfully executed, the current version number of the corresponding target object in the object storage unit is updated. The main OSD can be deleted in the distributed storage system through the process, so that the performance of the distributed storage system is effectively improved; meanwhile, the correct data and the correct operation can be guaranteed through the process of comparing the version numbers. The invention also provides a data operation device, data operation equipment and a computer readable storage medium, and the beneficial effects are also achieved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data operation method based on object storage, a data operation device based on object storage, and a computer-readable storage medium.
Background
With the continuous progress of science and technology in recent years, computers are almost full of the aspects of people such as daily life. Moreover, with the continuous progress of computer technology, the technology related to the distributed storage system makes a great breakthrough.
Compared with block storage and file storage, the object storage is used as a novel network storage architecture, and has the characteristics of SAN high-speed direct access disks and the distributed sharing characteristics of NAS. In the distributed storage system based on the object storage, the basic unit of the data storage is the object, and when the client operates the data, the basic unit corresponding to the operation is also the object.
In a distributed Storage system based on Object Storage, an Object is usually divided into a plurality of data blocks by erasure codes for redundancy at the present stage, and the plurality of data blocks are stored in different Object Storage units (OSD) respectively for Storage, so as to perform redundancy on the Object while storing, thereby preventing data loss.
In the prior art, a client sends data to a main OSD of a distributed storage system, and then the main OSD blocks the data by an erasure code, and sends the divided data blocks to corresponding OSDs. However, in the prior art, there is a problem that the performance of the distributed storage system is generally low, so how to improve the operation performance of the distributed storage system based on object storage is a problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The invention aims to provide a data operation method based on object storage, which can improve the operation performance of a distributed storage system based on object storage; another object of the present invention is to provide a data manipulation apparatus based on object storage, a data manipulation device based on object storage, and a computer readable storage medium, which can improve the operation performance of a distributed storage system based on object storage.
In order to solve the above technical problem, the present invention provides a data operation method based on object storage, including:
receiving a plurality of operation sub-requests sent by a client and target version numbers corresponding to the operation sub-requests; the operation sub-requests are a plurality of operation sub-requests into which the client divides the operation request corresponding to the target object through erasure codes;
sending a plurality of the operation sub-requests to corresponding object storage units;
judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is greater than a preset number or not; if yes, executing the operation sub-request;
and updating the current version number corresponding to the target object in the object storage unit after the operation sub-request is successfully executed.
Optionally, after determining whether the number of the current version number corresponding to the target object in the plurality of object storage units, which is the same as the target version number, is greater than a preset number, the method further includes:
and if not, not executing the operation sub-request.
Optionally, after the executing the operation sub-request, the method further includes:
replacing the current version number of the object storage unit with the target version number, wherein the current version number of the object storage unit is different from the target version number;
when the operation sub-request is successfully executed, updating the current version number of the target object in the object storage unit includes:
and after the operation sub-request is successfully executed, adding 1 to the current version number of the corresponding target object in the object storage unit.
Optionally, after the executing the operation sub-request, the method further includes:
when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with the current version number different from the target version number as an effective range;
after the determining whether the number of the current version number corresponding to the target object in the plurality of object storage units is equal to the target version number is greater than a preset number, the method further includes:
when the operation sub-request is a read operation sub-request, judging whether the data corresponding to the read operation sub-request is located in the effective range; and if so, executing the step of executing the operation sub-request.
Optionally, after the determining whether the data corresponding to the sub-request for read operation is located in the valid range, the method further includes:
and if not, redundantly reading corresponding data from the rest of the object storage units.
The invention also provides a data operation device based on object storage, which comprises:
a receiving module: the system comprises a client, a server and a server, wherein the server is used for receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests; the operation sub-requests are a plurality of operation sub-requests into which the client divides the operation request corresponding to the target object through erasure codes;
a sending module: the operation sub-requests are sent to corresponding object storage units;
a version number judging module: the device is used for judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is larger than a preset number or not; if yes, operating the operation execution module;
the operation execution module: for executing the operation sub-request;
and a version number updating module: and the operation sub-request is used for updating the current version number of the target object in the object storage unit after the operation sub-request is successfully executed.
Optionally, the version number determining module is further configured to:
judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is greater than a preset number or not; if not, the operation ends the module;
the end module is configured to:
the operation sub-request is not executed.
Optionally, the apparatus further comprises:
a recording module: when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with the current version number different from the target version number as an effective range;
the device further comprises:
an effective range judging module: the data processing device is used for judging whether the data corresponding to the read operation sub-request is located in the effective range or not when the operation sub-request is a read operation sub-request; and if so, operating the operation execution module.
The invention also provides a data operation device based on object storage, which comprises:
a memory: for storing a computer program;
a processor: for implementing the steps of the object store based data manipulation method of any one of the above when executing said computer program.
The invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method of operating on data based on object storage as defined in any one of the preceding claims.
The data operation method based on object storage provided by the invention can receive a plurality of operation sub-requests which are obtained by directly dividing operation requests corresponding to target objects into the operation sub-requests through erasure codes at a client side and target version numbers corresponding to the operation sub-requests, and send the operation sub-requests to corresponding object storage units; in order to ensure the correctness of data and the correctness of operation, whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not is judged; if yes, the previous step is correctly executed, the data is normal at the moment, and the operation sub-request can be executed; and after the operation sub-request is successfully executed, updating the current version number of the corresponding target object in the object storage unit. By dividing the operation request into a plurality of operation sub-requests directly through the erasure code in the client, the main OSD can be deleted in the distributed storage system, the steps of receiving the operation request in the distributed storage system, splitting the operation request through the erasure code and the like are avoided, and therefore the performance of the distributed storage system is effectively improved; meanwhile, the correct execution of data and operation can be guaranteed through the comparison and other processes of the version numbers, so that the operation performance of the distributed storage system based on object storage is improved.
The invention also provides a data operation device based on object storage, a data operation device based on object storage and a computer readable storage medium, which also have the beneficial effects and are not repeated herein.
Drawings
In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
Fig. 1 is a flowchart of a data operation method based on object storage according to an embodiment of the present invention;
fig. 2 is a flowchart of a specific data operation method based on object storage according to an embodiment of the present invention;
FIG. 3 is a flowchart of another specific method for operating data based on object storage according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating an embodiment of a data manipulation device based on object storage according to the present invention;
fig. 5 is a block diagram of a data operating apparatus based on object storage according to an embodiment of the present invention.
Detailed Description
The core of the invention is to provide a data operation method based on object storage. In the prior art, a client generally sends data to a main OSD of a distributed storage system, and then the main OSD blocks the data by an erasure code, and sends the divided data blocks to corresponding OSDs. This will make distributed storage system need to set up the main OSD structurally, need to carry on the receiving operation request in the procedure, carry on steps such as splitting to the operation request through the erasure code, thus make distributed storage system have lower performance.
The data operation method based on object storage provided by the invention receives a plurality of operation sub-requests which are obtained by directly dividing operation requests corresponding to target objects by erasure codes at a client and target version numbers corresponding to the operation sub-requests, and sends the operation sub-requests to corresponding object storage units; in order to ensure the correctness of data and the correctness of operation, whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not is judged; if yes, the previous step is correctly executed, the data is normal at the moment, and the operation sub-request can be executed; and after the operation sub-request is successfully executed, updating the current version number of the corresponding target object in the object storage unit. By dividing the operation request into a plurality of operation sub-requests directly through the erasure code in the client, the main OSD can be deleted in the distributed storage system, the steps of receiving the operation request in the distributed storage system, splitting the operation request through the erasure code and the like are avoided, and therefore the performance of the distributed storage system is effectively improved; meanwhile, the correct execution of data and operation can be guaranteed through the comparison and other processes of the version numbers, so that the operation performance of the distributed storage system based on object storage is improved.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating a data operation method based on object storage according to an embodiment of the present invention.
Referring to fig. 1, in an embodiment of the present invention, the method for operating data based on object storage includes:
s101: and receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests.
In the embodiment of the present invention, the operation sub-requests are a plurality of operation sub-requests into which the operation request corresponding to the target object is divided by the client through an erasure code.
It should be noted that the data operation method provided in the embodiment of the present invention is particularly suitable for a distributed file system based on object storage, and particularly relates to a specific operation flow for an operation request in the distributed file system. In the embodiment of the present invention, when a user needs to operate a target object at a client, an operation request corresponding to the target object is generated at the client, but in the embodiment of the present invention, the operation request is directly divided and redundant at the client by an Erasure Code (EC) to generate a plurality of operation sub-requests, and the operation sub-requests also correspond to the target object. Meanwhile, the client sends the operation sub-requests and the target version numbers corresponding to the operation sub-requests to the distributed storage system. For the details of the erasure codes, reference may be made to the prior art, and further description is omitted here.
In this step, a plurality of operation sub-requests sent by the client and the target version numbers corresponding to the operation sub-requests are received. The target version number actually corresponds to the target object, and the target version number is a version number that the target object should currently be at. In the embodiment of the present invention, when an object storage unit (OSD) is successfully operated, the version number corresponding to the operated object in the object storage unit is updated as the current version number of the object.
S102: and sending the plurality of operation sub-requests to the corresponding object storage units.
In this step, the operation sub-request acquired in S101 is sent to the corresponding object storage unit. For the specific sending process, reference may be made to the prior art, and details are not described herein. It should be noted that, after this step, it is usually necessary to determine whether the object storage unit receives the corresponding operation sub-request; if a certain object storage unit does not receive the corresponding operation sub-request, the operation sub-request corresponding to the object storage unit needs to be sent to the object storage unit from the new object storage unit. If the object storage units all receive the corresponding operation sub-requests, the subsequent steps can be executed.
S103: judging whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not; if not, executing S104; if yes, go to step S105.
In the embodiment of the present invention, the current version number of the target object corresponding to the data block stored in each object storage unit is recorded in advance. In this step, the current version number of the corresponding target object in the plurality of object storage units receiving the operation sub-request is usually obtained first; and comparing the obtained current version number with the target version number received in the step S101, and counting the comparison result.
Specifically, when the current version number of the corresponding target object in one object storage unit is the same as the target version number, it means that the previous operation on the target object in the object storage unit is successful, and at the same time, it means that the data stored in the object storage unit is correct and complete; when the current version number of the corresponding target object in one object storage unit is different from the target version number, the operation on the object storage unit for the target object is failed before, and the data stored in the object storage unit at the moment is wrong.
In this step, when the number of the current version number of the corresponding target object in the plurality of object storage units, which is the same as the target version number, is greater than the preset number, it means that the data of the target object is basically complete, and the target object can be continuously operated; when the number of the current version number of the corresponding target object in the plurality of object storage units, which is the same as the target version number, is less than or equal to the preset number, it means that the data of the target object is incomplete, and the target object cannot be continuously operated. The specific numerical values of the preset number can be set according to actual conditions, and are not particularly limited herein.
S104: the operation sub-request is not executed.
In this step, if the data of the target object is basically incomplete, the operation sub-request is not executed in this step, that is, the operation is cancelled.
S105: the operation sub-request is executed.
In this step, each object storage unit that receives the operation sub-request executes the corresponding operation sub-request to complete the operation on the target object.
S106: and when the operation sub-request is successfully executed, updating the current version number of the corresponding target object in the object storage unit.
In this step, after the operation sub-request is successfully executed, the current version number of the corresponding target object in the object storage unit is updated to prove that this operation is successfully executed, and the data stored in the object storage unit is complete, that is, the data of the target object is complete, so that the operation can be sequentially executed when the operation is executed on the target object next time. The updated version number of the target object, that is, the target version number received from the client when the target object is operated again next time, is described.
The detailed updating steps will be described in detail in the following embodiments of the present invention, and will not be described herein again.
The data operation method based on object storage provided by the embodiment of the invention can receive a plurality of operation sub-requests which are obtained by directly dividing operation requests corresponding to target objects into by erasure codes at a client and target version numbers corresponding to the operation sub-requests, and send the operation sub-requests to corresponding object storage units; in order to ensure the correctness of data and the correctness of operation, whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not is judged; if yes, the previous step is correctly executed, the data is normal at the moment, and the operation sub-request can be executed; and after the operation sub-request is successfully executed, updating the current version number of the corresponding target object in the object storage unit. By dividing the operation request into a plurality of operation sub-requests directly through the erasure code in the client, the main OSD can be deleted in the distributed storage system, the steps of receiving the operation request in the distributed storage system, splitting the operation request through the erasure code and the like are avoided, and therefore the performance of the distributed storage system is effectively improved; meanwhile, the correct execution of data and operation can be guaranteed through the comparison and other processes of the version numbers, so that the operation performance of the distributed storage system based on object storage is improved.
The following embodiments of the present invention will describe a specific process of a data operation method based on object storage in detail.
Referring to fig. 2, fig. 2 is a flowchart illustrating a specific data operation method based on object storage according to an embodiment of the present invention.
Referring to fig. 2, in an embodiment of the present invention, the method for operating data based on object storage includes:
s201: and receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests.
S202: and sending the plurality of operation sub-requests to the corresponding object storage units.
S203: judging whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not; if not, executing S204; if yes, go to S205.
S204: the operation sub-request is not executed.
S205: the operation sub-request is executed.
In the embodiment of the present invention, the above-mentioned S201 to S205 are substantially the same as the above-mentioned S101 to S105 in the embodiment of the present invention, and please refer to the above-mentioned embodiment for details, which will not be described herein again.
S206: and replacing the current version number of the object storage unit with the target version number, wherein the current version number of the object storage unit is different from the target version number.
This step needs to be executed after S203, but this step may not be executed after S205 any more, that is, S206 and S205 may be executed in parallel in the embodiment of the present invention.
When this step is executed, it means that the result of the determination in S203 is yes, which means that the data stored in most of the object storage units is correct and complete at this time, but there may be a small portion of the data stored in the object storage units that is incomplete; that is, the data of the target object is substantially complete at this time, but there may be a small amount of errors in the data stored in the object storage unit. Of course, since the target object is stored after being redundantly processed by the error correction code in the embodiment of the present invention, data errors stored in a small number of object storage units do not cause much influence.
However, in order to ensure the correct execution of the process provided in the embodiment of the present invention, in this step, the current version numbers of the corresponding target objects in the plurality of object storage units are unified, specifically, the current version number of the object storage unit whose current version number is different from the target version number is replaced with the target version number, that is, the current version number of the corresponding target object in the plurality of object storage units is replaced with the latest current version number, that is, the target version number.
S207: and after the operation sub-request is successfully executed, adding 1 to the current version number of the corresponding target object in the object storage unit.
In this step, after the operation sub-request is successfully executed, the current version number of the corresponding target object in the object storage unit is updated in a manner of adding 1 to the current version number of the corresponding target object in the object storage unit.
The rest of the steps have been described in detail in the above embodiments of the present invention, and are not described herein again.
In the data operation method based on object storage provided by the embodiment of the present invention, when the number of the current version number of the corresponding target object in the plurality of object storage units, which is the same as the target version number, is greater than the preset number, the current version number of the corresponding target object in the plurality of object storage units is unified into the latest version number; and when the operation sub-request is successfully executed, adding 1 to the current version number of the corresponding target object in the object storage unit. Through the steps, the situation that the data stored in the object storage unit is normal but the wrong version numbers are more and more can be avoided; and meanwhile, whether the operation is successfully executed in each object storage unit can be distinguished through the version number.
The following embodiments of the present invention will describe a specific process of a data operation method based on object storage in detail.
Referring to fig. 3, fig. 3 is a flowchart illustrating another specific object-based storage data manipulation method according to an embodiment of the present invention.
Referring to fig. 3, in an embodiment of the present invention, the method for operating data based on object storage includes:
s301: and receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests.
S302: and sending the plurality of operation sub-requests to the corresponding object storage units.
S303: judging whether the number of the current version numbers of the corresponding target objects in the plurality of object storage units, which are the same as the target version number, is greater than a preset number or not; if not, executing S304; if yes, S305 is performed.
S304: the operation sub-request is not executed.
In the embodiment of the present invention, the above-mentioned S301 to S304 are substantially the same as the above-mentioned S101 to S104, and S201 to S204 in the embodiment of the present invention, and the detailed contents refer to the above-mentioned embodiment, which is not repeated herein.
S305: when the operation sub-request is a read operation sub-request, judging whether data corresponding to the read operation sub-request is located in an effective range; if not, executing S306; if yes, go to step S307.
In this step, the specific operation type of the operation sub-request is determined. When the operation sub-request is a read operation sub-request, that is, when the operation request generated by the client is a read operation request, it is continuously determined whether data corresponding to the operation sub-request is in the valid range. Details regarding the effective range will be described in detail in the following paragraphs. When the data corresponding to the sub-request of the read operation is in the effective range, the data is the correct data, and the data can be read; when the data corresponding to the sub-request of the read operation is not in the valid range, it means that the data is usually erroneous and cannot be read from the object storage unit.
It should be noted that, when the operation sub-request is not a read operation sub-request, the step and the following step S306 do not need to be executed. Of course, if the determination result in S303 is yes, the following S307 needs to be executed. That is, when the operation sub-request is not a read operation sub-request, S305 and S306 may be skipped directly.
S306: and redundantly reading corresponding data from the rest object storage units.
When this step is performed, it means that if the data read from the invalid range of the target storage unit may be erroneous, the corresponding data needs to be redundantly read from the remaining target storage units in this step. For the specific step of redundantly reading the corresponding data from the remaining object storage units, reference may be made to the prior art, for example, related contents of Erasure Codes (EC), which are not described herein again.
S307: the operation sub-request is executed.
When the operation sub-request is a read operation sub-request, in this step, it means that the read operation sub-request is completed when the read data is read; in this step, the data corresponding to the operation sub-request is read from the valid range recorded in the object storage unit. When the operation sub-request is of another type, for example, a write operation sub-request, the corresponding data is operated according to the operation sub-request in this step.
The rest of the steps are substantially the same as S105 and S205 in the above embodiment of the invention, and the detailed contents may refer to the above embodiment of the invention, which is not described herein again.
S308: when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with a current version number different from the target version number as an effective range.
This step needs to be executed after S303, but this step may not be executed after S307, that is, S307 and S308 may be executed in parallel in the embodiment of the present invention.
When this step is performed, it means that the result of the determination in S303 is yes, which means that most of the data stored in the object storage units are correct and complete at this time, but there may be a case where a small part of the data stored in the object storage units are incomplete. Meanwhile, when the operation sub-request is a write operation sub-request, no matter whether the previously stored data in each object storage unit is complete or not, new data is written in the corresponding object storage unit according to the write operation sub-request in S307; in this step, the start position and the data length corresponding to the write operation sub-request in the object storage unit with the current version number different from the target version number, that is, the start position and the data length of the write data in the object storage unit in which the data is stored in error before recording, that is, the offset and the length of the write operation sub-request of this time, are recorded, and the start position and the data length of the data are used as the valid range of the object storage unit for the target object.
When the process provided by the embodiment of the present invention is executed next time, if the operation sub-request corresponding to the target object is the read operation sub-request, the steps from S305 to S307 need to be executed according to the valid range.
It should be noted that, in this step, the specific operation type of the operation sub-request is determined first. When the operation sub-request is a write operation sub-request, namely when the operation request generated by the client is a read operation request, the step is executed; when the operation sub-request is not the write operation sub-request, the step is not required to be executed, and the operation is skipped.
S309: and when the operation sub-request is successfully executed, updating the current version number of the corresponding target object in the object storage unit.
This step is substantially the same as S106 in the above embodiment of the present invention, and for details, reference is made to the above embodiment of the present invention, which is not repeated herein.
According to the data operation method based on object storage provided by the embodiment of the invention, when the data of a target object is basically complete, the effective range of writing data into an object storage unit with data errors stored before is recorded when a write operation sub-request is carried out; when a read operation sub-request is carried out on the same target object later, whether the read data is located in the effective range or not is judged, and when the read data is located in the effective range, the read data can be directly read; when the data is not in the effective range, redundant reading can be carried out from the rest object storage units, so that the correctness and the completeness of the read data are ensured.
In the following, a data operation apparatus based on object storage according to an embodiment of the present invention is introduced, and the data operation apparatus described below and the data operation method described above may be referred to correspondingly.
Fig. 4 is a block diagram of a data operation apparatus based on object storage according to an embodiment of the present invention, where the data operation apparatus according to fig. 4 may include:
the receiving module 100: the system comprises a client, a server and a server, wherein the server is used for receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests; the operation sub-requests are a plurality of operation sub-requests into which the client divides the operation request corresponding to the target object through erasure codes.
The sending module 200: and the operation sub-requests are used for sending the operation sub-requests to the corresponding object storage units.
Version number judging module 300: the device is used for judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is larger than a preset number or not; if yes, the operation execution module 400 is executed.
The operation execution module 400: for executing the operation sub-request.
Version number update module 500: and the operation sub-request is used for updating the current version number of the target object in the object storage unit after the operation sub-request is successfully executed.
Preferably, in this embodiment of the present invention, the version number determining module 300 may be further configured to:
judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is greater than a preset number or not; if not, the operation ends the module.
The end module is configured to:
the operation sub-request is not executed.
Preferably, in an embodiment of the present invention, the apparatus may further include:
a version number unification module: and the target version number is used for replacing the current version number of the object storage unit with the current version number different from the target version number.
The version number update module may be specifically configured to:
and after the operation sub-request is successfully executed, adding 1 to the current version number of the corresponding target object in the object storage unit.
Preferably, in an embodiment of the present invention, the apparatus may further include:
a recording module: and when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with the current version number different from the target version number as an effective range.
The apparatus may further include:
an effective range judging module: the data processing device is used for judging whether the data corresponding to the read operation sub-request is located in the effective range or not when the operation sub-request is a read operation sub-request; and if so, operating the operation execution module.
Preferably, in an embodiment of the present invention, the valid range determining module may be specifically configured to:
the data processing device is used for judging whether the data corresponding to the read operation sub-request is located in the effective range or not when the operation sub-request is a read operation sub-request; if not, the redundancy reading module is operated.
The redundancy read module is specifically configured to:
and redundantly reading corresponding data from the rest of the object storage units.
The data operation device of this embodiment is used to implement the foregoing data operation method, and therefore, the specific implementation of the data operation device may be found in the foregoing embodiments of the data operation method, for example, the receiving module 100, the sending module 200, the version number determining module 300, the operation executing module 400, and the version number updating module 500, which are respectively used to implement steps S101, S102, S103, S105, and S106 in the foregoing data operation method, so that the specific implementation thereof may refer to the description of the corresponding embodiments of each part, and will not be described herein again.
The following describes a data operation device based on object storage according to an embodiment of the present invention, and the data operation device described below, the data operation method described above, and the data operation apparatus described above may be referred to in correspondence with each other.
Referring to fig. 5, fig. 5 is a block diagram illustrating a data operation apparatus based on object storage according to an embodiment of the present invention.
Referring to fig. 5, the data manipulation device may include a processor 11 and a memory 12.
The memory 12 is used for storing a computer program; the processor 11 is configured to implement the data operation method based on object storage in the above embodiment of the invention when executing the computer program.
The processor 11 in the data operation device of this embodiment is used to install the data operation apparatus described in the above embodiment of the invention, and the processor 11 and the memory 12 are combined to implement the data operation method described in any of the above embodiments of the invention. Therefore, the specific implementation manner of the data operation device can be seen in the foregoing embodiment section of the data operation method, and the specific implementation manner thereof may refer to the description of each corresponding embodiment section, which is not described herein again.
The present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements a data operation method based on object storage as described in any of the embodiments of the present invention. The rest can be referred to the prior art and will not be described in an expanded manner.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The data operation method based on object storage, the data operation device based on object storage, the data operation equipment based on object storage and the computer readable storage medium provided by the invention are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (8)
1. A data operation method based on object storage is characterized by comprising the following steps:
receiving a plurality of operation sub-requests sent by a client and target version numbers corresponding to the operation sub-requests; the operation sub-requests are a plurality of operation sub-requests into which the client divides the operation request corresponding to the target object through erasure codes;
sending a plurality of the operation sub-requests to corresponding object storage units;
judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is greater than a preset number or not; if yes, executing the operation sub-request;
when the operation sub-request is successfully executed, updating the current version number of the target object in the object storage unit;
after the executing the operation sub-request, the method further comprises:
when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with the current version number different from the target version number as an effective range;
after the determining whether the number of the current version number corresponding to the target object in the plurality of object storage units is equal to the target version number is greater than a preset number, the method further includes:
when the operation sub-request is a read operation sub-request, judging whether the data corresponding to the read operation sub-request is located in the effective range; and if so, executing the operation sub-request.
2. The method of claim 1, wherein after determining whether a number of the object storage units corresponding to the target object having the same current version number as the target version number is greater than a preset number, the method further comprises:
and if not, not executing the operation sub-request.
3. The method of claim 1, wherein after said performing said operation sub-request, said method further comprises:
replacing the current version number of the object storage unit with the target version number, wherein the current version number of the object storage unit is different from the target version number;
when the operation sub-request is successfully executed, updating the current version number of the target object in the object storage unit includes:
and after the operation sub-request is successfully executed, adding 1 to the current version number of the corresponding target object in the object storage unit.
4. The method of claim 1, wherein after the determining whether the data corresponding to the sub-request of read operation is in the valid range, the method further comprises:
and if not, redundantly reading corresponding data from the rest of the object storage units.
5. An apparatus for operating data based on object storage, comprising:
a receiving module: the system comprises a client, a server and a server, wherein the server is used for receiving a plurality of operation sub-requests sent by the client and target version numbers corresponding to the operation sub-requests; the operation sub-requests are a plurality of operation sub-requests into which the client divides the operation request corresponding to the target object through erasure codes;
a sending module: the operation sub-requests are sent to corresponding object storage units;
a version number judging module: the device is used for judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is larger than a preset number or not; if yes, operating the operation execution module;
the operation execution module: for executing the operation sub-request;
and a version number updating module: the operation sub-request is used for updating the current version number of the target object in the object storage unit after the operation sub-request is successfully executed;
the device further comprises:
a recording module: when the operation sub-request is a write operation sub-request, recording a start position and a data length corresponding to the write operation sub-request in an object storage unit with the current version number different from the target version number as an effective range;
the device further comprises:
an effective range judging module: the data processing device is used for judging whether the data corresponding to the read operation sub-request is located in the effective range or not when the operation sub-request is a read operation sub-request; and if so, operating the operation execution module.
6. The apparatus of claim 5, wherein the version number determining module is further configured to:
judging whether the number of the current version number corresponding to the target object in the object storage units, which is the same as the target version number, is greater than a preset number or not; if not, the operation ends the module;
the end module is configured to:
the operation sub-request is not executed.
7. An apparatus for operating on data stored on an object basis, the apparatus comprising:
a memory: for storing a computer program;
a processor: for implementing the steps of the object store based data manipulation method of any one of claims 1 to 4 when executing said computer program.
8. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the object storage based data manipulation method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811513259.0A CN109634526B (en) | 2018-12-11 | 2018-12-11 | Data operation method based on object storage and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811513259.0A CN109634526B (en) | 2018-12-11 | 2018-12-11 | Data operation method based on object storage and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109634526A CN109634526A (en) | 2019-04-16 |
| CN109634526B true CN109634526B (en) | 2022-04-22 |
Family
ID=66072837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811513259.0A Active CN109634526B (en) | 2018-12-11 | 2018-12-11 | Data operation method based on object storage and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109634526B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113126923B (en) * | 2021-04-21 | 2022-05-27 | 山东英信计算机技术有限公司 | Additional uploading method, system, equipment and storage medium for distributed object storage |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101178660A (en) * | 2007-04-28 | 2008-05-14 | 四川长虹电器股份有限公司 | Memory data automatic update method |
| CN103558998A (en) * | 2013-11-07 | 2014-02-05 | 华为技术有限公司 | Data processing method and device |
| CN105426483A (en) * | 2015-11-19 | 2016-03-23 | 华为技术有限公司 | File reading method and device based on distributed system |
| CN106484321A (en) * | 2016-09-08 | 2017-03-08 | 华为数字技术（成都）有限公司 | A kind of date storage method and data center |
| CN106657356A (en) * | 2016-12-29 | 2017-05-10 | 郑州云海信息技术有限公司 | Data writing method and device for cloud storage system, and cloud storage system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9613052B2 (en) * | 2012-06-05 | 2017-04-04 | International Business Machines Corporation | Establishing trust within a cloud computing system |
| CN103678337B (en) * | 2012-09-06 | 2017-12-12 | 华为技术有限公司 | Data clearing method, apparatus and system |
| CN106462639B (en) * | 2014-06-24 | 2020-04-24 | 谷歌有限责任公司 | Handling changes to a remote database |
| CN105993013B (en) * | 2014-12-27 | 2018-05-04 | 华为技术有限公司 | A kind of data processing method apparatus and system |
| CN108388438B (en) * | 2018-02-07 | 2021-04-23 | 平安普惠企业管理有限公司 | System base table updating method and device, computer equipment and storage medium |
-
2018
- 2018-12-11 CN CN201811513259.0A patent/CN109634526B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101178660A (en) * | 2007-04-28 | 2008-05-14 | 四川长虹电器股份有限公司 | Memory data automatic update method |
| CN103558998A (en) * | 2013-11-07 | 2014-02-05 | 华为技术有限公司 | Data processing method and device |
| CN105426483A (en) * | 2015-11-19 | 2016-03-23 | 华为技术有限公司 | File reading method and device based on distributed system |
| CN106484321A (en) * | 2016-09-08 | 2017-03-08 | 华为数字技术（成都）有限公司 | A kind of date storage method and data center |
| CN106657356A (en) * | 2016-12-29 | 2017-05-10 | 郑州云海信息技术有限公司 | Data writing method and device for cloud storage system, and cloud storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109634526A (en) | 2019-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107643880B (en) | File data migration method and device based on distributed file system | |
| US8650439B2 (en) | Apparatus and method for fault tolerant FOTA update | |
| US20150213100A1 (en) | Data synchronization method and system | |
| EP2583176B1 (en) | Error detection for files | |
| CN110442473B (en) | Nonvolatile data storage method and device, electronic equipment and medium | |
| CN111857602B (en) | Data processing method, data processing device, data node and storage medium | |
| CN107479823B (en) | Data verification method and device in random read-write file test | |
| CN109189613A (en) | A kind of database data recovery method and relevant apparatus | |
| WO2017016282A1 (en) | Software upgrading method and apparatus, and computer storage medium | |
| US7849355B2 (en) | Distributed object sharing system and method thereof | |
| CN110083379A (en) | A kind of server component upgrade method and relevant apparatus | |
| CN110597779A (en) | Data reading and writing method in distributed file system and related device | |
| CN107608821B (en) | Data reading method, device and equipment based on erasure codes | |
| CN109558263A (en) | A kind of processing method and relevant apparatus of solid state hard disk data read errors | |
| CN110505314B (en) | Processing method for concurrent additional uploading request | |
| CN109634526B (en) | Data operation method based on object storage and related device | |
| CN109213450B (en) | Associated metadata deleting method, device and equipment based on flash memory array | |
| CN108196975B (en) | Data verification method and device based on multiple checksums and storage medium | |
| CN107329699B (en) | Erasure rewriting method and system | |
| CN107894900B (en) | MCU upgrading method and system | |
| CN108363635B (en) | Machine-readable storage medium, apparatus and method for rewinding | |
| US20120151005A1 (en) | Image file download method | |
| CN111026531B (en) | Task repeated transmission processing method and device, computer equipment and storage medium | |
| CN101977231A (en) | Method for downloading mapping file | |
| CN110119285A (en) | A kind of low side scm software upgrade method and relevant apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |