CN112181301A - Data export method and device of cloud hard disk and server equipment - Google Patents
Data export method and device of cloud hard disk and server equipment Download PDFInfo
- Publication number
- CN112181301A CN112181301A CN202011037414.3A CN202011037414A CN112181301A CN 112181301 A CN112181301 A CN 112181301A CN 202011037414 A CN202011037414 A CN 202011037414A CN 112181301 A CN112181301 A CN 112181301A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- hard disk
- storage unit
- cloud hard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 86
- 238000009795 derivation Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 description 25
- 238000013500 data storage Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data export method and device of a cloud hard disk and server-side equipment, relates to the technical field of data processing, and solves the technical problem of low data export efficiency of the cloud hard disk. The method comprises the following steps: receiving a derivation instruction aiming at all data of a target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a record table and is used for recording whether data exist in each storage unit or not; determining a target storage unit with data from a target record table corresponding to the target cloud hard disk based on the export instruction; and only carrying out data export on the target storage unit with the data.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a data export method and device for a cloud hard disk and server-side equipment.
Background
The data storage mode of the cloud hard disk generally adopts a distributed file system (Ceph) structure, and a data storage space required for storing each cloud hard disk is dispersed in a plurality of storage servers (physical machines) in the Ceph structure, so that the data of one cloud hard disk is prevented from being stored in a centralized manner.
At present, the operations on the cloud hard disk include two data writing operations and data exporting operations. For writing in the cloud hard disk, data in the cloud hard disk can be directly written in. The data export process of the cloud hard disk is complex, the data storage spaces of the cloud hard disk scattered in each storage server need to be searched through a network or a hard disk interface, and then data in all the data storage spaces are tried to be read, so that data export omission of the cloud hard disk is avoided. However, this also results in a long time-consuming data export process for the cloud hard disk, which makes the data export inefficient.
Disclosure of Invention
The invention aims to provide a data export method and device of a cloud hard disk and server-side equipment, so as to relieve the technical problem of low efficiency of exporting data of the cloud hard disk.
In a first aspect, an embodiment of the present application provides a data export method for a cloud hard disk, where the method includes:
receiving a derivation instruction aiming at all data of a target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a record table and is used for recording whether data exist in each storage unit or not;
determining a target storage unit with data from a target record table corresponding to the target cloud hard disk based on the export instruction;
and only carrying out data export on the target storage unit with the data.
In one possible implementation, after the step of receiving an export instruction for all data of the target cloud hard disk, the method further includes:
performing data snapshot processing on the target cloud hard disk to obtain a target cloud hard disk at the current moment; and the target cloud hard disk at the current moment is used for providing an export object for the export instruction.
In one possible implementation, a plurality of identifiers are recorded in the record table, and each identifier is a first identifier or a second identifier;
the first identification is used for representing that no data exists in the storage unit;
the second identifier is used for indicating that data exists in the storage unit.
In one possible implementation, the record table records the identifier in the form of a Bit Vector container Bit Vector; the identifications in all bits of the Bit Vector are in one-to-one correspondence with the storage units respectively;
the first flag is 0 and the second flag is 1.
In one possible implementation, the step of determining a target storage unit in which data exists from a target record table corresponding to the target cloud hard disk includes:
for each target storage unit, determining a target identifier corresponding to the target storage unit in the target record table;
performing AND operation on the target identifier and 1; if the AND operation result is 0, determining that no data exists in the target storage unit; and if the AND operation result is 1, determining that the data exists in the target storage unit.
In one possible implementation, the capacity of each target storage unit is a preset unit capacity; the method further comprises the following steps:
determining the total capacity of the target cloud hard disk, and dividing the total capacity by the preset unit capacity to obtain the unit number of the target storage unit;
and taking the number of the units as the number of bits, and determining the Bit number of the target Bit Vector according to the number of the bits.
In one possible implementation, the method further comprises:
and when the target storage unit is written with data, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 1.
In one possible implementation, the method further comprises:
and when the data in the target storage unit is deleted, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to be 0.
In one possible implementation, after the step of determining the target storage unit having data from the target record table corresponding to the target cloud hard disk, the method further includes:
and returning a data derivation result with empty content for the target storage unit without data.
In one possible implementation, after the step of performing data derivation only on the target storage unit in which the data exists, the method further includes:
and returning the data export result of the target cloud hard disk according to the sequence of the target storage units.
In a second aspect, a data exporting apparatus for a cloud hard disk is provided, which includes:
the receiving module is used for receiving export instructions of all data of the target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a record table and is used for recording whether data exist in each storage unit or not;
the determining module is used for determining a target storage unit with data from a target record table corresponding to the target cloud hard disk based on the derivation instruction;
and the export module is used for only exporting the data of the target storage unit with the data.
In a third aspect, an embodiment of the present application further provides a server device, which includes a memory and a processor, where the memory stores a computer program that is executable on the processor, and the processor implements the method of the first aspect when executing the computer program.
In a fourth aspect, this embodiment of the present application further provides a computer-readable storage medium storing machine executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method of the first aspect.
The embodiment of the application brings the following beneficial effects:
the method, the device and the server device for exporting data of the cloud hard disk provided by the embodiment of the application can receive an export instruction of all data of a target cloud hard disk, all data of the cloud hard disk in the scheme is stored in a plurality of storage units, the cloud hard disk corresponds to a data recording table capable of recording whether each storage unit exists or not, a storage server can determine the target storage unit with data from the target recording table corresponding to the target cloud hard disk based on the export instruction, so that data export can be carried out only aiming at the target storage unit with data, in the scheme, whether each storage unit of the cloud hard disk exists or not can be rapidly inquired through the recording table corresponding to the cloud hard disk, further, the storage unit with data exists is determined, and data export can be directly carried out only aiming at the target storage unit with data in the data export process of the cloud hard disk, the target storage unit without data does not need to be searched and read, so that the time for searching and reading the target storage unit without data is saved, and the whole data export efficiency of the target cloud hard disk is improved.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the detailed description of the present application or the technical solutions in the prior art, the drawings needed to be used in the detailed description of the present application or the prior art description will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic structural diagram of an application scenario according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data export method of a cloud disk according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a data export process of a cloud disk according to an embodiment of the present application;
FIG. 4 is an example of a bit vector container type record table provided by an embodiment of the present application;
FIG. 5 is a schematic flowchart illustrating a data reading process of a memory cell according to an embodiment of the present disclosure;
fig. 6 is a schematic flowchart of a process of creating a record table of a storage unit according to an embodiment of the present application;
FIG. 7 is a flowchart illustrating a process of deleting data from a storage unit according to an embodiment of the present disclosure;
fig. 8 provides a schematic structural diagram of a data export apparatus of a cloud disk;
fig. 9 shows a schematic structural diagram of a server device according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "comprising" and "having," and any variations thereof, as referred to in the embodiments of the present application, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In practical applications, a data storage manner of a cloud hard disk (virtual hard disk) generally adopts a block device storage structure of Ceph, and as shown in fig. 1, data storage spaces (e.g., four data storage spaces in fig. 1) required for storing each specific block device (i.e., one cloud hard disk) are distributed among a plurality of (e.g., four) storage servers (physical machines) in the Ceph structure.
At present, a specific block device (a cloud hard disk) in a Ceph block device storage structure rarely has a block storage export function, that is, it is difficult to export the whole data of a cloud hard disk, and the whole data of the cloud hard disk can only be read out from an application on the block storage to indirectly realize the export function. The method is relatively dependent on an application layer, can only be exported by a user, cannot be exported from a storage layer, has more uncontrollable factors and is relatively complex for user operation.
Certainly, some cloud disks also have an export function, and in the process of exporting data, whether storage units related to the cloud disks exist or not needs to be searched on all storage servers (physical machines) and then spliced. For example, a cloud hard disk with 4 Terabytes (TB) finally has millions of storage units, which are dispersedly stored in many storage servers, and if all the storage units are searched for once, the time is long, and the time for searching is easy to affect normal services, so that the data export efficiency is low.
Based on this, the embodiment of the application provides a data export method and device for a cloud hard disk and server equipment, and the technical problem of low efficiency in exporting data of the cloud hard disk can be solved through the method.
Embodiments of the present invention are further described below with reference to the accompanying drawings.
Fig. 2 is a schematic flowchart of a data export method of a cloud disk according to an embodiment of the present application. As shown in fig. 2, the method includes:
step S210, receiving an export instruction for all data of the target cloud hard disk.
All data of the cloud hard disk are stored in a plurality of storage units. The cloud hard disk corresponds to a record table for recording whether data exists in each storage unit. The record table can be stored in each storage server of the Ceph structure, and the storage space occupied in the storage server is very small due to the small data volume of the record table.
It should be noted that the storage unit is understood as a storage unit based on an elastic customizable data object. For example, the storage unit may be the smallest data storage unit defined for data object storage.
In practical applications, the data export instruction corresponds to a data write instruction, and may include any form of instruction requesting to obtain data, such as a data obtain instruction, a data read instruction, and the like.
In this embodiment, a storage server in the service end Ceph structure may receive a data export instruction sent by a client, and may also receive a data export instruction local to the service end.
Step S220, based on the export instruction, determines a target storage unit having data from a target record table corresponding to the target cloud hard disk.
It should be noted that each cloud hard disk corresponds to a record table for recording whether data exists in each storage unit in the cloud hard disk. The target record table in this step is a record table corresponding to the target cloud hard disk targeted by the export instruction.
Since whether data exists in each target storage unit in the target cloud hard disk is also recorded in the target record table, the storage server may find out the target storage unit in which the data exists in the target record table, and certainly, may also find out the target storage unit in which the data does not exist in the target record table.
In step S230, data is derived only for the target storage unit in which data exists.
In step S220, a target storage unit having data is determined from the plurality of target storage units of the target cloud hard disk, and in this step, the storage server may perform data derivation only for the target storage unit having data. Therefore, in the whole data export process of the target cloud hard disk, the storage server does not need to search each target storage unit through a network or a hard disk interface and the like, and does not need to try to read data when each target storage unit is issued to the rear-end storage server, so that the target storage unit without data is prevented from being read, the time consumption of the whole data export process is reduced, and the data export speed is increased.
In the embodiment of the application, the record table corresponding to the cloud hard disk is utilized, the storage server can quickly inquire whether data exist in each storage unit of the cloud hard disk, and then the storage unit with the data exists is determined, in the data export process of the cloud hard disk, data export can be carried out only aiming at the target storage unit with the data, the target storage unit without the data does not need to be searched and data read, so that the time for searching and data reading the target storage unit without the data is saved, and the whole data export efficiency of the target cloud hard disk is improved.
The above steps are described in detail below.
In some embodiments, the storage server may snapshot the target cloud hard disk first and then export the data of the snapshot, so that the data export process is not affected by the data writing and other services. As an example, after step S210, the method may further include the steps of:
performing data snapshot processing on the target cloud hard disk to obtain the target cloud hard disk at the current moment; the target cloud hard disk at the current moment is used for providing export objects for the data export instructions.
It should be noted that Snapshot (Snapshot) processing is to backup current cloud hard disk data at a certain time point, and the obtained copy has the same content as the cloud hard disk data at the time point. In practical application, an export object for exporting data from each cloud hard disk may be snapshot data at a certain time after receiving a data export instruction for the cloud hard disk, instead of an original cloud hard disk.
In this embodiment of the application, as shown in fig. 3, when a storage server receives an export instruction of a block device (cloud hard disk), a snapshot may be performed on the block device first, for example, a createSnap interface is called to perform snapshot processing to obtain a copy of the block device, then, a plurality of threads may be started to try to read contents of all storage units (objects) of the copy of the block device, and after response data is read, the contents are exported and written to a local file, such as a client and a server, which sends the export instruction, so as to complete an export process.
The snapshot data at a certain moment can be exported only through the snapshot, the phenomenon that the export process is influenced by data writing in the export process to lead the export data to be inconsistent is avoided, the cloud hard disk data are exported without being influenced by written business, and the consistency of the export data is guaranteed.
In some embodiments, the indication of whether data is present in the storage unit may be implemented by identification. As an example, a plurality of identifiers are recorded in the record table, and each identifier is a first identifier or a second identifier; the first identification is used for indicating that no data exists in the storage unit; the second identifier is used to indicate the presence of data in the memory location.
The storage unit record table (Object map) can distinguish whether data exists in each storage unit by recording the first identifier and the second identifier which are different in content. Of course, the existence of data in the storage unit may also be represented by the first identifier, and the absence of data in the storage unit may also be represented by the second identifier.
Whether data exist in the recording storage unit is distinguished through two different marks with different contents, whether the data exist in the recording storage unit is shown more clearly and more simply, and the storage server can quickly determine whether the data exist in the recording storage unit through recognizing the marks.
Based on this, the two kinds of identifiers different in the above-described contents may be 0 and 1 to make the identifier recognition process simpler. As an example, the record table records the identification through the form of a Bit Vector container Bit Vector; the identifiers in all bits of the Bit Vector correspond to the storage units one by one respectively; the first flag is 0 and the second flag is 1.
It should be noted that the Bit Vector container Bit Vector is a sequence container of one Bit (Bit) element, and has member functions like the Vector container. The Bit Vector saves more memory space, and one element only occupies one Bit (Bit) without occupying one byte.
As shown in fig. 4, the Bit Vector in the embodiment of the present application is a data structure of bits (Bit), and values (values) corresponding to elements are marked by one Bit (Bit), and it can also be understood that each element therein is a Bit value, the value takes 0 or 1, the value is continuously allocated in a Bit block with a Bit as a unit, and 0 or 1 of a numerical value on one Bit (Bit) indicates whether data exists in a storage unit corresponding to the Bit (Bit).
Through the data structure of bits (Bit) of the Bit Vector, 9 Bit values, namely element values, can be stored in one Byte (Byte), so that the storage space of the recording table of the storage unit can be greatly saved.
Based on this, in the process of identifying the bit value, the bit value can be rapidly identified to be 0 or 1 through AND operation. As an example, the step S220 may include the following steps:
step a), aiming at each target storage unit, determining a target identifier corresponding to the target storage unit in a target record table;
step b), performing AND operation on the target identifier and 1; if the AND operation result is 0, determining that no data exists in the target storage unit; if the and operation result is 1, it is determined that data exists in the target memory cell.
For example, as shown in fig. 5, in the process of deriving certain memory cell (Object) data, the memory cell record table (Object map) corresponding to the memory cell is read first, and the value of the corresponding bit is obtained, and then whether the value is 0 or 1 can be determined by and-operating the value with 1. If the value is 0, the data does not exist in the storage unit corresponding to the bit, and the empty content of the Client (Client) can be directly returned without reading the data; if the value is 1, it indicates that there is data in the storage unit corresponding to the bit, so that the content in the storage unit (object) can be directly exported from the bottom storage layer and returned to the client.
Whether data exist in the storage unit is judged by directly carrying out AND operation, so that the identification speed and accuracy of the storage server for 0 and 1 are improved, and time and efficiency are saved.
In some embodiments, during the creation of the memory cell record table, the number of bits (bits) of the record table may be determined using capacity calculations. As an example, the capacity of each target storage unit is a preset unit capacity; the method may further comprise the steps of:
step c), determining the total capacity of the target cloud hard disk, and dividing the total capacity by the preset unit capacity to obtain the unit number of the target storage unit;
and d), taking the number of the units as the number of the bits, and determining the Bit number of the target Bit Vector according to the number of the bits.
As shown in fig. 6, when creating the record table (Image), the capacity (Image size) of the record table needs to be specified first, how many bits (bits) are needed by the record table, that is, how many storage units (objects) are needed can be calculated according to the total capacity of the cloud hard disk and the capacity (object size) of the storage units, assuming that the number is N, then applying for BitVector of N bits (bits), that is, creating Bit Vector of size needed by N bits, and generating the record table in the form of Bit Vector.
The capacity of all storage units is the same, and the capacity of the storage unit may be a unit capacity preset by a worker, or may be a unit capacity specified by a distributed file system (ceph) configuration, for example, a unit capacity of 4 Megabytes (MB).
The total capacity of the cloud hard disk is divided by the same capacity of each storage unit, so that the number of the storage units can be rapidly calculated, the number of bits (bits) required by the record table can be further determined, and the creation efficiency of the record table is improved.
In some embodiments, the record table may be updated in real time according to the modification condition of the data in the storage unit, so as to maintain the consistency between the record table and the corresponding storage unit. Wherein the updating of the record table may include various implementations.
As an example, the method may further comprise the steps of:
and e), when the target storage unit is written with data, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 1.
As shown in fig. 6, when a user writes data into a certain storage unit (object), the data may be written into the back-end storage server, and meanwhile, the corresponding Bit in the Bit Vector of the record table is updated, and the value of the Bit is set to 1 to indicate that the data exists in the storage unit.
As another example, the method may further comprise the steps of:
and f), when the data in the target storage unit is deleted, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 0.
As shown in fig. 7, when a certain memory cell (object) is deleted, the memory cell record table (object map) is read first, then the specified bit value is set to 0, and then the data in the memory cell (object) is deleted from the underlying hard disk.
By updating the record table corresponding to the storage unit in real time according to the writing and deleting conditions of the data in the storage unit, the record table and the corresponding storage unit can be kept unified, and error information which does not accord with actual conditions is prevented from being acquired from the record table.
In some embodiments, a null value may be returned directly when there is no data in the memory location. As an example, after step S220, the method may further include the steps of:
and g), returning a data export result with empty content to the target storage unit without data.
For example, as shown in fig. 5, if the value is 0, it indicates that there is no data in the memory cell corresponding to the bit, and the empty content of the Client (Client) can be directly returned without reading the empty content. By directly returning the data export result of the null value when no data exists in the storage unit, the redundant processing process of the storage unit without data is saved, and the export efficiency of the whole data is improved.
In some embodiments, the derived data may be returned when the data is present in the storage unit. As an example, after step S230, the method may further include the steps of:
and h), returning the data export result of the target cloud hard disk according to the sequence of the target storage units.
For example, as shown in fig. 5, if the value is 1, it indicates that there is data in the storage unit corresponding to the bit, so that the content in the storage unit (object) can be directly derived from the storage bottom layer and returned to the client. In the process of returning the data export result, the content of the storage units exported from the storage back end can be written into the local file of the client by using the write interface of the system per se according to the sequence of the storage units.
When data exist in the storage units, the contents of the storage units which accord with the sequence of the plurality of storage units are used as export results, so that the export results form a more complete and complete cloud hard disk, and the exported cloud hard disk is convenient to use by a server side or a user side.
Fig. 8 provides a schematic structural diagram of a data export apparatus of a cloud disk. As shown in fig. 8, the data export apparatus 800 of the cloud hard disk includes:
a receiving module 801, configured to receive an instruction for exporting all data of a target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a recording table and is used for recording whether data exist in each storage unit or not;
a first determining module 802, configured to determine, based on the data derivation instruction, a target storage unit in which data exists from a target record table corresponding to a target cloud hard disk;
and a derivation module 803, configured to perform data derivation only on the target storage unit in which the data exists.
In some embodiments, the apparatus further comprises:
the snapshot module is used for performing data snapshot processing on the target cloud hard disk after receiving the export instruction of all data of the target cloud hard disk to obtain the target cloud hard disk at the current moment; the target cloud hard disk at the current moment is used for providing export objects for the data export instructions.
In some embodiments, a plurality of identifiers are recorded in the record table, and each identifier is a first identifier or a second identifier;
the first identification is used for indicating that no data exists in the storage unit;
the second identifier is used to indicate the presence of data in the memory location.
In some embodiments, the record table records the identification by the form of a Bit Vector container Bit Vector; the identifiers in all bits of the Bit Vector correspond to the storage units one by one respectively;
the first flag is 0 and the second flag is 1.
In some embodiments, the determining module 802 is specifically configured to:
for each target storage unit, determining a target identifier corresponding to the target storage unit in a target record table;
performing AND operation on the target identifier and 1; if the AND operation result is 0, determining that no data exists in the target storage unit; if the and operation result is 1, it is determined that data exists in the target memory cell.
In some embodiments, the capacity of each target storage unit is a preset unit capacity; the device also includes:
the second determining module is used for determining the total capacity of the target cloud hard disk and dividing the total capacity by the preset unit capacity to obtain the unit number of the target storage unit;
and the third determining module is used for taking the number of the units as the number of the bits and determining the Bit number of the target Bit Vector according to the number of the bits.
In some embodiments, the apparatus further comprises:
and the first updating module is used for updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 1 when the target storage unit is written with data.
In some embodiments, the apparatus further comprises:
and the second updating module is used for updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 0 when the data in the target storage unit is deleted.
In some embodiments, the apparatus further comprises:
the first returning module is used for returning a data export result with empty content to the target storage unit without data after the target storage unit with data is determined from the target record table corresponding to the target cloud hard disk.
In some embodiments, the apparatus further comprises:
and the second returning module is used for returning the data export result of the target cloud hard disk according to the sequence of the target storage units after only the target storage units with data are exported.
The data export device of the cloud hard disk provided by the embodiment of the application has the same technical characteristics as the data export method of the cloud hard disk provided by the embodiment of the application, so that the same technical problems can be solved, and the same technical effects can be achieved.
As shown in fig. 9, a server device 900 includes a memory 901 and a processor 902, where the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method provided in the foregoing embodiment.
Referring to fig. 9, the server device further includes: a bus 903 and a communication interface 904, the processor 902, the communication interface 904, and the memory 901 are connected by the bus 903; the processor 902 is used to execute executable modules, such as computer programs, stored in the memory 901.
The Memory 901 may include a high-speed Random Access Memory (RAM), and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 904 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
Bus 903 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 9, but this does not indicate only one bus or one type of bus.
The memory 901 is used for storing a program, and the processor 902 executes the program after receiving an execution instruction, and the method performed by the apparatus defined by the process disclosed in any of the foregoing embodiments of the present application may be applied to the processor 902, or implemented by the processor 902.
The processor 902 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 902. The Processor 902 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 901, and the processor 902 reads the information in the memory 901, and completes the steps of the above method in combination with the hardware thereof.
Corresponding to the data export method of the cloud hard disk, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores machine executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to execute the steps of the data export method of the cloud hard disk.
The data export device of the cloud hard disk provided by the embodiment of the application can be specific hardware on equipment or software or firmware installed on the equipment. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments where no part of the device embodiments is mentioned. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
For another example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or a part of the technical solution may be embodied in the form of a software product, where the computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the data export method of the cloud hard disk according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the scope of the embodiments of the present application. Are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (13)
1. A data export method of a cloud hard disk is characterized by comprising the following steps:
receiving a derivation instruction aiming at all data of a target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a record table and is used for recording whether data exist in each storage unit or not;
determining a target storage unit with data from a target record table corresponding to the target cloud hard disk based on the export instruction;
and only carrying out data export on the target storage unit with the data.
2. The method of claim 1, wherein after the step of receiving the export instruction for all data of the target cloud hard disk, the method further comprises:
performing data snapshot processing on the target cloud hard disk to obtain a target cloud hard disk at the current moment; and the target cloud hard disk at the current moment is used for providing an export object for the export instruction.
3. The method according to claim 1 or 2, wherein a plurality of identifiers are recorded in the record table, and each identifier is a first identifier or a second identifier;
the first identification is used for representing that no data exists in the storage unit;
the second identifier is used for indicating that data exists in the storage unit.
4. The method of claim 3, wherein the record table records the identifier in the form of a Bit Vector container Bit Vector; the identifications in all bits of the Bit Vector are in one-to-one correspondence with the storage units respectively;
the first flag is 0 and the second flag is 1.
5. The method according to claim 4, wherein the step of determining the target storage unit with data from the target record table corresponding to the target cloud hard disk comprises:
for each target storage unit, determining a target identifier corresponding to the target storage unit in the target record table;
performing AND operation on the target identifier and 1; if the AND operation result is 0, determining that no data exists in the target storage unit; and if the AND operation result is 1, determining that the data exists in the target storage unit.
6. The method of claim 4, wherein the capacity of each of the target storage units is a preset unit capacity; the method further comprises the following steps:
determining the total capacity of the target cloud hard disk, and dividing the total capacity by the preset unit capacity to obtain the unit number of the target storage unit;
and taking the number of the units as the number of bits, and determining the Bit number of the target Bit Vector according to the number of the bits.
7. The method of claim 4, further comprising:
and when the target storage unit is written with data, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to 1.
8. The method of claim 4, further comprising:
and when the data in the target storage unit is deleted, updating the value of the target Bit corresponding to the target storage unit in the target Bit Vector to be 0.
9. The method according to claim 1 or 2, wherein after the step of determining the target storage unit with data from the target record table corresponding to the target cloud hard disk, the method further comprises:
and returning a data derivation result with empty content for the target storage unit without data.
10. The method according to claim 1 or 2, wherein the step of data derivation only for the target storage unit where the data exists is further followed by:
and returning the data export result of the target cloud hard disk according to the sequence of the target storage units.
11. The utility model provides a data export device of cloud hard disk which characterized in that includes:
the receiving module is used for receiving export instructions of all data of the target cloud hard disk; all data of the cloud hard disk are stored in a plurality of storage units; the cloud hard disk corresponds to a record table and is used for recording whether data exist in each storage unit or not;
the determining module is used for determining a target storage unit with data from a target record table corresponding to the target cloud hard disk based on the derivation instruction;
and the export module is used for only exporting the data of the target storage unit with the data.
12. A server device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor implements the steps of the method according to any one of claims 1 to 10 when executing the computer program.
13. A computer readable storage medium having stored thereon machine executable instructions which, when invoked and executed by a processor, cause the processor to execute the method of any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011037414.3A CN112181301A (en) | 2020-09-27 | 2020-09-27 | Data export method and device of cloud hard disk and server equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011037414.3A CN112181301A (en) | 2020-09-27 | 2020-09-27 | Data export method and device of cloud hard disk and server equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112181301A true CN112181301A (en) | 2021-01-05 |
Family
ID=73944324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011037414.3A Pending CN112181301A (en) | 2020-09-27 | 2020-09-27 | Data export method and device of cloud hard disk and server equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181301A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030380A (en) * | 2002-06-27 | 2004-01-29 | Sony Corp | Data storage device, method for controlling the same, its program, recording medium with its program recorded, data processor, program for controlling the same and recording medium with its program recorded |
| CN103034735A (en) * | 2012-12-26 | 2013-04-10 | 北京讯鸟软件有限公司 | Big data distributed file export method |
| CN103064797A (en) * | 2012-12-21 | 2013-04-24 | 华为技术有限公司 | Data processing method and virtual machine management platform |
| CN104407820A (en) * | 2014-12-12 | 2015-03-11 | 华为技术有限公司 | Solid state disk storage system based data processing method, device and system |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN110795029A (en) * | 2019-10-08 | 2020-02-14 | 济南浪潮数据技术有限公司 | Cloud hard disk management method, device, server and medium |
| CN110807058A (en) * | 2018-08-01 | 2020-02-18 | 北京京东尚科信息技术有限公司 | Method and system for exporting data |
| US20200272348A1 (en) * | 2019-02-25 | 2020-08-27 | Liveramp, Inc. | Reducing Duplicate Data |
-
2020
- 2020-09-27 CN CN202011037414.3A patent/CN112181301A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030380A (en) * | 2002-06-27 | 2004-01-29 | Sony Corp | Data storage device, method for controlling the same, its program, recording medium with its program recorded, data processor, program for controlling the same and recording medium with its program recorded |
| CN103064797A (en) * | 2012-12-21 | 2013-04-24 | 华为技术有限公司 | Data processing method and virtual machine management platform |
| CN103034735A (en) * | 2012-12-26 | 2013-04-10 | 北京讯鸟软件有限公司 | Big data distributed file export method |
| CN104407820A (en) * | 2014-12-12 | 2015-03-11 | 华为技术有限公司 | Solid state disk storage system based data processing method, device and system |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN110807058A (en) * | 2018-08-01 | 2020-02-18 | 北京京东尚科信息技术有限公司 | Method and system for exporting data |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| US20200272348A1 (en) * | 2019-02-25 | 2020-08-27 | Liveramp, Inc. | Reducing Duplicate Data |
| CN110795029A (en) * | 2019-10-08 | 2020-02-14 | 济南浪潮数据技术有限公司 | Cloud hard disk management method, device, server and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765076B (en) | Data storage method, device, electronic equipment and storage medium | |
| CN109976669B (en) | Edge storage method, device and storage medium | |
| US11100047B2 (en) | Method, device and computer program product for deleting snapshots | |
| CN111061752B (en) | Data processing method and device and electronic equipment | |
| CN111475105A (en) | Monitoring data storage method, device, server and storage medium | |
| CN115840731A (en) | File processing method, computing device and computer storage medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN115857801A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN117577163A (en) | SSD bad block detection method, SSD bad block detection device, computer equipment and storage medium | |
| US10719497B2 (en) | Utilization of optimized ordered metadata structure for container-based large-scale distributed storage | |
| EP3264254B1 (en) | System and method for a simulation of a block storage system on an object storage system | |
| CN112181301A (en) | Data export method and device of cloud hard disk and server equipment | |
| CN110851437A (en) | Storage method, device and equipment | |
| CN115269347A (en) | Storage performance testing method, device, medium and electronic equipment | |
| CN115904211A (en) | Storage system, data processing method and related equipment | |
| CN114489481A (en) | Method and system for storing and accessing data in hard disk | |
| CN112015672A (en) | Data processing method, device, equipment and storage medium in storage system | |
| CN117271440B (en) | File information storage method, reading method and related equipment based on freeRTOS | |
| CN117762335B (en) | Writing method, device, equipment and storage medium for Ceph object | |
| CN117762336B (en) | Regular deletion method, device and equipment for Ceph object | |
| CN116974734A (en) | Object storage system, object modification method and related equipment | |
| CN118245446A (en) | Method, device and storage medium for managing metadata | |
| CN115344536A (en) | File processing method and device, server and readable storage medium | |
| CN117785825A (en) | Log file reading method, system, equipment and storage medium | |
| CN115033532A (en) | Reading method of bank file and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |