CN113194127B - Data storage method, data transmission method and computing equipment - Google Patents
Data storage method, data transmission method and computing equipment Download PDFInfo
- Publication number
- CN113194127B CN113194127B CN202110429687.0A CN202110429687A CN113194127B CN 113194127 B CN113194127 B CN 113194127B CN 202110429687 A CN202110429687 A CN 202110429687A CN 113194127 B CN113194127 B CN 113194127B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- data block
- stored
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 84
- 238000013500 data storage Methods 0.000 title claims abstract description 32
- 230000005540 biological transmission Effects 0.000 title claims abstract description 15
- 230000004044 response Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 description 17
- 238000010586 diagram Methods 0.000 description 13
- 238000013507 mapping Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000007723 transport mechanism Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
Abstract
The invention discloses a data storage method, a data transmission method and computing equipment, wherein the data storage method comprises the following steps: acquiring data to be stored, splitting a data object to be stored into a plurality of data blocks, assigning a unique name to each data block, storing each data block into a node of a preset storage structure, wherein, each node in the preset storage structure corresponds to a data server, each node is allocated with a unique node I D, the maximum storage capacity of each node is the same, all leaf nodes except the rightmost leaf node store data with the maximum capacity, each node in the preset storage structure is used for storing first-class data, the first-class data comprises one or more of a father node I D, a directory name, a data block name and a node I D, the directory name is the directory name of the data block stored by the current node corresponding to the child node, therefore, when the node information is modified, only the information of the node and the information of the associated node are modified, and the probability of the node conflict is greatly reduced.
Description
Technical Field
The invention relates to the technical field of cloud storage, in particular to a data storage method, a data transmission method, computing equipment and a storage medium.
Background
With the continuous development of social economy, data is growing explosively, various industries face the problem of huge data file storage, and enterprises want to store data and maintain data with high cost by means of own power and need professional teams. Thus, cloud storage becomes an option for enterprises to store data.
Cloud storage is an online data storage service mode provided by a network service provider, namely, data is stored in a plurality of virtual servers hosted by third parties. The third party operates a large-scale data center, and a user needs to purchase the operating storage space and access the storage space through an application program interface to store data.
Cloud storage is implemented based on a cloud storage architecture. Currently, a cloud storage structure mainly includes a central storage structure and a directory storage structure. The central storage structure, i.e. the directory entries of the files are all stored in one node (the node is a data server), and the directory storage structure, i.e. each node stores one directory, including the directory entries of all the files under the directory. However, the cloud storage structure modifies the central storage structure when a file is changed or deleted, and conflicts are likely to occur when multiple clients operate the file simultaneously.
Therefore, the invention provides a data storage method capable of avoiding conflict.
Disclosure of Invention
To this end, the present invention provides a data storage method which seeks to solve, or at least alleviate, the above-identified problems.
According to an aspect of the present invention, there is provided a data storage method, executed in a computing device, the method comprising the steps of:
acquiring data to be stored;
splitting a data object to be stored into a plurality of data blocks, and allocating a unique name to each data block;
and storing each data block into a node of a preset storage structure, wherein each node in the preset storage structure corresponds to a data server, each node is allocated with a unique node ID, the maximum storage capacity of each node is the same, all leaf nodes except the rightmost leaf node store data with the maximum capacity, each node in the preset storage structure is used for storing first-class data, the first-class data comprises one or more of a father node ID, a directory name, the data block, a data block name and a node ID, and the directory name is the directory name of the data block stored by the current node corresponding to the child node.
Optionally, the step of splitting the data object to be stored into a plurality of data blocks includes:
determining the current offset of a data pointer of a data object to be stored;
taking the determined current offset of the data pointer as an initial position, and reading data of a first preset numerical value from the data to be stored as a data block;
and updating the current offset of the data pointer to be the sum of the determined current offset of the data pointer and the first preset value.
Optionally, the step of reading data of a first preset value from the data to be stored with the determined current offset of the data pointer as a start position, as a data block, includes:
judging whether the product of the current data reading times and the first preset value is smaller than the size of the data to be stored, if so, reading the data of the first preset value by taking the determined current offset of the data pointer as an initial position, if not, determining the quantity of the data to be stored to be taken as a second value by taking the determined current offset of the data pointer as an initial position, and reading the data of the second value by taking the determined current offset of the data pointer as an initial position:
L＝m-(a-1)*b
wherein L is the number of the data to be stored, m is the size of the storage space occupied by the data to be stored, a is the current number of times of reading the data to be stored, and b is a first preset value;
the read data is used as a data block.
Optionally, the step of saving each data block to a node of the predetermined storage structure comprises:
and judging whether the current node has a child node, if so, storing the parent node ID, the current node ID and the directory name of the current node, and if not, storing the parent node ID, the data block name and the current node ID of the current node.
Optionally, comprising the steps of:
and judging whether each node in the preset storage structure has a left node or not, and if so, storing the left node ID in the current node.
Optionally, the method further comprises the steps of:
setting version information for each data block, wherein the version of the data block is distinguished through the version information of the data block;
the step of judging whether the current node has a child node comprises the following steps:
and judging whether the current node has a child node, if so, storing the father node ID, the current node ID, the left node ID, the directory name and the data block version information of the current node, and if not, storing the father node ID, the current node ID, the left node ID, the data block name and the data block version information of the current node.
Optionally, the method further comprises the steps of:
responding to the operation of replacing the data block in one leaf node of the preset storage structure by the target data block, judging whether the data block name of the target data block is the same as the data block name in the current node, if not, carrying out replacement, and if so, judging whether the data block version information of the target data block is the same as the data block version information in the current node;
and if the version information of the data block of the target data block is the same as that of the data block in the current node, replacing the data block in the current node with the target data block, and if the version information of the data block in the current node is not the same as that of the data block in the current node, not replacing.
Optionally, the method further comprises the steps of:
in response to a data block name operation saved in one node of the rename predetermined storage structure, updating a data block name stored in a current node to be a renamed name.
Optionally, the first preset value is equal to a maximum storage capacity of each node in the predetermined storage structure.
According to another aspect of the present invention, there is provided a data transmission method, executed in a computing device, the method comprising:
acquiring data to be transmitted, wherein the data to be transmitted is data stored by each node in a preset storage structure according to the data storage method;
carrying out Hash operation on data to be transmitted to obtain a first Hash value;
combining the data to be transmitted and the first hash value to serve as target data;
encrypting the target data through a data encryption algorithm to obtain encrypted data;
and sending the encrypted data to a receiving end.
Optionally, the operation steps after the receiving end receives the encrypted data are as follows:
decrypting the encrypted data through a data encryption algorithm to obtain target data;
splitting target data to obtain first data to be transmitted and the first hash value;
performing hash operation on the first data to be transmitted to obtain a second hash value;
and judging whether the second hash value is the same as the first hash value or not, if so, using the data to be transmitted, and if not, not using the data to be transmitted.
Optionally, the data encryption algorithm is an ABE encryption algorithm.
According to an aspect of the present invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method as described above.
According to an aspect of the present invention, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the method as described above.
According to the technical scheme of the invention, the data to be stored is stored in the preset storage structure, specifically, each node in the preset storage structure stores the father node ID, but not the father node ID stores the file directory (including all file directory items of the file directory), so that when the node information is modified, only the information of the node and the information of the associated node are required to be modified, the change of the father node is not involved, the probability of node collision is greatly reduced, and the modification complexity is also reduced. In addition, the node also stores the version information of the data block, so that an attacker cannot replace the file through different versions of the same file, the integrity of the data is ensured, and the safety of data storage is improved.
In addition, in the cloud storage structure, all leaf nodes except the rightmost leaf node store the maximum data (compared with the method that each leaf node only stores partial data and does not store the maximum data), the number of nodes is reduced, and therefore waste of storage space is reduced. And because the maximum capacity of the leaf node stored data is the same, when the size of the stored data is determined, the size of the stored data of each node does not need to be obtained and then summed, but the size of the stored data can be determined by multiplying the maximum capacity of the node by subtracting 1 from the number of the nodes and adding the data size of the rightmost leaf node, namely, the size of the stored data can be calculated by only accessing the data size of the rightmost leaf node, the size of the stored data can be determined quickly, and the time consumption is reduced.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a block diagram of a data storage system 100 according to one embodiment of the invention
FIG. 2 shows a schematic diagram of a computing device 200, according to one embodiment of the invention;
FIG. 3 shows a flow diagram of a data storage method 300 according to one embodiment of the invention;
FIG. 4 illustrates a schematic diagram of a predetermined storage structure according to one embodiment of the invention;
FIG. 5 is a diagram illustrating a mapping table corresponding to a predetermined storage structure according to one embodiment of the invention; and
fig. 6 shows a flow diagram of a data transmission method 600 according to one embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
With the continuous development of social economy, data is growing explosively, various industries face the problem of huge data file storage, and enterprises want to store data and maintain data with high cost by means of own power and need professional teams. Thus, cloud storage becomes an option for enterprises to store data.
Cloud storage is an online data storage service mode provided by a network service provider, namely, data is stored in a plurality of virtual servers hosted by third parties. The third party operates a large-scale data center, and a user needs to purchase the operating storage space and access the storage space through an application program interface to store data.
Cloud storage is implemented based on a cloud storage architecture. Currently, a cloud storage structure mainly includes a central storage structure and a directory storage structure. The central storage structure, that is, all directory entries of a file are stored in one node (the node is a data server), the central storage structure is modified when the file is changed or deleted, conflicts easily occur when multiple clients operate the file simultaneously, and the central node of the storage structure can be judged by observing the change of the file after the file is operated, so that node information is exposed. The directory storage structure comprises a directory storage structure, each node stores a directory, directory entries of all files in the directory are included, and all directory nodes are traversed and decrypted in path searching, so path searching is slow, and similarly, if a plurality of clients operate the files simultaneously, conflicts can be generated at parent nodes, so that only one version of nodes are reserved, and an orphan node can occur to the client which is not reserved.
To this end, the invention defines a memory structure defined as follows: the maximum storage capacity of each node in the storage structure is the same, all leaf nodes except the rightmost leaf child node in the predetermined storage structure store data with the maximum capacity, and each node in the storage structure stores a parent node ID, a current node ID, a directory name (the directory name is the directory name of a data block stored by the current node corresponding to the child node), a left node ID and data block version information (different versions of the data block are distinguished), or each node in the storage structure stores the parent node ID, the current node ID, the left node ID, the data block name and the data block version information.
On the basis of defining the storage structure, the invention provides a data storage method capable of avoiding conflict. The method is to store the data to be stored in the storage structure defined above. Specifically, each node in the storage structure stores a parent node ID, but not a child node ID stored by the parent node, so that when a data node is modified, only the information of the node and the information of the associated node are modified, and the change of the parent node is not involved, so that when a plurality of nodes are modified, the probability of node collision is greatly reduced, and the modification complexity is also reduced.
Since all leaf nodes except the rightmost leaf node store the maximum capacity of data (compared with the case that each leaf node stores only partial data but does not store the maximum capacity of data), the number of nodes is reduced, and thus the waste of storage space is reduced. And because the maximum capacity of the leaf node stored data is the same, when the size of the stored data is determined, the size of the stored data of each node does not need to be obtained, but the size of the stored data can be determined by multiplying the maximum capacity of the node by subtracting 1 from the number of the nodes and adding the data size of the rightmost leaf node, namely, the size of the stored data can be calculated by only accessing the data size of the rightmost leaf node, the size of the stored data can be determined quickly, and the time consumption is reduced.
In addition, the node also stores the version information of the data block, so that an attacker cannot replace the file through different versions of the same file, the integrity of the data is ensured, and the safety of data storage is improved.
The data storage method is realized based on a data storage system. Fig. 1 is a schematic diagram of a data storage system, where fig. 1 includes a control center server 110 and a plurality of data servers 120, the control center server 110 is respectively connected to the plurality of data servers 120 in a communication manner, and the connection manner between the ends in the system is not limited in the present invention. For example, one end can access the internet in a wired or wireless manner, and is connected with the other end through a data interface, so that interaction between the ends in the system is realized.
It should be noted that each data server 110 in the data storage system represents one node in the storage structure, that is, each data server 110 is configured to store a parent node ID, a current node ID, a directory name (the directory name is the directory name of a data block stored by a current node corresponding to a child node), a left node ID, and data block version information (to distinguish different versions of the data block), or each data server 110 is configured to store a parent node ID, a current node ID, a left node ID, a data block name, and data block version information. .
Control center server 110 and data server 120 may each be a computing device, and in one embodiment, FIG. 2 shows a block diagram of a computing device 200 in accordance with one embodiment of the invention. A block diagram of a computing device 200 as shown in fig. 2, in a basic configuration 202, the computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, process 204 may be any type of process, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include an operating system 120, one or more applications 222, and program data 224. In some embodiments, application 122 may be arranged to operate with program data 224 on an operating system. Program data 224 includes instructions, and in computing device 200 according to the present invention, program data 224 contains instructions for performing data storage method 300.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
FIG. 3 shows a flow diagram of a data storage method 300 according to one embodiment of the invention. The method 300 is adapted to be performed in the control center server 110. As shown in fig. 3, the data storage method 300 starts at step S310, and in executing step S310, the data to be stored is acquired, specifically: the method comprises the steps of obtaining data to be stored, searching for a mounting point, mounting the data to be stored to the mounting point, and distributing a unique node ID to the mounting point. Each node ID is unique, and the node ID may be a random number, a random character, a random combination of numbers and characters, etc., which is not limited in the present invention, for example, the node ID is a rootID, ID _ A, ID _ a _1, etc.
Subsequently, in step S320, the data to be stored is split into a plurality of data blocks, and a unique name is assigned to each data block, where the name of the data block may be a random number, a random character, a random combination of a number and a character, and the like, which is not limited in the present invention, for example, the name of the data block is file a 1.
Taking the splitting of one data block as an example, the following process is performed: determining the current offset of a data pointer of data to be stored, judging whether the product of the current data reading time and a first preset value is smaller than the size of the data to be encrypted, if so, reading the data of the first preset value by taking the determined current offset of the data pointer as an initial position, otherwise, determining the quantity of the data to be stored to be used as a second value by taking the determined current offset of the data pointer as the initial position, and reading the data of the second value by taking the determined current offset of the data pointer as the initial position:
L＝m-(a-1)*b
the method includes the steps that L is the number of data to be stored, m is the memory size occupied by the data to be stored, a is the number of times of currently reading the data to be stored, and b is a first preset value, wherein the first value is the same as the maximum storage capacity of each node in a preset storage structure, that is, the maximum capacity of leaf node storage data corresponding to the data to be stored in the preset storage structure is the same. The maximum storage capacity of each node in the predetermined storage structure may be set according to the storage capacity of each node, which is not limited in the present invention, and for example, the storage capacity of each node is 5G.
Because the maximum capacity of the leaf node storage data of the preset storage structure is the same, and the leaf node stores the data block, when the size of the stored data is obtained after the preset storage structure is used for storing the data to be stored, the size of the data stored by each node is not required to be obtained, the number of the nodes is reduced by 1 and then multiplied by the maximum capacity of the node, and the size of the rightmost leaf node data is added to determine the size of the data to be encrypted.
And then, taking the read data as a data block, and updating the current offset of the data pointer to be the sum of the determined current offset of the data pointer and a first preset value. And repeating the above process, namely dividing the data to be stored into a plurality of data blocks.
In the process of splitting data to be stored into a plurality of data blocks, the leaf nodes except the rightmost leaf node all store the data with the maximum capacity, and the number of the nodes can be reduced when the data with the same size is stored due to the fact that the other leaf nodes all store the data with the maximum capacity, so that the occupation of storage space is reduced in the data storage process, the system operation efficiency is improved, and the data storage efficiency is further improved.
After splitting the data to be stored into a plurality of data blocks, step S330 is executed to store each data block into a node of a predetermined storage structure, specifically:
and setting version information for each data block, judging whether each node in a preset storage structure has a left node or not, and if so, storing the left node ID in the current node. If the left node ID is not stored, which is equivalent to the sequence of the non-stored child nodes, the sequence of the data blocks cannot be determined, so that the left node ID is stored in each node, the storage sequence of the data blocks stored in the nodes of the same level can be distinguished, and a mapping table can be quickly constructed after traversing the block file.
And continuously judging whether the current node has a child node, if so (indicating that the current node is not a leaf node, storing the directory name of the data block stored by the child node corresponding to the current node in the node), storing the father node ID, the current node ID, the directory name, the left node ID and the data block version information in the current node, and if not (indicating that the current node is a leaf node, storing the data block in the node), storing the father node ID, the current node ID, the left node ID, the data block name and the data block version information in the current node.
The version of the data block is distinguished through the version information of the data block, the version information of the data block can be random numbers, random characters, random combinations of numbers and characters and the like, and the version information of the data block is not limited in the invention, and only the set version information is unique. In one embodiment, in response to an operation of replacing a data block in a leaf node of a predetermined storage structure by a target data block, whether the data block name of the target data block is the same as the data block name in a current node is determined, if not, the replacement is not performed, if yes, whether the data block version information of the target data block is the same as the data block version information in the current node is determined, if yes, the data block in the current node is replaced by the target data block, and if not, the replacement is not performed.
By storing the name and version information of the data block in the node, which is equivalent to standardizing the node format of the stored data, an attacker can be prevented from replacing the data block stored by the current node ID with the data block with different name, and can also be prevented from replacing the data block of the current version with the data block of the previous version, thereby ensuring the integrity of the data and improving the safety of data storage.
Taking an example of splitting a file into 4 data blocks, fig. 4 is a schematic diagram of a predetermined storage structure according to an embodiment. The storage structure in fig. 4 includes nodes rootID, ID _ A, ID _ B, ID _ C, ID _ a _1, ID _ B _1, and ID _ B _2, the node rootID is a root node of the storage structure, node ID _ A, ID _ B, ID _ C is a child node of the node rootID, node ID _ a _1 is a child node of node ID _ a, and node ID _ B _1 and ID _ B _2 are child nodes of node ID _ B, respectively. In the storage structure shown in fig. 4, nodes ID _ C, ID _ a _1, ID _ B _1, and ID _ B _2 are leaf nodes in which data blocks are stored, and since ID _ B _2 is the rightmost leaf node, nodes ID _ C, ID _ a _1 and ID _ B _1 may each store the largest capacity of data, whereas node ID _ B _2 may not store the largest capacity of data, where there may be a partially empty storage space.
In this embodiment, the node rootID is the root node, and since each node stores the parent node ID and the data block name, the parent node in the node rootID is empty, and the data block name is empty. Since the node ID _ A, ID _ B exists as a child node, the node ID _ A, ID _ B stores a parent node ID, a current node ID, a directory name, a left node ID, and data block version information. Since node ID _ C has no child node, node ID _ C stores a parent node ID, a current node ID, a left node ID, a data block name, and data block version information. Node ID _ a _1 is a child node of node ID _ a, and the node has no child node, so node ID _ a _1 stores a parent node ID, a current node ID, a left node ID, a data block name, and data block version information. Nodes ID _ B _1 and ID _ B _2 are children of node ID _ B and have no children, so nodes ID _ B _1 and ID _ B _2 respectively parent node ID, current node ID, left node ID, data block name and data block version information.
The mapping table may be mapped to a mapping table according to the storage structure, and the data to be stored is also mapped to a mapping table after being stored in the predetermined storage structure, taking the storage structure shown in fig. 4 as an example, the corresponding mapping table is shown in fig. 5, and fig. 5 is a schematic diagram of a mapping table of a storage structure corresponding to a file to be stored, taking the example of splitting the data to be stored into four data blocks (the names of the data blocks are file a1, file B1, file B2, and file C, respectively).
The node root ID in fig. 5 is the root node of the storage structure, and there is no left node, so that the node stores the node ID (root ID) and the data block version information (version), and the stored parent node ID, left node ID and data block name are all null. Node ID _ A, ID _ B is a child node of the node rootID, and there are child nodes and no left node in these two nodes, so node ID _ A, ID _ B holds the parent node ID, the current node ID, the directory name, the left node ID and the data block Version information, specifically, node ID _ A, ID _ B stores the node ID (ID _ A, ID _ B, respectively), the parent node ID (rootID), the directory name (directory a, directory B, respectively), the left node ID (null) and the data block Version information (Version), respectively. Node ID _ C is a child node of the node rootID, and the node has no child node and no left node, and therefore node ID _ C stores a parent node ID, a current node ID, a left node ID, a data block name, and data block Version information, and specifically, node ID _ C stores a node ID (ID _ C), a parent node ID (rootID), a left node ID (null), a data block (control), a data block name (file C), and data block Version information (Version). Node ID _ a _1 is a child node of node ID _ a, and the node has no child node and no left node, so node ID _ a _1 stores node ID (ID _ a _1), parent node ID (ID _ a), left node ID (null), data block (current), data block name (file a1), and data block Version information (Version). Node ID _ B _1 and ID _ B _2 are child nodes of node ID _ B, respectively, and two nodes have no child node, but node ID _ B _2 has a left node ID _ B _1, and therefore node ID _ B _1 and ID _ B _2 store node IDs (ID _ B _1 and ID _ B _2, respectively), parent node IDs (both ID _ B), left node IDs (null and ID _ B _1, respectively), data blocks (content), data block names (file B1 and file B2, respectively), and data block Version information (Version).
In one embodiment, in response to a data block name operation saved in one node of a renaming preset storage structure, the data block name stored in the current node is updated to be the renamed name without changing the parent node information of the current node, so that the renaming is low in execution complexity and high in efficiency after the data is stored based on the storage structure defined by the invention.
Based on the above, it can be seen that the data storage method provided by the present invention stores data to be stored in a predetermined storage structure, specifically, each node in the predetermined storage structure stores parent node information, and the parent node does not store child node information (including all file directory entries of a file directory), so that when modifying node information, only the information of the node itself and the information of an associated node (for example, right node information) need to be modified, and no change of the parent node is involved, so that when modifying node information at the same time by multiple clients, the probability of node collision is greatly reduced, and the modification complexity is also reduced. And the node also stores the left node ID, so that the orderliness of data can be ensured, and the mapping table can be quickly obtained by traversing a preset storage structure. In addition, the node also stores version information of the data block, so that an attacker cannot replace the file through different versions of the same file, the integrity of the data is ensured, and the safety of data storage is improved.
In addition, since all leaf nodes except the rightmost leaf node in the predetermined storage structure store the maximum capacity of data (compared with the case where each leaf node stores only partial data and does not store the maximum capacity of data), the number of nodes is reduced, thereby reducing the waste of storage space. And because the maximum capacity of the leaf node stored data is the same, when the size of the stored data is determined, the size of the stored data can be calculated by only accessing the data size of the rightmost leaf node without acquiring the size of the stored data of each node, so that the size of the stored data can be calculated quickly, and the time consumption is reduced.
However, the above-described predetermined storage structure does not store information of child nodes in parent nodes in order to reduce conflicts after storing data, and thus partial data sharing cannot be realized. In order to solve the problem, partial data sharing can be achieved through secondary encryption, namely data to be shared is encrypted and then sent to A, and the A shares the data to B.
In order to solve the above problems, the present invention provides a data transmission method. Fig. 6 shows a flow diagram of a data transmission method 600 according to one embodiment of the invention. The method 600 is adapted to be performed in the control center server 110. As shown in fig. 6, the data transmission method 600 begins at step S610. In step S610, data to be transmitted (shared data) is obtained, where the data to be transmitted is data stored in each node in a preset storage structure, that is, the data to be transmitted is partial data stored in the preset storage structure. Next, in step S620, performing hash (hash) operation on the data to be transmitted to obtain a first hash value, in step S620, combining the data to be transmitted and the first hash value as target data, then in step S630, encrypting the target data by using a data encryption algorithm to obtain encrypted data, and finally in step S640, sending the encrypted data to a receiving end. The data encryption algorithm may be an ABE (Attribute Based encryption system) encryption algorithm, and further, the data encryption algorithm may be a CP-ABE (Ciphertext Policy Attribute Based encryption system) encryption algorithm.
The operation steps of the receiving end after receiving the encrypted data are as follows: decrypting the encrypted data through a data encryption algorithm to obtain target data, splitting the target data to obtain first data to be transmitted and a first hash value, performing hash operation on the first data to be transmitted to obtain a second hash value, judging whether the second hash value is the same as the first hash value, if so, using the data to be transmitted, and if not, not using the data to be transmitted.
According to the data transmission method provided by the invention, in the data transmission process, the integrity check of partial data (shared data) stored in a preset storage structure in the transmission process is realized based on the hash abstract, and different access strategies can be realized after the shared data is processed by an ABE encryption algorithm in the transmission process, so that a one-to-many partial data sharing mode is realized, and the granularity of data sharing is refined.
A8 the method of any one of A1 to A7, further comprising the step of:
and updating the name of the data block stored in the current node to be the renamed name in response to the operation of renaming the name of the data block stored in one node of the preset storage structure.
A9 the method of a2, wherein the first preset number is equal to the maximum storage capacity of each node in the predetermined storage structure.
A11 the method as in a10, wherein the operation of the receiver after receiving the encrypted data is as follows:
decrypting the encrypted data through the data encryption algorithm to obtain the target data;
splitting the target data to obtain first data to be transmitted and the first hash value;
performing hash operation on the first data to be transmitted to obtain a second hash value;
and judging whether the second hash value is the same as the first hash value or not, if so, using the data to be transmitted, and if not, not using the data to be transmitted.
A12 the method of a10 or a11, wherein the data encryption algorithm is an ABE encryption algorithm.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the data storage method and the data transmission method of the present invention according to instructions in the program code stored in the memory.
By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may additionally be divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (14)
1. A data storage method, executed in a computing device, the method comprising the steps of:
acquiring data to be stored;
splitting a data object to be stored into a plurality of data blocks, and assigning a unique name to each data block;
and storing each data block into a node of a preset storage structure, wherein each node in the preset storage structure corresponds to a data server, each node is allocated with a unique node ID, the maximum storage capacity of each node is the same, all leaf nodes except the rightmost leaf node store data with the maximum capacity, each node in the preset storage structure is used for storing first-class data, the first-class data comprises one or more of a father node ID, a directory name, the data block, a data block name and a node ID, and the directory name is the directory name of the data block stored by the current node corresponding to the child node.
2. The method of claim 1, wherein the step of splitting the data object to be stored into a plurality of data blocks comprises:
determining the current offset of the data pointer of the data object to be stored;
taking the determined current offset of the data pointer as an initial position, and reading data of a first preset numerical value from the data to be stored as a data block;
and updating the current offset of the data pointer to be the sum of the determined current offset of the data pointer and the first preset numerical value.
3. The method of claim 2, wherein the step of reading the data of the first preset value from the data to be stored as a data block with the determined current offset of the data pointer as a starting position comprises:
judging whether the product of the current times of reading the data to be stored and a first preset value is smaller than the size of the data to be stored, if so, reading the data of the first preset value by taking the determined current offset of the data pointer as an initial position, if not, determining the quantity of the data to be stored to be taken as a second value by taking the determined current offset of the data pointer as the initial position, and reading the data of the second value by taking the determined current offset of the data pointer as the initial position:
L＝m-(a-1)*b
wherein, L is the number of data to be stored read, m is the size of the storage space occupied by the data to be stored, a is the current number of times of reading the data to be stored, and b is the first preset value;
the read data is used as a data block.
4. A method as claimed in any one of claims 1 to 3, wherein the step of saving each said data block to a node of a predetermined storage structure comprises:
and judging whether the current node has a child node, if so, storing the parent node ID, the current node ID and the directory name of the current node, and if not, storing the parent node ID, the data block name and the current node ID of the current node.
5. The method of claim 4, further comprising the steps of:
and judging whether each node in the preset storage structure has a left node or not, and if so, storing the left node ID in the current node.
6. The method of claim 5, further comprising the steps of:
setting version information for each data block, wherein the version of the data block is distinguished through the version information of the data block;
wherein, the step of judging whether the current node has a child node comprises the following steps:
and judging whether the current node has a child node, if so, storing the father node ID, the current node ID, the left node ID, the directory name and the data block version information of the current node, and if not, storing the father node ID, the current node ID, the left node ID, the data block name and the data block version information of the current node.
7. The method of claim 6, further comprising the steps of:
responding to the operation of replacing the data block in one leaf node of the preset storage structure by the target data block, judging whether the data block name of the target data block is the same as the data block name in the current node, if not, carrying out replacement, and if so, judging whether the data block version information of the target data block is the same as the data block version information in the current node;
and if the version information of the data block of the target data block is the same as the version information of the data block in the current node, replacing the data block in the current node with the target data block, and if the version information of the data block in the current node is not the same as the version information of the data block in the current node, not replacing.
8. The method of any of claims 1 to 3, further comprising the step of:
and updating the name of the data block stored in the current node to be the renamed name in response to the operation of renaming the name of the data block stored in one node of the preset storage structure.
9. The method of claim 2, wherein the first predetermined value is equal to a maximum storage capacity of each node in the predetermined storage structure.
10. A data transmission method, executed in a computing device, the method comprising:
acquiring data to be transmitted, wherein the data to be transmitted is stored according to the data storage method of any one of claims 1 to 9 and stored by each node in a preset storage structure;
performing hash operation on the data to be transmitted to obtain a first hash value;
combining the data to be transmitted and the first hash value to serve as target data;
encrypting the target data through a data encryption algorithm to obtain encrypted data;
and sending the encrypted data to a receiving end.
11. The method of claim 10, wherein the receiving end operates after receiving the encrypted data by:
decrypting the encrypted data through the data encryption algorithm to obtain the target data;
splitting the target data to obtain first data to be transmitted and the first hash value;
performing hash operation on the first data to be transmitted to obtain a second hash value;
and judging whether the second hash value is the same as the first hash value or not, if so, using the data to be transmitted, and if not, not using the data to be transmitted.
12. The method of claim 10 or 11, wherein the data encryption algorithm is an ABE encryption algorithm.
13. A computing device, comprising:
at least one processor; and
a memory storing program instructions, wherein the program instructions are configured to be adapted to be executed by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 1-12.
14. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method of any of claims 1-12.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110429687.0A CN113194127B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
| CN202210800036.2A CN115190136B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110429687.0A CN113194127B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210800036.2A Division CN115190136B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113194127A CN113194127A (en) | 2021-07-30 |
| CN113194127B true CN113194127B (en) | 2022-08-12 |
Family
ID=76977821
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210800036.2A Active CN115190136B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
| CN202110429687.0A Active CN113194127B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210800036.2A Active CN115190136B (en) | 2021-04-21 | 2021-04-21 | Data storage method, data transmission method and computing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN115190136B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1233023A (en) * | 1998-02-24 | 1999-10-27 | 佳能株式会社 | Data communication system, data communication method, data communication apparatus and digital interface |
| CN101421993A (en) * | 2006-04-12 | 2009-04-29 | 兄弟工业株式会社 | Node device, storage control program, and information storage method |
| CN103282899A (en) * | 2011-12-23 | 2013-09-04 | 华为技术有限公司 | File system data storage method and access method and device therefor |
| US9432192B1 (en) * | 2014-03-28 | 2016-08-30 | Emc Corporation | Content aware hierarchical encryption for secure storage systems |
| CN108628753A (en) * | 2017-03-24 | 2018-10-09 | 华为技术有限公司 | Memory headroom management method and device |
| CN111316251A (en) * | 2017-11-06 | 2020-06-19 | 海量数据有限公司 | Scalable storage system |
| CN111414426A (en) * | 2020-03-26 | 2020-07-14 | 北京云图科瑞科技有限公司 | Data processing method and system based on block chain |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243334B (en) * | 2015-09-17 | 2018-10-30 | 浪潮(北京)电子信息产业有限公司 | A kind of data storage protection method and system |
| CN110569213A (en) * | 2018-05-18 | 2019-12-13 | 北京果仁宝软件技术有限责任公司 | File access method, device and equipment |
| CN109254733B (en) * | 2018-09-04 | 2021-10-01 | 北京百度网讯科技有限公司 | Method, device and system for storing data |
| CN110162525B (en) * | 2019-04-17 | 2023-09-26 | 平安科技（深圳）有限公司 | B+ tree-based read-write conflict resolution method, device and storage medium |
| CN111858146A (en) * | 2019-04-26 | 2020-10-30 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for recovering data |
| CN110149373B (en) * | 2019-04-28 | 2022-01-07 | 平安科技（深圳）有限公司 | Data storage method, device, medium and server based on peer-to-peer network |
| CN110737658B (en) * | 2019-09-06 | 2020-12-18 | 平安国际智慧城市科技股份有限公司 | Data fragment storage method, device, terminal and readable storage medium |
| CN111309523A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Data reading and writing method, data remote copying method and device and distributed storage system |
| CN111309260B (en) * | 2020-02-16 | 2021-04-09 | 西安奥卡云数据科技有限公司 | Data storage node selection method |
-
2021
- 2021-04-21 CN CN202210800036.2A patent/CN115190136B/en active Active
- 2021-04-21 CN CN202110429687.0A patent/CN113194127B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1233023A (en) * | 1998-02-24 | 1999-10-27 | 佳能株式会社 | Data communication system, data communication method, data communication apparatus and digital interface |
| CN101421993A (en) * | 2006-04-12 | 2009-04-29 | 兄弟工业株式会社 | Node device, storage control program, and information storage method |
| CN103282899A (en) * | 2011-12-23 | 2013-09-04 | 华为技术有限公司 | File system data storage method and access method and device therefor |
| US9432192B1 (en) * | 2014-03-28 | 2016-08-30 | Emc Corporation | Content aware hierarchical encryption for secure storage systems |
| CN108628753A (en) * | 2017-03-24 | 2018-10-09 | 华为技术有限公司 | Memory headroom management method and device |
| CN111316251A (en) * | 2017-11-06 | 2020-06-19 | 海量数据有限公司 | Scalable storage system |
| CN111414426A (en) * | 2020-03-26 | 2020-07-14 | 北京云图科瑞科技有限公司 | Data processing method and system based on block chain |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113194127A (en) | 2021-07-30 |
| CN115190136A (en) | 2022-10-14 |
| CN115190136B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10394847B2 (en) | Processing data in a distributed database across a plurality of clusters | |
| US11823178B2 (en) | Optimization of high volume transaction performance on a blockchain | |
| US10255108B2 (en) | Parallel execution of blockchain transactions | |
| TW202016815A (en) | Method, apparatus and system for executing block chain transaction in parallelized manner | |
| JP2021526751A (en) | Secure consensus endorsement for self-monitoring blockchain | |
| TW202025685A (en) | Cross-chain evidence preservation method and access method, apparatus, and electronic device | |
| WO2021217863A1 (en) | Order identifier generation method and apparatus, server, and storage medium | |
| US9830333B1 (en) | Deterministic data replication with conflict resolution | |
| EP2858024A1 (en) | An asset management device and method in a hardware platform | |
| US10855637B2 (en) | Architecture for large data management in communication applications through multiple mailboxes | |
| CN112632071A (en) | Database primary key id generation method, device, equipment and storage medium | |
| WO2021174836A1 (en) | Differential package generation method and apparatus, computer device, and storage medium | |
| US11360966B2 (en) | Information processing system and method of controlling information processing system | |
| WO2022222350A1 (en) | Method for encrypting data, and computing device | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| WO2023236756A1 (en) | Multi-tenant data database allocation method, and program product and electronic device | |
| WO2023207529A1 (en) | Data processing method and apparatus, device, medium, and product | |
| CN112925954A (en) | Method and apparatus for querying data in a graph database | |
| CN113194127B (en) | Data storage method, data transmission method and computing equipment | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| WO2023142605A1 (en) | Blockchain-based data processing method and related apparatus | |
| CN110889040B (en) | Method and device for pushing information | |
| CN110417777A (en) | A kind of method and device communicated between micro services of optimization | |
| CN110995574A (en) | User information management method, acquisition method and system | |
| US20230046579A1 (en) | Information security system and method for secure data transmission among user profiles using a blockchain network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |