CN112256472A - Distributed data calling method and device, electronic equipment and storage medium - Google Patents
Distributed data calling method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112256472A CN112256472A CN202011127804.XA CN202011127804A CN112256472A CN 112256472 A CN112256472 A CN 112256472A CN 202011127804 A CN202011127804 A CN 202011127804A CN 112256472 A CN112256472 A CN 112256472A
- Authority
- CN
- China
- Prior art keywords
- data
- blocks
- constructed
- parts
- calling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 239000011159 matrix material Substances 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 22
- 230000004044 response Effects 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 8
- 229940050561 matrix product Drugs 0.000 claims description 4
- 238000000638 solvent extraction Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 2
- 230000006870 function Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000007620 mathematical function Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data processing, and discloses a distributed data calling method, which comprises the following steps: acquiring a position information set of an element block related to a calling instruction input by a user; calling all the element blocks according to the position information set, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value; and carrying out matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user. The invention also provides a distributed data retrieval device, electronic equipment and a computer readable storage medium. The invention also relates to a blockchain technique, the element blocks being storable in blockchain link points. The invention aims to solve the problem that when the distributed storage data is called, the overall calling speed is reduced because the individual data blocks are called too slowly.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a distributed data retrieving method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of computers, a distributed storage mode with higher utilization rate becomes the mainstream of the technology of storage products nowadays. The distributed storage mode is divided into two forms of multi-copy and erasure code, wherein the erasure code cuts initial data into X data blocks and constructs Y verification blocks. When data is extracted, X data blocks are called for use, and when a certain data block is lost, a corresponding check block is extracted, so that the data can be restored.
When the erasure code is called, the computer always calls the data block in a slow disk without calling the parity block when a certain data block is stored in the slow disk, so that the calling speed is reduced along with the reduction of the calling speed of the certain data block.
Disclosure of Invention
The invention provides a distributed data retrieval method, a distributed data retrieval device, electronic equipment and a computer-readable storage medium, and mainly aims to solve the problem that when retrieving distributed storage data, the overall retrieval speed is reduced due to too slow retrieval of individual data blocks.
In order to achieve the above object, the present invention provides a distributed data retrieving method, including:
acquiring a position information set of an element block related to a data calling instruction input by a user;
according to the position information set, simultaneously calling all the element blocks, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value;
and carrying out matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user. Optionally, before the obtaining of the location information set of the element block related to the data retrieval instruction according to the data retrieval instruction input by the user, the method includes:
acquiring original data, generating X data blocks, and generating Y check blocks through a pre-constructed erasure code;
and storing the X data blocks and the Y check blocks into each pre-constructed chunk node by using a data distribution algorithm, and returning the position information sets of the X data blocks and the Y check blocks.
And the element block is formed by the X parts of data blocks and the Y parts of check blocks.
Optionally, the obtaining of the original data and generating X parts of data blocks and generating Y parts of check blocks through the pre-constructed erasure codes include:
encoding the original data by using an encoder to obtain compiled data;
and partitioning the compiled data to obtain X parts of data blocks, and performing matrix product operation on the X parts of data blocks and a preset coding matrix by using the erasure codes to obtain Y parts of check blocks.
Optionally, the segmenting the compiled data to obtain X data blocks includes:
inserting a preset number of blank data nodes into the compiled data, copying preset byte data at the front end of the data nodes, and storing the preset byte data into the blank data nodes;
and dividing the front end of the data node as a dividing line to obtain X parts of the data blocks.
Optionally, the obtaining of the original data to generate X parts of data blocks, and before generating Y parts of check blocks through the pre-constructed erasure code, includes:
constructing a metadata server node and a chunk node, and connecting the metadata server node and the chunk node;
connecting the metadata server node with the pre-built database management system.
Optionally, the storing the X parts of data blocks and the Y parts of check blocks into each pre-constructed chunk node includes:
calculating a hash value by using a preset data distribution algorithm and the characteristics of the element block, and establishing a mapping relation between the hash value and the database management system;
and storing the element blocks corresponding to different hash values into each chunk node by using the mapping relation.
Optionally, the matrix processing is performed on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and the original data is returned to the user, where the matrix processing includes:
performing matrix multiplication on the data block and the check block by using the erasure code, and removing the coding matrix to obtain an initial data matrix;
and decoding the initial data matrix to obtain initial data, and transmitting the initial data to a user through a pre-constructed database management system.
In order to solve the above problem, the present invention further provides a distributed data retrieving apparatus, including:
the retrieval module is used for acquiring a position information set of an element block related to a data calling instruction input by a user;
the calling module is used for calling all the element blocks simultaneously according to the position information set, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value;
and the data processing module is used for carrying out matrix processing on the element blocks stored in the pre-constructed list by utilizing the pre-constructed erasure codes to obtain original data and returning the original data to the user.
In order to solve the above problem, the present invention also provides an electronic device, including: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the distributed data retrieval method described above.
In order to solve the above problem, the present invention further provides a computer-readable storage medium comprising a storage data area and a storage program area, wherein the storage data area stores created data, and the storage program area stores a computer program; wherein the computer program, when executed by a processor, implements the distributed data retrieval method described above.
In the embodiment of the invention, according to a data calling instruction input by a user, position information sets of all element blocks related to the calling instruction are acquired, according to the position information sets, all the element blocks are called simultaneously, and according to the response time of each element block, the element blocks are sequentially stored into a pre-constructed list until the number of the element blocks stored into the list is equal to a preset value without waiting for data calling in a slow disk. Therefore, the distributed data retrieval method, the distributed data retrieval device, the electronic device and the computer readable storage medium solve the problem that the whole process is affected by slow storage response of a certain data block in the distributed storage retrieval process.
Drawings
Fig. 1 is a schematic flowchart of a distributed data retrieval method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a detailed implementation of one step in the distributed data retrieval method provided in FIG. 1;
fig. 3 is a schematic diagram illustrating an erasure code generating check block according to an embodiment of the present invention;
fig. 4 is a block diagram of a distributed data retrieval apparatus according to an embodiment of the present invention;
fig. 5 is a schematic internal structural diagram of an electronic device implementing a distributed data retrieval method according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a distributed data retrieval method. The execution subject of the distributed data retrieval method provided by the embodiment of the present application includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the distributed data retrieval method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Referring to fig. 1, a distributed data retrieving method according to an embodiment of the present invention includes:
and S1, acquiring a position information set of the element block related to the calling instruction according to the data calling instruction input by the user.
In the embodiment of the present invention, the element block includes a data block and a check block, where the data block and the check block are both generated from original data.
In the embodiment of the invention, when a user needs to call original data, query information is input, a call instruction corresponding to the query information is generated through a pre-constructed database management system, and a position information set of the data block and the check block is obtained according to the call instruction. In this embodiment of the present invention, the location information may be physical information.
In one embodiment of the present invention, before S1, the method includes:
constructing a metadata server node and a chunk node, and connecting the metadata server node and the chunk node; further connecting the metadata server node with the pre-built database management system.
The chunk nodes are built in each hard disk and serve as final places for storing the data blocks or the check blocks, and the metadata server nodes are internally provided with erasure codes and serve as transfer places of the data blocks and the check blocks.
Wherein the erasure code creates a mathematical function to describe the original data to check the accuracy of the original data, and can also perform recovery on the original data once there is data loss in the original data.
In detail, as shown in fig. 2, in one embodiment of the present invention, before S1, the method further includes:
and S11, acquiring the original data, generating X data blocks, and generating Y check blocks through the pre-constructed erasure codes.
In an embodiment of the present invention, the erasure code uses Reed-Solomon (hereinafter abbreviated as RS) code, and the RS code is a type of erasure code commonly used in a storage system, and has two parameters X and Y, which are denoted as RS (X, Y). X represents the number of original data blocks, and Y represents the number of check blocks.
Referring to fig. 3, taking X as 5, Y as 3, that is, 5 original data blocks and 3 parity blocks as an example, the 5 data blocks (that is, the portion denoted by D) are multiplied by a matrix of (n + m) × n (that is, the portion denoted by B) to obtain a result matrix of (n + m) × 1, and according to the characteristics of the result matrix, the former 5 values in the result matrix are equal to the values of the original 5 data blocks, and the last 3 are calculated parity blocks.
In detail, in the embodiment of the present invention, the S11 further includes:
acquiring the original data, and encoding the original data by using an encoder to obtain compiled data; and partitioning the compiled data to obtain X parts of data blocks, and performing matrix product operation on the X parts of data blocks and a preset coding matrix by using the erasure codes to obtain Y parts of check blocks.
Further, in this embodiment of the present invention, the dividing the compiled data includes:
inserting a preset number of blank data nodes into the compiled data, copying preset byte data at the front end of the data nodes, and storing the preset byte data into the blank data nodes; and dividing the front end of the data node as a dividing line to obtain X parts of the data blocks.
In the embodiment of the invention, the front end of the data node is taken as a parting line to obtain X parts of the data blocks, so that each part of the data blocks has a little data information related to other data.
S12, storing the X data blocks and the Y check blocks into each pre-constructed chunk node, and returning the position information sets of the X data blocks and the Y check blocks.
Calculating a hash value by using a preset data distribution algorithm and the characteristics of the element block, and establishing a mapping relation between the hash value and the database management system; and storing the element blocks corresponding to different hash values into each chunk node by using the mapping relation.
In the embodiment of the invention, the data distribution algorithm uses a consistent hash (consistent hash) algorithm, and the consistent hash introduces the idea of virtual nodes. That is, virtual nodes representing a certain chunk node are interspersed among the chunk nodes, the hash function output value range is a closed ring, and the maximum value of the hash function output is the preamble of the minimum value.
In the embodiment of the invention, the metadata server node is used for distributing the data block and the check block to different chunk nodes, wherein the chunk nodes are in different hard disks, and the hard disks exist in computers distributed in all corners of a company, so that the aim of safely storing data is fulfilled. And returning the position information sets of the X data blocks and the Y check blocks, and ensuring that the X data blocks and the Y check blocks can be inquired when the database management system is used.
In another embodiment of the present invention, the chunk nodes may be divided into a hot area and a cold area, and data in the hot area (about 70%) has a higher access frequency, so to ensure performance, the nodes in the hot area are always in an active state; the data usage of the cold zone (about 30%) is very low, so the nodes of the cold zone will enter the power saving mode. Therefore, in the embodiment of the invention, X parts of element blocks can be stored in the hot area, and the rest Y parts of element blocks can be stored in the cold area, and when the element blocks in the hot area are damaged or lost, the element blocks with the fastest response in the cold area can be supplemented to the hot area, so as to further improve the data reading speed.
S2, calling all the element blocks according to the position information set, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset value.
The embodiment of the invention firstly constructs a list, acquires the physical addresses of the data block and the check block and extracts the data block and the check block. Because the response speed of the hard disk where the chunk nodes stored in the data block and the check block are located is different, the embodiment of the invention stores the data block and the check block into the list one by one according to the response speed.
In the embodiment of the invention, when the quantity stored in the list is equal to a preset value, the element blocks are stopped to be stored in the list.
In the embodiment of the present invention, the preset value may be set to X.
In an embodiment of the present invention, RS erasure code is applied to distributed storage, and in an erasure code rule, there are X parts of the data block and Y parts of the check block, and in the X + Y parts, the original data can be generated through a reverse process of the erasure code generation check block principle of fig. 3 as long as X parts of any data block or check block are obtained.
And S3, performing matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user.
In detail, in an embodiment of the present invention, the S3 includes:
performing matrix multiplication on the data block and the check block by using the erasure code, and removing the coding matrix to obtain an initial data matrix; and decoding the initial data matrix to obtain initial data, and transmitting the initial data to a user through a pre-constructed database management system.
In the embodiment of the invention, according to a data calling instruction input by a user, position information sets of all element blocks related to the calling instruction are acquired, according to the position information sets, all the element blocks are called simultaneously, and according to the response time of each element block, the element blocks are sequentially stored into a pre-constructed list until the number of the element blocks stored into the list is equal to a preset value without waiting for data calling in a slow disk. Therefore, the embodiment of the invention solves the problem that the distributed storage extraction process influences the data retrieval process because the storage response of a certain data block is slow.
Fig. 4 is a schematic block diagram of the distributed data retrieval apparatus according to the present invention.
The distributed data retrieval apparatus 100 according to the present invention may be installed in an electronic device. According to the implemented functions, the distributed data retrieval apparatus 100 may include a retrieval module 101, a retrieval module 102, and a data processing module 103. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the retrieval module 101 is configured to obtain, according to a data retrieval instruction input by a user, position information sets of all element blocks related to the retrieval instruction.
In the embodiment of the present invention, the element block includes a data block and a check block, where the data block and the check block are both generated from original data.
In the embodiment of the invention, when a user needs to call original data, query information is input, a call instruction corresponding to the query information is generated through a pre-constructed database management system, and a position information set of the data block and the check block is obtained according to the call instruction. In this embodiment of the present invention, the location information may be physical information.
In one embodiment of the present invention, the retrieving module 101 is further configured to:
constructing a metadata server node and a chunk node, and connecting the metadata server node and the chunk node; further connecting the metadata server node with the pre-built database management system.
The chunk nodes are built in each hard disk and serve as final places for storing the data blocks or the check blocks, and the metadata server nodes are internally provided with erasure codes and serve as transfer places of the data blocks and the check blocks.
Wherein the erasure code creates a mathematical function to describe the original data to check the accuracy of the original data, and can also perform recovery on the original data once there is data loss in the original data.
In detail, in one embodiment of the present invention, the retrieving module 101 is further configured to: acquiring the original data, generating X data blocks, and generating Y check blocks by using a pre-constructed erasure code; and storing the X data blocks and the Y check blocks into each pre-constructed chunk node by using a data distribution algorithm, and returning the position information sets of the X data blocks and the Y check blocks.
In an embodiment of the present invention, the erasure code uses Reed-Solomon (hereinafter abbreviated as RS) code, and the RS code is a type of erasure code commonly used in a storage system, and has two parameters X and Y, which are denoted as RS (X, Y). X represents the number of original data blocks, and Y represents the number of check blocks.
Referring to fig. 3, taking X as 5, Y as 3, that is, 5 original data blocks and 3 parity blocks as an example, the 5 data blocks (that is, the portion denoted by D) are multiplied by a matrix of (n + m) × n (that is, the portion denoted by B) to obtain a result matrix of (n + m) × 1, and according to the characteristics of the result matrix, the former 5 values in the result matrix are equal to the values of the original 5 data blocks, and the last 3 are calculated parity blocks.
In detail, in the embodiment of the present invention, when the original data is acquired, X data blocks are generated, and Y check blocks are generated by using a pre-constructed erasure code, the retrieval module 101 is specifically configured to:
acquiring the original data, and encoding the original data by using an encoder to obtain compiled data; and partitioning the compiled data to obtain X parts of data blocks, and performing matrix product operation on the X parts of data blocks and a preset coding matrix by using the erasure codes to obtain Y parts of check blocks.
Further, in this embodiment of the present invention, the dividing the compiled data includes:
inserting a preset number of blank data nodes into the compiled data, copying preset byte data at the front end of the data nodes, and storing the preset byte data into the blank data nodes; and dividing the front end of the data node as a dividing line to obtain X parts of the data blocks.
In the embodiment of the invention, the front end of the data node is taken as a parting line to obtain X parts of the data blocks, so that each part of the data blocks has a little data information related to other data.
Calculating a hash value by using a preset data distribution algorithm and the characteristics of the element block, and establishing a mapping relation between the hash value and the database management system; and storing the element blocks corresponding to different hash values into each chunk node by using the mapping relation.
In the embodiment of the invention, the data distribution algorithm uses a consistent hash (consistent hash) algorithm, and the consistent hash introduces the idea of virtual nodes. That is, virtual nodes representing a certain chunk node are interspersed among the chunk nodes, the hash function output value range is a closed ring, and the maximum value of the hash function output is the preamble of the minimum value.
The embodiment of the invention utilizes the metadata server node to distribute the data block and the check block to different chunk nodes, wherein the chunk nodes are in different hard disks, and the hard disks exist in all computers distributed in all corners of a company, so that the aim of safely storing data is fulfilled. And returning the position information sets of the X data blocks and the Y check blocks, and ensuring that the X data blocks and the Y check blocks can be inquired when the database management system is used.
Further, in another embodiment of the present invention, the chunk nodes may be divided into a hot area and a cold area, and data in the hot area (about 70%) has a higher access frequency, so to ensure performance, the nodes in the hot area are always in an active state; the data usage of the cold zone (about 30%) is very low, so the nodes of the cold zone will enter the power saving mode.
The retrieval module 101 according to the embodiment of the present invention may store X parts of the element blocks in the hot area and store the remaining Y parts of the element blocks in the cold area, and when an element block in the hot area is damaged or lost, the element block with the fastest response in the cold area may be supplemented to the hot area, so as to further improve the data reading speed.
The retrieving module 102 is configured to retrieve all the element blocks according to the location information set, and sequentially store the element blocks in a pre-constructed list according to response time of each element block until the number of the element blocks stored in the list is equal to a preset value.
In the embodiment of the present invention, the invoking module 102 is utilized to construct a list, obtain the physical addresses of the data block and the check block, and extract the data block and the check block at the same time. Because the response speed of the hard disk where the chunk nodes stored in the data block and the check block are located is different, the embodiment of the invention stores the data block and the check block into the list one by one according to the response speed.
In this embodiment of the present invention, when the number stored in the list is equal to a preset value, the invoking module 102 stops storing the element block in the list.
In the embodiment of the present invention, the preset value may be set to X.
In an embodiment of the present invention, RS erasure code is applied to distributed storage, and in an erasure code rule, there are X parts of the data block and Y parts of the check block, and in the X + Y parts, the original data can be generated through a reverse process of the erasure code generation check block principle of fig. 3 as long as X parts of any data block or check block are obtained.
The data processing module 103 is configured to perform matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and return the original data to the user.
In detail, in the embodiment of the present invention, the data processing module 103 performs matrix processing on the element blocks stored in the pre-constructed list by the following operations to obtain original data, and returns the original data to the user:
performing matrix multiplication on the data block and the check block by using the erasure code, and removing the coding matrix to obtain an initial data matrix; and decoding the initial data matrix to obtain initial data, and transmitting the initial data to a user through a pre-constructed database management system.
Fig. 5 is a schematic structural diagram of an electronic device implementing the distributed data retrieval method according to the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a distributed data retrieval program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of the distributed data retrieval program 12, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (for example, executing a distributed data retrieval program and the like) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 5 only shows an electronic device with components, and it will be understood by a person skilled in the art that the structure shown in fig. 5 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or a combination of certain components, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The distributed data retrieval program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
acquiring a position information set of an element block related to a data calling instruction input by a user;
according to the position information set, simultaneously calling all the element blocks, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value;
and carrying out matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
Further, the computer usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any accompanying claims should not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data calling, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method for calling distributed data is characterized by comprising the following steps:
acquiring a position information set of an element block related to a data calling instruction input by a user;
calling the element blocks according to the position information set, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value;
and carrying out matrix processing on the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user.
2. The method for calling distributed data according to claim 1, wherein before acquiring the location information set of the element block associated with the call instruction according to the data call instruction input by the user, the method comprises:
acquiring original data, generating X data blocks, and generating Y check blocks through a pre-constructed erasure code;
storing the X data blocks and the Y check blocks into each pre-constructed chunk node, and returning the position information sets of the X data blocks and the Y check blocks;
and the element block is formed by the X parts of data blocks and the Y parts of check blocks.
3. The method for retrieving distributed data according to claim 2, wherein the obtaining of original data to generate X parts of data blocks and generating Y parts of check blocks through a pre-constructed erasure code comprises:
encoding the original data by using an encoder to obtain compiled data;
and partitioning the compiled data to obtain X parts of data blocks, and performing matrix product operation on the X parts of data blocks and a preset coding matrix by using the erasure codes to obtain Y parts of check blocks.
4. The method of claim 3, wherein the splitting the compiled data into X data blocks comprises:
inserting a preset number of blank data nodes into the compiled data, copying preset byte data at the front end of the data nodes, and storing the preset byte data into the blank data nodes;
and dividing the front end of the data node as a dividing line to obtain X parts of the data blocks.
5. The method for retrieving distributed data according to claim 3, wherein before the obtaining of the original data to generate X parts of data blocks and generating Y parts of check blocks through the pre-constructed erasure codes, the method comprises:
constructing a metadata server node and a chunk node, and connecting the metadata server node and the chunk node;
connecting the metadata server node with the pre-built database management system.
6. The method for invoking distributed data according to claim 2, wherein said storing said X parts of data blocks and said Y parts of check blocks into each pre-constructed chunk node comprises:
calculating a hash value by using a preset data distribution algorithm and the characteristics of the element block, and establishing a mapping relation between the hash value and the database management system;
and storing the element blocks corresponding to different hash values into each chunk node by using the mapping relation.
7. The method according to any one of claims 1 to 6, wherein the matrix processing the element blocks stored in the pre-constructed list by using the pre-constructed erasure codes to obtain original data, and returning the original data to the user includes:
performing matrix multiplication on the data block and the check block by using the erasure code, and removing the coding matrix to obtain an initial data matrix;
and decoding the initial data matrix to obtain initial data, and transmitting the initial data to a user through a pre-constructed database management system.
8. A distributed data retrieval apparatus, the apparatus comprising:
the retrieval module is used for acquiring a position information set of an element block related to a data calling instruction input by a user;
the calling module is used for calling all the element blocks simultaneously according to the position information set, and sequentially storing the element blocks into a pre-constructed list according to the response time of each element block until the number of the element blocks stored into the list is equal to a preset numerical value;
and the data processing module is used for carrying out matrix processing on the element blocks stored in the pre-constructed list by utilizing the pre-constructed erasure codes to obtain original data and returning the original data to the user.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores computer program instructions executable by the at least one processor to enable the at least one processor to perform the distributed data retrieval method of any one of claims 1 to 7.
10. A computer-readable storage medium comprising a storage data area and a storage program area, wherein the storage data area stores created data, and the storage program area stores a computer program; wherein the computer program, when executed by a processor, implements the distributed data retrieval method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011127804.XA CN112256472B (en) | 2020-10-20 | 2020-10-20 | Distributed data retrieval method and device, electronic equipment and storage medium |
| PCT/CN2020/131985 WO2021189905A1 (en) | 2020-10-20 | 2020-11-27 | Distributed data retrieval method and apparatus, and electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011127804.XA CN112256472B (en) | 2020-10-20 | 2020-10-20 | Distributed data retrieval method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112256472A true CN112256472A (en) | 2021-01-22 |
| CN112256472B CN112256472B (en) | 2024-06-25 |
Family
ID=74245227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011127804.XA Active CN112256472B (en) | 2020-10-20 | 2020-10-20 | Distributed data retrieval method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112256472B (en) |
| WO (1) | WO2021189905A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114594911B (en) * | 2022-03-13 | 2024-03-29 | 西安电子科技大学 | Block chain data storage system and method based on under-chain erasure code distributed storage |
| CN117349075A (en) * | 2022-06-27 | 2024-01-05 | 华为技术有限公司 | Data processing method and related equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095352A (en) * | 2015-06-19 | 2015-11-25 | 曙光信息产业股份有限公司 | Data processing method and apparatus applied to distributed system |
| CN110278222A (en) * | 2018-03-15 | 2019-09-24 | 华为技术有限公司 | The method, system and relevant device of data management in distributed file storage system |
| CN110674175A (en) * | 2019-09-26 | 2020-01-10 | 浪潮商用机器有限公司 | Object storage method, device and equipment and readable storage medium |
| CN110990189A (en) * | 2019-11-29 | 2020-04-10 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
| CN111046014A (en) * | 2019-12-24 | 2020-04-21 | 广州众鑫达科技有限公司 | File distributed storage system, method and device and electronic equipment |
| CN111541753A (en) * | 2020-04-16 | 2020-08-14 | 深圳市网心科技有限公司 | Distributed storage system, method, computer device and medium for block chain data |
| CN111682874A (en) * | 2020-06-11 | 2020-09-18 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, equipment and readable storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106469100B (en) * | 2015-08-17 | 2019-04-05 | 华为技术有限公司 | A kind of method that data are restored, the corresponding apparatus and system of the method for storage |
| CN109491968B (en) * | 2018-11-13 | 2021-01-22 | 恒生电子股份有限公司 | File processing method, device, equipment and computer readable storage medium |
| CN110175212B (en) * | 2019-05-22 | 2021-07-06 | 杭州复杂美科技有限公司 | Block chain distributed storage method, data reading method, device and storage medium |
-
2020
- 2020-10-20 CN CN202011127804.XA patent/CN112256472B/en active Active
- 2020-11-27 WO PCT/CN2020/131985 patent/WO2021189905A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095352A (en) * | 2015-06-19 | 2015-11-25 | 曙光信息产业股份有限公司 | Data processing method and apparatus applied to distributed system |
| CN110278222A (en) * | 2018-03-15 | 2019-09-24 | 华为技术有限公司 | The method, system and relevant device of data management in distributed file storage system |
| CN110674175A (en) * | 2019-09-26 | 2020-01-10 | 浪潮商用机器有限公司 | Object storage method, device and equipment and readable storage medium |
| CN110990189A (en) * | 2019-11-29 | 2020-04-10 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
| CN111046014A (en) * | 2019-12-24 | 2020-04-21 | 广州众鑫达科技有限公司 | File distributed storage system, method and device and electronic equipment |
| CN111541753A (en) * | 2020-04-16 | 2020-08-14 | 深圳市网心科技有限公司 | Distributed storage system, method, computer device and medium for block chain data |
| CN111682874A (en) * | 2020-06-11 | 2020-09-18 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112256472B (en) | 2024-06-25 |
| WO2021189905A1 (en) | 2021-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112052370A (en) | Message generation method and device, electronic equipment and computer readable storage medium | |
| CN112506486A (en) | Search system establishing method and device, electronic equipment and readable storage medium | |
| CN112256472B (en) | Distributed data retrieval method and device, electronic equipment and storage medium | |
| CN111651453A (en) | User historical behavior query method and device, electronic equipment and storage medium | |
| CN114491047A (en) | Multi-label text classification method and device, electronic equipment and storage medium | |
| CN114185895A (en) | Data import and export method and device, electronic equipment and storage medium | |
| CN113890712A (en) | Data transmission method and device, electronic equipment and readable storage medium | |
| CN112580079A (en) | Authority configuration method and device, electronic equipment and readable storage medium | |
| CN111985194A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN114398346A (en) | Data migration method, device, equipment and storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN113722533A (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN112464619A (en) | Big data processing method, device and equipment and computer readable storage medium | |
| CN112948380A (en) | Data storage method and device based on big data, electronic equipment and storage medium | |
| CN111538768A (en) | Data query method and device based on N-element model, electronic equipment and medium | |
| CN115687384A (en) | UUID (user identifier) identification generation method, device, equipment and storage medium | |
| CN115827115A (en) | H5 page jump method, device, equipment and storage medium of WeChat applet | |
| CN114840388A (en) | Data monitoring method and device, electronic equipment and storage medium | |
| CN115033605A (en) | Data query method and device, electronic equipment and storage medium | |
| CN114547011A (en) | Data extraction method and device, electronic equipment and storage medium | |
| CN114448930A (en) | Short address generation method and device, electronic equipment and computer readable storage medium | |
| CN114238233A (en) | Automatic file cleaning method, device, equipment and storage medium | |
| CN113076069A (en) | Double-cache method and device, electronic equipment and computer readable storage medium | |
| CN113590856B (en) | Label query method and device, electronic equipment and readable storage medium | |
| CN114756564B (en) | Data processing method, device, equipment and medium for stream computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |