WO2014075273A1 - Procédé, dispositif et système de lecture/écriture de données - Google Patents

Procédé, dispositif et système de lecture/écriture de données Download PDF

Info

Publication number
WO2014075273A1
WO2014075273A1 PCT/CN2012/084684 CN2012084684W WO2014075273A1 WO 2014075273 A1 WO2014075273 A1 WO 2014075273A1 CN 2012084684 W CN2012084684 W CN 2012084684W WO 2014075273 A1 WO2014075273 A1 WO 2014075273A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
index
storage medium
physical address
index block
Prior art date
Application number
PCT/CN2012/084684
Other languages
English (en)
Chinese (zh)
Inventor
雷晓松
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2012/084684 priority Critical patent/WO2014075273A1/fr
Priority to CN201280003247.8A priority patent/CN103403709B/zh
Publication of WO2014075273A1 publication Critical patent/WO2014075273A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation

Definitions

  • the present invention relates to the field of computers, and in particular, to a method, device and system for reading and writing data. Background technique
  • a storage system using a variable length Key-Va lue storage method is stored by means of LSM-Tree, and is updated in an additional manner.
  • the system uniquely determines the Key value for the target data according to a preset rule (such as a file name), and then writes the target data into the memory.
  • a preset rule such as a file name
  • the target data is small, it is all written to the memory at a time, and is sorted.
  • the data in the memory reaches a certain scale or after a certain period of time, the data is written to the disk.
  • the target data is large, the system writes the data into the memory in batches, and the data written into the memory each time is sorted according to certain rules.
  • the data in the memory reaches a certain scale or after a certain period of time, the data in the memory is written to the disk to form a small ordered file.
  • Bloom Fier is a binary vector data structure that detects whether an element is a member of a collection. However, due to the constraints of its data structure, it cannot make a completely correct judgment. That is, the filtered data includes target data, but it also includes data other than the target data. Thus, as the amount of raw data increases, the error rate increases. To avoid the occurrence of an error, the system must make a secondary selection of the data that has been filtered.
  • Embodiments of the present invention provide a method, device, and system for reading and writing data, which solve the problem that the existing data reading and writing method has large system overhead, low efficiency, and poor resistance to external interference.
  • a method for data writing including:
  • the method further includes:
  • the index and the physical address are written to the storage medium.
  • the selecting an index block corresponding to the index according to the index further includes:
  • a method for data reading including:
  • the read instruction includes a data feature value, and the data feature value is unique Characterizing the data to be read;
  • index block corresponding to the index is selected, the index block is stored in a storage medium, and the index block records at least one index and a physical address corresponding to each index of the record;
  • the index block that is loaded into the memory, detecting whether the index corresponding to the data feature value is only when the detection result is unique, searching for the physical address corresponding to the index in the index block;
  • the physical address reads the data from the storage medium.
  • the method further includes:
  • the selecting an index block corresponding to the index includes:
  • the preset algorithm is: when the data is stored in the storage medium, calculating, according to the data feature value of the data The data feature value corresponds to an algorithm of the index.
  • an apparatus for data writing including: a write instruction acquisition unit, configured to acquire a write instruction, where the write instruction includes data to be written;
  • a write data feature value extracting unit configured to extract a data feature value according to the data, where the data feature value is a parameter that uniquely represents the data;
  • a write index obtaining unit configured to calculate an index corresponding to the data feature value according to the data feature value, where the index corresponds to the data feature value; wherein, the storage medium stores at least one index block;
  • a physical address obtaining unit configured to acquire a physical address of the data in the storage medium
  • Index block unit configured to write the index and the physical address together into the index block of the memory
  • the writing data unit is further configured to: when the data is written into the storage medium, write the index and the physical address In the storage medium.
  • the write index block obtaining unit further includes:
  • Writing to a calculation subunit configured to calculate the index according to a preset rule; writing an index block determining subunit, configured to, according to the calculated value, an index block stored in the storage medium The index block corresponding to the index is determined.
  • the fourth aspect provides a device for reading data, including:
  • a read instruction acquisition unit configured to acquire a read instruction, where the read instruction includes a data feature value, and the data feature value uniquely represents data to be read
  • a read data feature value extracting unit configured to extract a data feature value according to the read instruction
  • a read index obtaining unit configured to calculate, according to the data feature value, a data feature value corresponding to the data according to a preset algorithm index of;
  • a read index block obtaining unit configured to select an index block corresponding to the index, where the index block is stored in a storage medium, where the index block records at least one index and a physical address corresponding to each index of the record;
  • a detecting unit configured to detect, in the index block that is loaded into the memory, whether an index corresponding to the data feature value is unique
  • a physical address obtaining unit configured to search for a physical address corresponding to the index in the index block when the detection result is unique
  • Reading a data unit configured to read the data from the storage medium according to a physical address corresponding to the index.
  • the reading physical address obtaining unit is further used to search for a physical address corresponding to each index in the index block. Wherein each physical address points to a different data in the storage medium;
  • the detecting unit is further configured to: search data that is the same as the data feature value included in the read command from the data pointed to by each physical address;
  • the read data unit is further configured to read data having the same data characteristic value as that of the read command.
  • the reading index block acquiring unit specifically includes:
  • a reading calculation unit configured to perform an algorithm for selecting the index block according to an index of the data when the data is stored in the storage medium
  • a read index block determining subunit configured to determine, according to the calculated result, the index block corresponding to the index in an index block stored in the storage medium.
  • the preset algorithm used by the read index acquiring unit is: when the data is stored in the storage medium, according to the An algorithm for calculating an index corresponding to the data feature value by the data feature value of the data.
  • an apparatus for data writing including:
  • a processor configured to acquire a read instruction, where the read instruction includes a data feature value, the data feature value uniquely represents data to be read; and extracting a data feature value according to the read instruction;
  • An index value corresponding to the data feature value is calculated according to a preset algorithm; an index block corresponding to the index is selected, the index block is stored in a storage medium, and the index block records at least one index and Recording the physical address corresponding to each index; transferring the index block into the memory; detecting, in the index block that is loaded into the memory, whether the index corresponding to the data feature value is unique; when the detection result is unique, Searching, in the index block, a physical address corresponding to the index; reading the data from the storage medium according to a physical address corresponding to the cable 1.
  • the memory is used to assist the processor to read data
  • a storage medium is used to store data that the processor needs to read
  • a bus is used to connect the processor, the memory and the storage medium.
  • the processor is further configured to: when the data is written to the storage medium, write the index and the physical address into the In the storage medium.
  • the processor is further configured to perform calculation on the index according to a preset rule;
  • the index block corresponding to the index is determined in an index block in the storage medium.
  • a data reading apparatus includes:
  • a processor configured to acquire a read instruction, where the read instruction includes a data feature value, the data feature value uniquely represents data to be read; and extracting a data feature value according to the read instruction;
  • An index value corresponding to the data feature value is calculated according to a preset algorithm; an index block corresponding to the index is selected, the index block is stored in a storage medium, and the index block records at least one index and The physical address corresponding to each index recorded; the index block is loaded into the In the index block that is loaded into the memory, detecting whether the index corresponding to the data feature value is unique; and when the detection result is unique, searching for the physical address corresponding to the index in the index block;
  • the physical address corresponding to the cable 1 reads the data from the storage medium.
  • the memory is used to assist the processor to read data
  • a storage medium is used to store data that the processor needs to read
  • a bus is used to connect the processor, the memory and the storage medium.
  • the processor is further configured to: when the detection unit detects that at least two indexes corresponding to the data feature value exist, Locating, in the index block, a physical address corresponding to each index, where each physical address points to different data in the storage medium; and searching for feature values and data features included in the read instruction from data pointed to by each physical address Data with the same value; reads the same data as the data feature value contained in the read command.
  • the processor is further configured to: when the data is stored in the storage medium, select the index according to an index of the data The algorithm of the block performs calculation; according to the calculated result, the index block corresponding to the index is determined in an index block stored in the storage medium.
  • the preset algorithm used by the processor is: when the data is stored in the storage medium, according to the data An algorithm for calculating an index corresponding to the data feature value by using the data feature value.
  • a computer system including: a processor, a memory, a storage medium, and a bus;
  • the processor includes any one of the possible devices described in the third aspect and any one of the possible devices described in the fourth aspect;
  • the memory is for assisting the processor to write data to and read data from the storage medium
  • the storage medium is configured to store data that the processor needs to write and data that the processor needs to read;
  • a bus is used to connect the processor, the memory, and the storage medium.
  • Embodiments of the present invention provide a method, apparatus, and system for reading and writing data.
  • the index corresponding to the data feature value is first obtained according to the data feature value, and then the index block is obtained; then the physical address stored in the storage medium of the index and the index corresponding data is written into the index block, and finally the index block is finally obtained.
  • Write to the storage medium When the data is read, the index corresponding to the data feature value is first obtained according to the data feature value, then the physical address corresponding to the index is determined, and finally the data is read from the physical address.
  • the invention is primarily used in computer systems. Through the above solution, the system overhead during data reading and writing is reduced, the reading and writing efficiency is increased, and the interference performance against external interference is improved.
  • Embodiment 1 is a flow chart showing a data writing method in Embodiment 1 of the present invention.
  • FIG. 2 is a schematic diagram of an internal structure of an index block in Embodiment 1 of the present invention.
  • Embodiment 3 is a schematic diagram of writing an index and a physical address in Embodiment 1 of the present invention.
  • Embodiment 4 is a flow chart showing a data reading method in Embodiment 1 of the present invention.
  • Embodiment 5 is a flow chart of a data writing method in Embodiment 2 of the present invention.
  • FIG. 6 is a flow chart of a data reading method in Embodiment 2 of the present invention.
  • Figure ⁇ is a block diagram of a data writing device in Embodiment 3 of the present invention.
  • FIG. 8 is a schematic diagram of a write index block acquiring unit in Embodiment 3 of the present invention.
  • Figure 9 is a block diagram of a data reading apparatus in Embodiment 3 of the present invention.
  • FIG. 10 is a schematic diagram of a read index block acquiring unit in Embodiment 3 of the present invention.
  • FIG. 1 is a schematic diagram of a data writing device in Embodiment 4 of the present invention.
  • FIG. 12 is a schematic diagram of a data reading apparatus according to Embodiment 4 of the present invention.
  • FIG. 13 is a schematic diagram of a computer system according to Embodiment 4 of the present invention.
  • the technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope are the scope of the present invention.
  • One embodiment of the present invention provides a method of reading and writing data.
  • a method of writing data as shown in FIG. 1, the method includes the following steps:
  • the data to be written described in step 101 includes data that needs to be stored in the storage medium and various parameters related to the data.
  • Example 1 the data to be written is the song A. mp3, the song name is A. mp3 : the required storage space is 5MB (Mega Byte, 100 megabytes), then the song itself is the to-be-written as described in step 101.
  • the entered data, the song name A and the required storage space 5MB are the parameters related to the data.
  • the parameters related to the data include but are not limited to the parameters listed above.
  • step 102 The parameters uniquely characterizing the data described in step 102 are also illustrated by way of example 1. If the stored data song A.mp3 name A.mp3 can uniquely represent the data, it means that the name of any data is also A.mp3, then A.mp3 can be used as the data feature value of the data.
  • the calculation described in step 103 may be a hash algorithm, and the index is a value obtained by calculating a data feature value according to a hash algorithm, that is, a hash value.
  • the index is contained in an index block, which is stored in a storage medium, and each index block contains a preset number of indexes.
  • Figure 2 shows the internal structure of an index block stored in a storage medium.
  • 201 is a storage medium
  • 202 is an index block stored in the storage medium.
  • Each of the index blocks includes a plurality of preset entries 205.
  • the entry 205 is composed of 203 and 204, wherein an index is recorded in 203, and the physical address in which the data is stored in the storage medium is recorded. It can be clearly seen from the figure that 203 and 204 have a corresponding relationship.
  • 104 Select, according to the index, an index block corresponding to the index, where at least one index block is stored in the storage medium.
  • step 104 may specifically be:
  • Calculating the index according to a preset rule determining, according to the calculated value, the index block corresponding to the index in an index block stored in the storage medium.
  • the index block number is first determined (this process is determined when designing the system), and then the index is subjected to a remainder operation (the remainder operation is a preset rule. Obviously, other algorithms that have the same effect as the remainder operation can also be used as The preset rule, the embodiment takes the remainder operation as an example, but is not limited.) Finally, the index block corresponding to the index is determined according to the value obtained after the operation. Assume that the index block stored in the storage medium has a total of 3 blocks with an index of 10. The index block is numbered consecutively from integer to small, and the numbered index block is: index block 0, index block 1 and index block 2. Then, the index 10 is divided by the number of index blocks. 3 The result of the remainder is 1, and the index block corresponding to the index is the number 1 index block.
  • the selected index block 302 is loaded into the memory 301, and then the index and the physical address are written into the record item 305 in the index block 302, that is, the index is written into the storage space 303.
  • the physical address is written in the storage space 304 corresponding to 303.
  • Embodiments of the present invention provide a method of data writing. First, an index corresponding to the data feature value is obtained according to the data feature value, and then the index block is obtained; then the physical address stored in the storage medium of the index and index corresponding data is written into the index block, and finally the index block is written into the storage medium.
  • the system overhead at the time of data writing is reduced, and the writing efficiency is increased.
  • an embodiment of the present invention further provides a data reading method, as shown in FIG. 4, The method includes the following steps:
  • the data feature value described in step 401 can be the data name of the data or other parameter that can uniquely characterize the data.
  • the data feature value is mainly used to select data to be read from data stored in the storage medium.
  • the data feature value extracted in step 402 is A. mp3.
  • the preset algorithm described in step 403 is an algorithm for calculating an index corresponding to the data feature value according to the data feature value of the data when the data is stored in the storage medium.
  • the written data is the song A. mp3
  • the extracted data feature value is A. mp3
  • the A. mp3 is calculated by the hash algorithm to obtain an index.
  • the algorithm for obtaining an index when reading data should be the same as the algorithm for obtaining an index when writing data. That is, if the data is written using the X algorithm to obtain an index, when the data is read, the X algorithm is also used to obtain the index.
  • index block 404 Select an index block corresponding to the index, where the index block is stored in a storage medium, where the index block records at least one index and a physical address corresponding to each index of the record.
  • step 404 The specific process selected in step 404 is as follows:
  • the storage medium The index block in the quality is a total of 3 blocks, numbered as index block 0, index block 1 and index block 2, and the algorithm is used to determine the index block, that is, the index 10 is divided by the number of index blocks 3 to obtain the remainder 1 , to determine that the index block corresponding to the index 10 is the index block No. 1, then, when reading the data, the index block should also be obtained according to the remainder algorithm, that is, the data song A. mp3 is converted into the index 10 by calculation, and then used The index 10 is divided by the number of index blocks 3 to obtain a remainder of 1, so that the index block corresponding to the index of the data to be read is determined to be the index block No. 1.
  • the detection described in step 406 is to match the index obtained in step 403 with the index stored in the index block loaded into memory. Since the index is calculated for the data feature value, although the data feature value can uniquely represent the data, after the data feature value is operated, different data feature values may have the same calculation result, that is, the same index is generated. Specifically, when the hash algorithm is used as the calculation method for obtaining the index, a hash conflict may occur, resulting in the occurrence of the same index. Therefore, it is possible that there are two identical indexes in the same index block.
  • Embodiments of the present invention provide a method of data reading. First, an index corresponding to the data feature value is obtained according to the data feature value, then the physical address corresponding to the index is determined, and finally the data is read from the physical address.
  • the invention is primarily used in computer systems. Through the above scheme, the system overhead during data reading and writing is reduced, the reading and writing efficiency is increased, and the interference performance against external interference is improved.
  • Embodiments of the present invention provide a method of reading and writing data.
  • a method of writing data as shown in FIG. 5, the method includes the following steps:
  • Step 506 may be implemented by storing an index and a physical address on a storage address of the stored data, or storing the data in the storage. At one address of the space, the index and the physical address are stored at another address of the storage medium. For convenience of description, an index and a physical address are stored on the storage address of the stored data as an example. It should be noted that, in order to save storage space, after performing step 506, the system automatically deletes the last stored index and physical address, that is, only one index and the physical address corresponding to the index are reserved at the address where the data is stored. .
  • index block After the system stops working unexpectedly, it returns to normal.
  • the index and the physical address stored in the storage medium are read out, and the corresponding index block in the storage medium is obtained according to the index, and then the index and the physical address are written and stored in the storage medium. Index block.
  • the physical address and the index of the data have a corresponding relationship (the corresponding relationship is determined at the time of writing, and the specific implementation is as described in Embodiment 1 and will not be described here). Therefore, when reading data, it is only necessary to find the data to be read.
  • the index can find the data.
  • the step of writing data is to first write data to the storage medium, and then write the index block to the storage medium (the index and the physical address are stored in the index block and the two are corresponding).
  • an unexpected situation occurs suddenly, such as a power outage, etc., which has the following consequences: Since there is no index in the index block stored in the storage medium.
  • step 510 is to eliminate the above-mentioned unexpected situation: After the system resumes working, the index and physical address of the storage medium written in step 506 are obtained, and the corresponding index block stored in the storage medium is determined according to the index, and then the index and the physical are The address is written to an index block stored in a storage medium.
  • the address where the data song A. mp3 is written to the storage medium is a and the index is 10.
  • address a index 10 and song A. mp3 are stored at an address on the storage medium.
  • steps 507 to 509 are not performed, the index 10 and the address a do not exist in the index block in the storage medium.
  • the index 10 and the address a are first taken out from the storage medium, and then the index block corresponding to the index 10 is obtained according to the index 10, and then the index 10 and the address a are written into the index block.
  • Embodiments of the present invention provide a method of data writing. First, an index corresponding to the data feature value is obtained according to the data feature value, and then the index block is obtained; then the index and the physical address are simultaneously written into the storage medium and the index and the index are corresponding data. The physical address stored in the storage medium is written to the index block, and finally the index block is written to the storage medium.
  • the system overhead during data writing is reduced, the writing efficiency is increased, and the system's ability to resist interference is increased.
  • FIG. 6 a method for reading data is provided, as shown in FIG. 6, including:
  • index block corresponding to the index, where the index block is stored in a storage medium, where the index block records at least one index and a physical address corresponding to each index of the record.
  • index block that is loaded into the memory, check whether an index corresponding to the data feature value is unique. Finding a physical address corresponding to each index in the index block, where each physical address points to the storage Different data in the media.
  • steps 607 to 608 The data to be read is the song A.mp3, the data feature value is A. mp3, the index is 10, and the index block corresponding to the index 10 is the index block No. 1.
  • the index block No. 1 three indexes are recorded, that is, the first index, the index is 10, the second index, the index is 10, and the third index, the index is 11.
  • the data on the physical address corresponding to the first index and the second index are respectively taken out, the data corresponding to the first index is the song A.mp3, and the data corresponding to the second index is the movie B. avi. Since the data feature value of the song A. mp3 is the same as the data feature value A. mp3 included in the read command, the song A. mp3 is taken out and returned to the user.
  • An embodiment of the present invention provides a method for reading data. First, an index corresponding to a data feature value is obtained according to a data feature value. When there are at least two indexes corresponding to the data feature value, the data pointed to by each physical address is used. And searching for the data with the same data feature value as the data feature value included in the read instruction and reading. Through the above scheme, the interference performance against external interference is improved, the system overhead during data reading and writing is reduced, and the reading and writing efficiency is increased.
  • One embodiment of the present invention provides an apparatus for reading and writing data.
  • a device for writing data as shown in FIG. 7, the device includes:
  • the write instruction acquisition unit 71 is configured to acquire a write instruction, the write instruction including data to be written.
  • the write data feature value extracting unit 72 is configured to extract a data feature value according to the data, and the data feature value is a parameter that uniquely represents the data.
  • the write index obtaining unit 73 is configured to calculate an index corresponding to the data feature value according to the data feature value, where the index corresponds to the data feature value.
  • the write index block obtaining unit 74 is configured to select an index block corresponding to the index according to the index, wherein at least one index block is stored in the storage medium.
  • the write index block call unit 75 is configured to load the index block into the memory.
  • a write data unit 76 is operative to write the data to the storage medium.
  • the write physical address obtaining unit 77 is configured to acquire a physical address of the data in the storage medium.
  • the write index block unit 78 is configured to write the index and the physical address together into the index block of the memory.
  • the write storage medium unit 79 is configured to write the index block loaded into the memory into the storage medium.
  • write data unit 76 is further configured to write the index and the physical address into the storage medium when the data is written to the storage medium.
  • the write index block obtaining unit 75 further includes:
  • the write calculation sub-unit 751 is configured to calculate the index according to a preset rule.
  • the write index block determining subunit 752 is configured to determine the index block corresponding to the index in an index block stored in the storage medium according to the calculated value.
  • the write instruction acquisition unit 71 the write data feature value extraction unit 72, the write index acquisition unit 73, the write index block acquisition unit 74, the write index block call unit 75, and the write data unit 76.
  • the write physical address acquisition unit 77 writes the index block unit 78, and the write storage medium unit 79 can be integrated in a CPU (Central Processing Unit).
  • Embodiments of the present invention provide an apparatus for data writing. First, an index corresponding to the data feature value is obtained according to the data feature value, and then the index block is obtained; then the index and the physical address are simultaneously written into the storage medium and the index and the index are corresponding data. The physical address stored in the storage medium is written to the index block, and finally the index block is written to the storage medium.
  • the system overhead during data writing is reduced, the writing efficiency is increased, and the system's ability to resist interference is increased.
  • a data reading device as shown in Figure 9, includes:
  • the read instruction obtaining unit 81 is configured to acquire a read instruction, where the read instruction includes a data feature value, and the data feature value uniquely represents data to be read.
  • the read data feature value extracting unit 82 is configured to extract a data feature value according to the read instruction, and the read index obtaining unit 83 is configured to calculate according to the data feature value according to a preset algorithm. An index corresponding to the data feature value.
  • the read index block obtaining unit 84 is configured to select an index block corresponding to the index, where the index block is stored in a storage medium, and the index block records at least one index and a physical address corresponding to each index of the record.
  • the read index block call unit 85 is configured to load the index block into the memory.
  • the detecting unit 86 is configured to detect, in the index block that is loaded into the memory, whether an index corresponding to the data feature value is unique.
  • the read physical address obtaining unit 87 is configured to find a physical address corresponding to the index in the index block when the detection result is unique.
  • the read data unit 88 is configured to read the data from the storage medium according to a physical address corresponding to the index. When the value corresponds to the index,
  • the read physical address obtaining unit 87 is further configured to search, in the index block, a physical address corresponding to each index, where each physical address points to different data in the storage medium.
  • the detecting unit 86 is further configured to search for data having the same feature value and a data feature value included in the read command from the data pointed to by each physical address.
  • the read data unit 88 is further configured to read data that is the same as the data feature value included in the read command.
  • the read index block obtaining unit 84 specifically includes: a read calculation subunit 841, configured to select, according to the index of the data, when the data is stored in the storage medium according to the data. The algorithm of the index block is determined for calculation.
  • the read index block determining subunit 842 is configured to determine, according to the calculated result, the index block corresponding to the index in an index block stored in the storage medium.
  • the preset algorithm used by the read index acquiring unit 83 is that when the data is stored in the storage medium, the data corresponding to the data feature value is calculated according to the data feature value of the data.
  • the algorithm of the index It should be noted that the read instruction acquisition unit 81, the read data feature value extraction unit 82, the read index acquisition unit 83, the read index block acquisition unit 84, the read index block call unit 85, the detection unit 86, and the read The physical address obtaining unit 87 is taken, and the read data unit 88 can be integrated in a CPU (Centra l Proces s Uni).
  • An embodiment of the present invention provides an apparatus for reading data, first acquiring an index corresponding to a data feature value according to a data feature value, and when there is an index corresponding to at least two data feature values, from the data pointed to by each physical address And searching for the data with the same data feature value as the data feature value included in the read instruction and reading.
  • One embodiment of the present invention provides an apparatus for reading and writing data.
  • a device for writing data including:
  • the processor 91 is configured to acquire a write command, where the write command includes data to be written, and according to the data, extract a data feature value, where the data feature value is a parameter that uniquely represents the data; An index corresponding to the data feature value is calculated, the index corresponding to the data feature value; an index block corresponding to the index is selected according to the index, wherein at least one index is stored in the storage medium And loading the index block into a memory; acquiring a physical address of the data in the storage medium; writing the index and the physical address together into the index block of the memory; Used to write the data to the storage medium; write the index block loaded into the memory into the storage medium.
  • a memory 92 is provided to assist the processor in writing data to the storage medium.
  • a storage medium 93 is configured to store data that the processor needs to write.
  • a bus 94 is configured to connect the processor, the memory and the storage medium.
  • processor 91 is further configured to write the index and the physical address into the storage medium when the data is written to the storage medium.
  • the processor 91 is further configured to: calculate, according to a preset rule, the index; determine, according to the calculated value, an index block stored in the storage medium The corresponding index block is referenced.
  • Embodiments of the present invention provide an apparatus for data writing. First, an index corresponding to the data feature value is obtained according to the data feature value, and then the index block is obtained; then the index and the physical address are simultaneously written into the storage medium and the index and the index are corresponding data. The physical address stored in the storage medium is written to the index block, and finally the index block is written to the storage medium.
  • the system overhead during data writing is reduced, the writing efficiency is increased, and the system's ability to resist interference is increased.
  • a device for providing data reading includes:
  • the processor 1001 is configured to acquire a read instruction, where the read instruction includes a data feature value, where the data feature value uniquely represents data to be read; and extracting a data feature value according to the read instruction; a data feature value, an index corresponding to the data feature value is calculated according to a preset algorithm; an index block corresponding to the index is selected, the index block is stored in a storage medium, and the index block records at least one index And the physical address corresponding to each index of the record; the index block is transferred into the memory; in the index block called into the memory, detecting whether the index corresponding to the data feature value is unique; when the detection result is unique Searching, in the index block, a physical address corresponding to the index; and reading the data from the storage medium according to a physical address corresponding to the index.
  • the memory 1002 is configured to assist the processor to read data.
  • the storage medium 1003 is configured to store data that the processor needs to read.
  • the bus 1004 is configured to connect the processor, the memory and the storage medium.
  • the processor 1001 is further configured to: when the detection unit detects that there are at least two indexes corresponding to the data feature values, search for the physical addresses corresponding to the indexes in the index block, Each physical address points to different data in the storage medium; from the data pointed to by each physical address, the data whose feature value is the same as the data feature value included in the read instruction is found; the data features included in the read and read instructions The same value of data.
  • the processor 1001 is further configured to perform an algorithm for selecting the index block according to an index of the data when the data is stored in the storage medium; according to the calculated result, The index block corresponding to the index is determined in an index block stored in the storage medium.
  • the preset algorithm used by the processor 1001 is an algorithm for calculating an index corresponding to the data feature value according to the data feature value of the data when the data is stored in the storage medium. .
  • An embodiment of the present invention provides an apparatus for reading data, first acquiring an index corresponding to a data feature value according to a data feature value, and when there is an index corresponding to at least two data feature values, from the data pointed to by each physical address And searching for the data with the same data feature value as the data feature value included in the read instruction and reading.
  • a computer system including: a processor 1101, a memory 1102, a storage medium 1103, and a bus 1104;
  • the processor 1101 includes any one of the devices of claims 8 to 10 and any of the devices of claims 11 to 14.
  • the memory 1102 is configured to assist the processor to write data to and read data from the storage medium
  • the storage medium 1103 is configured to store data that the processor needs to write and data that the processor needs to read.
  • the bus 1104 is configured to connect the processor 1101, the memory 1102, and the storage medium 1103.
  • processor 91 the processor 1001, and the processor 1101 in this embodiment may be a CPU (Centra l Proce s ing Uni t, a central processing unit) or other devices having the functions described above.
  • Embodiments of the present invention provide a system for reading and writing data.
  • the index corresponding to the data feature value is first obtained according to the data feature value, and then the index block is obtained; then the physical address stored in the storage medium of the index and the index corresponding data is written into the index block, and finally the index block is finally obtained.
  • Write to the storage medium When the data is read, the index corresponding to the data feature value is first obtained according to the data feature value, then the physical address corresponding to the index is determined, and finally the data is read from the physical address.
  • the invention is primarily used in computer systems. Through the above scheme, reduce system overhead when reading and writing data, and increase read and write efficiency Rate, improve the interference performance against the outside world.
  • the present invention can be implemented by means of software plus necessary general hardware, and of course, by hardware, but in many cases, the former is a better implementation. .
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a readable storage medium, such as a floppy disk of a computer.
  • a hard disk or optical disk or the like includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention concerne un procédé, un dispositif et un système de lecture/écriture de données, qui se rapportent au domaine de l'informatique et permettent de résoudre les problèmes de surdébits élevés des systèmes, de faible efficacité et de performances médiocres pour résister à un brouillage externe que présente le procédé traditionnel concerné par la présente invention. Le procédé d'écriture de données comprend les étapes consistant à : en fonction d'une valeur de caractéristique de données, acquérir un indice correspondant à la valeur de caractéristique de données, puis acquérir un bloc d'indices ; et écrire des adresses physiques de l'indice et des données correspondant à l'indice qui sont stockées dans un support de stockage à l'intérieur du bloc d'indices, et enfin, écrire le bloc d'indices à l'intérieur du support de stockage. Le procédé de lecture de données comprend les étapes consistant à : en fonction d'une valeur de caractéristique de données, acquérir un indice correspondant à la valeur de caractéristique de données ; ensuite, déterminer une adresse physique correspondant à l'indice ; et enfin, lire des données à partir de l'adresse physique. La présente invention est principalement utilisée dans un système informatique.
PCT/CN2012/084684 2012-11-15 2012-11-15 Procédé, dispositif et système de lecture/écriture de données WO2014075273A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2012/084684 WO2014075273A1 (fr) 2012-11-15 2012-11-15 Procédé, dispositif et système de lecture/écriture de données
CN201280003247.8A CN103403709B (zh) 2012-11-15 2012-11-15 一种数据读写的方法、装置和系统

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/084684 WO2014075273A1 (fr) 2012-11-15 2012-11-15 Procédé, dispositif et système de lecture/écriture de données

Publications (1)

Publication Number Publication Date
WO2014075273A1 true WO2014075273A1 (fr) 2014-05-22

Family

ID=49565847

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/084684 WO2014075273A1 (fr) 2012-11-15 2012-11-15 Procédé, dispositif et système de lecture/écriture de données

Country Status (2)

Country Link
CN (1) CN103403709B (fr)
WO (1) WO2014075273A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104809237B (zh) * 2015-05-12 2018-12-14 百度在线网络技术（北京）有限公司 LSM-tree索引的优化方法和装置
CN106445416B (zh) * 2016-09-22 2019-08-09 飞天诚信科技股份有限公司 一种数据记录的存储、查询和检索的方法及装置
CN108614668B (zh) * 2016-12-12 2022-12-06 北京忆恒创源科技股份有限公司 基于kv模型的数据访问方法与固态存储设备
CN108614671B (zh) * 2016-12-12 2023-02-28 北京忆恒创源科技股份有限公司 基于命名空间的键-数据访问方法与固态存储设备
CN108614669B (zh) * 2016-12-12 2023-02-17 北京忆恒创源科技股份有限公司 解决哈希冲突的键-数据访问方法与固态存储设备
CN107315543B (zh) * 2017-07-03 2020-10-09 合肥联宝信息技术有限公司 一种对ec的存储器中数据的读写方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101221538A (zh) * 2008-01-24 2008-07-16 杭州华三通信技术有限公司 实现对缓存中数据快速查找的系统和方法
CN101546342A (zh) * 2009-05-08 2009-09-30 阿里巴巴集团控股有限公司 实现搜索服务的方法与系统
CN101639833A (zh) * 2008-08-01 2010-02-03 财团法人资讯工业策进会 中介装置、中介方法、计算机程序产品及数据储存系统
WO2012032727A1 (fr) * 2010-09-09 2012-03-15 Nec Corporation Système de stockage

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101686209B (zh) * 2008-09-24 2013-10-09 阿里巴巴集团控股有限公司 消息转发系统中存储消息的方法和装置
CN102567434B (zh) * 2010-12-31 2013-04-10 百度在线网络技术（北京）有限公司 一种数据块处理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101221538A (zh) * 2008-01-24 2008-07-16 杭州华三通信技术有限公司 实现对缓存中数据快速查找的系统和方法
CN101639833A (zh) * 2008-08-01 2010-02-03 财团法人资讯工业策进会 中介装置、中介方法、计算机程序产品及数据储存系统
CN101546342A (zh) * 2009-05-08 2009-09-30 阿里巴巴集团控股有限公司 实现搜索服务的方法与系统
WO2012032727A1 (fr) * 2010-09-09 2012-03-15 Nec Corporation Système de stockage

Also Published As

Publication number Publication date
CN103403709B (zh) 2016-10-05
CN103403709A (zh) 2013-11-20

Similar Documents

Publication Publication Date Title
WO2014075273A1 (fr) Procédé, dispositif et système de lecture/écriture de données
TWI720491B (zh) 縮減機率篩選查詢延時
JP5500309B2 (ja) ストレージ装置
US8370315B1 (en) System and method for high performance deduplication indexing
US20170364414A1 (en) System and method for data deduplication using log-structured merge trees
US8392423B2 (en) Data set index record preservation
JP2005267600A5 (fr)
US20110040763A1 (en) Data processing apparatus and method of processing data
WO2016041401A1 (fr) Procédé et dispositif pour l'écriture de données dans une mémoire cache
EP2770446A1 (fr) Procédé et dispositif de traitement de données
US10007436B2 (en) Storage control apparatus and control method of storage control apparatus
US20070156778A1 (en) File indexer
CN116450656B (zh) 数据处理方法、装置、设备及存储介质
CN111522502B (zh) 数据去重方法、装置、电子设备及计算机可读存储介质
CN113535670B (zh) 一种虚拟化资源镜像存储系统及其实现方法
CN110021345B (zh) 基于spark平台的基因数据分析方法
CN106528830B (zh) 一种恢复文件索引目录的方法和装置
CN107133334B (zh) 基于高带宽存储系统的数据同步方法
JP6113816B1 (ja) 情報処理システム、情報処理装置、及びプログラム
US20130046736A1 (en) Recovering method and device for linux using fat file system
WO2016122318A1 (fr) Procédé mis en œuvre par ordinateur pour générer une variante de fichier d'appel
US8341376B1 (en) System, method, and computer program for repartitioning data based on access of the data
US20220100716A1 (en) Selection and movement of data between nodes of a distributed storage system
CN107301073B (zh) 一种基于固态硬盘系统的配置信息检索方法及其装置
CN111143110A (zh) 一种逻辑卷管理中基于元数据的raid数据恢复方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12888358

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12888358

Country of ref document: EP

Kind code of ref document: A1