CN110764705A - Data reading and writing method, device, equipment and storage medium - Google Patents
Data reading and writing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110764705A CN110764705A CN201911007543.5A CN201911007543A CN110764705A CN 110764705 A CN110764705 A CN 110764705A CN 201911007543 A CN201911007543 A CN 201911007543A CN 110764705 A CN110764705 A CN 110764705A
- Authority
- CN
- China
- Prior art keywords
- key value
- storage position
- key
- target
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a data reading and writing method, a data reading and writing device, data reading and writing equipment and a storage medium. The method comprises the following steps: acquiring target data from a partition, and acquiring a target key name and a target key value in the target data; according to the current reading sequence number and/or writing sequence number, the target key value and the target key value index are adjacently stored in a first array in a ring writing mode, and the target key value index are adjacently stored in a second array; the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array; and circularly reading the data in the first array and the second array according to the current reading sequence number. By using the technical scheme of the embodiment of the invention, data can be directly read and written in the memory, thereby avoiding the discrete I/O problem of a disk and improving the data processing performance.
Description
Technical Field
The present invention relates to data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for reading and writing data.
Background
In the current big data processing framework, the streaming big data processing framework spark streaming has two characteristics of high throughput and strong fault-tolerant capability when processing batch streaming service.
However, when spark streaming handles bulk streaming services, there is a one-to-one relationship between RDDs (flexible distributed data sets) at this stage and RDD partitions upstream, and at this time, the dependency is wide, and shuffle operations need to be performed. According to the flow chart of spark streaming shuffle operation in the prior art shown in fig. 1, partitioned data are aggregated, data in an aggregated data set are sorted according to a key, when the data amount in a memory reaches a certain threshold, the data set is written into a disk, and then the data set is pulled out from the disk, and aggregation is performed again to obtain a final result. The process of writing to disk in a shuffle operation may cause random I/O, resulting in reduced performance of the stream processing. In order to solve the problem of Disk discrete I/O caused by reading and writing, most of the current schemes use a Solid State Disk (SSD), but the cost is too high in large-scale large data clusters when the SSD is configured.
Disclosure of Invention
The embodiment of the invention provides a data reading and writing method, a device, equipment and a storage medium, which are used for directly reading and writing data in a memory.
In a first aspect, an embodiment of the present invention provides a data reading and writing method, where the method includes:
acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
according to the current reading sequence number and/or writing sequence number, the target key value and the target key value index are adjacently stored in a first array in a ring writing mode, and the target key value index are adjacently stored in a second array;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and circularly reading the data in the first array and the second array according to the current reading sequence number.
In a second aspect, an embodiment of the present invention further provides a data reading and writing device, where the device includes:
the target data acquisition module is used for acquiring target data from the partitions and acquiring a target key name and a target key value in the target data;
the target data writing module is used for adjacently storing the target key value and the target key value index in a first array and adjacently storing the target key value and the target key value index in a second array in a ring writing mode according to the current reading sequence number and/or writing sequence number;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and the target data reading module is used for circularly reading the data in the first array and the second array according to the current reading sequence number.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements a method for reading and writing data provided in any embodiment of the present invention when executing the program.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a computer processor, implements a method for reading and writing data as provided in any embodiment of the present invention.
According to the embodiment of the invention, the key value and the key value index of the target data are stored in the first array, the key name and the key value index are stored in the second array, and the target data are read and written in an annular writing and annular reading mode. The method solves the technical problem of discrete I/O of the disk caused by the fact that data needs to be written into the disk in the traditional data stream processing process. The stream processing execution performance reduction caused by random I/O (input/output) caused by the process of writing data into a disk is avoided, and the technical effect of improving the data processing performance is realized.
Drawings
FIG. 1 is a flow chart of a spark streaming shuffle operation in the prior art;
FIG. 2 is a flowchart of a method for reading and writing data according to a first embodiment of the present invention;
FIG. 3 is a flowchart of a method for reading and writing data according to a second embodiment of the present invention;
fig. 4 is a flowchart of a data reading and writing method according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data read/write apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data read-write device in a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 2 is a flowchart of a data read-write method according to an embodiment of the present invention, where this embodiment is applicable to a situation that a shuffle operation needs to be performed when a streaming big data processing framework spark streaming processes streaming data, and the method may be executed by a data read-write device, where the device may be implemented by software and/or hardware and is generally integrated in a data read-write server. The method specifically comprises the following steps:
and step 110, acquiring target data from the partition, and acquiring a target key name and a target key value in the target data.
RDD is a fault tolerant, parallel data structure that allows users to store data explicitly to disk and memory and to control the partitioning of data. The data sets within an RDD are logically (and physically) divided into a plurality of small sets, each such small set being referred to as a partition, and an RDD may contain multiple partitions. The target data in one partition is expressed in the form of (key, value), where key is the target key name and value is the target key value.
And 120, adjacently storing the target key value and the target key value index in a first array and adjacently storing the target key value and the target key value index in a second array in a ring writing mode according to the current reading sequence number and/or writing sequence number.
The target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array.
The reading sequence number is used for indicating a current reading position on the first array, and the writing sequence number is used for indicating a current writing position on the first array. And the annular writing is to return to execute a new round of writing again after performing a round of writing on the first array and the second array by the writing sequence number. The dimensionality of the first array is the same as that of the second array, a target key value and a target key name index are stored in the first array, the target key name index is a result obtained by carrying out Hash calculation on the target key name and is used for indicating the storage position of the target key name in the second array, and the target key name index is stored at the next position of the target key value; and the second array stores a target key name and a target key value index, and the target key value index is used for indicating the storage position of the target key value in the first array. The target key value index is stored at the next location of the target key name.
And step 130, performing annular reading on the data in the first array and the data in the second array according to the current reading sequence number.
The reading sequence number is used for indicating the current reading position on the first array, and the annular reading is that after one round of reading is carried out on the first array and the second array by the reading sequence number, a new round of reading is returned and executed.
According to the technical scheme of the embodiment, the technical means that the target key value in the target data, the storage position of the target key name and the target key value are respectively written into the first array and the second array in an annular writing mode and read in an annular reading mode are adopted, the technical problem that when the shuffle operation exceeds the memory threshold value, data needs to be written into a disk, so that the disk discrete I/O is caused is solved, the data is directly processed in the memory, and the technical effect of improving the execution performance of data processing is achieved.
Example two
Fig. 3 is a flowchart of a data reading and writing method in a second embodiment of the present invention, in this embodiment, based on the foregoing embodiment, a writing flow of data is further refined, and the method in this embodiment specifically includes the following steps:
step 210, acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
step 220, taking the hash value of the target key name as the target key name index, locating a first key name storage position and a first key value index storage position in the second array, and acquiring the first key value index at the first key value index storage position;
and taking the hash value of the target key name as the target key name index, wherein the target key name index is used for indicating the storage position of the target key name in the second array. And in the second array, acquiring a storage position matched with the target key name index as the first key name storage position. The target key value index is stored at the next position of the target key name, so that the next storage position of the first key name storage position is used as the first key value index storage position;
step 230, judging whether the first key value index is empty;
if the first key value index is not empty, the data is stored in the position.
Step 240, if it is determined that the first key value index is not empty, locating a first key value storage location and a first key name index storage location in a first array according to the first key value index, and obtaining a first key value at the first key value storage location;
if the first key value index is determined not to be empty, the position is indicated to have previously stored data, and the stored data can be stored in the first array and the second array and is not read by the reading sequence number in the current round; or the current round is read by the reading sequence number and is overwritten by new data; it may also be that the current round has been read by the read sequence number but not yet overwritten by new data, where the first key index is not empty but the first key is empty. Therefore, the data which is accessed before needs to be acquired and matched with the target key name of the target data, and if the key names are the same, the data can be directly accumulated; if the key names are different, they cannot be directly accumulated. And in the first array, acquiring a storage position matched with the first key value index as a first key value storage position, and taking a later storage position of the first key value storage position as the first key name index storage position. And acquiring a first key value at the first key value storage position.
Step 250, determining whether a hash-free conflict storage condition is met according to one or more items of the first key value storage position, the read sequence number, the first key value, the first key name and the target key name;
the hash collision indicates that a result obtained by performing hash calculation according to the target key name, that is, the first key name in the second group pointed by the target key name index is different from the target key name.
If the position number of the first key value storage position is smaller than the reading sequence number and the first key value is empty, the Hash collision-free storage condition is determined to be met, at this time, it is indicated that data stored in the position before is read according to the reading sequence number but is not covered by new data, the first key value and the first key name are empty, and the target data can be directly stored in the position.
If it is determined that the position number of the first key value storage position is smaller than the read sequence number, the first key value is not null, and the target key name is consistent with the first key name, it is determined that the hash collision-free storage condition is satisfied, which indicates that data previously stored at the position is read according to the read sequence number but is covered by new data, and the key name (i.e., the first key name) of the new data is the same as the target key name of the target data, and the target key value of the target data can be directly accumulated with the key value of the new data.
If the position number of the first key value storage position is smaller than the reading sequence number, the first key value is not null, and the target key name is inconsistent with the first key name, it is determined that the no-hash-conflict storage condition is not met, which indicates that the data previously stored at the position is read according to the reading sequence number and is covered by new data, but the key name (i.e., the first key name) of the new data is different from the target key name of the target data, that is, a hash conflict exists, and the target data cannot be stored at the position again at this time.
If the position number of the first key value storage position is determined to be larger than the reading sequence number and the target key name is consistent with the first key name, the no-hash-conflict storage condition is determined to be met, at this time, it is indicated that data stored at the position before is not read according to the reading sequence number, the key name (namely the first key name) of the stored data is the same as the target key name of the target data, and the key value of the target data can be directly accumulated with the key value of the stored data.
And if the position number of the first key value storage position is determined to be larger than the reading sequence number and the target key name is inconsistent with the first key name, determining that the no-hash-conflict storage condition is not met. At this time, it is explained that the data stored in the position before is not read according to the read sequence number, but the key name (i.e. the first key name) of the stored data is different from the target key name of the target data, that is, there is a hash collision, and at this time, the target data cannot be stored in the position any more.
Step 260, if yes, storing the target key name index in the first key name index storage position, storing the target key name in the first key name storage position, and accumulating and storing the target key value in the first key value storage position.
If the hash-conflict-free storage condition is met, the target data can be stored at the position, the target key values of the target data are accumulated and stored at the first key value storage position on the first array, and the target key names of the target data are stored at the first key name storage position on the second array.
Step 270, if not, in the second array, taking the next key name storage position after the first key name storage position as a starting point, acquiring a second key name storage position which is closest to the first key name storage position and does not store the key name, and determining a second key name index according to the second key name storage position;
if the hash-conflict-free storage condition is not met, the hash conflict exists at the moment, and the target data can not be stored at the position, so that the first key name storage position which is empty is obtained as the second key name storage position by taking the next key name storage position of the first key name storage position in the second array as a starting point. And determines a second key name index according to the second key name storage location.
Step 280, determining a second key value storage position in the first array according to the write sequence number, and determining a second key value index according to the second key value storage position;
and taking the position of the first array pointed by the current write sequence number as a second key value storage position for storing a target key value of the target data. The second key value storage location is taken as a second key value index.
Step 290, storing the target key value in the second key value storage location, storing the target key name in the second key name storage location, storing the second key name index in a subsequent storage location of the second key value storage location, and storing the second key value index in a subsequent storage location of the second key name storage location.
After the second key name storage location, the second key name index, the second key value storage location and the second key value index are obtained according to the steps 270 and 280, the target key value is stored in the second key value storage location, and the second key name index is stored in the next storage location of the second key value storage location, wherein the target key value and the second key name index are both stored in the first array; and storing the target key name in the second key name storage position, and storing a second key value index in a later storage position of the second key name storage position, wherein the target key name and the second key value index are both stored on a second array.
Step 2100, if the first key value index is determined to be empty, determining a third key value index according to the first key name storage position;
if the first key value index is empty, it indicates that no data has been written in the location, no matter in the current round of writing process or in the previous writing process, and the target data can be directly written in the location. And acquiring a third key name index according to the storage position of the first key name.
Step 2110, determining a third key value storage position in the first array according to the write sequence number, and determining a third key value index according to the third key value storage position;
and determining the storage position of a third key value according to the position in the first array pointed by the current write sequence number, and determining a third key value index according to the storage position of the third key value.
Step 2120 of storing the target key value in the third key value storage location, storing the target key name in the first key name storage location, storing the third key value index in a subsequent storage location of the third key value storage location, and storing the third key value index in a subsequent storage location of the first key name storage location.
Storing the target key value to a third key value storage position, and storing a third key index to a later storage position of the third key value storage position, wherein the target key value and the third key index are both stored on a first array; storing the target key name in the first key name storage location, and storing a third key value index in a subsequent storage location of the first key name storage location, wherein the target key name and the third key value index are both stored on the second array.
And 2130, performing annular reading on the data in the first array and the data in the second array according to the current reading sequence number.
According to the technical scheme of the embodiment, the target key value, the target key name and the target key name index of the target data are obtained, the index position of the first key value is determined according to the target key name index, and the data are written into the data by judging whether the index position of the first key value is empty or not and whether the storage condition without hash conflict is met or not, so that different writing situations are distinguished. The method and the device solve the technical problem that in the prior art, when the shuffle operation exceeds the memory threshold, data needs to be written into a disk, so that the disk discrete I/O is caused. The data can be directly processed in the memory by annular writing of the data without writing the data into a disk, so that the technical effect of improving the execution performance of data processing is achieved.
EXAMPLE III
Fig. 4 is a flowchart of a data reading and writing method in a third embodiment of the present invention, in this embodiment, a reading flow of data is further refined on the basis of the foregoing embodiment, and the method in this embodiment specifically includes the following steps:
step 310, acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
step 320, according to the current reading sequence number and/or writing sequence number, storing the target key value and the target key value index adjacently in a first array in a ring writing mode, and storing the target key value and the target key value index adjacently in a second array;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
step 330, determining a current key value storage position pointed by the read sequence number in the first array according to the current read sequence number, acquiring a current key value, and emptying the current key value storage position;
and acquiring a current key value according to the position in the first array pointed by the current reading sequence number, and leaving the position empty to indicate that the position is read, so that other data can be conveniently written.
And 340, determining a current key name index according to a next storage position of the current key value storage position, determining the storage position of the current key name in the second array according to the current key name index, acquiring the current key name, and emptying the storage position of the current key name.
The position in the first array pointed by the reading sequence number is the storage position of the current key value, and the current key name index is stored in the next storage position of the current key value. And according to the current key name index, positioning the storage position of the current key name in the second array, acquiring the current key name, and leaving the storage position of the current key name empty to indicate that the position is read, thereby facilitating the writing of other data.
The embodiment of the invention carries out annular writing on target data in a first array and a second array, nulls a key value position pointed by a current reading sequence number according to the reading sequence number, nulls a key name position in the second array according to a key name index at a position behind the key value position, and reserves a key name index storage position in the first array and a key value index storage position in the second array. The method and the device solve the technical problem that in the prior art, when the shuffle operation exceeds the memory threshold, data needs to be written into a disk, so that the disk discrete I/O is caused. The annular writing and annular reading of the data are carried out, and the key value position and the key name position are vacant, so that the new data can be written conveniently, the data can be directly processed in the memory without being written into a magnetic disk, and the technical effect of improving the execution performance of data processing is achieved.
Example four
Fig. 5 is a schematic structural diagram of a data read-write apparatus according to a fourth embodiment of the present invention, where the data read-write apparatus includes: a target data acquisition module 410, a target data write module 420, and a target data read module 430. Wherein:
a target data obtaining module 410, configured to obtain target data from a partition, and obtain a target key name and a target key value in the target data;
a target data writing module 420, configured to store the target key value and the target key value index adjacently in a first array and store the target key value and the target key value index adjacently in a second array in a ring writing manner according to a current read sequence number and/or write sequence number;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and a target data reading module 430, configured to perform circular reading on data in the first array and the second array according to the current read sequence number.
According to the technical scheme of the embodiment, the technical means that the target key value in the target data, the storage position of the target key name and the target key value are respectively written into the first array and the second array in an annular writing mode and read in an annular reading mode are adopted, the technical problem that when the shuffle operation exceeds the memory threshold value, data needs to be written into a disk, so that the disk discrete I/O is caused is solved, the data is directly processed in the memory, and the technical effect of improving the execution performance of data processing is achieved.
On the basis of the above embodiment, the target data writing module 420 includes:
a first key value index obtaining unit, configured to use a hash value of the target key name as the target key name index, locate a first key name storage location and a first key value index storage location in the second number group, and obtain the first key value index at the first key value index storage location;
a first key value obtaining unit, configured to, if it is determined that the first key value index is not empty, locate a first key value storage location and a first key name index storage location in a first array according to the first key value index, and obtain a first key value at the first key value storage location;
a condition judging unit, configured to determine whether a hash-collision-free storage condition is satisfied according to one or more of the first key value storage location, the read sequence number, the first key value, the first key name, and the target key name;
and the satisfied condition execution unit is used for storing the target key name index in the first key name index storage position, storing the target key name in the first key name storage position and accumulating and storing the target key value in the first key value storage position if the hash-free conflict storage condition is satisfied.
On the basis of the above embodiment, the target data writing module 420 further includes:
the unsatisfied condition execution unit is used for acquiring a second key name storage position which is closest to the first key name storage position and does not store the key name from a next key name storage position behind the first key name storage position as a starting point in the second array if the hash-conflict-free storage condition is not satisfied, and determining a second key name index according to the second key name storage position;
a second key value index determining unit, configured to determine a second key value storage location in the first array according to the write sequence number, and determine a second key value index according to the second key value storage location;
the first target data storage unit is configured to store the target key value in the second key value storage location, store the target key name in the second key name storage location, store the second key name index in a subsequent storage location of the second key value storage location, and store the second key value index in a subsequent storage location of the second key name storage location.
On the basis of the foregoing embodiment, the condition determining unit specifically includes:
a first satisfaction condition determining unit, configured to determine that the hash collision free storage condition is satisfied if it is determined that the location number of the first key value storage location is smaller than the read sequence number and the first key value is null;
a second satisfaction condition determining unit, configured to determine that the hash-collision-free storage condition is satisfied if it is determined that the location number of the first key value storage location is smaller than the read sequence number, the first key value is non-empty, and the target key name is consistent with the first key name;
a first unsatisfied condition determining unit, configured to determine that the hash-collision-free storage condition is unsatisfied if it is determined that the location number of the first key value storage location is smaller than the read sequence number, the first key value is non-empty, and the target key name is inconsistent with the first key name; and
a third satisfaction condition determining unit, configured to determine that the hash-collision-free storage condition is satisfied if it is determined that the location number of the first key value storage location is greater than the read sequence number and the target key name is consistent with the first key name; and
a second unsatisfied condition determining unit, configured to determine that the hash-collision-free storage condition is not satisfied if it is determined that the location number of the first key-value storage location is greater than the read sequence number and the target key name is inconsistent with the first key name.
On the basis of the foregoing embodiment, the first key value index obtaining unit is specifically configured to:
taking the hash value of the target key name as the target key name index, acquiring a storage position matched with the target key name index in the second array as the first key name storage position, and taking the latter storage position of the first key name storage position as the first key value index storage position;
on the basis of the foregoing embodiment, the first key value obtaining unit is specifically configured to:
if the first key value index is determined not to be empty, in the first array, a storage position matched with the first key value index is obtained to serve as a first key value storage position, a later storage position of the first key value storage position is used as the first key value index storage position, and a first key value is obtained at the first key value storage position.
On the basis of the above embodiment, the target data writing module 420 further includes:
a third key name index determining unit, configured to determine a third key name index according to the first key name storage location if it is determined that the first key value index is empty;
a third key value index determining unit, configured to determine a third key value storage location in the first array according to the write sequence number, and determine a third key value index according to the third key value storage location;
the second target data storage unit is configured to store the target key value in the third key value storage location, store the target key name in the first key name storage location, store the third key value index in a storage location subsequent to the third key value storage location, and store the third key value index in a storage location subsequent to the first key name storage location.
On the basis of the above embodiment, the target data reading module 430 includes:
a key value storage position vacancy unit, configured to determine, according to the current read sequence number, a current key value storage position pointed to by the read sequence number in the first array, obtain a current key value, and vacancy the current key value storage position;
and the key name storage position vacancy unit is used for determining the current key name index according to the next storage position of the current key value storage position, determining the storage position of the current key name in the second array according to the current key name index, acquiring the current key name and vacancy the storage position of the current key name.
The data reading and writing device provided by the embodiment of the invention can execute the data reading and writing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 6 is a schematic structural diagram of an apparatus according to a fifth embodiment of the present invention, as shown in fig. 6, the apparatus includes a processor 50, a memory 51, an input device 52, and an output device 53; the number of processors 50 in the device may be one or more, and one processor 50 is taken as an example in fig. 6; the processor 50, the memory 51, the input device 52 and the output device 53 in the apparatus may be connected by a bus or other means, as exemplified by the bus connection in fig. 6.
The memory 51 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data reading and writing method in the embodiment of the present invention (for example, the target data obtaining module 410, the target data writing module 420, and the target data reading module 430 in the data reading and writing device). The processor 50 executes various functional applications of the device and data processing, i.e., implements the above-described data reading and writing method, by executing software programs, instructions, and modules stored in the memory 51. The method comprises the following steps:
acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
according to the current reading sequence number and/or writing sequence number, the target key value and the target key value index are adjacently stored in a first array in a ring writing mode, and the target key value index are adjacently stored in a second array;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and circularly reading the data in the first array and the second array according to the current reading sequence number.
The memory 51 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 51 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 51 may further include memory located remotely from the processor 50, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 52 is operable to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus. The output device 53 may include a display device such as a display screen.
EXAMPLE six
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, where the computer-executable instructions are executed by a computer processor to perform a method for reading and writing data, and the method includes:
acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
according to the current reading sequence number and/or writing sequence number, the target key value and the target key value index are adjacently stored in a first array in a ring writing mode, and the target key value index are adjacently stored in a second array;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and circularly reading the data in the first array and the second array according to the current reading sequence number.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the data reading and writing method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the data reading and writing device, each unit and each module included in the embodiment are only divided according to functional logic, but are not limited to the above division, as long as the corresponding function can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for reading and writing data, comprising:
acquiring target data from a partition, and acquiring a target key name and a target key value in the target data;
according to the current reading sequence number and/or writing sequence number, the target key value and the target key value index are adjacently stored in a first array in a ring writing mode, and the target key value index are adjacently stored in a second array;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and circularly reading the data in the first array and the second array according to the current reading sequence number.
2. The method of claim 1, wherein storing the target key value and the target key value index adjacently in a first array and storing the target key value and the target key value index adjacently in a second array according to a current read sequence number and/or write sequence number in a circular write manner comprises:
taking the hash value of the target key name as the target key name index, positioning a first key name storage position and a first key value index storage position in the second array, and acquiring a first key value index at the first key value index storage position;
if the first key value index is determined not to be empty, a first key value storage position and a first key name index storage position are located in a first array according to the first key value index, and a first key value is obtained at the first key value storage position;
determining whether a hash-conflict-free storage condition is met according to one or more items of the first key value storage position, the read sequence number, the first key value, the first key name and the target key name;
if yes, storing the target key name index in the first key name index storage position, storing the target key name in the first key name storage position, and accumulating and storing the target key value in the first key value storage position.
3. The method of claim 2, after determining whether a hash collision free storage condition is satisfied based on one or more of the first key value storage location, the read sequence number, the first key value, the first key name, and the target key name, further comprising:
if not, acquiring a second key name storage position which is closest to the first key name storage position and is not stored with the key name from a next key name storage position behind the first key name storage position as a starting point in the second array, and determining a second key name index according to the second key name storage position;
determining a second key value storage position in the first array according to the write sequence number, and determining a second key value index according to the second key value storage position;
storing the target key value at the second key value storage position, storing the target key name at the second key name storage position, storing the second key name index at a subsequent storage position of the second key value storage position, and storing the second key value index at a subsequent storage position of the second key name storage position.
4. The method of claim 2 or 3, wherein determining whether a hash collision free storage condition is satisfied based on one or more of the first key value storage location, the read sequence number, the first key value, the first key name, and the target key name comprises at least one of:
if the position number of the first key value storage position is smaller than the reading sequence number and the first key value is empty, determining that the hash-collision-free storage condition is met;
if the position number of the first key value storage position is smaller than the reading sequence number, the first key value is not empty, and the target key name is consistent with the first key name, determining that the no-hash-collision storage condition is met;
if the position number of the first key value storage position is smaller than the reading sequence number, the first key value is not empty, and the target key name is inconsistent with the first key name, determining that the no-hash conflict storage condition is not met; and
if the position number of the first key value storage position is larger than the reading sequence number and the target key name is consistent with the first key name, determining that the Hash-free conflict storage condition is met; and
and if the position number of the first key value storage position is larger than the reading sequence number and the target key name is inconsistent with the first key name, determining that the no-hash-conflict storage condition is not met.
5. The method of claim 2, wherein using the hash value of the target key name as the target key name index, locating a first key name storage location in the second number group, and a first key value index storage location comprises:
in the second array, acquiring a storage position matched with the target key name index as the first key name storage position, and taking a later storage position of the first key name storage position as the first key value index storage position;
locating a first key value storage location and a first key name index storage location in a first array according to the first key value index, comprising:
and in the first array, acquiring a storage position matched with the first key value index as a first key value storage position, and taking a later storage position of the first key value storage position as the first key name index storage position.
6. The method of claim 2, after obtaining the first key-value index at the first key-value index storage location, further comprising:
if the first key value index is determined to be empty, determining a third key value index according to the first key name storage position;
determining a third key value storage position in the first array according to the writing sequence number, and determining a third key value index according to the third key value storage position;
storing the target key value at the third key value storage position, storing a target key name at the first key name storage position, storing the third key value index at a subsequent storage position of the third key value storage position, and storing the third key value index at a subsequent storage position of the first key name storage position.
7. The method of any of claims 1-6, wherein performing a circular read of data in the first array and the second array based on the current read sequence number comprises:
determining a current key value storage position pointed by the reading sequence number in the first array according to the current reading sequence number, acquiring a current key value, and emptying the current key value storage position;
and determining a current key name index according to a next storage position of the current key value storage position, determining the storage position of the current key name in the second array according to the current key name index, acquiring the current key name, and emptying the storage position of the current key name.
8. An apparatus for reading and writing data, comprising:
the target data acquisition module is used for acquiring target data from the partitions and acquiring a target key name and a target key value in the target data;
the target data writing module is used for adjacently storing the target key value and the target key value index in a first array and adjacently storing the target key value and the target key value index in a second array in a ring writing mode according to the current reading sequence number and/or writing sequence number;
the target key name index is used for indicating the storage position of a target key name in the second array, and the target key value index is used for indicating the storage position of a target key value in the first array;
and the target data reading module is used for circularly reading the data in the first array and the second array according to the current reading sequence number.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements a method for reading and writing data according to any of claims 1-7 when executing the program.
10. A storage medium containing computer-executable instructions for performing a method of reading and writing data according to any one of claims 1 to 7 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911007543.5A CN110764705B (en) | 2019-10-22 | 2019-10-22 | Data reading and writing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911007543.5A CN110764705B (en) | 2019-10-22 | 2019-10-22 | Data reading and writing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110764705A true CN110764705A (en) | 2020-02-07 |
| CN110764705B CN110764705B (en) | 2023-08-04 |
Family
ID=69331308
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911007543.5A Active CN110764705B (en) | 2019-10-22 | 2019-10-22 | Data reading and writing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110764705B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579003A (en) * | 2020-12-15 | 2021-03-30 | 浙江大华技术股份有限公司 | Key value pair adjustment method, key value pair adjustment device, storage medium and electronic device |
| CN113467716A (en) * | 2021-06-11 | 2021-10-01 | 苏州浪潮智能科技有限公司 | Data storage method, device, equipment and readable medium |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102323947A (en) * | 2011-09-05 | 2012-01-18 | 东北大学 | Generation method of pre-join table on ring-shaped schema database |
| CN102722449A (en) * | 2012-05-24 | 2012-10-10 | 中国科学院计算技术研究所 | Key-Value local storage method and system based on solid state disk (SSD) |
| CN103186668A (en) * | 2013-03-11 | 2013-07-03 | 北京京东世纪贸易有限公司 | Method and device for processing data as well as data storage system based on key value data base |
| CN105320775A (en) * | 2015-11-11 | 2016-02-10 | 中科曙光信息技术无锡有限公司 | Data access method and apparatus |
| US20160110292A1 (en) * | 2014-10-21 | 2016-04-21 | Samsung Electronics Co., Ltd. | Efficient key collision handling |
| US20160321294A1 (en) * | 2015-04-30 | 2016-11-03 | Vmware, Inc. | Distributed, Scalable Key-Value Store |
| CN106096023A (en) * | 2016-06-24 | 2016-11-09 | 腾讯科技（深圳）有限公司 | Method for reading data, method for writing data and data server |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108563532A (en) * | 2018-02-28 | 2018-09-21 | 深圳和而泰数据资源与云技术有限公司 | Data processing method and relevant apparatus |
| CN108572958A (en) * | 2017-03-07 | 2018-09-25 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN108595268A (en) * | 2018-04-24 | 2018-09-28 | 咪咕文化科技有限公司 | A kind of data distributing method, device and computer readable storage medium based on MapReduce |
| CN108664487A (en) * | 2017-03-28 | 2018-10-16 | Tcl集团股份有限公司 | A kind of write-in of hash table data, read method and system |
| CN109543080A (en) * | 2018-12-04 | 2019-03-29 | 北京字节跳动网络技术有限公司 | A kind of caching data processing method, device, electronic equipment and storage medium |
| CN109558084A (en) * | 2018-11-29 | 2019-04-02 | 文华学院 | A kind of data processing method and relevant device |
| CN109656923A (en) * | 2018-12-19 | 2019-04-19 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, electronic equipment and storage medium |
| CN109710190A (en) * | 2018-12-26 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110049091A (en) * | 2019-01-10 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Date storage method and device, electronic equipment, storage medium |
-
2019
- 2019-10-22 CN CN201911007543.5A patent/CN110764705B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102323947A (en) * | 2011-09-05 | 2012-01-18 | 东北大学 | Generation method of pre-join table on ring-shaped schema database |
| CN102722449A (en) * | 2012-05-24 | 2012-10-10 | 中国科学院计算技术研究所 | Key-Value local storage method and system based on solid state disk (SSD) |
| CN103186668A (en) * | 2013-03-11 | 2013-07-03 | 北京京东世纪贸易有限公司 | Method and device for processing data as well as data storage system based on key value data base |
| US20160110292A1 (en) * | 2014-10-21 | 2016-04-21 | Samsung Electronics Co., Ltd. | Efficient key collision handling |
| US20160321294A1 (en) * | 2015-04-30 | 2016-11-03 | Vmware, Inc. | Distributed, Scalable Key-Value Store |
| CN105320775A (en) * | 2015-11-11 | 2016-02-10 | 中科曙光信息技术无锡有限公司 | Data access method and apparatus |
| CN106096023A (en) * | 2016-06-24 | 2016-11-09 | 腾讯科技（深圳）有限公司 | Method for reading data, method for writing data and data server |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108572958A (en) * | 2017-03-07 | 2018-09-25 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN108664487A (en) * | 2017-03-28 | 2018-10-16 | Tcl集团股份有限公司 | A kind of write-in of hash table data, read method and system |
| CN108563532A (en) * | 2018-02-28 | 2018-09-21 | 深圳和而泰数据资源与云技术有限公司 | Data processing method and relevant apparatus |
| CN108595268A (en) * | 2018-04-24 | 2018-09-28 | 咪咕文化科技有限公司 | A kind of data distributing method, device and computer readable storage medium based on MapReduce |
| CN109558084A (en) * | 2018-11-29 | 2019-04-02 | 文华学院 | A kind of data processing method and relevant device |
| CN109543080A (en) * | 2018-12-04 | 2019-03-29 | 北京字节跳动网络技术有限公司 | A kind of caching data processing method, device, electronic equipment and storage medium |
| CN109656923A (en) * | 2018-12-19 | 2019-04-19 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, electronic equipment and storage medium |
| CN109710190A (en) * | 2018-12-26 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110049091A (en) * | 2019-01-10 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Date storage method and device, electronic equipment, storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579003A (en) * | 2020-12-15 | 2021-03-30 | 浙江大华技术股份有限公司 | Key value pair adjustment method, key value pair adjustment device, storage medium and electronic device |
| CN112579003B (en) * | 2020-12-15 | 2022-06-14 | 浙江大华技术股份有限公司 | Key value pair adjustment method, key value pair adjustment device, storage medium and electronic device |
| CN113467716A (en) * | 2021-06-11 | 2021-10-01 | 苏州浪潮智能科技有限公司 | Data storage method, device, equipment and readable medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110764705B (en) | 2023-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110096336B (en) | Data monitoring method, device, equipment and medium | |
| CN105786405B (en) | A kind of online upgrading method, apparatus and system | |
| CN104951340A (en) | Information processing method and device | |
| CN110764705B (en) | Data reading and writing method, device, equipment and storage medium | |
| CN109547807B (en) | Information processing method and device based on live broadcast and server | |
| CN111124254B (en) | Method, electronic device and program product for scheduling memory space reclamation requests | |
| CN109359060B (en) | Data extraction method, device, computing equipment and computer storage medium | |
| CN108304272B (en) | Data IO request processing method and device | |
| CN110708369B (en) | File deployment method and device for equipment nodes, scheduling server and storage medium | |
| CN113836184A (en) | Service persistence method and device | |
| CN111813517A (en) | Task queue allocation method and device, computer equipment and medium | |
| US8543722B2 (en) | Message passing with queues and channels | |
| CN109524031B (en) | Conflict detection method and system | |
| CN113347238A (en) | Message partitioning method, system, device and storage medium based on block chain | |
| CN111400241B (en) | Data reconstruction method and device | |
| CN111046004B (en) | Data file storage method, device, equipment and storage medium | |
| CN112650449A (en) | Release method and release system of cache space, electronic device and storage medium | |
| CN116089477B (en) | Distributed training method and system | |
| CN112181662B (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN108958967B (en) | Data processing method and server | |
| CN108182241A (en) | A kind of optimization method of data interaction, device, server and storage medium | |
| CN116185649A (en) | Storage control method, storage controller, storage chip, network card, and readable medium | |
| CN111131512B (en) | Equipment information processing method and device, storage medium and processor | |
| CN113076197A (en) | Load balancing method and device, storage medium and electronic equipment | |
| CN112612790A (en) | Card number configuration method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |