CN109491602A - A kind of Hash calculation method and system for the storage of Key-Value data - Google Patents

A kind of Hash calculation method and system for the storage of Key-Value data Download PDF

Info

Publication number
CN109491602A
CN109491602A CN201811284607.1A CN201811284607A CN109491602A CN 109491602 A CN109491602 A CN 109491602A CN 201811284607 A CN201811284607 A CN 201811284607A CN 109491602 A CN109491602 A CN 109491602A
Authority
CN
China
Prior art keywords
hash
key value
key
value
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN201811284607.1A
Other languages
Chinese (zh)
Inventor
张跃进
李波
李光辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongxiang Bo Qian Mdt Infotech Ltd
Original Assignee
Zhongxiang Bo Qian Mdt Infotech Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhongxiang Bo Qian Mdt Infotech Ltd filed Critical Zhongxiang Bo Qian Mdt Infotech Ltd
Priority to CN201811284607.1A priority Critical patent/CN109491602A/en
Publication of CN109491602A publication Critical patent/CN109491602A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The present invention relates to a kind of Hash computing systems for the storage of Key-Value data, comprising: upstream fifo module, upstream fifo module is for reading key value and key value length；Hash computing module, Hash computing module carry out hash calculating according to key value and key value length, and obtaining three groups of bit wides is respectively 28bit, the hash value of 24bit, 5bit；And downstream fifo module, the data that downstream fifo module is used to for 3 hash values being integrated into 57bit bit wide are stored, are called for data flow rear end.The invention has the benefit that storing the mode that reading manner and jackknife method combine by using FIFO calculates cryptographic Hash, the time in the period of reading data is saved, the efficiency of reading data is improved, to keep the calculating of cryptographic Hash more efficient.

Description

A kind of Hash calculation method and system for the storage of Key-Value data
Technical field
The invention belongs to technical field of data storage, and in particular to a kind of based on the Hash of Key-Value data storage Calculate method and system.
Background technique
With the continuous development of the new technologies such as big data, artificial intelligence, traditional relationship type storing data library more cannot Meet the demand that new application scenarios store data, and Key-Value storage utilizes simplest key-value pair storage mode, tool There is amount of storage big, the characteristics of read or write speed is fast, high concurrent and scalability, is answered in more and more application scenarios With.Key-Value storage has become the important component of distributive data center now.
With the increase of internet scale, the data volume stored in internet is also growing day by day.Guaranteeing availability Under the premise of, the data access demand for how meeting most users to the maximum extent also becomes that Database Systems are urgently to be resolved to ask Topic, and be exactly to solve the problems, such as this effective means to the acceleration technique of Key-Value storage.
And Key-Value storage is a kind of data storage technology of data-stream form, each ring in data flow all influences The performance of whole system, search request of the every progress of system be employed to Hash function to realize, the response speed of system The speed of Hash lookup will be directly depended on.Therefore, the speed for how improving Hash lookup, which just becomes, to solve the above problems It is crucial.
Summary of the invention
In order to solve the problems, such as that Hash search speed of the existing technology is slow, the present invention provides one kind to be used for Key- The Hash calculation method and system of Value data storage, have the characteristics that the high calculating speed of reading efficiency is fast.
The object of the present invention is to provide a kind of efficient Hash calculation method is stored for Key-Value data and be System, to improve Key-Value data storage storage and search speed.
A kind of Hash computing system for the storage of Key-Value data of specific embodiment according to the present invention, packet It includes:
Upstream fifo module, the upstream fifo module is for reading key value and key value length；
Hash computing module, the Hash computing module carry out hash calculating according to the key value and key value length, obtain It is respectively 28bit, the hash value of 24bit, 5bit to three groups of bit wides；And
Downstream fifo module, the downstream fifo module are used to for 3 hash values being integrated into the data of 57bit bit wide Storage is called for data flow rear end.
Further, the key value width that the upstream fifo module is read every time is 128bit.
It further, is 1 the time required to the upstream fifo module reads the key value that width is 128bit every time Clock cycle.
Further, the upstream fifo module and the downstream fifo module are real by the IP kernel in Vivado design kit It is existing, and using the FIFO reading mode of " First Word Fall Through ".
Further, the Hash computing module is finite state machine, and key value is by the finite state machine with 128bit Unit carries out the accumulation calculating of exclusive or, obtains the key value of 128bit, then by XOR operation rule respectively obtain 28bit, The hash value of 24bit, 5bit.
Further, the width for the key value length that the upstream fifo module is read is 8bit.
Further, the finite state machine is programmed by Verilog hardware description language and is run on FPGA and realizes.
Further, the upstream fifo module and the downstream fifo module are asynchronous work mode.
A kind of Hash calculation method for the storage of Key-Value data of specific embodiment according to the present invention, it is first It first obtains key value and key value length, the Hash calculation method includes:
The accumulation calculating that key value is carried out to exclusive or as unit of 128bit, obtains the key value of 128bit；
The key value of the 128bit is subjected to exclusive or calculating three times respectively, respectively obtains 28bit, 24bit, 5bit Hash value；
Three hash values are fused to the storage of 57bit data, are called for data flow rear end.
Further, clock periodicity needed for the Hash calculation method are as follows:
Clock periodicity=(key value length -1)/16+3.
The invention has the benefit that storing the mode that reading manner and jackknife method combine by using FIFO calculates Kazakhstan Uncommon value, saves the time in the period of reading data, improves the efficiency of reading data, to make the calculating of cryptographic Hash more Efficiently.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with It obtains other drawings based on these drawings.
Fig. 1 is the Hash computing system for the storage of Key-Value data that a specific embodiment provides according to the present invention Structural schematic diagram；
Fig. 2 is the reading timing diagram of the fifo module provided according to a specific embodiment；
Fig. 3 is the flow diagram of the finite state machine provided according to a specific embodiment；
Fig. 4 is that the Hash calculation method process for the storage of Key-Value data provided according to a specific embodiment is shown It is intended to.
Appended drawing reference
The upstream 1- fifo module；2-Hash computing module；The downstream 3- fifo module.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, technical solution of the present invention will be carried out below Detailed description.Obviously, described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.Base Embodiment in the present invention, those of ordinary skill in the art are obtained all without making creative work Other embodiment belongs to the range that the present invention is protected.
Shown in referring to Fig.1, the embodiment provides one kind to be used for Key-Value (a kind of distributed memory system) The Hash computing system of data storage, comprising:
Upstream fifo module 1 (data buffer of first in first out), upstream fifo module 1 is for reading key value and key value Length；
Hash computing module 2, Hash computing module 2 carry out hash calculating according to key value and key value length, obtain three groups Bit wide is respectively 28bit, the hash value of 24bit, 5bit；And
Downstream fifo module 3, the data that downstream fifo module 3 is used to for 3 hash values being integrated into 57bit bit wide store, It is called for data flow rear end.
The method combined using RDMA with programmable, intelligent network interface card has unloaded part Key-Value storage network, will be former Come a series of activities that Xie Wangluobao, Hash for needing CPU to participate in are calculated, it is unloaded in programmable network interface card and executes, thus significantly The concurrent executive capability of single memory node is improved, the throughput of system is improved.
Wherein, RDMA (Remote DMA) is long-distance inner access technique, is a kind of net similar to ICP/IP protocol stack Network transport protocol has bypassed CPU during unpacking, has accelerated network transfer speeds.
Programmable NIC, the i.e. network interface card with network offloading functions, wherein NIC is the network interface card of a support rdma protocol, packet Contain Hash locating function, to alleviate the burden of cpu data processing, accelerates the throughput of system.
The difficult point of the hardware-accelerated system first is that Hash search hardware unloading, and Hash search pith Hash Function exactly influences the key component of entire hardware realization performance.
It in a specific embodiment of the invention, is the portability of lifting system, upstream fifo module 1 and downstream FIFO Module 3 is realized using asynchronous FIFO, i.e., during this module and other modules interconnect without the concern for across when The problem of clock domain.Two groups of data: Key and KeyLen are mainly exported from upstream fifo module 1.Wherein, the length of Key is 64bit to 1024bit is variable, and the interface width for storing the FIFO of Key is 128, and therefore, it is necessary to KeyLen to indicate Key The byte number possessed.The bit wide of KeyLen is fixed 8bit, it represents the length of Key, i.e. byte number.If KeyLen's is big Small is 16, then represents the length of Key as 16 bytes, i.e. 128bit, then FIFO only needs a clock cycle that can export One Key；Similarly, if the size of KeyLen is 32, the length of Key is represented as 2 bytes, i.e. 256bit, then FIFO needs Two clock cycle can just export a Key.Mainly exported from downstream fifo module 3 three bit wides be respectively 5bit, The hash value of 24bit and 28bit.
Upstream fifo module 1 and downstream FIFO as a kind of feasible implementation of above-described embodiment, in the present invention Module 3 is realized using the IP kernel provided in Vivado.For the read or write speed for promoting FIFO, present invention employs " First The FIFO reading mode of Word Fall Through ", the mode and mode standard timing are referring to shown in attached drawing 2.Under this mode, During reading data fifo, when reading to enable in rising edge, the read port of FIFO can read data.Therefore, Compared with standard mode, which saves the time of a reading clock cycle when reading data, improves the effect of reading data Rate.
In a specific embodiment of the invention, Hash computing module 2 is for calculating hash value corresponding to keyword Key Size.In view of the input mode of keyword Key data flow, to reduce the clock cycle needed for Hash is calculated, using folding The mode of method realizes Hash, as long as in this way, Hash calculating can be carried out when obtaining part Key value, therefore, in Key input Meanwhile Hash calculating can be carried out to obtained Key value.The calculation formula for the clock periodicity that wherein whole process needs Are as follows: clock periodicity=(key value length -1)/16+3.
Referring to shown in Fig. 3, as a kind of feasible implementation of above-described embodiment, Hash computing module 2 is by one A finite state machine is come what is realized, and the state of the finite state machine is first in the S_1 state of initialization, if KeyLen is always Sky is then always maintained at this state.When detecting KeyLen is non-empty, then it is transformed into S_2 state, this state is following It generates output signal and does data preparation, while also having read the data of preceding 128 bit in Key.If storing the upstream of keyword Fifo module 2 is sky, then state machine is always maintained at this state, when detecting non-empty, utilizes a kind of KeyCnt (counting change Amount) judge that fully entering Key also needs several clock cycle.If only needing a clock cycle, S_4 state is jumped directly to； If desired it is more than or equal to a clock cycle, then jumps to S_3.S_3 state is to judge whether Key input terminates and to input Part Key carries out the state of exclusive or calculating, and the state is identical as the decision condition that S_2 state shifts, i.e., when detecting non-empty, Key, which is fully entered, using KeyCnt judgement also needs several clock cycle.If only needing a clock cycle, S_ is jumped directly to 4 states；If desired it is more than or equal to a clock cycle, then maintains the original state.S_4 state be to input obtain it is folded 128 hash values are translated by the state that hash value standardizes again by the method for two operation of mould The hash value of 28bit, 24bit, 5bit.When downstream fifo module 3 is non-empty, S_4 state can be converted to S_1, and this is initial State, so that the Hash for carrying out next Key is calculated.
In a specific embodiment of the invention, Hash computing module 2 (i.e. finite state machine) is developed based on NetFPGA , using it is hard-wired, applied to Key-Value storage in Hash computing module.Entire module utilizes Verilog hardware Description language is write.
As a kind of feasible implementation of above-described embodiment, by Test Bench to Verilog Hardware description language Say that the program write is verified.
TestBench (testing stand) module simulation upstream fifo module 1, for Hash computing module 2 provide Key with It is 50MHZ that KeyLen, FIFO, which read clock frequency,；It simulates downstream fifo module 3 and reads interface, read three hash values calculated, It is 20MHZ that downstream fifo module 3, which reads clock frequency,；The CLK and RST signal for simulating 50MHZ, provide clock for Hash computing module Input and reset signal.The test logic of entire module is as follows: providing firstly, Hash computing module 2 reads TestBench module Key and two groups of data of KeyLen, tri- groups of hash values of 28bit, 24bit and 5bit are calculated, and store data in downstream In fifo module 3；When in KeyLen data be all read finish when, TestBench start read downstream fifo module 3 in Hash value, until the reading of all hash values terminates.Wherein, the key length of use is identical, i.e. data in one group of KeyLen It is all identical.It is of different sizes according to two groups of data volumes of Key and KeyLen, 3 groups of different tests have been carried out altogether, test result is as follows table It is shown:
Due to using the FIFO reading mode of First Word Fall Through, i.e., in the initial state, each State conversion is withouted waiting for when just having had reading on the data line that FIFO is read, and having read first data, therefore, Every time in test, two clock cycle can be saved when reading first data.Therefore, each data are arranged in following table third to add The time (40ns) of upper two clock cycle, then divided by keyword number, so that the read access time of averagely each keyword is obtained, As shown in the 5th column in table.
It can thus be seen that the present invention stores the mode meter that reading manner and finite state machine combine by using FIFO Cryptographic Hash is calculated, the time in the period of reading data is saved, improves the efficiency of reading data, to make the calculating of cryptographic Hash more What is added is efficient.
Referring to shown in Fig. 4, the embodiments of the present invention also provide a kind of Hash calculating for the storage of Key-Value data Method, first acquisition key value and key value length, Hash calculation method include:
101, the accumulation calculating that key value is carried out to exclusive or as unit of 128bit obtains the key value of 128bit；
102, the key value of 128bit is subjected to exclusive or calculating three times respectively, respectively obtains 28bit, 24bit, 5bit Hash value；
103, three hash values are fused to the storage of 57bit data, are called for data flow rear end.
Wherein, clock periodicity needed for Hash calculation method are as follows:
Clock periodicity=(key value length -1)/16+3.
In a specific embodiment of the invention, to reduce the clock cycle needed for Hash is calculated, using the side of jackknife method Formula realizes Hash, as long as in this way, can carry out Hash calculating when obtaining part Key value, it therefore, can while Key input To carry out Hash calculating to obtained Key value.Its specific Hash calculating process: firstly, the part that will be obtained from FIFO Keyword Key carries out the accumulating operation of exclusive or as unit of 128bit；Next, when Key is fully entered and finished, by 128 Keyword again in the way of exclusive or, the hash value of 28bit, 24bit, 5bit is calculated, to obtain final 3 groups Hash value.The clock periodicity that whole process needs are as follows: clock periodicity=(key value length -1)/16+3.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any Those familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all contain Lid is within protection scope of the present invention.Therefore, protection scope of the present invention should be based on the protection scope of the described claims.

Claims (10)

1. a kind of Hash computing system for the storage of Key-Value data characterized by comprising
Upstream fifo module, the upstream fifo module is for reading key value and key value length；
Hash computing module, the Hash computing module carry out hash calculating according to the key value and key value length, obtain three Group bit wide is respectively 28bit, the hash value of 24bit, 5bit；And
Downstream fifo module, the downstream fifo module is for depositing the data that 3 hash values are integrated into 57bit bit wide Storage is called for data flow rear end.
2. system according to claim 1, which is characterized in that the key value width that the upstream fifo module is read every time For 128bit.
3. system according to claim 2, which is characterized in that it is 128bit that the upstream fifo module reads width every time The key value the time required to be 1 clock cycle.
4. system according to claim 3, which is characterized in that the upstream fifo module and the downstream fifo module by IP kernel in Vivado design kit is realized, and uses the FIFO reading mode of " First Word Fall Through ".
5. system according to claim 4, which is characterized in that the Hash computing module is finite state machine, described to have Key value is carried out the accumulation calculating of exclusive or by limit state machine as unit of 128bit, obtains the key value of 128bit, then pass through exclusive or Algorithm respectively obtains the hash value of 28bit, 24bit, 5bit.
6. system according to claim 5, which is characterized in that the width for the key value length that the upstream fifo module is read Degree is 8bit.
7. system according to claim 6, which is characterized in that the finite state machine is by Verilog hardware description language It programs and is run on FPGA and realized.
8. system according to any one of claims 1 to 7, which is characterized in that the upstream fifo module and the downstream Fifo module is asynchronous work mode.
9. a kind of Hash calculation method for the storage of Key-Value data, which is characterized in that obtain key value and key value is long Degree, the Hash calculation method include:
The accumulation calculating that key value is carried out to exclusive or as unit of 128bit, obtains the key value of 128bit；
The key value of the 128bit is subjected to exclusive or calculating three times respectively, respectively obtains the hash value of 28bit, 24bit, 5bit；
Three hash values are fused to the storage of 57bit data, are called for data flow rear end.
10. according to the method described in claim 9, it is characterized in that, clock periodicity needed for the Hash calculation method are as follows:
Clock periodicity=(key value length -1)/16+3.
CN201811284607.1A 2018-10-31 2018-10-31 A kind of Hash calculation method and system for the storage of Key-Value data Withdrawn CN109491602A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811284607.1A CN109491602A (en) 2018-10-31 2018-10-31 A kind of Hash calculation method and system for the storage of Key-Value data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811284607.1A CN109491602A (en) 2018-10-31 2018-10-31 A kind of Hash calculation method and system for the storage of Key-Value data

Publications (1)

Publication Number Publication Date
CN109491602A true CN109491602A (en) 2019-03-19

Family

ID=65692079

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811284607.1A Withdrawn CN109491602A (en) 2018-10-31 2018-10-31 A kind of Hash calculation method and system for the storage of Key-Value data

Country Status (1)

Country Link
CN (1) CN109491602A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116257521A (en) * 2023-01-18 2023-06-13 深存科技(无锡)有限公司 KV storage method based on FPGA
WO2024060934A1 (en) * 2022-09-22 2024-03-28 北京火山引擎科技有限公司 Data processing method and apparatus

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024060934A1 (en) * 2022-09-22 2024-03-28 北京火山引擎科技有限公司 Data processing method and apparatus
CN116257521A (en) * 2023-01-18 2023-06-13 深存科技(无锡)有限公司 KV storage method based on FPGA
CN116257521B (en) * 2023-01-18 2023-11-17 深存科技(无锡)有限公司 KV storage method based on FPGA

Similar Documents

Publication Publication Date Title
Dharmapurikar et al. Fast and scalable pattern matching for content filtering
WO2020220743A1 (en) Computer data processing method and apparatus
WO2017000756A1 (en) Data processing method and processor based on 3072-pointfast fourier transformation, and storage medium
Lei et al. An FPGA implementation for solving the large single-source-shortest-path problem
Min Analysis on bubble sort algorithm optimization
CN109491602A (en) A kind of Hash calculation method and system for the storage of Key-Value data
CN110175184A (en) A kind of lower drill method, system and the electronic equipment of data dimension
Kulkarni et al. Hyperloglog sketch acceleration on fpga
Usui et al. A cost-effective and scalable merge sorter tree on FPGAs
CN106789733A (en) A kind of device and method for improving large scale network flow stream searching efficiency
Papaphilippou et al. An adaptable high-throughput FPGA merge sorter for accelerating database analytics
US9135984B2 (en) Apparatuses and methods for writing masked data to a buffer
CN108959168A (en) SHA512 full-flow water circuit based on-chip memory and implementation method thereof
CN110019205A (en) A kind of data storage, restoring method, device and computer equipment
CN108664518A (en) A kind of method and device for realizing processing of tabling look-up
US9570125B1 (en) Apparatuses and methods for shifting data during a masked write to a buffer
CN102129419B (en) Based on the processor of fast fourier transform
CN102411557B (en) Multi-granularity parallel FFT (Fast Fourier Transform) computing device
CN102541813B (en) Method and corresponding device for multi-granularity parallel FFT (Fast Fourier Transform) butterfly computation
CN101741743B (en) Network address sort-based bidirectional stream combining method
Puranik et al. Key-Value Store using High Level Synthesis Flow for Securities Trading System
CN106603442A (en) Cross-clock-domain high-speed data communication interface circuit of network on chip
CN104678815B (en) The interface structure and collocation method of fpga chip
CN105337759A (en) Internal and external ratio measurement method based on community structure, and community discovery method
KR20120069731A (en) Quad-data rate controller and realization method thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20190319

WW01 Invention patent application withdrawn after publication