CN113377819B

CN113377819B - Hash table rolling searching method, system and chip

Info

Publication number: CN113377819B
Application number: CN202110769814.1A
Authority: CN
Inventors: 张洪柳; 司浩
Original assignee: Qingdao Fangcun Microelectronic Technology Co ltd; Shandong Fangcun Microelectronics Technology Co ltd
Current assignee: Qingdao Fangcun Microelectronic Technology Co ltd; Shandong Fangcun Microelectronics Technology Co ltd
Priority date: 2021-07-07
Filing date: 2021-07-07
Publication date: 2023-02-03
Anticipated expiration: 2041-07-07
Also published as: CN113377819A

Abstract

The invention provides a hash table rolling searching method, a hash table rolling searching system and a chip. The method comprises the steps of calling compressed historical data and data to be compressed into a compression core, and respectively correspondingly assembling and updating the compressed historical data and the data to be compressed into a linear shift register window and a data window to be compressed; performing hash calculation on data in a data window to be compressed, and rolling to establish a first array according to a calculation result, wherein key values to be compressed are stored in the first array; sequentially rolling the linear shift register window backwards in a flow manner in the historical data buffer and constructing a second array through Hash calculation, wherein the second array stores the positions of key values in the searched historical window, and the positions of the key values are the same as the Hash calculation results of the key values to be compressed in the first array; performing pipeline parallel primary fuzzy matching on the first array and the second array to obtain a possible maximum candidate position of the primary fuzzy matching; and performing second-stage flow accurate matching based on the historical data buffer and the data window to be compressed to obtain a second-stage accurate matching candidate position, namely a maximum string matching result.

Description

Hash table rolling searching method, system and chip

Technical Field

The invention belongs to the field of software thread data processing, and particularly relates to a hash table rolling searching method, a hash table rolling searching system and a hash table rolling searching chip.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

According to the traditional design of the hash query compression method based on the software thread, the compression speed is low, the hardware implementation depends on an adopted CPU and a fixed third-party hardware platform corresponding to peripheral equipment by means of an adopted CPU processor and an integral framework, a large number of CPU cycles are consumed for software hash data compression, the compression speed depends on the fastest natural frequency of the CPU, and the hash compression thread occupies a CPU processing time slice. Namely, the traditional software hash compression processing method is inconvenient to apply to a data compression processing scene facing Internet of Things (IoT) application pursuing high speed and real time, or corresponding data compression embedded hardware equipment and hardware system, and a hardware bottom platform constructed by such embedded hardware. Meanwhile, the traditional hash software compression processing thread has no capability of IP self-adaptive transplantation to a hardware SoC; the software hash process cannot meet the low latency requirements of real-time compression/decompression.

Disclosure of Invention

In order to solve the technical problems in the background art, the invention provides a hash table rolling search method, a hash table rolling search system and a hash table rolling search chip, which can reduce the processing time cost of searching and matching compressed historical data, so that the compression rate is not lost.

In order to achieve the purpose, the invention adopts the following technical scheme:

a first aspect of the present invention provides a hash table rolling lookup method.

A hash table rolling lookup method, comprising:

calling the compressed historical data and the data to be compressed into a compression core, and respectively and correspondingly assembling and updating the compressed historical data and the data to be compressed into a linear shift register window and a data window to be compressed;

performing hash calculation on data in a data window to be compressed, and rolling to establish a first array according to a calculation result, wherein the first array stores key values to be compressed;

sequentially rolling the linear shift register window backwards in a flow manner in the historical data buffer, and constructing a second array through Hash calculation, wherein the second array stores the positions of key values in the searched historical window, which are the same as the Hash calculation results of the key values to be compressed in the first array;

performing pipeline parallel primary fuzzy matching on the first array and the second array to obtain a possible maximum candidate position of the primary fuzzy matching;

and for the first-stage matching result, performing second-stage flow accurate matching based on the historical data buffer and the data window to be compressed to obtain a second-stage accurate matching candidate position, namely a maximum string matching result.

A second aspect of the invention provides a hash table rolling lookup system.

A hash table rolling lookup system, comprising:

the array assembling and updating module is used for calling the compressed historical data and the data to be compressed into the compression core and respectively and correspondingly assembling and updating the compressed historical data and the data to be compressed into the linear shift register window and the data window to be compressed;

the data to be compressed Hash calculation module is used for carrying out Hash calculation on the data in the data window to be compressed and establishing a first array in a rolling mode according to a calculation result, and the first array stores key values to be compressed;

the compressed historical data Hash calculation module is used for sequentially rolling the linear shift register windows backwards in the historical data buffer in a flowing mode and constructing a second array through Hash calculation, and the second array stores the positions of key values in the searched historical windows, wherein the key values are the same as the Hash calculation results of the key values to be compressed in the first array;

the first-stage fuzzy matching module is used for performing pipeline parallel first-stage fuzzy matching on the first array and the second array to obtain a possible maximum candidate position of the first-stage fuzzy matching;

and the second-stage accurate matching module is used for performing second-stage flow accurate matching on the first-stage matching result based on the historical data buffer and the data window to be compressed to obtain a second-stage accurate matching candidate position, namely a maximum string matching result.

A third aspect of the invention provides a chip comprising a hash table rolling lookup system as described above.

A fourth aspect of the invention provides a computer-readable storage medium.

A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps in the hash table rolling lookup method as described above.

A fifth aspect of the invention provides a computer apparatus.

A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the hash table rolling lookup method as described above when executing the program.

Compared with the prior art, the invention has the beneficial effects that:

performing pipeline parallel primary fuzzy matching on a first array and a second array to obtain a possible maximum candidate position of the primary fuzzy matching; for the first-stage matching result, second-stage pipeline accurate matching is carried out based on the historical data buffer and the data window to be compressed to obtain a second-stage accurate matching candidate position, namely a maximum string matching result, and by utilizing a pipeline processing technology, the hash table rolling updating is included, and the first-stage fuzzy and second-stage accurate pipeline matching is carried out, so that on one hand, the running frequency of processing special hardware can be greatly improved, and on the other hand, the array rolling establishment and pipeline matching speed are accelerated.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are included to illustrate an exemplary embodiment of the invention and not to limit the invention.

FIG. 1 is a flowchart of a hash table rolling lookup method according to an embodiment of the present invention;

FIG. 2 is a block diagram of a MUX-FF based linear shift register window (Line _ shift _ window) according to an embodiment of the present invention;

FIG. 3 is a table building of a rolling-in hash with adjustable PWS and HTD parameters according to an embodiment of the present invention;

FIG. 4 is a diagram of an embodiment of a shift register window (Line _ shift _ window) located on a larger History buffer (History _ buffer).

Detailed Description

The invention is further described with reference to the following figures and examples.

It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

Example one

The embodiment provides a hash table rolling lookup method aiming at embedded hardware equipment or systems with limited physical storage spaces and network bandwidths such as the internet and the internet of things and the like, and meeting the application requirements of continuously traversing, searching and matching an LZ lossless dictionary on iterative processing, in order to reduce the performance requirements of transmission and storage of data packets on the embedded hardware equipment or systems, and in order to reduce the processing time cost of searching and matching of LZ dictionary compression historical data.

Referring to fig. 1, the hash table rolling lookup method of the present embodiment includes:

step 1: and calling the compressed historical data and the data to be compressed into a compression core, and correspondingly assembling and updating the compressed historical data and the data to be compressed into a linear shift register window and a data window to be compressed respectively.

Specifically, the compressed historical data and the data waiting to be compressed are called from the outside of the chip to the inside of the chip compression core. The compressed history data is called from the DDR memory to the compression core for assembly and updating to the Line _ shift _ window linear shift register window. And calling the data window to be compressed (needing to be compressed) from the DDR memory to the compression core for assembling and updating the look _ ahead _ buffer.

Step 2: and performing hash calculation on the data in the data window to be compressed, and rolling to establish a first array according to a calculation result, wherein the first array stores the key values to be compressed. Wherein, the first array storage is realized by adopting a Head _ RAM.

And 3, step 3: and sequentially rolling the linear shift register windows backwards in the historical data buffer in a running mode, and constructing a second array through Hash calculation, wherein the second array stores the positions of key values in the searched historical windows, and the positions of the key values are the same as the Hash calculation results of the key values to be compressed in the first array. The second array storage is implemented using Pre _ RAM.

The method comprises the steps of carrying out Hash flow calculation on a plurality of continuous key values in a linear shift register window, and obtaining addresses in a double-block address range corresponding to the head key values of the continuous key values through Hash function calculation.

For hash running calculation of 3 continuous key values in a Line _ shift _ window linear shift register window, the obtained continuous key values are overlapped, and the corresponding hv is obtained through hash function calculation: a hash value (for 3 consecutive key values) and an address (first _ key _ address) within a two-block address range of the first key of the 3 consecutive overlapping key values. Wherein, hv is a hash result obtained by executing hash function calculation; the result of this calculation may also be referred to as: addresses are pointed to because the hash function result value is used as an address, and the location where the next key value with the same hash function result is stored is linked, thus forming the location information into a chain.

A series of hash-RAMs comprising: a unique Head hash-Head-RAM, and subsequent series of hash-Pre-RAMs; and the series hash-RAMs are: (1) addressed by the calculated hv; (2) For the same hv, the Head hash-Head-RAM and the subsequent series of hash-Pre-RAMs can address the same hv as the RAM address corresponding to the addressing index; (3) The first hash-Head-RAM and the subsequent series hash-Pre-RAMs store addresses first _ key _ address in the double-block address range of the first key of 3 continuous overlapped key values; (4) For the same hv, all addressed hash-RAMs form a shift chain (shift _ chain (hv)), and when a new hv corresponding to first _ key _ address is inserted, the shift chain moves integrally along the direction from the Head _ RAM to the Pre _ RAM, namely, the Update of the series of hash-RAMs is completed. The number of RAMs of the series Hash-RAMs forms the HTD (Hash-Table-Depth), PWS (Parallel-Window-Size, the number of location windows, i.e., the total number of indices of Head _ RAM and Pre _ RAM). The 2 parameters HTD, PWS can be flexibly adjusted to the architecture.

And (hv) of the calculated one (hv), namely the hash value of the new continuous 3 key values rolled in, wherein the first key address is the first key of the continuous 3 key values. (hv, first _ key _ address) is inserted into the unique Head hash-Head-RAM and the subsequent series of hash-Pre-RAMs once, then one Update operation of the whole hash-RAMs is completed, and the process is completed in a single cycle; that is, the designed hash-RAMs are updated on a rolling basis.

The Line _ shift _ window linear shift register windows sequentially roll backward in a pipeline on the history _ buffer.

And 4, step 4: and performing pipeline parallel primary fuzzy matching on the first array and the second array to obtain the possible maximum candidate position of the primary fuzzy matching.

And 5: and for the first-stage matching result, performing second-stage flow accurate matching based on the historical data buffer and the data window to be compressed to obtain a second-stage accurate matching candidate position, namely a maximum string matching result.

The data in the Look _ ahead _ buffer has been previously computed in parallel for all corresponding hash _ Look _ positions. Wherein, the hash _ look _ position is the hash function calculation result (i.e. pointing address). The hash function calculation result of Look _ ahead _ buffer is used as the head pointing address to unlink the same location in the Line _ shift _ window as the hash function result. Namely, the positions of the key values in the history _ buffer, which are the same as the hash calculation result, are linked and stored in 2 groups of Head _ RAM and Pre _ RAM; the Head _ RAM stores the compressed keyboard value, and the Pre _ RAM stores the position of the key value in the searched history window, which is the same as the hash function calculation result of the compressed keyboard value stored in the Head _ RAM.

Then, performing flow-parallel primary fuzzy matching on the only Head _ RAM and the series Pre _ RAM which are well established by the calculated hv _ look derotation to obtain a possible maximum candidate position of the primary fuzzy matching, and then performing secondary flow-line accurate matching on the history _ buffer and the look _ ahead _ buffer according to the first-stage matching result to obtain a final secondary accurate matching candidate position, namely obtaining a maximum string matching result. The whole process is from fuzzy matching to accurate matching, and is from first-level to second-level, and the process of eliminating candidate positions is divided into two levels, and finally, the maximum matching character string is obtained.

As shown in fig. 2, the MUX-FF structure, the MUX-FF sign and the linear shift window 3 are respectively shown at the upper left, the upper right and the lower side;

in this embodiment, based on the linear shift register window in fig. 2, 2 processing of Line _ shift _ window data can be implemented, and the functional modes include:

data parallel loading mode: loading a plurality of unit data of a linear shift register window into a window unit in parallel;

serial shift mode: the shifted-in data moves forward along the linear shifter window chain.

The present embodiment proposes the use of a linear shift register window composition and structure constructed based on MUX-FF, as shown in FIG. 3. As shown in fig. 3, explanation of 2 adjustable parameters PWS (Position-Window-Size) and HTD (Hash-Table-Depth) in the upper part of fig. 3;

the top right corner of FIG. 3 is for the linear shift window Line _ shift _ window (i.e., SB: the portion of the history lookup buffer), which can be seen to implement Hash value calculation for 8 consecutive overlapping 3 keys of the partial history buffer (larger compressed history data keys, on which Line _ shift _ window slides sliding backwards) (see the explanation of the top, bottom right corner of FIG. 2), the unique Head Hash-Head-RAM inserted to the corresponding hv (Hash value) address and series Hash-Pre-RAMs (Hash-RAMs) stored inside which are the Block address assignment of the Block (Block _ current) where the current processing data is located and the previous Block (Block _ Pre), and the Hash-Head-RAM and series Hash-Pre-RAMs are the same to facilitate the reading of the corresponding hv-Hash-Chiv (content) Block according to hv.

As shown in FIG. 4, when the Line _ shift _ window rolls backward in the larger history buffer (history _ buffer) to obtain a new key value key, and after hv is calculated, the Head Hash-Head-RAM and the series Hash-Pre-RAMs (Hash-RAMs) Update in a shift chain manner.

This embodiment proposes that, based on the two-stage matching method, based on the hash values of 3 keys that are continuously overlapped in the look-ahead buffer (to-be-compressed data, see the look-ahead buffer on the upper right part of fig. 3), the established hash-RAMs are scrolled to perform pipeline matching: parallel first-stage fuzzy matching match is included to find the most probable candidate matching substring (the first stage may have false match); then, executing second-stage accurate matching, namely reading the candidate maximum-length substrings from the corresponding historical history _ buffers, and performing maximum substring determination comparison with substrings in the look-ahead buffers; at the moment, the hash-RAMs are continuously updated in a pipeline rolling mode, meanwhile, the first-stage matching and the second-stage matching of the pipeline are executed in parallel, the last unique and accurate determined maximum substring is obtained through screening aiming at the current look-ahead buffer data to be compressed, in this way, the traversal search of the maximum substring of the history _ buffer is completed once, the unique final maximum matching substring is obtained, and then (offset address offset, maximum matching substring length) lossless dictionary coding processing is carried out.

In this embodiment, for the established Head Hash-Head-RAM and series Hash-Pre-RAMs, it is proposed to use interleaved stream read and stream write operations (i.e. before the update of the write of the Hash-RAMs, the whole memory contents of the Hash-RAMs are read in a stream manner to perform the first-level fuzzy screening of the best matching candidate substring), and the following 2 stream processing operations need to be completed in parallel:

rolling in hv corresponding to the newly moved continuous keys in the line _ shift _ window, inserting the hv into the hash-RAMs, inserting the new hv position and completing the update of the hash-RAMs for newly calculating hv by using a shift chain (for the same hv) method, as shown in fig. 2; wherein, the new 'entering' new key value for Hash calculation is rolled in the running water; so as to perform hash search matching and perform rolling update of the search hash table.

According to hv in the look-ahead buffer, executing parallel read hash-RAMs content (the block where the current compressed data is located and the previous block form uniform address distribution, namely neighbor double-block address), and performing a first-stage pipeline matching fuzzy and a second-stage accurate matching process.

In this embodiment, in order to reduce the problem of pseudo-matching of different consecutive key values keys, a parallel Filter IDs array is formed based on a series of 3 consecutive overlapped keys inside a look-ahead buffer, and as shown in fig. 2, a pre-filtering process is performed on a new consecutive key rolled into a line _ shift _ window to Filter out 3 consecutive keys that are not present in a look-buffer window (to-be-compressed data window) at all and are newly moved into the line _ shift _ window, so as to reduce the hash redundancy calculation amount and the primary hash-RAMs redundancy update. Meanwhile, the redundant matching calculation of the redundant hash-RAMs Update to the subsequent pipeline secondary matching process is greatly reduced.

In the embodiment, part of the initial continuous keys or all of the continuous keys in the look-ahead buffer are selected as the parallel Filter IDs, and the more the selected Filter IDs cover the continuous keys in the look-ahead buffer, the more the parallel hardware pre-filtering logic resources are needed, but the better the redundancy removing effect is. On the contrary, part of continuous keys is selected, namely the Filter IDs array covers the look-ahead buffer part of continuous keys, so that less hardware logic resources are needed, and the effect of removing redundant hv calculation and the redundant hash table rolling-in is poor. The pre-filtering Filter IDs array has array expansion and expansion capabilities.

The embodiment makes full use of the pipeline processing technology, including hash table rolling update, and the first-stage fuzzy and second-stage precise pipeline matching, so that on one hand, the running frequency of processing special hardware can be greatly improved, and on the other hand, the hash-RAMs rolling establishment and pipeline matching speed are accelerated. In the embodiment, the design flexibility of building the hash-RAMs is improved by reasonably arranging 2 adjustable PWS and HTD experience parameters related to the hash-RAMs, so that the design can flexibly adjust parameters according to actual application experience; the PWS parameters related to the hash tables (i.e., the hash-Head-RAM and the hash-Pre-RAMs) proposed in this embodiment cannot be too small, otherwise, the calculated hv completes the truncation process to adapt to the PWS, and then the contents corresponding to a certain hv index of the hash-RAMs may be full of the hash-RAM-child (hv), which causes the hash-RAMs to be unevenly filled and distributed, and the resulting part of hv of the hash-RAMs is overfilled, and the resulting part of hv of the hash-RAMs is over-idle, resulting in the loss of the matching contents, and further causing the reduction of the compression rate.

In this embodiment, when the hash-RAMs are updated, the scrolled hv (for a 3-ge continuous overlapped key located on the Line _ shift _ window) is from the Line _ shift _ window, and the Line _ shift _ window may be from a key of a new shift in or from a key located at the other position, and the data tuning mode switching (parallel loading mode or serial shifting mode) is performed through a chip select SE signal of a MUX-FF unit of the Line _ shift _ window, so that the data tuning control of the Line _ shift _ window can be completed, and the design flexibility of data tuning is greatly increased.

Example two

The embodiment provides a hash table rolling lookup system, which specifically includes the following modules:

the array assembling and updating module is used for calling the compressed historical data and the data to be compressed into the compression core, and respectively and correspondingly assembling and updating the compressed historical data and the data to be compressed into the linear shift register window and the data to be compressed;

It should be noted that, each module in the hash table rolling lookup system of this embodiment corresponds to each step in the first embodiment one to one, and the specific implementation process is the same, which is not described here again.

EXAMPLE III

The present embodiment provides a chip, which specifically includes the hash table rolling lookup system as described in embodiment two.

Example four

The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the hash table rolling lookup method as described in the first embodiment above.

EXAMPLE five

The present embodiment provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps in the hash table rolling lookup method according to the first embodiment.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hash table rolling lookup method, comprising:

sequentially rolling the linear shift register windows backwards in a flow manner in the historical data buffer, and constructing a second array through Hash calculation, wherein the second array stores the positions of key values in the searched historical windows, and the positions of the key values are the same as the Hash calculation results of the key values to be compressed in the first array;

2. The hash table rolling lookup method of claim 1 wherein hash pipeline calculations are performed on a plurality of consecutive key values located within the linear shift register window, and addresses within the double block address range corresponding to the top key value of the consecutive key values are obtained by hash function calculations.

3. The hash table rolling lookup method as claimed in claim 1 wherein the linear shift register window is constructed using a multiplexer.

4. The hash table rolling lookup method as claimed in claim 1 wherein the first array and the second array are updated in a shift chain manner.

5. The hash table rolling lookup method of claim 1 wherein interleaved pipelined read and pipelined write operations are used during the one-level fuzzy matching.

6. The hash table rolling lookup method of claim 5 wherein the entire first array memory contents are read in a pipelined manner prior to updating the pipelined write operation, and then a first-level fuzzy screening of the best matching candidate substring is performed.

7. A hash table rolling lookup system, comprising:

8. A chip comprising the hash table rolling lookup system of claim 7.

9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the hash table rolling lookup method according to any one of claims 1-6.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the hash table rolling lookup method according to any one of claims 1-6 when executing the program.