CN109815234B - Multiple cuckoo filter under STREAMING computational model - Google Patents
Multiple cuckoo filter under STREAMING computational model Download PDFInfo
- Publication number
- CN109815234B CN109815234B CN201811635873.4A CN201811635873A CN109815234B CN 109815234 B CN109815234 B CN 109815234B CN 201811635873 A CN201811635873 A CN 201811635873A CN 109815234 B CN109815234 B CN 109815234B
- Authority
- CN
- China
- Prior art keywords
- cuckoo
- data
- cuckoo filter
- query
- filter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a multiple cuckoo filter under a flow type calculation model. The multiple cuckoo filter mainly comprises a plurality of standard cuckoo filters which are the same as the total number of data streams, one standard cuckoo filter is arranged for each data stream, each standard cuckoo filter processes the respective data stream, and the representation and query of the multiple data sets of the data streams are decomposed into the representation and query of a plurality of single data sets; and establishing a sliding window for each cuckoo filter, simultaneously carrying out filtering query on each standard cuckoo filter, dividing the data stream along a time boundary by the sliding window, and querying whether the same specified object exists between different data streams. The method can well inherit the advantages of the cuckoo filter, greatly simplifies the operation amount and the space occupancy rate aiming at the processing of the large data stream, reduces the false positive rate, is convenient for accurate data query, and has obvious technical effect.
Description
Technical Field
The invention relates to a cuckoo filter in the field of computer big data, in particular to a multiple cuckoo filter for massive multi-dimensional data index under a flow type computing model.
Background
With the rapid development of mobile internet, web2.0, smart devices, and other related industries, the amount of data generated by human beings is rapidly increasing in an exponential order. The mass data gradually presents the characteristics of large data such as huge scale, diversified types, high flow speed and the like. The multidimensional characteristics of data become more and more obvious, and the aspects of storage, real-time calculation and analysis, large-scale data indexing and searching and the like of massive multidimensional data bring severe challenges to an information system.
Unlike low-dimensional data, multidimensional data enables systems to record large amounts of comprehensive information and provide users with richer services through applications. However, the performance of a distributed data management system for multidimensional data is rapidly reduced in terms of indexing and the like, and particularly, the occupied memory space is rapidly increased along with the increase of the dimension.
Disclosure of Invention
The invention mainly aims to provide a multi-cuckoo filter for massive multi-dimensional data indexes under a stream type calculation model, thereby laying an operation foundation for establishing a multi-path data stream association relation.
The technical scheme adopted by the invention is as follows:
the invention designs a data structure of a multiple cuckoo filter according to the cuckoo filter, the multiple cuckoo filter mainly comprises a plurality of standard cuckoo filters which are the same as the total number of data streams, each data stream needing to be processed is provided with a corresponding standard cuckoo filter, each standard cuckoo filter processes respective data stream, the representation and query of the multiple data sets of the data streams are decomposed into the representation and query of a plurality of single data sets, and the false positive rate of the multiple cuckoo filter is reduced by controlling and increasing the value of the fingerprint size f of the data elements stored in the index.
When the query is carried out at any time, the standard cuckoo filters of all the data streams simultaneously carry out filtering query, whether the same specified objects exist among different data streams or not is queried, if yes, True is returned, and otherwise, False is returned.
And respectively establishing a sliding window for each cuckoo filter, segmenting the data streams from the heads of the corresponding data streams along a time boundary by the sliding windows, obtaining the quantity of the segmented source data corresponding to each data stream in any time period, comparing the fingerprints in each sliding window, and checking whether a certain specified element exists among the data streams or not.
The sliding window stores indexes of all entries of the cuckoo filter, and uses a queue data structure; and when the plurality of sliding windows simultaneously contain the element x, acquiring the final storage positions of the HashMap result of the element x in all the standard cuckoo filters, and searching whether the HashMap result is in the position index stored in the corresponding sliding window in the process of continuously moving and comparing the sliding windows along with the change of time.
The standard cuckoo filter for each data stream starts at the head of the data stream.
The designated object is a data segment or a characteristic numerical value of the data segment after processing.
The data of the standard cuckoo filter is dynamically generated by the data stream number or is set in advance.
The method mainly comprises the steps of designing a multiple cuckoo filter based on a cuckoo filter data structure and an algorithm, and decomposing representation and query of multiple data sets into representation and query of a plurality of single data sets.
In the invention, a query algorithm of multiple cuckoo filters is realized based on cuckoo filter data structure source C + + codes. And comparing the results of Hash mapping of the designated object at any moment, and inquiring whether the designated object is in the position indexes stored in a plurality of sliding windows corresponding to a plurality of standard cuckoo filters.
The false positive rate of the multiple cuckoo filters is implemented and analyzed, and the false positive rate of the multiple cuckoo filters is related to the size of a barrel, the number of the cuckoo filters, the total number of the set elements, the size of a sliding window, the size of each movement of the window and the size of a fingerprint.
The fingerprint in the invention refers to a digital fingerprint, i.e. a unique character value of a data segment, such as an MD5 value.
The present invention proposes a multiple cuckoo filter that decomposes representations and queries of multiple data streams into representations and queries for multiple single data streams. How many data streams are generated, and how many standard cuckoo filters represent the objects in the query data streams, respectively.
The input data stream of the invention is not limited to a large-capacity data set, such as a file stream.
The program codes for constructing the data structure of the multiple cuckoo filters are not limited to C + + language; the caller execution script is not limited to just Linux Shell languages, such as Python scripts. The Hash function used in the programming process is not limited to the MurmurHash, such as bobhhash, SuperFastHash, MD5Hash, SHA1Hash, etc.
In the specific implementation of the invention, the number of standard cuckoo filters is not limited to be dynamically generated along with the number of data streams, k cuckoo filters can be given in advance, when a certain cuckoo filter fails to be inserted, other elements in the set are continuously inserted into the next cuckoo filter, and after all elements are inserted, the space occupied by the idle cuckoo filter is released.
In the invention, the sliding window stores the indexes of all the entries of the cuckoo filter, namely, the barrel and the entries of the cuckoo filter are numbered from 0, thereby greatly simplifying the operation amount and the space occupancy rate.
The invention has the beneficial effects that:
the invention designs the multiple cuckoo filters based on the cuckoo filter, and can also well inherit the advantages of the cuckoo filter, namely dynamic addition and reliable deletion operation of support elements, better query performance, relevance of storage positions and smaller space utilization rate under certain conditions.
The invention not only can greatly simplify the operation amount and the space occupancy rate, but also can greatly reduce the false positive rate and facilitate accurate data query.
Compared with the conventional cuckoo filter, the multi-cuckoo filter can support the searching of objects meeting the specified relation in a plurality of data streams, and has wider application prospect compared with the conventional cuckoo filter.
Drawings
FIG. 1 is a schematic diagram of the data structure query logic of the multiple cuckoo filter according to the present invention;
FIG. 2 is a graph of Checkup query time versus sliding window size;
FIG. 3 is a graph of Checkup query time versus total number of collection elements.
Detailed Description
The invention is described in further detail below with reference to the figures and the embodiments.
The invention provides a self-adaptive cuckoo filter which dynamically changes along with the change of data flow number under a flow type calculation model, and in order to make the purpose, technical scheme and effect of the invention clearer and clearer, the invention is further described in detail below by referring to the attached drawings and taking examples.
1. Data structure
As shown in fig. 1, the multiple cuckoo filter is composed of a plurality of standard cuckoo filters having the same total number of data streams, each corresponding to a standard cuckoo filter, i.e., the representation and query of the multiple data sets are decomposed into a plurality of representations and queries of single data sets. How many data streams are generated, and how many standard cuckoo filters represent the objects in the query data streams, respectively.
Table 1: symbolic illustration of multiple cuckoo filters
Simulating a large data flow environment, the invention designs a multiple cuckoo filter data structure as shown in fig. 1.
Assuming that n data streams are used as n data sets, each data set comprises at least millions of data objects as elements, namely, n identical sets with at least millions of elements are designed, and n standard cuckoo filters are correspondingly and dynamically generated.
As shown in fig. 1, for each cuckoo filter, Windowing (Windowing) processing is performed in the following manner, which can solve the technical problem that an infinite data source never ends: each cuckoo filter establishes a sliding window, the sliding window starts from the head of each data stream, the sliding window divides the data streams along a time boundary, n pieces of segmented source data (namely query data in the sliding window) can be obtained in each time period, and then fingerprints in each sliding window are compared to check whether a certain specified element exists among a plurality of data streams or not. Fingerprints are identical and are considered to be identical designated elements.
2. Query algorithm
The element query process of the multiple cuckoo filter algorithm specifically comprises the following steps:
inputting data: num and x respectively representing the number of data streams and the element to be queried
The query process is as follows:
And 2, starting the query.
Case 1: for a single filter, if the current sliding window contains the element x, the filter is indicated to contain the element x, and true is returned;
case 2: for a single filter, if the element x is not contained in the current sliding window and the sliding window has not slid to the last piece of data in the filter, the sliding window moves downwards to continue the query;
case 3: for a single filter, if the element x is not contained in the current sliding window and the sliding window has reached the last piece of data in the filter, it indicates that the element x is not present in the filter, and false is returned;
the sliding window stores the indexes of all the entries of the cuckoo filter, the index data is the fingerprint generated by the hash, and the sliding window uses a queue data structure. And when the plurality of sliding windows simultaneously contain the element x, acquiring the final storage positions of the Hash mapping result of the element x in all the standard cuckoo filters, searching whether the Hash mapping result is in the position index stored in the corresponding sliding window in the continuous movement and comparison of the sliding windows along with the change of time, and timing to evaluate the performance under the condition of successful query.
And if the retrieval hash mapping result is not in the position index stored in the corresponding sliding window, namely the query fails, false is returned, and the failure does not indicate that the element x is not successfully inserted into the cuckoo filter, but indicates that the element x is not included in the sliding windows of the multiple cuckoo filters at any time in the sliding windows which dynamically move along with the time.
And retrieving the hash mapping result in the position index stored in the corresponding sliding window, namely, if the query is successful, returning true, and considering that the same element x exists. Meanwhile, the false positive rate is considered to exist, hash fingerprints of other elements and hash fingerprints of x may collide, and the hash fingerprints just fall into the sliding window. False positive rate analysis was performed as follows.
3. False positive rate analysis:
for a standard cuckoo filter, consider the worst query case-query one does not belong to an element in the set, then the query must retrieve all 2b entries in both buckets.
In each entry, the probability of returning a misjudgment query after matching the stored fingerprint is at most 1/2f, and after 2b times of fingerprint comparison, the upper limit of the misjudgment rate of the fingerprint is as follows:
∈CF＝1-(1-1/2f)2b≈2b/2f
the multiple cuckoo filters, without considering the dynamic window, require retrieval of all cuckoo filters for each elemental query according to the collective query operation of the multiple cuckoo filters. The false positive rate of the multiple cuckoo filters is the probability that at least one cuckoo filter in all cuckoo filters misjudges x.
The false positive rate of each cuckoo filter is equal to ECFIn all s cuckoo filters, the probability of no false positive is (1-e ∈)CF)s. The upper limit of the false positive rate of the combination of the s cuckoo filters is as follows:
1-(1-∈CF)s＝1-(1-1/2f)2bs≈2bs/2f
if the dynamic window change of the cuckoo filter is considered, assuming that a data set has m elements (i.e. data objects), the sliding window size is w, and each time k elements are moved, the two elements are generated
A sliding window.
In the dynamic window change of s cuckoo filters, the total comparison is carried out
Next, the false positive rate for the multiple cuckoo filters was calculated as:
according to the above formula, the false positive rate of the multiple cuckoo filters is related to the size of the bucket, the number of cuckoo filters, the total number of data set elements, the size of the sliding window, the size of each window movement, and the size of the fingerprint. Where a set refers to all data within a dynamic window.
In particular, increasing the value of the fingerprint size f can significantly reduce the false positive rate eMCFAnd under a larger data set, a larger value of the fingerprint size f is constructed, so that the false positive rate of the multiple cuckoo filters can be reduced.
Therefore, in the multiple cuckoo filter, a plurality of standard cuckoo filters are independent from one another, and the false positive rate is smaller than that of the standard cuckoo filters.
In the specific implementation of the invention, a cuckoo filter data structure source C + + code is used for realizing a multiple cuckoo filter.
The experiment is divided into three groups, and the relationship between the Checkup query time and the total number of cuckoo filters, the size of a sliding window and the total number of collection elements is analyzed respectively. As shown in FIG. 2, the total number of elements in the experiment is initially set to 1000000, the initial setting value of the sliding window is controlled between 50000 and 100000, and the size of each movement of the sliding window is designated as 2000.
And (4) simultaneously moving the sliding windows downwards by a plurality of cuckoo filters, and returning True if the existence of the specified elements is inquired in the corresponding sliding windows at any time, otherwise, returning False. When one of the sliding windows has reached its maximum downward, it is necessary to fix its window waiting for the other windows to move to the end of the filter in turn.
Specific time-consuming data are shown in tables 2-6 below. It is found that when the number of data streams exceeds 4, it is difficult to find fingerprints containing a certain element in 4 dynamic windows at the same time, so that the number of cuckoo filters is increased from 1 to 5, and the time and query times consumed by searching for a sliding window in which a specified element exists in a plurality of cuckoo filters at a certain moment are tested.
As can be seen from tables 2 to 5, when the number of cuckoo filters is increased from 1 to 4, the number of successful queries is greatly reduced, and the query time is linearly increased. As shown in table 6, when the number of cuckoo filters is 5 or more, it is very difficult to successfully search the fingerprint of the designated element existing in the sliding window at any time.
Table 2: 1 cuckoo filter, 50 queries per group
Table 3: 2 cuckoo filters, 100 queries per group
Table 4: 3 cuckoo filters, 100 queries per group
Table 5: 4 cuckoo filters, 100 queries per group
Table 6: 5 cuckoo filters, 100 queries per group
And further implementing the relationship between the Checkup query time and the size of the sliding window, wherein the size of the sliding window is not randomly generated. When 2 cuckoo filters are specified, the sliding window of one window is set to be the maximum, and the size of the other window is set to be a value between 20000 and 160000 of 20000 in each increase. When 3 cuckoo filters were specified, the first sliding window was set to maximum, and the other two windows were sequentially incremented at a rate of 10000 each time. As shown in fig. 3, it can be seen that the increase of the number of cuckoo filters leads to more expenditure of query time, and in general, the query time consumption gradually decreases as the sliding window is larger among the cuckoo filters with the same number.
And the total number of the data set elements is respectively and successively increased, and the influence on the Checkup query time is tested. As can be seen in FIG. 3, the larger the total number of data sets, the more time consuming the query.
Therefore, the implementation of the method can well inherit the advantages of the cuckoo filter, greatly simplifies the operation amount and the space occupancy rate aiming at the processing of the large data stream, reduces the false positive rate, is convenient for accurate data query, and has obvious technical effect.
Claims (5)
1. A multiple cuckoo filter under a flow computing model, comprising:
the multiple cuckoo filter mainly comprises a plurality of standard cuckoo filters with the same total number as the data streams, each standard cuckoo filter is arranged for each data stream, each standard cuckoo filter processes the respective data stream, the representation and query of the multiple data sets of the data streams are decomposed into the representation and query of a plurality of single data sets, and the false positive rate of the multiple cuckoo filters is controlled and reduced by increasing the value of the fingerprint size f of the data elements stored in the index;
when the query is carried out at any time, the standard cuckoo filters of all data streams simultaneously carry out filtering query, whether the same specified objects exist among different data streams or not is queried, if yes, True is returned, and otherwise, False is returned;
establishing a sliding window for each cuckoo filter, dividing the data stream along a time boundary from the head of the corresponding data stream by the sliding window, obtaining the quantity of the segmented source data corresponding to each data stream in any time period, comparing the fingerprints in each sliding window, and checking whether a certain specified element exists among a plurality of data streams;
in the dynamic window change of s cuckoo filters, the total comparison is carried out
The false positive rate of the multiple cuckoo filter belongs toMCFThe calculation is as follows:
wherein s represents the total number of cuckoo filters in the multiple cuckoo filters, w is the size of the sliding window, m represents the total number of elements in the single dataset, 2b represents the number of fingerprint comparisons, k represents the step size of the sliding window, 1/2f represents the maximum probability of returning a false positive query by matching to a stored fingerprint, eCFRepresenting the false positive rate of a single cuckoo filter.
2. The multi-cuckoo filter under flow computing model of claim 1, wherein: the sliding window stores indexes of all entries of the cuckoo filter, and uses a queue data structure; and when the plurality of sliding windows simultaneously contain the element x, acquiring the final storage positions of the HashMap result of the element x in all the standard cuckoo filters, and searching whether the HashMap result is in the position index stored in the corresponding sliding window in the process of continuously moving and comparing the sliding windows along with the change of time.
3. The multi-cuckoo filter under flow computing model of claim 1, wherein: the standard cuckoo filter for each data stream starts at the head of the data stream.
4. The multi-cuckoo filter under flow computing model of claim 1, wherein: the designated object is a data segment or a characteristic numerical value of the data segment after processing.
5. The multi-cuckoo filter under flow computing model of claim 1, wherein: the data of the standard cuckoo filter is dynamically generated by the data stream number or is set in advance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811635873.4A CN109815234B (en) | 2018-12-29 | 2018-12-29 | Multiple cuckoo filter under STREAMING computational model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811635873.4A CN109815234B (en) | 2018-12-29 | 2018-12-29 | Multiple cuckoo filter under STREAMING computational model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109815234A CN109815234A (en) | 2019-05-28 |
| CN109815234B true CN109815234B (en) | 2021-01-08 |
Family
ID=66602770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811635873.4A Active CN109815234B (en) | 2018-12-29 | 2018-12-29 | Multiple cuckoo filter under STREAMING computational model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109815234B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111478769A (en) * | 2020-03-18 | 2020-07-31 | 西安电子科技大学 | Distributed credible identity authentication method, system, storage medium and terminal |
| CN111339058B (en) * | 2020-03-24 | 2023-05-16 | 中国人民解放军国防科技大学 | Aggregation synchronization method and device |
| CN111552692B (en) * | 2020-04-30 | 2023-04-07 | 南方科技大学 | Plus-minus cuckoo filter |
| CN111552693B (en) * | 2020-04-30 | 2023-04-07 | 南方科技大学 | Tag cuckoo filter |
| CN112149416B (en) * | 2020-09-09 | 2023-08-22 | 南京大学 | Method for detecting hot academic research topics in distributed academic data warehouse |
| CN111858651A (en) * | 2020-09-22 | 2020-10-30 | 中国人民解放军国防科技大学 | Data processing method and data processing device |
| CN112597345B (en) * | 2020-10-30 | 2023-05-12 | 深圳市检验检疫科学研究院 | Automatic acquisition and matching method for laboratory data |
| CN112507689B (en) * | 2021-01-20 | 2023-08-01 | 中国地质大学（武汉） | Space range-keyword query method under distributed subscription and release mode |
| CN113535706B (en) * | 2021-08-03 | 2023-05-23 | 佛山赛思禅科技有限公司 | Two-stage cuckoo filter and repeated data deleting method based on two-stage cuckoo filter |
| CN114844638B (en) * | 2022-07-03 | 2022-09-20 | 浙江九州量子信息技术股份有限公司 | Big data volume secret key duplication removing method and system based on cuckoo filter |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103116599A (en) * | 2012-11-30 | 2013-05-22 | 浙江工商大学 | Urban mass data flow fast redundancy elimination method based on improved Bloom filter structure |
| US9584533B2 (en) * | 2014-11-07 | 2017-02-28 | Arbor Networks, Inc. | Performance enhancements for finding top traffic patterns |
| CN105989061B (en) * | 2015-02-09 | 2019-11-26 | 中国科学院信息工程研究所 | Multidimensional data repeats detection fast indexing method under a kind of sliding window |
-
2018
- 2018-12-29 CN CN201811635873.4A patent/CN109815234B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109815234A (en) | 2019-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109815234B (en) | Multiple cuckoo filter under STREAMING computational model | |
| Ding et al. | Tsunami: A learned multi-dimensional index for correlated data and skewed workloads | |
| Tan et al. | Clost: a hadoop-based storage system for big spatio-temporal data analytics | |
| Zoumpatianos et al. | Indexing for interactive exploration of big data series | |
| US7797265B2 (en) | Document clustering that applies a locality sensitive hashing function to a feature vector to obtain a limited set of candidate clusters | |
| CN110413611B (en) | Data storage and query method and device | |
| US9529912B2 (en) | Metadata querying method and apparatus | |
| US8037059B2 (en) | Implementing aggregation combination using aggregate depth lists and cube aggregation conversion to rollup aggregation for optimizing query processing | |
| US7711668B2 (en) | Online document clustering using TFIDF and predefined time windows | |
| US10521441B2 (en) | System and method for approximate searching very large data | |
| Yagoubi et al. | Dpisax: Massively distributed partitioned isax | |
| JP2008225575A (en) | Computer load estimation system and method | |
| CN113407785B (en) | Data processing method and system based on distributed storage system | |
| US11567952B2 (en) | Systems and methods for accelerating exploratory statistical analysis | |
| Wang et al. | Fast and adaptive indexing of multi-dimensional observational data | |
| CN113010484A (en) | Log file management method and device | |
| CN110795469B (en) | Spark-based high-dimensional sequence data similarity query method and system | |
| Liu et al. | HAP: an efficient hamming space index based on augmented pigeonhole principle | |
| Chen et al. | Ranked window query retrieval over video repositories | |
| Schuh et al. | Mitigating the curse of dimensionality for exact knn retrieval | |
| Chung et al. | Multiple k nearest neighbor search | |
| CN114925067A (en) | Data processing method, device, equipment and storage medium | |
| CN114610960A (en) | Real-time recommendation method based on item2vec and vector clustering | |
| Tang et al. | Exploit every bit: Effective caching for high-dimensional nearest neighbor search | |
| Ragavan et al. | A Novel Big Data Storage Reduction Model for Drill Down Search. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 310000 Room 501, building 9, No. 20, kekeyuan Road, Baiyang street, Hangzhou Economic and Technological Development Zone, Zhejiang Province Patentee after: Hangzhou Zhongke advanced technology development Co.,Ltd. Address before: 310026 Room 501, building 9, 20 kejiyuan Road, Baiyang street, Hangzhou Economic and Technological Development Zone, Zhejiang Province Patentee before: HANGZHOU ZHONGKE ADVANCED TECHNOLOGY RESEARCH INSTITUTE Co.,Ltd. |