WO2019242374A1 - 数据结构、数据索引方法、装置及设备、存储介质 - Google Patents
数据结构、数据索引方法、装置及设备、存储介质 Download PDFInfo
- Publication number
- WO2019242374A1 WO2019242374A1 PCT/CN2019/081205 CN2019081205W WO2019242374A1 WO 2019242374 A1 WO2019242374 A1 WO 2019242374A1 CN 2019081205 W CN2019081205 W CN 2019081205W WO 2019242374 A1 WO2019242374 A1 WO 2019242374A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- bloom filter
- mapping
- positioning array
- hash
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
Definitions
- the present invention relates to the field of communications technologies, and in particular, to a data structure, a data indexing method, apparatus, and device, and a computer-readable storage medium.
- the information center network adopts a communication mode driven by a content requester.
- Two types of data packets are used in the communication process: Interest packets and Data packets.
- three types of data structures are deployed in the forwarding plane: CS (Content Store), PIT (Pending Interest Table), and FIB (Forwarding Information Base), which enables data packets to be forwarded.
- CS Content Store
- PIT Point Interest Table
- FIB Forwarding Information Base
- the forwarding plane will only find the matching entry in the PIT and forward the Data packet to these interfaces according to the interface list displayed in the entry. After the forwarding is completed, the corresponding PIT entry is deleted and the Data packet is stored in the CS.
- the communication mode in which the information center network processes each data packet places high requirements on the performance of the forwarding plane, especially the storage capacity of the router and the processing speed of the data packet. For example, when the information of multiple Interest packets is stored in the PIT table of the forwarding plane, these Interest packets each correspond to a communication request, and this information will be stored in the PIT table until the response Data packet is returned. Therefore, the PIT table The amount of data in will be huge. Research shows that in the PIT table of a router's forwarding plane, the magnitude of the record entry can be expressed as: bandwidth * RTT / P. Among them, P represents the average size of the Data packet, and RTT represents the average waiting time of the Interest recorded in the PIT table.
- the router's PIT table will contain at least 1,000,000,000 records.
- the content of each record in the PIT table is far more complicated than the records in the IP routing table.
- the size of the forwarding plane PIT table will also increase by an order of magnitude. Therefore, how to further increase the storage capacity of the router, speed up the retrieval of data, and realize the sharing of routing data at the same time has become a hot research issue in the information center network forwarding plane.
- an object of the embodiments of the present invention is to provide a data structure, a data indexing method, a device and a device, and a computer-readable storage medium to solve the performance problem of the information center network forwarding plane.
- a data structure includes: a compressed Bloom filter containing m bits and a positioning array containing j bits, where m> j; the positioning array and There is a mapping relationship between the compressed bloom filters;
- the compressed bloom filter is used to perform a hash mapping operation on the input data and compress the data after the hash mapping operation;
- the value of the positioning array is used as an offset address of the input data in the memory for storage access.
- a data indexing method is provided, and the method includes steps:
- the value of the i-th bit of the positioning array is set to the first binary radix, where i ⁇ j ;
- the value of the positioning array finally obtained is used as an offset address of the input data in the memory for storage access.
- a data indexing apparatus includes a hash mapping operation module and a spatial address mapping module;
- the hash mapping operation module is configured to perform k hash mapping operations on input data in a compressed Bloom filter, and compress the data after k hash mapping operations;
- the spatial address mapping module is configured to set the value of the i-th bit of the positioning array to be the first if the input data has a hash map in the i-th part of the compressed bloom filter.
- Binary radix, where i ⁇ j; the value of the positioning array finally obtained as the offset address of the input data in the memory is used for storage access.
- a data indexing device including: a memory, a processor, and a data indexing program stored on the memory and executable on the processor.
- the data indexing program is executed by the processor, the steps of the data indexing method are implemented.
- a computer-readable storage medium stores a data indexing program, and the data indexing program is implemented by a processor to implement the foregoing data indexing method. A step of.
- the data structure has the capability of transmitting and sharing the indicated information on the network, and can directly index data. It can be directly deployed on on-chip high-speed memory and effectively implements storage compression of the index structure.
- FIG. 1 is a schematic diagram of a data structure according to a first embodiment of the present invention
- FIG. 2 is a schematic diagram of a data indexing process according to a second embodiment of the present invention.
- FIG. 3 is another schematic flowchart of data indexing according to the second embodiment of the present invention.
- FIG. 4 is a schematic structural diagram of a data indexing device according to a third embodiment of the present invention.
- FIG. 5 is another schematic structural diagram of a data indexing device according to a third embodiment of the present invention.
- FIG. 6 is a schematic structural diagram of a data indexing device according to a fourth embodiment of the present invention.
- a first embodiment of the present invention provides a data structure.
- the data structure includes: a compressed bloom filter 10 (Compressed Bloom filter) containing m bits and a positioning array 20 (Mapping) containing j bits.
- Array (MA for short), where m> j; the positioning array 20 has a mapping relationship with the compressed bloom filter 10;
- the compressed bloom filter 10 is configured to perform a hash mapping operation on the input data and compress the data after the hash mapping operation.
- the compressed bloom filter 10 includes a bloom filter 11 and a compression unit 12;
- the bloom filter 11 is configured to perform a hash mapping operation on the input data
- the compression unit 12 is configured to compress data after the hash mapping operation of the bloom filter 11.
- the value of the positioning array 20 is used as an offset address of the input data in the memory for storage access.
- the compression unit 12 is equally divided into j parts, and each part corresponds to one bit of the positioning array 20.
- the positioning array 20 is an array containing 6 bits
- the compression unit 12 is set to 18 bits
- the compression unit 12 is equally divided into 6 parts, each of which corresponds to the positioning array.
- One bit of 20, for example, 123 of compression unit 12 in the figure is the first part (shown as 1st in the figure), which corresponds to the first bit of the positioning array 20 (shown as 1st in the figure).
- the data compression function of the data structure can effectively reduce the amount of data transmission in the network; by transmitting the bit array of the data structure in the network, the data structure has the functions of network data transmission and data sharing.
- the positioning array 20 For each data element x operating in the data structure, the positioning array 20 will be based on the k hash map values h i (x), i ⁇ [1, k] of the element x in the compressed Bloom filter 10, Map a value M (x) ⁇ [0, j-1] again. If the element x ⁇ S, that is, x exists in the set S represented by the data structure, the value M (x) of the positioning array 20 will be used as the actual offset address of the element in the memory for storage access.
- the data structure of the embodiment of the present invention has a data structure capable of transmitting and sharing the indicated information in a network, and can implement direct indexing of data.
- the data indexing efficiency is high, and it can be directly deployed on on-chip high-speed memory and effectively implement the index structure Storage compression.
- a second embodiment of the present invention provides a method for indexing data in a data structure according to the first embodiment.
- the method includes the following steps:
- S21 Perform k hash mapping operations on the input data in the compressed Bloom filter, and compress the data after the k hash mapping operations.
- the input data may be variable-length character string data, such as variable-length character string name data, which is not specifically limited herein.
- the value of the positioning array finally obtained is used as an offset address of the input data in the memory for storage access.
- the k-hash mapping operation is performed on the input data in the compressed Bloom filter, and the data after the k-hash mapping operation is compressed. Including steps:
- the first binary radix may be 0 or 1; the second binary radix may also be 0 or 1.
- the size of the Bloom filter is set to 32 bits
- the compression unit is set to 18 bits
- the positioning array is an array containing 6 bits.
- the data structure can be static Indexes 2 to 6 elements in the physical storage unit.
- the three elements X, Y, and Z are entered in sequence. Before each element is entered, the positioning array is initialized to all 0 states. When X is input, bits 1, 6, and 11 in the Bloom filter are hash-mapped, and all three bits are 1, indicating that the element X exists in the data structure.
- the values of the first, second, and third bits of the positioning array will be set to 1, and the other positions will be 0.
- the value of the positioning array is 111000, that is, the offset address of the X element in the static physical storage unit is 111000.
- the elements Y and Z also exist in this data structure, and their offset addresses are: 010101 and 100011, respectively.
- the misjudgment probability is a key performance of describing a data structure.
- a misjudgment of a data structure can be defined as a data structure that incorrectly judges that an element that does not belong to the set S exists in the set S, or When an element performs spatial address mapping, a mapping conflict occurs. Assume that all hash functions used by the compressed Bloom filter are random, uniform, and independent of each other.
- P CoMBF P cbf + P MA -P (cbf ⁇ MA)
- P cbf The probability of misjudgement of the compressed bloom filter
- P MA is the probability of a mapping conflict when spatial address mapping is performed in the positioning array
- P (cbf ⁇ MA) is the occurrence of the compressed bloom filter.
- P cbf (1- ⁇ ) k ,
- n the size of the set S
- m the length of the compressed Bloom filter bit array
- k the number of hash functions used in the data structure.
- P MA can be expressed as Among them, j represents the size of the positioning array.
- P (cbf ⁇ MA) can be expressed as
- the data structure has the ability to transmit and share the indicated information in the network.
- the hash mapping process of the name data in the compressed Bloom filter skillfully solves the problem of processing variable-length string name data.
- the storage space occupied by the data structure is (m + j) bits, and the number of data contents that can be indexed can be expressed as:
- the data structure has the capability of transmitting and sharing the indicated information in the network, and can be directly deployed in the on-chip high-speed memory, and effectively realizes the storage and compression of the index structure; Data indexing is efficient.
- a third embodiment of the present invention provides a data indexing apparatus, where the apparatus includes a hash mapping operation module 31 and a spatial address mapping module 32;
- the hash mapping operation module 31 is configured to perform k hash mapping operations on input data in a compressed Bloom filter, and compress the data after k hash mapping operations;
- the input data may be variable-length character string data, such as variable-length character string name data, which is not specifically limited herein.
- the spatial address mapping module 32 is configured to set a value of the i-th bit of the positioning array to be the first if the input data has a hash map in the i-th part of the compressed Bloom filter.
- a binary radix, where i ⁇ j; the value of the positioning array finally obtained as the offset address of the input data in the memory is used for storage access.
- the device further includes an initial module 30;
- the initial module 30 is configured to initialize each bit of the positioning array to a second binary radix.
- the first binary radix may be 0 or 1; the second binary radix may also be 0 or 1.
- the size of the Bloom filter is set to 32 bits
- the compression unit is set to 18 bits
- the positioning array is an array containing 6 bits.
- the data structure can be static Indexes 2 to 6 elements in the physical storage unit.
- the three elements X, Y, and Z are entered in sequence. Before each element is entered, the positioning array is initialized to all 0 states. When X is input, bits 1, 6, and 11 in the Bloom filter are hash-mapped, and all three bits are 1, indicating that the element X exists in the data structure.
- the values of the first, second, and third bits of the positioning array will be set to 1, and the other positions will be 0.
- the value of the positioning array is 111000, that is, the offset address of the X element in the static physical storage unit is 111000.
- the elements Y and Z also exist in this data structure, and their offset addresses are: 010101 and 100011, respectively.
- the misjudgment probability is a key performance of describing a data structure.
- a misjudgment of a data structure can be defined as a data structure that incorrectly judges that an element that does not belong to the set S exists in the set S, or When an element performs spatial address mapping, a mapping conflict occurs. Assume that all hash functions used by the compressed Bloom filter are random, uniform, and independent of each other.
- P CoMBF P cbf + P MA -P (cbf ⁇ MA)
- P cbf The probability of misjudgement of the compressed bloom filter
- P MA is the probability of a mapping conflict when spatial address mapping is performed in the positioning array
- P (cbf ⁇ MA) is the occurrence of the compressed bloom filter.
- P cbf (1- ⁇ ) k ,
- n the size of the set S
- m the length of the compressed Bloom filter bit array
- k the number of hash functions used in the data structure.
- P MA can be expressed as Among them, j represents the size of the positioning array.
- P (cbf ⁇ MA) can be expressed as
- the data structure has the ability to transmit and share the indicated information in the network.
- the hash mapping process of the name data in the compressed Bloom filter cleverly solves the problem of processing variable-length string name data.
- the storage space occupied by the data structure is (m + j) bits, and the number of data contents that can be indexed can be expressed as:
- the data structure has the capability of transmitting and sharing the indicated information in the network, and can be directly deployed in the on-chip high-speed memory, and effectively realizes the storage and compression of the index structure; Data indexing is efficient.
- a fourth embodiment of the present invention provides a data indexing device.
- the device includes: a memory 41, a processor 42, and data stored in the memory 41 and operable on the processor 42.
- An indexing program when the data indexing program is executed by the processor 42, is used to implement the steps of the data indexing method described below:
- the value of the i-th bit of the positioning array is set to the first binary radix, where i ⁇ j ;
- the value of the positioning array finally obtained is used as an offset address of the input data in the memory for storage access.
- the data indexing program When executed by the processor 42, it is further configured to implement the steps of the data indexing method described below:
- Each bit of the positioning array is initialized to a second binary radix.
- the data indexing program When executed by the processor 42, it is further configured to implement the steps of the data indexing method described below:
- P CoMBF P cbf + P MA -P (cbf ⁇ MA) , where P cbf is the probability of misjudgment of the compressed bloom filter, and P MA is in the positioning array.
- P (cbf ⁇ MA) is the probability of a misjudgment of the compressed bloom filter and a mapping conflict occurring when spatial address mapping is performed in the positioning array.
- the data structure has the capability of transmitting and sharing the indicated information in the network, and can be directly deployed on the on-chip high-speed memory, and effectively realizes the storage and compression of the index structure; Data indexing is efficient.
- a fifth embodiment of the present invention provides a computer-readable storage medium.
- the computer-readable storage medium stores a data index program, and the data index program is used by a processor to implement the data described in the second embodiment. Steps in the indexing method.
- the data structure has the capability of transmitting and sharing the indicated information in the network, and can be directly deployed in the on-chip high-speed memory, and effectively implements the storage and compression of the index structure; the direct data Indexing, data indexing is efficient.
- the technical solution of the present invention in essence, or a part that contributes to the existing technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, The optical disc) includes a plurality of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the embodiments of the present invention.
- a terminal device which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Information Transfer Between Computers (AREA)
Abstract
本发明公开一种数据结构、数据索引方法、装置及设备、计算机可读存储介质,所述数据结构包括:含有m比特的压缩的布隆过滤器以及含有j比特的定位数组;所述定位数组与所述压缩的布隆过滤器存在映射关系;所述压缩的布隆过滤器,用于对输入数据进行哈希映射操作,并对哈希映射操作后的数据进行压缩;所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。本发明数据结构具备了所表示信息在网络中传输与共享的能力,可实现数据的直接索引,数据索引效率高,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩。
Description
本申请要求享有2018年6月21日提交的名称为“数据结构、数据索引方法、装置及设备、存储介质”的中国专利申请CN201810644706.X的优先权,其全部内容通过引用并入本文中。
本发明涉及通信技术领域,尤其涉及一种数据结构、数据索引方法、装置及设备、计算机可读存储介质。
为了实现面向内容的通信方式,信息中心网络采用了内容请求者驱动的通信模式。在通信过程中使用两种类型的数据包:Interest(兴趣)包和Data包。同时,在转发平面中部署3种数据结构:CS(Content Store,内容存储池)、PIT(Pending Interest Table,待定兴趣表)和FIB(Forwarding Information Base,转发信息表),实现了数据包在转发平面的快速检索、智能转发和高效率内容缓存。在转发平面中,当多个Interest包同时请求相同数据时,转发平面仅转发收到的第一个Interest包,并将这些请求存储在PIT中。当Data包沿着Interest包的反向路径回传时,转发平面只会在PIT中找到与之匹配的条目,并根据条目中显示的接口列表,分别向这些接口转发Data包。转发完成后,则会删除相应的PIT条目,并将Data包储存在CS中。
信息中心网络对每个数据包都进行处理的通信模式,对转发平面的性能提出了较高的要求,特别是路由器的存储容量以及数据包的处理速度。例如,当转发平面的PIT表中存储多条Interest包的信息时,这些Interest包会分别对应一个通信请求,且直到响应的Data包返回前,该信息都会一直储存在PIT表中,因此PIT表中的数据量将非常庞大。研究表明,在一个路由器转发平面的PIT表中,记录条目的数量级可表示为:带宽*RTT/P。其中,P表示Data包的平均大小,RTT表示Interest记录在PIT表中的平均等待时间。对于一个10Gbps的链路而言,若路由器包含10个端口,时间为100ms,Data包的平均大小P为1000Bytes,则该路由器的PIT表至少会包含1000,000,000条记录。虽然,当前基于 TCP/IP技术的路由器可以支撑百万级别的记录数目,但在PIT表中每条记录的内容都要远复杂于IP路由表的记录。此外,随着网络链路速度的提升以及路由端口数目的增加,转发平面PIT表的大小也将成数量级的增长。因此,如何进一步提高路由器的存储容量、加快数据的检索速度,同时实现路由数据共享就成为信息中心网络转发平面的一个热点研究问题。
发明内容
有鉴于此,本发明实施例的目的在于提供一种数据结构、数据索引方法、装置及设备、计算机可读存储介质,以解决信息中心网络转发平面的性能问题。
本发明实施例解决上述技术问题所采用的技术方案如下:
根据本发明实施例的一个方面,提供的一种数据结构,所述数据结构包括:含有m比特的压缩的布隆过滤器以及含有j比特的定位数组,其中m>j;所述定位数组与所述压缩的布隆过滤器存在映射关系;
所述压缩的布隆过滤器,用于对输入数据进行哈希映射操作,并对哈希映射操作后的数据进行压缩;
所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
根据本发明实施例的另一个方面,提供的一种数据索引方法,所述方法包括步骤:
在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;
若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;
最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
根据本发明实施例的另一个方面,提供的一种数据索引装置,所述装置包括哈希映射操作模块和空间地址映射模块;
所述哈希映射操作模块,用于在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;
所述空间地址映射模块,用于若所述输入数据在所述压缩的布隆过滤器的第i个部分 存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
根据本发明实施例的另一个方面,提供的一种数据索引设备,所述设备包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的数据索引程序,所述数据索引程序被所述处理器执行时实现上述的数据索引方法的步骤。
根据本发明实施例的另一个方面,提供的一种计算机可读存储介质,所述计算机可读存储介质上存储有数据索引程序,所述数据索引程序被处理器执行时实现上述的数据索引方法的步骤。
本发明实施例的数据结构、数据索引方法、装置及设备、计算机可读存储介质,数据结构具备了所表示信息在网络中传输与共享的能力,可实现数据的直接索引,数据索引效率高,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩。
图1为本发明第一实施例的数据结构示意图;
图2为本发明第二实施例的数据索引流程示意图;
图3为本发明第二实施例的数据索引另一流程示意图;
图4为本发明第三实施例的数据索引装置结构示意图;
图5为本发明第三实施例的数据索引装置另一结构示意图;
图6为本发明第四实施例的数据索引设备结构示意图。
本发明目的的实现、功能特点及优点将结合实施例,参照附图做进一步说明。
为了使本发明所要解决的技术问题、技术方案及有益效果更加清楚、明白,以下结合附图和实施例,对本发明进行进一步详细说明。应当理解,此处所描述的具体实施例仅仅用以解释本发明,并不用于限定本发明。
第一实施例
如图1所示,本发明第一实施例提供一种数据结构,所述数据结构包括:含有m比特 的压缩的布隆过滤器10(Compressed Bloom filter)以及含有j比特的定位数组20(Mapping Array,简称MA),其中m>j;所述定位数组20与所述压缩的布隆过滤器10存在映射关系;
所述压缩的布隆过滤器10,用于对输入数据进行哈希映射操作,并对哈希映射操作后的数据进行压缩。
在本实施例中,所述压缩的布隆过滤器10包括布隆过滤器11和压缩单元12;
所述布隆过滤器11,用于对所述输入数据进行哈希映射操作;
所述压缩单元12,用于对所述布隆过滤器11哈希映射操作后的数据进行压缩。
所述定位数组20的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
在本实施例中,所述压缩单元12等分为j个部分,且每一个部分对应所述定位数组20的一个比特。
作为示例地,请参考图1所示,假设定位数组20为含有6比特的数组,压缩单元12被设置为18比特,将压缩单元12等分为6个部分,每一个部分对应所述定位数组20的一个比特,例如图中压缩单元12的123为第1个部分(图中的1st所示),对应所述定位数组20的第一个比特(图中的1st所示)。
在本实施例中,数据结构可以表示一个含有n个元素的集合S,其中S={x
1,x
2,x
3,...,x
n},可实现元素检索的基本功能,即确定一个数据元素是否在数据结构所表示的集合S中。同时,数据结构的数据压缩功能,可有效降低网络中的数据传输量;通过在网络中传输数据结构的比特数组,使数据结构具备网络数据传输、数据共享功能。对于每一个在数据结构中操作的数据元素x,定位数组20都会根据元素x在压缩的布隆过滤器10中的k个哈希映射值h
i(x),i∈[1,k],再次映射一个数值M(x)∈[0,j-1]。若元素x∈S,即x存在于数据结构所表示的集合S中,那么定位数组20的数值M(x)将作为该元素在存储器中的实际偏移地址用于存储访问。
本发明实施例的数据结构,数据结构具备了所表示信息在网络中传输与共享的能力,可实现数据的直接索引,数据索引效率高,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩。
第二实施例
如图2所示,本发明第二实施例提供一种在第一实施例所述的数据结构中进行数据索引的方法,所述方法包括步骤:
S21、在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩。
在本实施例中,输入数据可以为变长字符串数据,例如变长字符串名称数据,具体地在此不作限制。
S22、若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j。
S23、最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
请参考图3所示,在一种实施方式中,所述在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩之前还包括步骤:
S20、将所述定位数组的每一位都初始为第二二进制基数。
在本实施例中,第一二进制基数可以为0或1;第二二进制基数也可以为0或1。
作为示例地,假设数据结构使用K=3个哈希函数,布隆过滤器的大小被设置为32比特,压缩单元被设置为18比特,定位数组为含有6比特的数组,数据结构可在静态物理存储单元中索引2
6个元素。三个元素X、Y、Z被依次输入,每次元素输入前,定位数组都会被初始化为全0状态。当X输入时,布隆过滤器中的第1、6、11比特位被哈希映射,且这三个比特位均为1,即表示元素X存在于该数据结构中。经过压缩感知,压缩单元的第1、2、3部分存在哈希映射,因此定位数组的第1、2、3比特位的数值将被设定为1,其他位置数值为0。最终得到定位数组的值为111000,即X元素在静态物理存储单元中的偏移地址为111000。同理,元素Y和Z也均存在于该数据结构中,其偏移地址分别为:010101和100011。
在本实施例中,误判概率是描述数据结构的关键性能,可将数据结构发生误判定义为数据结构将一个不属于集合S的元素错误地判断为存在于集合S中,或者在对某一元素进行空间地址映射时,发生映射冲突的现象。假设压缩的布隆过滤器使用的所有哈希函数都是随机、均匀且相互独立的,数据结构发生误判的概率P
CoMBF=P
cbf+P
MA-P
(cbf∩MA),其中P
cbf为所述压缩的布隆过滤器发生误判的概率,P
MA为在所述定位数组 中进行空间地址映射时发生映射冲突的概率,P
(cbf∩MA)为所述压缩的布隆过滤器发生误判、且同时在所述定位数组中进行空间地址映射时发生映射冲突的概率。
在本实施例中,P
cbf可表示为P
cbf=(1-ρ)
k,
其中ρ=(1-1/m)
kn,n代表集合S的大小,m表示压缩的布隆过滤器比特数组的长度,k表示数据结构中使用哈希函数的个数。
因此,数据结构发生误判的概率可表示为:
综上,数据结构具备了所表示信息在网络中传输与共享的能力,同时名称数据在压缩的布隆过滤器中的哈希映射过程巧妙地解决了变长字符串名称数据的处理问题。对于每一个访问数据结构的名称数据,只需要进行一次片内k个哈希函数的映射,即可实现数据的直接索引。因此数据结构的数据索引效率会远优于布隆过滤器与哈希表结合的数据索引方式。此外,数据结构所占存储空间为(m+j)比特,能索引数据内容的数量可表示为:
当数据结构应用于面向内容的信息中心网络数据平面时,其存储容量需求应约为:2MB~3MB。即数据结构可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩。
本发明实施例的数据索引方法,数据结构具备了所表示信息在网络中传输与共享的能力,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩;实现了数据的直接索引,数据索引效率高。
第三实施例
如图4所示,本发明第三实施例提供一种数据索引装置,所述装置包括哈希映射操作模块31和空间地址映射模块32;
所述哈希映射操作模块31,用于在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;
在本实施例中,输入数据可以为变长字符串数据,例如变长字符串名称数据,具体地在此不作限制。
所述空间地址映射模块32,用于若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
请参考图5所示,在一种实施方式中,所述装置还包括初始模块30;
所述初始模块30,用于将所述定位数组的每一位都初始为第二二进制基数。
在本实施例中,第一二进制基数可以为0或1;第二二进制基数也可以为0或1。
作为示例地,假设数据结构使用K=3个哈希函数,布隆过滤器的大小被设置为32比特,压缩单元被设置为18比特,定位数组为含有6比特的数组,数据结构可在静态物理存储单元中索引2
6个元素。三个元素X、Y、Z被依次输入,每次元素输入前,定位数组都会被初始化为全0状态。当X输入时,布隆过滤器中的第1、6、11比特位被哈希映射,且这三个比特位均为1,即表示元素X存在于该数据结构中。经过压缩感知,压缩单元的第1、2、3部分存在哈希映射,因此定位数组的第1、2、3比特位的数值将被设定为1,其他位置数值为0。最终得到定位数组的值为111000,即X元素在静态物理存储单元中的偏移地址为111000。同理,元素Y和Z也均存在于该数据结构中,其偏移地址分别为:010101和100011。
在本实施例中,误判概率是描述数据结构的关键性能,可将数据结构发生误判定义为数据结构将一个不属于集合S的元素错误地判断为存在于集合S中,或者在对某一元素进行空间地址映射时,发生映射冲突的现象。假设压缩的布隆过滤器使用的所有哈希函数都是随机、均匀且相互独立的,数据结构发生误判的概率P
CoMBF=P
cbf+P
MA-P
(cbf∩MA),其中P
cbf为所述压缩的布隆过滤器发生误判的概率,P
MA为在所述定位数组中进行空间地址映射时发生映射冲突的概率,P
(cbf∩MA)为所述压缩的布隆过滤器发生误判、且同时在所述定位数组中进行空间地址映射时发生映射冲突的概率。
在本实施例中,P
cbf可表示为P
cbf=(1-ρ)
k,
其中ρ=(1-1/m)
kn,n代表集合S的大小,m表示压缩的布隆过滤器比特数组的长度,k表示数据结构中使用哈希函数的个数。
因此,数据结构发生误判的概率可表示为:
综上,数据结构具备了所表示信息在网络中传输与共享的能力,同时名称数据在压缩的布隆过滤器中的哈希映射过程巧妙地解决了变长字符串名称数据的处理问题。对于每一个访问数据结构的名称数据,只需要进行一次片内k个哈希函数的映射,即可实现数据的直接索引。因此数据结构的数据索引效率会远优于布隆过滤器与哈希表结合的数据索引方式。此外,数据结构所占存储空间为(m+j)比特,能索引数据内容的数量可表示为:
当数据结构应用于面向内容的信息中心网络数据平面时,其存储容量需求应约为:2MB~3MB。即数据结构可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩。
本发明实施例的数据索引装置,数据结构具备了所表示信息在网络中传输与共享的能力,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩;实现了数据的直接索引,数据索引效率高。
第四实施例
如图6所示,本发明第四实施例提供一种数据索引设备,所述设备包括:存储器41、处理器42及存储在所述存储器41上并可在所述处理器42上运行的数据索引程序,所述 数据索引程序被所述处理器42执行时,用于实现以下所述的数据索引方法的步骤:
在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;
若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;
最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
所述数据索引程序被所述处理器42执行时,还用于实现以下所述的数据索引方法的步骤:
将所述定位数组的每一位都初始为第二二进制基数。
所述数据索引程序被所述处理器42执行时,还用于实现以下所述的数据索引方法的步骤:
数据结构发生误判的概率P
CoMBF=P
cbf+P
MA-P
(cbf∩MA),其中P
cbf为所述压缩的布隆过滤器发生误判的概率,P
MA为在所述定位数组中进行空间地址映射时发生映射冲突的概率,P
(cbf∩MA)为所述压缩的布隆过滤器发生误判、且同时在所述定位数组中进行空间地址映射时发生映射冲突的概率。
本发明实施例的数据索引设备,数据结构具备了所表示信息在网络中传输与共享的能力,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩;实现了数据的直接索引,数据索引效率高。
第五实施例
本发明第五实施例提供一种计算机可读存储介质,所述计算机可读存储介质上存储有数据索引程序,所述数据索引程序被处理器执行时用于实现第二实施例所述的数据索引方法的步骤。
本发明实施例的计算机可读存储介质,数据结构具备了所表示信息在网络中传输与共享的能力,可以直接部署于片内高速存储器,并有效实现索引结构的存储压缩;实现了数据的直接索引,数据索引效率高。
需要说明的是,上述装置实施例与方法实施例属于同一构思,其具体实现过程详见方法实施例,且方法实施例中的技术特征在装置实施例中均对应适用,这里不再赘述。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件来实现,但很多情况下前者是更佳的实施方式。基于这样的理解,本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,空调器,或者网络设备等)执行本发明各个实施例所述的方法。
以上参照附图说明了本发明的优选实施例,并非因此局限本发明的权利范围。本领域技术人员不脱离本发明的范围和实质,可以有多种变型方案实现本发明,比如作为一个实施例的特征可用于另一实施例而得到又一实施例。凡在运用本发明的技术构思之内所作的任何修改、等同替换和改进,均应在本发明的权利范围之内。
Claims (11)
- 一种数据结构,其中,所述数据结构包括:含有m比特的压缩的布隆过滤器以及含有j比特的定位数组,其中m>j;所述定位数组与所述压缩的布隆过滤器存在映射关系;所述压缩的布隆过滤器,用于对输入数据进行哈希映射操作,并对哈希映射操作后的数据进行压缩;所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
- 根据权利要求1所述的数据结构,其中,所述压缩的布隆过滤器包括布隆过滤器和压缩单元;所述布隆过滤器,用于对所述输入数据进行哈希映射操作;所述压缩单元,用于对所述布隆过滤器哈希映射操作后的数据进行压缩。
- 根据权利要求2所述的数据结构,其中,所述压缩单元等分为j个部分,且每一个部分对应所述定位数组的一个比特。
- 一种在权利要求1-3任一所述的数据结构中进行数据索引的方法,其中,所述方法包括步骤:在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
- 根据权利要求4所述的方法,其中,所述在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩之前还包括步骤:将所述定位数组的每一位都初始为第二二进制基数。
- 根据权利要求4所述的方法,其中,数据结构发生误判的概率P CoMBF=P cbf+P MA-P (cbf∩MA),其中P cbf为所述压缩的布隆过滤器发生误判的概率,P MA为在所述定位数组中进行空间地址映射时发生映射冲突的概率,P (cbf∩MA)为所述压缩的布隆过滤器发生 误判、且同时在所述定位数组中进行空间地址映射时发生映射冲突的概率。
- 一种数据索引装置,其中,所述装置包括哈希映射操作模块和空间地址映射模块;所述哈希映射操作模块,用于在压缩的布隆过滤器中对输入数据进行k次哈希映射操作,并对k次哈希映射操作后的数据进行压缩;所述空间地址映射模块,用于若所述输入数据在所述压缩的布隆过滤器的第i个部分存在哈希映射,则将所述定位数组的第i个比特的值设为第一二进制基数,其中i≤j;最终得到的所述定位数组的数值作为所述输入数据在存储器中的偏移地址用于存储访问。
- 根据权利要求7所述的装置,其中,所述装置还包括初始模块;所述初始模块,用于将所述定位数组的每一位都初始为第二二进制基数。
- 根据权利要求7所述的装置,其中,数据结构发生误判的概率P CoMBF=P cbf+P MA-P (cbf∩MA),其中P cbf为所述压缩的布隆过滤器发生误判的概率,P MA为在所述定位数组中进行空间地址映射时发生映射冲突的概率,P (cbf∩MA)为所述压缩的布隆过滤器发生误判、且同时在所述定位数组中进行空间地址映射时发生映射冲突的概率。
- 一种数据索引设备,其中,所述设备包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的数据索引程序,所述数据索引程序被所述处理器执行时实现如权利要求4至6中任一项所述的数据索引的方法的步骤。
- 一种计算机可读存储介质,其中,所述计算机可读存储介质上存储有数据索引程序,所述数据索引程序被处理器执行时实现如权利要求4至6中任一项所述的数据索引的方法的步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810644706.X | 2018-06-21 | ||
CN201810644706.XA CN110704419A (zh) | 2018-06-21 | 2018-06-21 | 数据结构、数据索引方法、装置及设备、存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019242374A1 true WO2019242374A1 (zh) | 2019-12-26 |
Family
ID=68983237
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/081205 WO2019242374A1 (zh) | 2018-06-21 | 2019-04-03 | 数据结构、数据索引方法、装置及设备、存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110704419A (zh) |
WO (1) | WO2019242374A1 (zh) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113329048B (zh) * | 2021-04-13 | 2023-04-07 | 网络通信与安全紫金山实验室 | 基于交换机的云负载均衡方法、装置及存储介质 |
CN115422142B (zh) * | 2022-08-22 | 2024-07-09 | 北京羽乐创新科技有限公司 | 数据压缩方法及装置 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104408069A (zh) * | 2014-10-30 | 2015-03-11 | 浪潮电子信息产业股份有限公司 | 一种基于布隆过滤器思想的一致性目录设计方法 |
US20160188623A1 (en) * | 2014-12-29 | 2016-06-30 | International Business Machines Corporation | Scan optimization using bloom filter synopsis |
CN107832343A (zh) * | 2017-10-13 | 2018-03-23 | 天津大学 | 一种基于位图的mbf数据索引结构对数据快速检索的方法 |
CN107908357A (zh) * | 2017-10-13 | 2018-04-13 | 天津大学 | 命名数据网转发平面pit存储结构及其数据检索方法 |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101956031B1 (ko) * | 2012-10-15 | 2019-03-11 | 삼성전자 주식회사 | 데이터 압축 장치 및 방법, 데이터 압축 장치를 포함하는 메모리 시스템 |
US9608863B2 (en) * | 2014-10-17 | 2017-03-28 | Cisco Technology, Inc. | Address autoconfiguration using bloom filter parameters for unique address computation |
CN105989061B (zh) * | 2015-02-09 | 2019-11-26 | 中国科学院信息工程研究所 | 一种滑动窗口下多维数据重复检测快速索引方法 |
-
2018
- 2018-06-21 CN CN201810644706.XA patent/CN110704419A/zh not_active Withdrawn
-
2019
- 2019-04-03 WO PCT/CN2019/081205 patent/WO2019242374A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104408069A (zh) * | 2014-10-30 | 2015-03-11 | 浪潮电子信息产业股份有限公司 | 一种基于布隆过滤器思想的一致性目录设计方法 |
US20160188623A1 (en) * | 2014-12-29 | 2016-06-30 | International Business Machines Corporation | Scan optimization using bloom filter synopsis |
CN107832343A (zh) * | 2017-10-13 | 2018-03-23 | 天津大学 | 一种基于位图的mbf数据索引结构对数据快速检索的方法 |
CN107908357A (zh) * | 2017-10-13 | 2018-04-13 | 天津大学 | 命名数据网转发平面pit存储结构及其数据检索方法 |
Also Published As
Publication number | Publication date |
---|---|
CN110704419A (zh) | 2020-01-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1581875B1 (en) | Using direct memory access for performing database operations between two or more machines | |
US7177912B1 (en) | SCSI transport protocol via TCP/IP using existing network hardware and software | |
EP3771169A1 (en) | Message processing method and related device | |
EP3474146B1 (en) | Data processing method, storage system and exchange device | |
US20230153264A1 (en) | Data transmission method, chip, and device | |
US20240039995A1 (en) | Data access system and method, device, and network adapter | |
EP2216955A1 (en) | Network interface device | |
US20200274943A1 (en) | Data Processing Method and Apparatus, Server, and Controller | |
US20200274832A1 (en) | Rdma transport with hardware integration and out of order placement | |
CN110235098A (zh) | 存储系统访问方法及装置 | |
CN111314480B (zh) | 负载自适应跨平台文件传输协议分布式服务实现方法 | |
EP3804244B1 (en) | Systems and methods for transport layer processing of server message block protocol messages | |
US8539089B2 (en) | System and method for vertical perimeter protection | |
WO2019242374A1 (zh) | 数据结构、数据索引方法、装置及设备、存储介质 | |
CN114885045B (zh) | 一种在高速智能网卡/dpu内节约dma通道资源方法和装置 | |
CN106294191B (zh) | 处理表的方法、访问表的方法和装置 | |
WO2020187124A1 (zh) | 数据处理方法及装置 | |
CN113691466A (zh) | 一种数据的传输方法、智能网卡、计算设备及存储介质 | |
US8090832B1 (en) | Method and apparatus for allocating network protocol operation resources | |
CN107615259B (zh) | 一种数据处理方法及系统 | |
CN113923259A (zh) | 数据处理方法及系统 | |
CN105230074A (zh) | 视频缓存切换处理方法、装置和系统 | |
CN118034615B (zh) | 数据访问方法以及装置 | |
WO2024140068A2 (zh) | 报文传输方法及系统、网络装置 | |
US11314414B2 (en) | Methods, devices, and computer program products for storage management |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19823755 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.05.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19823755 Country of ref document: EP Kind code of ref document: A1 |